How to scheduled jobs in a cloudera cluster without oozie

How-to scheduled jobs in a
Cloudera Cluster without
oozie
Cloudera Manager + Kerberos + Airflow

Introduction
This presentation, it’s for everyone that is looking for an oozie alternative to scheduled
jobs in a secured Cloudera Cluster.
With this, you will be able to add and configure the Airflow Service an manage it with in
Cloudera Manager.

Architecture
Airflow will be the in the same layer of oozie.

Architecture
The Airflow architecture will be the following:

Pre-Assumptions
1. Environment with Centos 7
2. user root or with sudo
3. Cloudera CDH 6.3
4. MariaDB 10.3
5. Kerberos Installed
6. HiveServer with High Availability
7. Know the RabbitMQ and Erlang/OTP Compatibility Matrix

Prepare Airflow Installation
Download EPEL Repo
yum install –y epel-release;
Install Erlang
wget https://bintray.com/rabbitmq-erlang/rpm/download_file?file_path=erlang%2F21%2Fel%2F7%2Fx86_64%2Ferlang-
21.3.8.7-1.el7.x86_64.rpm;
mv download_file?file_path=erlang%2F21%2Fel%2F7%2Fx86_64%2Ferlang-21.3.8.7-1.el7.x86_64.rpm erlang-21.3.8.7-
1.el7.x86_64.rpm;
rpm -Uvh erlang-21.3.8.7-1.el7.x86_64.rpm;
Note: just need to change the highlighted values for your own values.

Test Erlang
erl
Erlang/OTP 22 [erts-10.4.4] [source] [64-bit] [smp:6:6] [ds:6:6:10]
[async-threads:1] [hipe]
Eshell V10.4.4 (abort with ^G)
1>
Install RabbitMQ via YUM
nano /etc/yum.repos.d/rabbitmq.repo;
[bintray-rabbitmq-server]
name=bintray-rabbitmq-rpm
baseurl=https://dl.bintray.com/rabbitmq/rpm/rabbitmq-server/v3.7.x/el/7/
gpgcheck=0
repo_gpgcheck=0
enabled=1
yum -y install rabbitmq-server-3.7.17-1.el7.noarch.rpm;

Modify Firewall Rules
firewall-cmd --zone=public --permanent --add-port=4369/tcp --add-port=25672/tcp --add-port=5671-5672/tcp --add-
port=15672/tcp --add-port=61613-61614/tcp --add-port=1883/tcp --add-port=8883/tcp;
firewall-cmd --reload;
Start the RabbitMQ server and enable it
systemctl start rabbitmq-server.service;
systemctl enable rabbitmq-server.service;
Check the status of RabbitMQ
rabbitmqctl status;
Enable and use the RabbitMQ management console
rabbitmq-plugins enable rabbitmq_management;
chown -R rabbitmq:rabbitmq /var/lib/rabbitmq/;

Create User
rabbitmqctl add_user mqadmin password;
rabbitmqctl set_user_tags mqadmin administrator;
rabbitmqctl set_permissions -p / mqadmin ".*" ".*" ".*";
Rabbit UI
http://cm630.localdomain:15672/

Airflow Installation
There’s many ways to do the Airflow installation, such as via pip. But i've found that the most useful it’s via
parcels with the advantage to be in the Cloudera Manager administration scope.
Create User and Database on MariaDB
mysql -uroot –ppassword;
CREATE DATABASE airflow DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;
GRANT ALL ON airflow.* TO 'airflow'@'localhost' IDENTIFIED BY 'password';
GRANT ALL ON airflow.* TO 'airflow'@'%' IDENTIFIED BY 'password';
Add AirFlow Parcel
Hosts > Parcels > Configuration
Add url: http://archive.clairvoyantsoft.com/airflow/parcels/latest/
Download and Distribute

Airflow Installation
Install the Airflow CSD
cd /opt/ckoudera/csd;
wget http://archive.clairvoyantsoft.com/airflow/csd/AIRFLOW-2.2.0.jar;
chmod 755 AIRFLOW-2.2.0.jar;
chown cloudera-scm:cloudera-scm cloudera-scm;
Restart Cloudera Server
systemctl restart cloudera-scm-server;

Enable Kerberos in Airflow
This process can be a bit tricky, please take special attention while configuring the airflow.cfg
AirFlow ProxyUser
HDFS > Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml
<property>
<name>hadoop.proxyuser.airflow.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.airflow.users</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.airflow.hosts</name>
<value>*</value>
</property>

Edit Airflow Config File
nano /etc/airflow/conf.cloudera.airflow/airflow.cfg;
[core]
[...]
executor = CeleryExecutor
sql_alchemy_conn = mysql://airflow:password@cm630.localdomain:3306/airflow
default_impersonation =
security = Kerberos
[celery]
celery_app_name = airflow.executors.celery_executor
broker_url = pyamqp://mqadmin:password@cm630.localdomain:4369/
result_backend = db+mysql://airflow:password@cm630.localdomain:3306/airflow
[kerberos]
ccache = /var/lib/airflow/airflow_krb5_ccache
principal = airflow
reinit_frequency = 3600
kinit_path = kinit
keytab = /var/lib/airflow/airflow.keytab

Create Airflow Kerberos Principal
kadmin.local <<eoj
addprinc -pw password airflow@DOMAIN.COM
modprinc -maxrenewlife 1week +allow_renewable airflow@DOMAIN.COM
xst -norandkey -k /var/lib/airflow/airflow.keytab airflow@DOMAIN.COM
eoj
chown airflow:airflow /var/lib/airflow/airflow.keytab;
Airflow Impersonation
nano /ets/sudoers;
## Same thing without a password
airflow ALL=(ALL) NOPASSWD: ALL
Note 1: For impersonations to work, Airflow must be run with sudo as subtasks are run with sudo -u and permissions of files are changed.
Furthermore, the unix user needs to exist on the worker. Here is what a simple sudoers file entry could look like to achieve this, assuming as airflow
is running as the airflow user. Which it means that the airflow user must be trusted and treated the same way as the root user.
Note 2: just need to change the highlighted values for your own values.

Add Airflow as a Cloudera Service
After all configuration, now we will add Airflow as a Service on Cloudera Manager.
Add Airflow Service
Add Services -> Airflow

Airflow Configurations

Test AirFlow ticket renewer
airflow Kerberos;

Interact with Airflow
On an initial phase the authentication on Airflow UI will de disabled, therefore you will not need any user/pass.
The Airflow UI url is: http://cm630.localdomain:8080/admin/
Create DAG and Plugin Dir
mkdir -p /var/lib/airflow/dags && mkdir -p /var/lib/airflow/plugins && chown airflow:airflow -R /var/lib/airflow/;
Login Airflow UI

Test MySQL/MariaDB Connection
Edit Airflow MySQL/MariaDB Connection

Test HiveServer2 Connection
Edit HiveServer2 Connection
Note: For test purpose it was created a table in Hive (tab_air);

Create Hello World DAG
from __future__ import print_function
import datetime
from airflow import models
from airflow.operators import bash_operator
from airflow.operators import python_operator
default_dag_args = {
'start_date': datetime.datetime(2019, 10, 10),
}
with models.DAG(
'composer_sample_simple_greeting',
schedule_interval=datetime.timedelta(days=1),
default_args=default_dag_args) as dag:
def greeting():
import logging
logging.info('Hello World!')
hello_python = python_operator.PythonOperator(
task_id='hello',
python_callable=greeting)
goodbye_bash = bash_operator.BashOperator(
task_id='bye',
bash_command='echo Goodbye.')
hello_python >> goodbye_bash
Test Hello World DAG
airflow test composer_sample_simple_greeting hello 15/10/2019

Issues with Airflow Service
I've found Airflow Service quite stable, but sometimes the Airflow Scheduler instance got stuck in a Starting/Stopping status.
Airflow Scheduler in Stopping/Starting Status
mysql -uroot -ppassword;
use scm;
select role_id, name, configured_status from ROLES where configured_status in ("STARTING","BUSY","STOPPING");
update ROLES set configured_status = "STOPPED" where role_id = "63";

Thanks
Big Data Engineer
Tiago Simões

How to scheduled jobs in a cloudera cluster without oozie

More Related Content

What's hot (20)

Similar to How to scheduled jobs in a cloudera cluster without oozie (20)

Recently uploaded (20)

How to scheduled jobs in a cloudera cluster without oozie