Airflow techtonic template
- 1. Learning and Development 2019 Apache Airflow For Data Engineering - Sampath Kumar, Principle Engineer
- 2. Agenda Goal of this session is to provide an overview of Airflow capabilities & how to use it for Data Engineering ● Introduction (DE, Airflow) ● Key Concepts (Key Words, Demo) ● Architecture (System Design) ● Features (Challenges, Loggins, Analytics) ● Challenges & Recommendations ● Q & A
- 3. DATA ENGINEERING Data Engineering is the aspect of data science that focuses on practical applications of data collection and analysis. Data Engineers are tasked with ● Designing, building, testing, integrating, managing, and optimizing data from a variety of sources ● Build the infrastructure and architecture that enable data generation ● Primary focus is to build free-flowing data pipelines by combining a variety of big data technologies that enable real-time analytics ● Data engineers also write complex queries to ensure that data is easily accessible
- 4. AIRFLOW ● Open-source workflow automation and scheduling system that can be used to author and manage your data pipelines. ● It started at Airbnb in October 2014 as a solution to manage the company's increasing complex workflows. ● License: Apache License 2.0 ● Written in: Python ● Operating system: Microsoft Windows, macOS, Linux ● Stable release: 1.10.5 / August 30, 2019; 3 months ago
- 5. CRON CRON: Derived from work CRONOS(means time), is a software for unix-like systems to schedule jobs based on time. Airflow is cron on steroids: it allows you to schedule tasks to run, run them in a particular order, and monitor / manage all of your tasks.
- 6. KEY CONCEPTS ● DAGS ○ Directed acyclic graphs that represent tasks workflow ● Task ○ Operators - Bash, Python, SSH, Http, MySql, SparkSubmit, Sensors(s3), Docker, Hive, Slack,.. ● Hooks ○ To store credentials for services - AWS, GCP, DataBase, Email,.. ● Vars & XCom ○ For sharing any global values or inter-task communication
- 7. Sample - DAG Start End S3Sensor Build VM Goal: As and when user files come into AWS S3 bucket, start a high-spec docker VM and process the job. Prior to start and post process, job status to be notified. Solutions required to be cost efficient and need to auto-scale if required.
- 8. DEMO - UI
- 9. Sample - DAG A data engineering pipeline, where an S3 sensor is used to identify the arrival of input file, following by several validation checks and then load into ElasticSearch which will be used for serving clients. Start (S3Sensor) Schema Validation Inputs Validation Data Validation Load to ElasticSearch End Failures
- 10. ARCHITECTURE ● MetaDB ● Message Broker ● Airflow Webserver ● Airflow Scheduler ● Airflow Workers
- 11. FEATURES ● Airflow Workers are Horizontally scalable ● Airflow Messaging Broker - Celery ● Airflow Integrations - GCP, Azure, AWS, Qubole & Databricks ● Hooks, Connections & Pools - Environment(dev/test/prod) friendly ● DAG - Dynamic sub dags & Branching
- 12. ANALYTICS ● Part of being productive with data is having the right weapons to profile the data you are working with. ● Airflow provides a simple query interface to write SQL and get results quickly, and a charting application letting you visualize data.
- 13. LOGGING ● Logging is visible from UI
- 14. CHALLENGES ● Airflow is not Apache NIFI or Apache Spark ○ is not - data routing or data transformation system. ● Airflow Workers ○ requires identical access(network, authentication & authorisation) ○ requires similar hardware capabilities ● Airflow Webserver & Scheduler ○ not scalable (Work-around - supervisor, docker health checks)
- 15. RECOMMENDATIONS Airflow is a best suited where ● Agility is important ● Portability ● Segregation b/w compute and workflow mgmt ● Scalability on demand ● Pool/Connection management ● Job Analytics ● Logging visibility
- 17. REFERENCES ● Roles of Data Engineer - https://inlovewithcode.wordpress.com/2019/05/15/roles-of-data-engineer-required-skill/ ● System Design - https://inlovewithcode.wordpress.com/system-design/ ● Airflow Tutorials - https://airflow-tutorial.readthedocs.io/en/latest/airflow-intro.html ● Airflow Documentation - https://airflow.apache.org/docs/stable/ ● Airflow Dockers - https://towardsdatascience.com/getting-started-with-airflow-using-docker-cd8b44dbff98
- 18. Thank you