Airflow at lyft for Airflow summit 2020 conference
2 likes1,359 views
1) Lyft uses Airflow for ETL workflows to move data from mobile apps and events to data warehouses. 2) Lyft has customized Airflow with features like UI auditing, DAG dependency graphs, and integrating Amundsen for data lineage. 3) Current focuses at Lyft include an ETL expiration system, upgrading DAGs to Python 3, and leveraging new Airflow features in a multi-tenant cluster.
Airflow at lyft for Airflow summit 2020 conference
- 1. July 2020 Tao Feng | @feng-tao | Engineer, Lyft Data Platform Blog: go.lyft.com/airflowblog Airflow @ Lyft
- 2. 2 Who ● Engineer at Lyft Data Platform and Tools ● Apache Airflow PMC and Committer ● Working on different data products (Airflow, Amundsen, etc) ● Previously at Linkedin, Oracle
- 3. Agenda • Data Platform @ Lyft • Airflow Customization @ Lyft • Current Focus For Airflow @ Lyft • Summary 3
- 4. Data Platform @ Lyft 4
- 5. About Lyft MISSION: Improve people's life with the world's best transportation
- 6. Lyft’s data analytics platform architecture Backend Services Mobile app PubSub Events Batch ETL Presto, Hive Client, and BI Tools
- 7. Airflow main use cases @ Lyft 7
- 8. Airflow usage @ Lyft 8 ● Two Clusters ● Celery Executors
- 10. Airflow customization @ Lyft • UI auditing • DAG dependency graph 10
- 11. Airflow customization @ Lyft • Extra link for task instance UI panel 11 ● Hive query log ● Dr elephant report for performance tuning ● Hive job analysis dashboard
- 12. Airflow customization @ Lyft • Amundsen is an open-sourced data discovery portal. • It is integrated with Airflow to show the task and table lineage. • It is currently used by 18+ companies. 12
- 13. Current Focus For Airflow @ Lyft 13
- 14. ETL Expiration System 14 • Lots of ETLs are not well maintained with no clear ownership. • Built an ETL Expiration system to: ‒ Disabled DAGs with expired TTLs (DAG owner needs to renew the TTL every six months). ‒ Disabled DAGs that produced unused datasets ‒ Disabled DAGs that are failing for a long time
- 15. PY2 -> PY3 • Built a dashboard to understand PY3 issue. ‒ Most issues are related to string encoding or string and integer comparison. • DAG loading time is higher in py3 compared to py2 ‒ Cherry pick a few performance improvement patches from upstream 15
- 16. Airflow Upgrade • Leverage new features: ‒ DAG serialization ‒ RBAC ‒ Data Lineage ‒ Performance Improvements • Current status: ‒ Built a new multi-tenant cluster to onboard new use cases. ‒ Finishing PY3 upgrade for legacy DAGs. ‒ Converting the existing legacy mono DAG repo as another tenant on the new cluster. 16
- 17. Summary 17
- 18. Summary 18 • Covers Lyft data platform in general • Discusses about Airflow customization at Lyft • Discusses about Airflow current work at Lyft
- 19. Acknowledgement 19 • Members who maintain Airflow at Lyft ‒ Andrew Stahlman ‒ Bhanu Renukuntla ‒ Chao-han Tsai (committer) ‒ Jinhyuk Chang ‒ Junda Yang ‒ Max Payton ‒ Sherry Zhao ‒ Shenghu Yang (EM) ‒ Tao Feng (committer) • Thanks Maxime for his guidance
- 20. Tao Feng | @feng-tao Blog at go.lyft.com/airflowblog 20