What's Coming in Apache Airflow 2.0 - PyDataWarsaw 2019
- 3. Polidea
- 4. Polidea Apache Airflow Airflow is a platform to programmatically author, schedule and monitor workflows. Dynamic/Elegant Extensible Scalable
- 6. Polidea What is the presentation about ? ● The team @ Polidea ● What the Airflow ? ● Where Apache Airflow is now? ● What’s coming in Apache Airflow 2.0.
- 8. Polidea Logo or mockup Hi! Jarek Potiuk Principal Software Engineer @Polidea Apache Airflow PMC member Certified GCP Architect ex-Googler, ex-CTO, ex-choir member @higrys
- 9. Polidea Apache Airflow Development team@ Polidea Jarek Potiuk Kamil Breguła Tomasz Urbaszek Karolina Rosół Dariusz Aniszewski Szymon Przedwojski Antoni Smoliński Tobiasz Kędzierski Michał Słowikowski PMC Past:
- 10. Polidea Apache Airflow Website team @ Polidea Kamil Breguła Zuzanna Rykowska Kamil Gabryjelski Magdalena WęgrzyńskaMarta StrzałkowskaTomasz Urbaszek
- 11. Polidea 70+ TALENTS 100+ PROJECTS DELIVERED 3m USERS OF OUR APPS 75% OF BUSINESS THROUGH REFERRALS Team @Polidea
- 13. Polidea August 2018 2 people Timeline December 2019 6 (9) people
- 14. Polidea Our tasks ● 130+ operators ● 18+ GCP services ● Oozie-To-Airflow ● New Apache Airflow Website
- 15. Polidea What we delivered extra ● Documentation improvements ● Breeze - improved dev environment ● Py2 -> Py3 ● Pylint compatibility ● Pre-commit framework introduction ● CI environment reimplemented ● Operator scaffolding ● Convert tests to pytests 2 Apache Airflow Committers Apache Airflow PMC member
- 18. Polidea Why Apache Airflow and not one of these? And many, many, many more ....
- 19. Polidea Airflow is an Orchestrator ● Tells others what/when to do ● Synchronizes work between others ● Monitors what’s going on ● Intervenes if needed ● Mostly does not do much
- 21. Polidea Arbitrary complex workflows as a program
- 22. Polidea Airflow has usable UI
- 24. Polidea What Airflow shines at ? ● Regular batch ETL jobs (think CRON) ● Processing fixed intervals of data ● Managing complex dependencies ● Backfilling data ● Interfacing to hundreds of different systems ● Platform for others to generate DAG files
- 25. Polidea Apache Airflow 1.10 state of the pinwheel
- 26. Polidea Current versions ● 1.10.2, 1.10.3, 1.10.4, 1.10.5, 1.10.6 …. ● 1.10.7 in the making ● Deployed in thousands of companies ● On the rise of usage ● 2.0 - in master
- 27. Polidea How to stay relevant ? ● Cloud Native is coming ● APIs are backbone of modern software ● User Interface matters ● Performance and reliability matter ● Many services, many changes ● Community over code
- 28. Polidea End of 2019 survey: 300 responses(!) ● Started by Tomasz Urbaszek ● Run for the last 2 weeks ● Fresh off-the press ● Some surprises found ● Going in the right direction
- 29. Polidea
- 30. Polidea What do you use Airflow for? Data processing (ETL) 97% Artificial Intelligence and Machine Learning Pipelines 29% Automating DevOps operations 21%
- 31. Polidea What can be improved ? Scheduler performance 61% Web UI 58% Logging, monitoring and alerting 47% Examples, howtos, onboarding documentation 46% Technical documentation 44% Reliability 36% REST API 31% Authentication and authorization 29%
- 32. Polidea What would be the most interesting feature for you ? Production-ready docker image 56% Declarative way of writing DAGs 50% Horizontal autoscaling 40% Examples, howtos, onboarding documentation 46% Asynchronous Operators 31% Stateless web server 26% Knative Executor 16% I already have all I need 4%
- 34. Polidea Cloud Native is coming
- 35. Polidea
- 36. Polidea No - we do not plan to use Kubernetes near term 29% Yes - setup on our own via Helm Chart or similar 21% Not yet - but we use Kubernetes in our organization and we could move 20% Yes - via managed service in the cloud (Composer/Astronomer etc.) 15% Not yet - but we plan to deploy Kubernetes in our organization soon 14% Other 2% Either use or can use Kubernetes in foreseeable future 69% Do not have plans to use Kubernetes 29% Do you use Kubernetes-based deployments for Airflow?
- 37. Polidea Cloud Native is coming: Scalability ● Knative Executor ● SIG-Knative => SIG Scalability ● Native Airflow Executor (WIP) ● Pub/Sub communication ● Horizontally auto-scalable
- 38. Polidea Cloud Native is coming: Deployability ● Native worker deployable at different providers ● “As a service” and “on-premises” friendly ● Generic Pub/Sub architecture for communication ● No DB communication between components ● Production-optimised docker image
- 39. Polidea Cloud Native is coming: Monitoring ● Integrate with standard monitoring tools ● More metrics exported using stats ● Integration with Prometheus on Kubernetes ● Horizontal Scalability approach based on metrics
- 40. Polidea APIs are backbone of modern software
- 41. Polidea APIs are taking over the world ● Modern API ● HTTP-based API used by CLI, webserver ● Pub/Sub API for communication Scheduler <> Workers ● Generic APIs - not tied to Kubernetes/other deployment options ● Better Authentication/Authorization ● Opens up multi-tenancy capabilities
- 43. Polidea Original Airflow Graphical User Interface 97% CLI 40% API (experimental) 20% Custom Own Created UI 8% Which interface(s) of Airflow do you use as part of your current role?
- 44. Polidea UIs are getting better ● Make UI refresh like it’s 2020 ● Modern design (possibly) ● Use APIs for communication not DB/file access ● Better authentication and authorisation ● Stateless web-server ● Better responsiveness
- 46. Polidea Performance and reliability is important ● Automated performance testing (CI - targeted) ● Monitoring performance characteristics ● Improve Webserver/Scheduler Performance ● Internal instrumentation and optimisations
- 47. Polidea Many services, fast changes
- 48. Polidea Fast evolving services ● Currently operators bound to releases of Airflow ● Migration to 2.0 will take time ● Introducing new approach ○ move operators to new path/namespaces ○ change import paths ○ backporting to 1.10 ○ backportable to 1.10 (!) ○ future: per-provider packaging
- 50. Polidea
- 51. Polidea Community over code: Documentation ● Google Season of Docs - great programme! ● Onboarding, best practices, architecture, deployment options ● Better, clearer structure ● Both user and developer documentation improved ● Worked with technical writers from India and Russia
- 52. Polidea Community over code: New website: airflow.apache.org Work sponsored by Google Cloud
- 53. Polidea Community over code: Development environment ● It’s a Breeze to develop Apache Airflow ● Get your environment up in 10 minutes ● Integration with IDE ● Well documented ● Team-work enabler ● Allows to run and debug DAGs ● Fully debuggable: DebugExecutor - cooperation with Databand.ai
- 55. Friday, December 13, 2019 5:30 PM to 9:30 PM Polidea Sp. z o.o. Przeskok 2 IV p. · Warsaw https://t.co/TmWdWwfemI First Warsaw Apache Airflow Workshop
- 57. Polidea Thanks!
- 58. Prototypes Widely Used UX Design Personal Growth Individuality Trust Beter, Fester Ideation API Design Shipping Building blocks Backend VR Android Learn more Firmware Decision Coffee CollaborationContactCVSave moneyOpen Source Management React Native Testing Team Quality iOS Maintain Security Front End Flutter Task Seamless UXResearchMobile AR