It's a Breeze to develop Apache Airflow (Apache Con Berlin)
- 2. Polidea
- 3. Polidea Airflow is a platform to programmatically author, schedule and monitor workflows. Dynamic/Elegant Extensible Scalable
- 4. Polidea
- 5. Polidea ● Developing Airflow is (was !) hard ● Road taken to developer productivity ● Improving first time experience for the developers ● Focus on teamwork
- 6. Polidea
- 7. Polidea
- 8. Polidea Hi! Principal Software Engineer @Polidea Apache Airflow PMC member Certified GCP Architect ex-Googler, ex-CTO, ex-choir member
- 9. Polidea TALENTS PROJECTS DELIVERED USERS OF OUR APPS OF BUSINESS THROUGH REFERRALS
- 10. Polidea
- 11. Polidea
- 12. Polidea
- 13. Polidea ● 100+ operators ● 18+ GCP services ● Oozie-To-Airflow
- 14. Polidea ● 1 Apache Airflow committer, 1 PMC member ● Documentation improvements ● Breeze - improved development environment ● Py2 -> Py3 ● Pylint compatibility ● CI environment reimplemented ● Operator scaffolding
- 15. Polidea
- 16. Polidea
- 17. Polidea ● Multiple backends: postgres, mysql, sqlite ● Multiple python versions (2.7) - 3.5, 3.6. 3.7 ● Multiple executors: Local/Sequential/Kubernetes ● Automated static code analysis ● Automated documentation building
- 18. Polidea ● Long time to set it up ● Frustrations of fresh developer experience ● High friction/learning curve for Airflow development environment ● Slow iteration speed ● Complicated Development Environment
- 19. Polidea ● Scripts only designed for CI, not local environment ● Dependencies installed every time you start the environment ● Always full database reset ● Minutes to run one test ● No guidance how to iterate over tests
- 20. Polidea
- 21. Polidea
- 22. Polidea ● Focus on developer productivity ● Faster development cycle ● Decrease developer frustration ● Improve the teamwork ● Easy for ad-hoc contributors to code & test
- 23. Polidea ● AIP-10: Multi-layered and multi-stage official Airflow image ● AIP-7: Simplified Development Workflow ● AIP-26: Production-ready Airflow Docker Image and helm chart ● AIP-23: Migrate out of Travis CI ● AIP-4: Support for System Tests for external systems
- 24. Polidea ● Local virtualenv ● Own Travis CI fork ● Docker compose (Travis CI equivalent)
- 25. Polidea ● Total time: 7 minutes ● Running one test only ● Failure at the end (!) ● Re-run - 10-20 seconds for DB ● Re-enter - same time (!) ● No bash history
- 26. Polidea
- 27. Polidea ● Docker images built from master automatically (DockerHub) ● Local images use cached images ● Tests and static checks run using Docker Compose/Docker environment ● Can be run on Kubernetes Cluster (Docker-In-Docker) ● CI system - independent ● Base to build production image
- 28. Polidea
- 29. Polidea ● works out-of-the-box ● initializes DB when needed ● environment variables set ● sub-second test overhead ● ipdb debugging ● verbose output
- 30. Polidea
- 31. Polidea ● entering the environment: ./breeze --backend sqlite --python 3.5 ● last-used environment: ./breeze ● automated image management ● autocomplete of options ● sub-second test execution overhead ● host sources mounted to Docker container ● ports forwarded ● hints for ad-hoc developers
- 32. Polidea ● run-tests tests.core<TAB><TAB> autocomplete ● bash history across sessions ● run static checks with Breeze ● build documentation with Breeze ● run licence checks with Breeze ● easy debugging (including debugging with IDE) ● pre-commit checks
- 33. Polidea
- 34. Polidea
- 35. Polidea
- 36. Polidea ● Docker image management ● Run-tests with DB initialisation ● Travis CI integration ● Run all tasks (docs/static/licence check ...) ● Pre-commit checks ● Comprehensive documentation - Google Season of Docs YAY!
- 37. Polidea
- 38. Polidea
- 39. Polidea ● easy to use ○ pre-commit install ○ pre-commit run ○ pre-commit run mypy ○ pre-commit run --all-files ● run only for changed files (fast) ● catches errors early ● make committers time efficient ● promotes good practices
- 40. Polidea
- 41. Polidea ● Production-ready Apache Airflow official image ● Simplifications (less images, easier scripts) ● Migrating out of Travis CI ○ GitLab CI (only CI) or GitHub Actions ○ Kubernetes Cluster on Google Kubernetes Engine (Thanks Google!) ● Automation of Performance Tests ● Automation of Release Tests