Bootstrapping a ML platform at Bluevine [Airflow Summit 2020]
1 like167 views
The document discusses the implementation of an ML platform using Apache Airflow at Bluevine, highlighting the migration from cron jobs to a more structured workflow. It details the challenges faced, including performance issues with the Airflow scheduler and strategies like standalone triggers and monitoring solutions to enhance efficiency. Ultimately, it emphasizes improvements in task execution times and robust monitoring capabilities to manage the system effectively.
Bootstrapping a ML platform at Bluevine [Airflow Summit 2020]
- 1. From Zero to AirïŹow bootstrapping a ML platform 1
- 2. 2 About us Bluevine â Fintech startup up based in Redwood City, CA and Tel Aviv, Israel â Provides working capital (loans) to small & medium sized businesses â Over $2 BN funded to date â Over 3.5$ BN delivered in Payment Protection Program Me â Noam Elfanbaum (@noamelf), Data Engineering team lead @ BlueVine â Live in Tel-Aviv with my wife, kid and dog. â My colleague Ido Shlomo created the original presentation for OSDC 2019 conference.
- 3. 3 Building a ML analytics platform into production using Apache Airflow at Bluevine. This includes: â Migrating our ML workload to Airflow â Hacking at Airflow to provide a semi-streaming solution â Monitoring business sensitive processes Case study
- 5. 5 What was in place? â Lots (and lots) of cron-jobs on a single server! â Every logic ran as an independent cron â Every logic / cron figured out its own triggering mechanism â Every logic / cron figured out its own dependencies â No communication between logics
- 6. 6 Desired â Ability to process one client end-to-end â Decision within a few minutes â Map and centrally control dependencies â Easy and simple monitoring â Easy to scale â Efficient error recovery Goals Existing â Scope defined by # of clients in data batch â Over 15 minutes â Hidden and distributed dependencies â Hard and confusing monitoring â Impractical to scale â âAll or nothingâ error recovery
- 7. 7 AirïŹow brief intro â Core component is the scheduler / executor â Uses dedicated metadata DB to figure out current status of tasks â Uses workers to execute new ones â Web server allows live interaction and monitoring
- 8. 8 What is a DAG? DAG: Directed Acyclic Graph â Basically a map of tasks run in a certain dependency structure â Each DAG has a run frequency (e.g. every 10 seconds) â Both DAGs and tasks can run concurrently
- 9. 9 Infrastructure setup â We run on AWS - and prefer managed services â Celery is the executor â Flower proved very useful for monitoring workers state â No thrills setup!
- 10. 10 Isolated environments â Isolation between Airflow environment and our scripts â BashOperator is executing the script under the correct virtual environment
- 11. 11 Phasing out cron jobs â Spin up Airflow alongside existing Data DBs, servers and cron jobs. â Translate every cron job into DAG with one task that points to same python script (Bash Operator). â For each cron (200 of them): â Turn off cron job â Turn on âSingletonâ DAG â When all crons off â Kill old servers
- 13. 13 User onboarding â Airflow is built for batch processing â We needed to support streaming user processing â Airflow is not a good fit for that! â Nevertheless, due to time constraints and familiarity, we chose to start with it
- 14. 14 THE Onboarding DAG (sort of)
- 15. 15 Onboarding âstreamingâ Design Logic executed Related functionality is executed, as the user progress through the application form User signup A ânew userâ event is sent. As user goes through the application forms the relevant events are sent Sensor poll on queue Onboarding DAG poll for the events using the SensorOperator. Once a ânew userâ event is received, the user ID is saved in XCOM to share it between the tasks
- 18. 18 Airflow scheduler took up to 30 seconds to compute the next task to run (i.e. step)!
- 19. 19 Hack #1 - standalone trigger Problem â Airflow scheduler is creating all tasks objects on DAG start â The onboarding DAG has ~40 tasks, and the scheduler works hard to figure out each task dependencies â A new DAG run starts on interval and a sensor is polling for new user â This creates a lot of âliveâ pending DAGs Solution â Have a triggering DAG that only contains a sensor and a triggering task â It triggers the large on-boarding DAG
- 20. 20 Hack #1 - standalone trigger
- 21. Solution â Archive DB data to keep 1 week of history â Gotcha! Also make sure to keep a DAG last run, not doing so will make Airflow think it didnât run and rerun it. 21 Hack #2: Archive DB tables Problem â Big DB â slower queries â slower scheduling & execution â DB contains metadata for all dag / task runs â High dag frequency + many DAGs + many tasks == many rows â Under our setup, within first two months, the DB was over 15 GB in size
- 22. 22 Hack #3 - Patch scheduler DAGâs state queries Problem â In order to determine if a task met its dependencies, the scheduler query the DB for each task in the DAG â The Onboarding Dag has 40 tasks and can have 20 parallel runs. â This means ~800 (!) DB queries every pass just for this one Dag. Solution â Patch Airflow to query the DAG state by sending one query per DAG instead of a query per DAG task. â PR made to Airflow team: AIRFLOW-3607, to be released in Airflow 2.0 â Results: â 90th percentile delay was decreased by 30% â DB CPU usage decreased by 20% â Avg delay was decreased 18%
- 23. 23 Hack #4 - Create a dedicated âfastâ AirïŹow Solution â Spin up a 2nd Airflow just for time-sensitive processes! â Dedicated instance â less dags / tasks â faster scheduling â Approx 60% reduction in average time spent on transitions between tasks. Problem â Scheduler has to continually parse all DAGs â Not all DAGs are equally latency sensitive but all are given the same scheduling resources
- 24. 24 Final results â Time between dependent tasks is consistently under 3 seconds â Overall runtime is under 3 minutes for 95% of the cases
- 26. 26 Plugin to match users with runs â Locates the Airflow DAG run for a given user ID â Helps to track down issues found with users
- 27. Track scheduler latencies â Query Airflow DB from Grafana â Query the delta between a time that a task finishes and the time the next one starts 27
- 28. Scheduler outage alerts 28 â Airflow most critical component is the scheduler - nothing happens without it â The scheduler sends a heartbeat to the DB â Grafana polls on that table to and sends us an alert if the scheduler is down
- 29. Track ïŹow latencies â Airflow UI is great! But, it doesnât allow to view aggregated data â Querying the DB allows to extract great aggregated view that can show the state of the system in a glance â Grafana is great! 29
- 30. Questions? 30