Resilient Predictive Data Pipelines (QCon London 2016)
6 likes2,014 views
This document discusses building resilient predictive data pipelines. It begins by distinguishing between ETL and predictive data pipelines, noting that predictive pipelines require high availability with downtimes of less than an hour. The document then outlines design goals for resilient data pipelines, including being scalable, available, instrumented/monitored/alert-enabled, and quickly recoverable. It proposes using AWS services like SQS, SNS, S3, and Auto Scaling Groups to build such pipelines. The document also recommends using Apache Airflow for workflow automation and scheduling to reliably manage pipelines as directed acyclic graphs. It presents an architecture using these techniques and assesses how well it meets the outlined design goals.
![About Me
2
Work [ed | s] @
Maintainer on
Report to
Co-Chair for
airbnb/airο¬ow](https://image.slidesharecdn.com/resilientpredictivedatapipelinesv4-160308161426/85/Resilient-Predictive-Data-Pipelines-QCon-London-2016-2-320.jpg)
![13
β’ Scalable
β’ Build your pipelines using [inο¬nitely] scalable components
β’ The scalability of your system is determined by its least-scalable
component
β’ Available
β’ Instrumented, Monitored, & Alert-enabled
β’ Quickly Recoverable
Desirable Qualities of a Resilient
Data Pipeline](https://image.slidesharecdn.com/resilientpredictivedatapipelinesv4-160308161426/85/Resilient-Predictive-Data-Pipelines-QCon-London-2016-13-320.jpg)
![Desirable Qualities of a Resilient
Data Pipeline
14
β’ Scalable
β’ Build your pipelines using [inο¬nitely] scalable components
β’ The scalability of your system is determined by its least-scalable
component
β’ Available
β’ Ditto
β’ Instrumented, Monitored, & Alert-enabled
β’ Quickly Recoverable](https://image.slidesharecdn.com/resilientpredictivedatapipelinesv4-160308161426/85/Resilient-Predictive-Data-Pipelines-QCon-London-2016-14-320.jpg)
![Desirable Qualities of a Resilient
Data Pipeline
17
β’ Scalable
β’ Build your pipelines using [inο¬nitely] scalable components
β’ The scalability of your system is determined by its least-scalable
component
β’ Available
β’ Ditto
β’ Instrumented, Monitored, & Alert-enabled
β’ Quickly Recoverable](https://image.slidesharecdn.com/resilientpredictivedatapipelinesv4-160308161426/85/Resilient-Predictive-Data-Pipelines-QCon-London-2016-17-320.jpg)
Resilient Predictive Data Pipelines (QCon London 2016)
- 1. Resilient Predictive Data Pipelines Sid Anand (@r39132) QCon London 2016 1
- 2. About Me 2 Work [ed | s] @ Maintainer on Report to Co-Chair for airbnb/airο¬ow
- 3. Motivation Why is a Data Pipeline talk in this High Availability Track? 3
- 4. Different Types of Data Pipelines 4 ETL β’ used for : loading data related to business health into a Data Warehouse β’ user-engagement stats (e.g. social networking) β’ product success stats (e.g. e-commerce) β’ audience : Business, BizOps β’ downtime? : 1-3 days Predictive β’ used for : β’building recommendation products (e.g. social networking, shopping) β’updating fraud prevention endpoints (e.g. security, payments, e-commerce) β’ audience : Customers β’ downtime? : < 1 hour VS
- 5. 5 ETL β’ used for Data Warehouse β> Reports β’ user-engagement stats ( social networking β’ product success stats ( e-commerce β’ audience β’ downtime? Predictive β’ used for : β’building recommendation products (e.g. social networking, shopping) β’updating fraud prevention endpoints (e.g. security, payments, e-commerce) β’ audience : Customers β’ downtime? : < 1 hour VS Different Types of Data Pipelines
- 6. Why Do We Care About Resilience? 6 d4d5d6d7d8 DB Search Engines Recommenders d1 d2 d3
- 7. DB Why Do We Care About Resilience? 7 d4 d6d7d8 Search Engines Recommenders d1 d2 d3 d5
- 8. Why Do We Care About Resilience? 8 d7d8 DB Search Engines Recommenders d1 d2 d3 d4 d5 d6
- 9. Why Do We Care About Resilience? 9 d7d8 DB Search Engines Recommenders d1 d2 d3 d4 d5 d6
- 10. Any Take-aways? 10 d7d8 DB Search Engines Recommenders d1 d2 d3 d4 d5 d6 β’ Bugs happen! β’ Bugs in Predictive Data Pipelines have a large blast radius β’ The bugs can affect customers and a companyβs proο¬ts & reputation!
- 11. Design Goals Desirable Qualities of a Resilient Data Pipeline 11
- 12. 12 β’ Scalable β’ Available β’ Instrumented, Monitored, & Alert-enabled β’ Quickly Recoverable Desirable Qualities of a Resilient Data Pipeline
- 13. 13 β’ Scalable β’ Build your pipelines using [inο¬nitely] scalable components β’ The scalability of your system is determined by its least-scalable component β’ Available β’ Instrumented, Monitored, & Alert-enabled β’ Quickly Recoverable Desirable Qualities of a Resilient Data Pipeline
- 14. Desirable Qualities of a Resilient Data Pipeline 14 β’ Scalable β’ Build your pipelines using [inο¬nitely] scalable components β’ The scalability of your system is determined by its least-scalable component β’ Available β’ Ditto β’ Instrumented, Monitored, & Alert-enabled β’ Quickly Recoverable
- 15. Instrumented 15 Instrumentation must reveal SLA metrics at each stage of the pipeline! What SLA metrics do we care about? Correctness & Timeliness β’ Correctness β’ No Data Loss β’ No Data Corruption β’ No Data Duplication β’ A Deο¬ned Acceptable Staleness of Intermediate Data β’ Timeliness β’ A late result == a useless result β’ Delayed processing of now()βs data may delay the processing of future data
- 16. Instrumented, Monitored, & Alert- enabled 16 β’ Instrument β’ Instrument Correctness & Timeliness SLA metrics at each stage of the pipeline β’ Monitor β’ Continuously monitor that SLA metrics fall within acceptable bounds (i.e. pre-deο¬ned SLAs) β’ Alert β’ Alert when we miss SLAs
- 17. Desirable Qualities of a Resilient Data Pipeline 17 β’ Scalable β’ Build your pipelines using [inο¬nitely] scalable components β’ The scalability of your system is determined by its least-scalable component β’ Available β’ Ditto β’ Instrumented, Monitored, & Alert-enabled β’ Quickly Recoverable
- 18. Quickly Recoverable 18 β’ Bugs happen! β’ Bugs in Predictive Data Pipelines have a large blast radius β’ Optimize for MTTR
- 19. Implementation Using AWS to meet Design Goals 19
- 21. SQS - Overview 21 AWSβs low-latency, highly scalable, highly available message queue Inο¬nitely Scalable Queue (though not FIFO) Low End-to-end latency (generally sub-second) Pull-based How it Works! Producers publish messages, which can be batched, to an SQS queue Consumers consume messages, which can be batched, from the queue commit message contents to a data store ACK the messages as a batch
- 22. visibility timer SQS - Typical Operation Flow 22 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 1: A consumer reads a message from SQS. This starts a visibility timer!
- 23. visibility timer SQS - Typical Operation Flow 23 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 2: Consumer persists message contents to DB
- 24. visibility timer SQS - Typical Operation Flow 24 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 3: Consumer ACKs message in SQS
- 25. visibility timer SQS - Time Out Example 25 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 1: A consumer reads a message from SQS
- 26. visibility timer SQS - Time Out Example 26 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 2: Consumer attempts persists message contents to DB
- 27. visibility time out SQS - Time Out Example 27 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 3: A Visibility Timeout occurs & the message becomes visible again.
- 28. visibility timer SQS - Time Out Example 28 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 m1 SQS Step 4: Another consumer reads and persists the same message
- 29. visibility timer SQS - Time Out Example 29 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 5: Consumer ACKs message in SQS
- 30. SQS - Dead Letter Queue 30 SQS - DLQ visibility timer Producer Producer Producer m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Redrive rule : 2x m1
- 32. SNS - Overview 32 Highly Scalable, Highly Available, Push-based Topic Service Whereas SQS ensures each message is seen by at least 1 consumer SNS ensures that each message is seen by every consumer Reliable Multi-Push Whereas SQS is pull-based, SNS is push-based There is no message retention & there is a ο¬nite retry count No Reliable Message Delivery Can we work around this limitation?
- 33. SNS + SQS Design Pattern 33 m1m2 m1m2 m1m2 SQS Q1 SQS Q2 SNS T1 Reliable Multi Push Reliable Message Delivery
- 34. SNS + SQS 34 Producer Producer Producer m1m2 Consumer Consumer Consumer DB m1 m1m2 m1m2 SQS Q1 SQS Q2 SNS T1 Consumer Consumer Consumer ES m1
- 35. S3 + SNS + SQS Design Pattern 35 m1m2 m1m2 m1m2 SQS Q1 SQS Q2 SNS T1 Reliable Multi Push Transactions S3 d1 d2
- 36. Batch Pipeline Architecture Putting the Pieces Together 36
- 37. Architecture 37
- 38. Architectural Elements 38 A Schema-aware Data format for all data (Avro) The entire pipeline is built from Highly-Available/Highly-Scalable components S3, SNS, SQS, ASG, EMR Spark (exception DB) The pipeline is never blocked because we use a DLQ for messages we cannot process We use queue-based auto-scaling to get high on-demand ingest rates We manage everything with Airο¬ow Every stage in the pipeline is idempotent Every stage in the pipeline is instrumented
- 40. ASG - Overview 40 What is it? A means to automatically scale out/in clusters to handle variable load/trafο¬c A means to keep a cluster/service always up Fulο¬lls AWSβs pay-per-use promise! When to use it? Feed-processing, web trafο¬c load balancing, zone outage, etcβ¦
- 41. ASG - Data Pipeline 41 importer importer importer importer Importer ASG scaleout/in SQS DB
- 42. ASG : CPU-based 42 Sent CPU ACKd/Recvd CPU-based auto-scaling is good at scaling in/out to keep the average CPU constant
- 43. ASG : CPU-based 43 Sent CPU Recv Premature Scale-in Premature Scale-in: The CPU drops to noise-levels before all messages are consumed. This causes scale in to occur while the last few messages are still being committed resulting in a long time- to-drain for the queue!
- 44. ASG - Queue-based 44 Scale-out: When Visible Messages > 0 (a.k.a. when queue depth > 0) Scale-in: When Invisible Messages = 0 (a.k.a. when the last in-ο¬ight message is ACKβd) This causes the ASG to grow This causes the ASG to shrink
- 45. Architecture 45
- 46. Reliable Hourly Job Scheduling Workο¬ow Automation & Scheduling 46
- 47. 47 Historical Context Our ο¬rst cut at the pipeline used cron to schedule hourly runs of Spark Problem We only knew if Spark succeeded. What if a downstream task failed? We needed something smarter than cron that Reliably managed a graph of tasks (DAG - Directed Acyclic Graph) Orchestrated hourly runs of that DAG Retried failed tasks Tracked the performance of each run and its tasks Reported on failure/success of runs Our Needs
- 48. Airο¬ow Workο¬ow Automation & Scheduling 48
- 49. Airο¬ow - DAG Dashboard 49 Airο¬ow: Itβs easy to manage multiple DAGs
- 50. Airο¬ow - Authoring DAGs 50 Airο¬ow: Visualizing a DAG
- 51. 51 Airο¬ow: Author DAGs in Python! No need to bundle many conο¬g ο¬les! Airο¬ow - Authoring DAGs
- 52. Airο¬ow - Performance Insights 52 Airο¬ow: Gantt chart view reveals the slowest tasks for a run!
- 53. 53 Airο¬ow: β¦And we can easily see performance trends over time Airο¬ow - Performance Insights
- 54. Airο¬ow - Alerting 54 Airο¬ow: β¦And easy to integrate with Ops tools!
- 56. 56 Airο¬ow - Join the Community With >30 Companies, >100 Contributors , and >2500 Commits, Airο¬ow is growing rapidly! We are looking for more contributors to help support the community! Disclaimer : Iβm a maintainer on the project
- 57. Design Goal Scorecard Are We Meeting Our Design Goals? 57
- 58. 58 β’ Scalable β’ Available β’ Instrumented, Monitored, & Alert-enabled β’ Quickly Recoverable Desirable Qualities of a Resilient Data Pipeline
- 59. 59 β’ Scalable β’ Build using scalable components from AWS β’ SQS, SNS, S3, ASG, EMR Spark β’ Exception = DB (WIP) β’ Available β’ Build using available components from AWS β’ Airο¬ow for reliable job scheduling β’ Instrumented, Monitored, & Alert-enabled β’ Airο¬ow β’ Quickly Recoverable β’ Airο¬ow, DLQs, ASGs, Spark & DB Desirable Qualities of a Resilient Data Pipeline