Building highly reliable data pipeline @datadog par Quentin François
3 likes567 views
The document discusses the construction of highly reliable data pipelines at Datadog, focusing on architecture, monitoring, and handling failures. It emphasizes the importance of breaking down jobs into smaller, manageable pieces to enhance fault tolerance and performance while monitoring various metrics to ensure timely completion and data accuracy. Additionally, it highlights strategies for quick recovery and graceful degradation during failures.
Building highly reliable data pipeline @datadog par Quentin François
- 1. Building highly reliable data pipelines @ Datadog Quentin FRANCOIS Team Lead, Data Engineering 1 Data Engineers Meetup Paris 26 Février 2019
- 2. 2
- 3. 3
- 4. 4
- 5. 5
- 6. Building highly reliable data pipelines @ Datadog 6 Quentin FRANCOIS Team Lead, Data Engineering Data Engineers Meetup Paris 26 Février 2019
- 7. Reliability is the probability that a system will produce correct outputs up to some given time t. Source: E.J. McClusky & S. Mitra (2004). "Fault Tolerance" in Computer Science Handbook 2ed. ed. A.B. Tucker. CRC Press. 7
- 8. 1. Architecture Highly reliable data pipelines 8
- 9. 1. Architecture 2. Monitoring Highly reliable data pipelines 9
- 10. 1. Architecture 2. Monitoring 3. Failures handling Highly reliable data pipelines 10
- 11. Historical metric queries Time series data metric system.load.1 timestamp 1526382440 value 0.92 tags host:i-xyz,env:dev,... 11
- 12. Historical metric queries 12 1 point/second
- 13. Historical metric queries 13 1 point/day
- 14. Historical metric queries 14 High resolution data Low resolution data 1pt /min 1pt /hour 1pt /day 1pt /sec AWS S3 • Runs once a day. • Dozens of TBs of input data. • Trillions of points processed. Rollups pipeline
- 15. 1. Architecture 2. Monitoring 3. Failures handling Highly reliable data pipelines 15
- 16. Our big data platform architecture CLUSTERS DATA WORKERS USERS Luigi Spark Datadog monitoring S3 EMR Web SchedulerCLI 16 EMR EMR EMR
- 17. Our big data platform architecture CLUSTERS DATA WORKERS USERS Luigi Spark Datadog monitoring S3 EMR Web SchedulerCLI 17 EMR EMR EMR
- 18. Many ephemeral clusters • New cluster for every pipeline. • Dozens of clusters at a time. • Median lifetime of ~3 hours. 18
- 19. Total isolation We know what is happening and why. 19
- 20. Pick the best hardware for each job For CPU-bound jobs c3 For memory-bound jobs r3 20
- 21. Scale up/down clusters • If we are behind. • Scale as we grow. • No more waiting on loaded clusters. 21
- 22. Safer upgrades of EMR/Hadoop/Spark 5.13 5.12 5.125.125.12 22
- 23. Spot-instance clusters Ridiculous savings (up to 80% off the on-demand price) Nodes can die at any time 23 - +
- 24. Spot-instance clusters Ridiculous savings (up to 80% off the on-demand price) Nodes can die at any time 24 - +
- 25. How can we build highly reliable data pipelines with instances killed randomly all the time? 25 ?
- 26. No long running jobs • The longer the job, the more work you lose on average. • The longer the job, the longer it takes to recover. 26
- 27. No long running jobs 0 9 27 Pipeline A Pipeline B Time (hours)
- 28. No long running jobs Pipeline A Pipeline B 0 Time (hours)97 10 16 Job failure 28
- 29. Break down jobs into smaller pieces Vertically - persist intermediate data between transformations. Horizontally - partition the input data. 29
- 30. Example 30 Input data Output data Rollups pipeline Aggregated time series data (custom file format) Raw time series data
- 31. Example 31 Input data Output data Rollups pipeline Raw time series data Aggregated time series data (custom file format) 1. Aggregate high resolution data. 2. Store the aggregated data in our custom file format. 1 2
- 32. Example 32 Output data Checkpoint data Input data Vertical split Aggregated time series data (Parquet format) Raw time series data Aggregated time series data (custom file format) 1 2 1. Aggregate high resolution data. 2. Store the aggregated data in our custom file format. 1 2
- 33. Example 33 Output data Checkpoint data Horizontal split Aggregated time series data (Parquet format) Raw time series data Aggregated time series data (custom file format) 1 2 A B C D 1. Aggregate high resolution data. 2. Store the aggregated data in our custom file format. 1 2
- 34. Example 34 A A Horizontal split B B C C D D Aggregated time series data (Parquet format) Raw time series data Aggregated time series data (custom file format) 1 2 1. Aggregate high resolution data. 2. Store the aggregated data in our custom file format. 1 2
- 35. Break down jobs into smaller pieces 35 Fault tolerance Performance
- 36. Lessons • Many clusters for better isolation. • Break down jobs into smaller pieces (no longer than ~3 hours). • Trade-off between performance and fault tolerance. 36
- 37. 1. Architecture 2. Monitoring 3. Failures handling Highly reliable data pipelines 37
- 38. Reliability is the probability that a system will produce correct outputs up to some given time t. 38
- 39. Reliability is the probability that a system will produce correct outputs up to some given time t. 39
- 40. Monitoring data pipelines 40 We monitor actively 3 types of metrics: • Data lags metrics. • Cluster health metrics. • Job health metrics. 1. Is the data pipeline going to finish before the deadline?
- 41. Monitoring data pipelines 41 We monitor actively 3 types of metrics: • Data lags metrics. • Cluster health metrics. • Job health metrics. 1. Is the data pipeline going to finish before the deadline?
- 42. Monitoring data pipelines 42 1. Is the data pipeline going to finish before the deadline?
- 43. Monitoring data pipelines 43 We monitor actively 3 types of metrics: • Data lags metrics. • Cluster health metrics. • Job health metrics. 1. Is the data pipeline going to finish before the deadline?
- 44. Monitoring data pipelines 44 1. Is the data pipeline going to finish before the deadline?
- 45. Monitoring data pipelines 45 1. Is the data pipeline going to finish before the deadline?
- 46. Monitoring data pipelines 46 We monitor actively 3 types of metrics: • Data lags metrics. • Cluster health metrics. • Job health metrics. 1. Is the data pipeline going to finish before the deadline?
- 47. Monitoring data pipelines 47 1. Is the data pipeline going to finish before the deadline?
- 48. Monitoring data pipelines 48 1. Is the data pipeline going to finish before the deadline?
- 49. Monitoring data pipelines 49 1. Is the data pipeline going to finish before the deadline?
- 50. Monitoring data pipelines 50 ● Add custom counters throughout the pipelines. ○ Count records. ○ Count duplicates. ○ Count records that can’t join. ● Ad hoc checks on the output data. 2. Is the data produced correct?
- 51. Lessons • Monitoring = will we finish before t? + is the data correct? • Measure, measure and measure! • Alert on meaningful and actionable metrics. 51
- 52. 1. Architecture 2. Monitoring 3. Failures handling Highly reliable data pipelines 52
- 53. 53
- 54. Data pipelines will break Hardware failures Bad code changes 54 Upstream delays Increasing volume of data
- 55. Data pipelines will break 1. Recover fast We want to fix the issues ASAP. 2. Degrade gracefully We want to limit the customer-facing impact. 55
- 56. Recover fast • No long running job. • Switch from spot to on-demand clusters. • Increase cluster size. • Easy ways to rerun jobs (not always trivial!). 56
- 57. Example: rerun the rollups pipeline 2018-01 2018-02 2018-03 2018-04 2018-05 s3://bucket/ 57
- 58. Example: rerun the rollups pipeline as-of_2018-05-01 as-of_2018-05-02 ... as-of_2018-05-21 s3://bucket/2018-05/ 58
- 59. Example: rerun the rollups pipeline as-of_2018-05-01 as-of_2018-05-02 ... as-of_2018-05-21 s3://bucket/2018-05/ Active location 59
- 60. Example: rerun the rollups pipeline as-of_2018-05-01 as-of_2018-05-02 ... as-of_2018-05-21 as-of_2018-05-22 s3://bucket/2018-05/ Active location 60
- 61. Example: rerun the rollups pipeline as-of_2018-05-01 as-of_2018-05-02 ... as-of_2018-05-21 as-of_2018-05-22 s3://bucket/2018-05/ Active location 61
- 62. Example: rerun the rollups pipeline as-of_2018-05-01 as-of_2018-05-02 ... as-of_2018-05-21 as-of_2018-05-22 s3://bucket/2018-05/ Active location 62
- 63. Example: rerun the rollups pipeline as-of_2018-05-01 as-of_2018-05-02 ... as-of_2018-05-21 as-of_2018-05-22 as-of_2018-05-22_run-2 s3://bucket/2018-05/ Active location 63
- 64. Degrade 64 A A B B C C D D gracefully • Isolate issues to a limited number of customers thanks to horizontal sharding. • Keep the functionalities operational at the cost of performance/accuracy.
- 65. Lessons • Think about potential issues ahead of time. • Have knobs ready to recover fast. • Have knobs ready to limit the customer facing impact. 65
- 66. Conclusion 66 Building highly reliable data pipelines
- 67. Conclusion • Know your time constraints. 67 Building highly reliable data pipelines
- 68. Conclusion • Know your time constraints. • Break down jobs into small survivable pieces. 68 Building highly reliable data pipelines
- 69. Conclusion • Know your time constraints. • Break down jobs into small survivable pieces. • Monitor cluster metrics, job metrics and data lags. 69 Building highly reliable data pipelines
- 70. Conclusion • Know your time constraints. • Break down jobs into small survivable pieces. • Monitor cluster metrics, job metrics and data lags. • Think about failures ahead of time and get prepared. 70 Building highly reliable data pipelines