Flink Forward SF 2017: Malo Deniélou - No shard left behind: Dynamic work rebalancing and Autoscaling in Apache Beam
0 likes271 views
Apache Beam is a unified programming model for efficient and portable data processing pipelines, supporting various SDKs and runners. It addresses challenges like dynamic work rebalancing and the straggler problem, which affect processing time and resource utilization. Beam's abstractions empower runners to optimize performance through adaptive execution based on workload characteristics.
![Common straggler mitigation techniques
● Split files into equal sizes?
● Pre-emptively over split?
● Detect slow workers and reexecute?
● Sample the data and split based on partial execution
All have major costs, but do not solve completely the problem.
Workers
Time
« The most straightforward way to tune the number of partitions is experimentation:
Look at the number of partitions in the parent RDD and then keep multiplying that
by 1.5 until performance stops improving. »
From [blog]how-to-tune-your-apache-spark-jobs](https://image.slidesharecdn.com/flinkforwardsf2017malodenielounoshardsleftbehind-170414104330/85/Flink-Forward-SF-2017-Malo-Denielou-No-shard-left-behind-Dynamic-work-rebalancing-and-Autoscaling-in-Apache-Beam-10-320.jpg)
Flink Forward SF 2017: Malo Deniélou - No shard left behind: Dynamic work rebalancing and Autoscaling in Apache Beam
- 1. Confidential + Proprietary No shard left behind Dynamic Work Rebalancing and other adaptive features in Apache Beam Malo Denielou (malo@google.com)
- 2. Apache Beam is a unified programming model designed to provide efficient and portable data processing pipelines.
- 3. Apache Beam 1. The Beam Programming Model 2. SDKs for writing Beam pipelines -- Java/Python/... 3. Runners for existing distributed processing backends ○ Apache Flink ○ Apache Spark ○ Apache Apex ○ Dataflow ○ Direct runner (for testing) Beam Model: Fn Runners Apache Flink Apache Spark Beam Model: Pipeline Construction Other LanguagesBeam Java Beam Python Execution Execution Cloud Dataflow Execution
- 4. 1.Classic Batch 2. Batch with Fixed Windows 3. Streaming 5. Streaming With Retractions 4. Streaming with Speculative + Late Data 6. Streaming With Sessions Apache Beam use cases
- 5. Data processing for realistic workloads Workload Time Streaming pipelines have variable input Batch pipelines have stages of different sizes
- 6. The curse of configuration Workload Time Workload Time Over-provisioning resources? Under-provisioning on purpose? A considerable effort is spent to finely tune all the parameters of the jobs.
- 7. Ideal case Workload Time A system that adapts.
- 8. The straggler problem in batchWorkers Time Tasks do not finish evenly on the workers. ● Data is not evenly distributed among tasks ● Processing time is uneven between tasks ● Runtime constraints Effects are cumulative per stage!
- 9. Common straggler mitigation techniques ● Split files into equal sizes? ● Pre-emptively over split? ● Detect slow workers and reexecute? ● Sample the data and split based on partial execution All have major costs, but do not solve completely the problem. Workers Time
- 10. Common straggler mitigation techniques ● Split files into equal sizes? ● Pre-emptively over split? ● Detect slow workers and reexecute? ● Sample the data and split based on partial execution All have major costs, but do not solve completely the problem. Workers Time « The most straightforward way to tune the number of partitions is experimentation: Look at the number of partitions in the parent RDD and then keep multiplying that by 1.5 until performance stops improving. » From [blog]how-to-tune-your-apache-spark-jobs
- 11. No amount of upfront heuristic tuning (be it manual or automatic) is enough to guarantee good performance: the system will always hit unpredictable situations at run-time. A system that's able to dynamically adapt and get out of a bad situation is much more powerful than one that heuristically hopes to avoid getting into it. Fine-tuning execution parameters goes against having a truly portable and unified programming environment.
- 12. Beam abstractions empower runners A bundle is group of elements of a PCollection processed and committed together. APIs (ParDo/DoFn): • setup() • startBundle() • processElement() n times • finishBundle() • teardown() Streaming runner: • small bundles, low-latency pipelining across stages, overhead of frequent commits. Classic batch runner: • large bundles, fewer large commits, more efficient, long synchronous stages. Other runner strategies may strike a different balance.
- 13. Beam abstractions empower runners Efficiency at runner’s discretion “Read from this source, splitting it 1000 ways” ➔ user decides “Read from this source” ➔ runner decides APIs for portable Sources: • long getEstimatedSize() • List<Source> splitIntoBundles(size)
- 14. Beam abstractions empower runners Efficiency at runner’s discretion “Read from this source, splitting it 1000 ways” ➔ user decides “Read from this source” ➔ runner decides APIs: • long getEstimatedSize() • List<Source> splitIntoBundles(size) Runner Source gs://logs/* Size?
- 15. Beam abstractions empower runners Efficiency at runner’s discretion “Read from this source, splitting it 1000 ways” ➔ user decides “Read from this source” ➔ runner decides APIs: • long getEstimatedSize() • List<Source> splitIntoBundles(size) Runner Source gs://logs/* 50TB
- 16. Beam abstractions empower runners Efficiency at runner’s discretion “Read from this source, splitting it 1000 ways” ➔ user decides “Read from this source” ➔ runner decides APIs: • long getEstimatedSize() • List<Source> splitIntoBundles(size) Runner (cluster utilization, quota, bandwidth, throughput, concurrent stages, …) Source gs://logs/* Split in chunks of 500GB
- 17. Beam abstractions empower runners Efficiency at runner’s discretion “Read from this source, splitting it 1000 ways” ➔ user decides “Read from this source” ➔ runner decides APIs: • long getEstimatedSize() • List<Source> splitIntoBundles(size) Runner Source gs://logs/* List<Source>
- 18. Solving the straggler problem: Dynamic Work Rebalancing
- 19. Solving the straggler problem: Dynamic Work Rebalancing Workers Time Done work Active work Predicted completion Now Average
- 20. Solving the straggler problem: Dynamic Work Rebalancing Workers Time Done work Active work Predicted completion Now Average
- 21. Solving the straggler problem: Dynamic Work Rebalancing Workers Time Done work Active work Predicted completion Now Average Workers Time Now Average
- 22. Solving the straggler problem: Dynamic Work Rebalancing Workers Time Done work Active work Predicted completion Now Average Workers Time Now Average
- 23. Dynamic Work Rebalancing in the wild A classic MapReduce job (read from Google Cloud Storage, GroupByKey, write to Google Cloud Storage), 400 workers. Dynamic Work Rebalancing disabled to demonstrate stragglers. X axis: time (total ~20min.); Y axis: workers
- 24. Dynamic Work Rebalancing in the wild A classic MapReduce job (read from Google Cloud Storage, GroupByKey, write to Google Cloud Storage), 400 workers. Dynamic Work Rebalancing disabled to demonstrate stragglers. X axis: time (total ~20min.); Y axis: workers Same job, Dynamic Work Rebalancing enabled. X axis: time (total ~15min.); Y axis: workers
- 25. Dynamic Work Rebalancing in the wild A classic MapReduce job (read from Google Cloud Storage, GroupByKey, write to Google Cloud Storage), 400 workers. Dynamic Work Rebalancing disabled to demonstrate stragglers. X axis: time (total ~20min.); Y axis: workers Same job, Dynamic Work Rebalancing enabled. X axis: time (total ~15min.); Y axis: workers Savings
- 26. Dynamic Work Rebalancing with Autoscaling Initial allocation of 80 workers, based on size Multiple rounds of upsizing, enabled by dynamic work rebalancing Upscales to 1000 workers. ● tasks are balanced ● no oversplitting or manual tuning
- 27. Apache Beam enable dynamic adaptation Beam Source Readers provide simple progress signals, which enable runners to take action based on execution-time characteristics. All Beam runners can implement Autoscaling and Dynamic Work Rebalancing. APIs for how much work is pending. • bounded: double getFractionConsumed() • unbounded: long getBacklogBytes() APIs for splitting: • bounded: • Source splitAtFraction(double) • int getParallelismRemaining() • unbounded: • Coming soon ...
- 28. Apache Beam is a unified programming model designed to provide efficient and portable data processing pipelines.
- 29. To learn more Read our blog posts! • No shard left behind: Dynamic work rebalancing in Google Cloud Dataflow https://cloud.google.com/blog/big-data/2016/05/no-shard-left-behind-dyna mic-work-rebalancing-in-google-cloud-dataflow • Comparing Cloud Dataflow autoscaling to Spark and Hadoop https://cloud.google.com/blog/big-data/2016/03/comparing-cloud-dataflow- autoscaling-to-spark-and-hadoop Join the Apache Beam community! https://beam.apache.org/