Till Rohrmann - Dynamic Scaling - How Apache Flink adapts to changing workloads
2 likes1,578 views
The document discusses Apache Flink's dynamic scaling capabilities, focusing on adapting to changing workloads and service level agreements (SLAs). It details the processes for scaling both stateless and stateful jobs, including the management of keyed and non-keyed state. Future developments aim to enhance automatic rescaling and integration with various auto-scaling policies.
Till Rohrmann - Dynamic Scaling - How Apache Flink adapts to changing workloads
- 1. Till Rohrmann trohrmann@apache.org @stsffap Dynamic Scaling: How Apache Flink® Adapts to Changing Workloads
- 2. Changing Workloads And SLAs 2
- 4. What Is This Talk About? § Flink’s approach to dynamic scaling § Current state with demo § Outlook on next development steps 4
- 6. Basic Idea 6 • Spread work across more workers to decrease workload
- 7. Scaling Stateless Jobs 7 Scale Up Scale Down Source Mapper Sink • Scale up: Deploy new tasks • Scale down: Cancel running tasks
- 8. Scaling Stateful Jobs 8 ? • Problem: Which state to assign to new task?
- 9. State in Apache Flink 9
- 10. Keyed vs. Non-keyed State 10 • State bound to a key • E.g. Keyed UDF and window state • State bound to a subtask • E.g. Source state Keyed Non-keyed
- 11. Repartitioning Keyed State § Similar to consistent hashing § Split key space into key groups § Assign key groups to tasks 11 Key space Key group #1 Key group #2 Key group #3Key group #4
- 12. Repartitioning Keyed State contd. § Rescaling changes key group assignment § Maximum parallelism defined by #key groups 12
- 13. Repartitioning Non-keyed state § User defined merge and split functions • Most general approach § Breaking non-keyed state up into finer granularity • State has to contain multiple entries • Automatic repartitioning wrt granularity 13 #1 #2 #3
- 14. Repartitioning Non-keyed State contd. § Non-keyed state entries gathered at the job manager § Repartitioning schemes • Repartition & send • Union & broadcast 14
- 15. Example: Kafka Source 15 partitionId: 1, offset: 42 partitionId: 3, offset: 10 partitionId: 6, offset: 27 • Store offset for each partition • Individual entries are repartitionable partitionId: 1, offset: 42 partitionId: 3, offset: 10 partitionId: 6, offset: 27
- 16. Rescaling: Why is That so Hard? § Handling of state § Repartitioning of keyed & non-keyed state § Unique among open source stream processors, afaik 16
- 17. Demo Time 17
- 18. Demo Topology 18 Kafka Source Counter KeyBy
- 19. Current State and next Steps 19
- 20. Current State § Manual rescaling 1. Take savepoint 2. Stop the job 3. Restart job with adjusted parallelism and savepoint 20
- 21. Next Steps § Integrate savepoint with stop signal § Rescaling individual operators w/o restart § Dynamic container de-/allocation • “Running Flink Everywhere” by Stephan Ewen, 16:45 at Kesselhaus 21
- 22. Auto Scaling Policies 22 • Latency • Throughput • Resource utilization • Kubernetes on GCE, EC2 and Mesos (marathon- autoscale) already support auto-scaling
- 23. Conclusion § Scaling of keyed and non-keyed state § Flink supports manual rescaling with restart (WIP branch: https://github.com/tillrohrmann/flink/tree/partitionable-op-state) § Future versions might support scaling on the fly and automatic rescaling policies 23