State Management in Apache Flink : Consistent Stateful Distributed Stream Processing
7 likes2,498 views
The document presents an overview of state management in Apache Flink, focusing on consistent stateful distributed stream processing. It covers topics such as the system architecture, pipelined consistent snapshots, and operations with snapshots aimed at large-scale deployments. Additionally, it discusses the implications of state reconfiguration, version control, and the importance of maintaining end-to-end processing guarantees.
State Management in Apache Flink : Consistent Stateful Distributed Stream Processing
- 1. Paris Carbone<parisc@kth.se> - KTH Royal Institute of Technology Stephan Ewen<stephan@data-artisans.com> - data Artisans Gyula FĆ³ra<gyula.fora@king.com> - King Digital Entertainment Ltd Seif Haridi<haridi@kth.se> - KTH Royal Institute of Technology Stefan Richter<s.richter@data-artisans.com> - data Artisans Kostas Tzoumas<kostas@data-artisans.com> - data Artisans 1 State Management in Apache FlinkĀ® Consistent Stateful Distributed Stream Processing @vldb17
- 2. Overview ā¢ The Apache Flink System Architecture ā¢ Pipelined Consistent Snapshots ā¢ Operations with Snapshots ā¢ Large Scale Deployments and Evaluation 2
- 3. The Apache Flink Framework Cluster Backend Metrics Dataļ¬ow Runtime DataStream DataSet SQL Table CEP Graphs ML Libraries Core API Runner Setup 3
- 4. Distributed Architecture Cluster Backend Metrics Dataļ¬ow Runtime DataStream DataSet SQL Table CEP Graphs ML Libraries Core API Runner Setup Job Manager Task Manager Task Manager ā¦. Client 4
- 5. Distributed Architecture Cluster Backend Metrics Dataļ¬ow Runtime DataStream DataSet SQL Table CEP Graphs ML Libraries Core API Runner Setup Job Manager Task Manager Task Manager ā¦. Client 4
- 6. Distributed Architecture Cluster Backend Metrics Dataļ¬ow Runtime DataStream DataSet SQL Table CEP Graphs ML Libraries Core API Runner Setup Job Manager Task Manager Task Manager ā¦. Client optimised logical graph 4
- 7. Distributed Architecture Cluster Backend Metrics Dataļ¬ow Runtime DataStream DataSet SQL Table CEP Graphs ML Libraries Core API Runner Setup Job Manager Task Manager Task Manager ā¦. ā¢ scheduling ā¢ state partitioning ā¢ snapshot coordination Client optimised logical graph 4
- 8. Zookeeper ā¢ passive failover ā¢ snapshot metadata Distributed Architecture Cluster Backend Metrics Dataļ¬ow Runtime DataStream DataSet SQL Table CEP Graphs ML Libraries Core API Runner Setup Job Manager Task Manager Task Manager ā¦. ā¢ scheduling ā¢ state partitioning ā¢ snapshot coordination Client optimised logical graph 4
- 9. Zookeeper ā¢ passive failover ā¢ snapshot metadata Distributed Architecture Cluster Backend Metrics Dataļ¬ow Runtime DataStream DataSet SQL Table CEP Graphs ML Libraries Core API Runner Setup Job Manager Task Manager Task Manager ā¦. ā¢ scheduling ā¢ state partitioning ā¢ snapshot coordination Client optimised logical graph ā¢ memory management ā¢ local snapshot execution ā¢ ļ¬ow control physical long-running tasks 4
- 10. Zookeeper ā¢ passive failover ā¢ snapshot metadata Distributed Architecture Cluster Backend Metrics Dataļ¬ow Runtime DataStream DataSet SQL Table CEP Graphs ML Libraries Core API Runner Setup Job Manager Task Manager Task Manager ā¦. ā¢ scheduling ā¢ state partitioning ā¢ snapshot coordination Client optimised logical graph ā¢ memory management ā¢ local snapshot execution ā¢ ļ¬ow control physical long-running tasks locally managed state 4
- 11. Zookeeper ā¢ passive failover ā¢ snapshot metadata Distributed Architecture Cluster Backend Metrics Dataļ¬ow Runtime DataStream DataSet SQL Table CEP Graphs ML Libraries Core API Runner Setup Job Manager Task Manager Task Manager ā¦. ā¢ scheduling ā¢ state partitioning ā¢ snapshot coordination Client optimised logical graph ā¢ memory management ā¢ local snapshot execution ā¢ ļ¬ow control physical long-running tasks locally managed state External Snapshot Store (e.g., hdfs) partial snapshots 4
- 12. 1. End-to-End Guarantees Snapshots 2. Reconļ¬guration 3. Version Control 4. Isolation Snapshots 5
- 13. 1. End-to-End Guarantees Snapshots 2. Reconļ¬guration 3. Version Control 4. Isolation Snapshots 6
- 15. Stateful Processing tasktasktask invoke per input record 7
- 16. Stateful Processing tasktasktask readwrite managed state logical operations (collections) invoke per input record 7
- 17. Local State Backend physical operations In-Memory(Heap) Embedded Off-heap+Disk Key-Value Store (RocksDB) Stateful Processing tasktasktask readwrite managed state logical operations (collections) invoke per input record 7
- 18. Local State Backend physical operations In-Memory(Heap) Embedded Off-heap+Disk Key-Value Store (RocksDB) Stateful Processing tasktasktask readwrite managed state logical operations (collections) invoke per input record state = f(input) 7
- 19. 8
- 23. local statesinput streams capture all local states after completing an epoch divide computation into epochs stream processor 8
- 24. local statesinput streams capture all local states after completing an epoch divide computation into epochs stream processor can rollback input and state to captured point in the past 8
- 25. Snapshot Store copy states A Synchronous Approach master 9
- 26. drain epoch 1 Snapshot Store copy states A Synchronous Approach master 9
- 27. drain epoch 1 Snapshot Store copy states A Synchronous Approach master 9
- 28. drain epoch 1 Snapshot Store copy states A Synchronous Approach master 9
- 29. drain epoch 2 Snapshot Store copy states A Synchronous Approach master 9
- 30. drain epoch 2 Snapshot Store copy states A Synchronous Approach master 9
- 31. drain epoch 2 Snapshot Store copy states A Synchronous Approach master 9
- 32. ā¢ In use: Storm Trident and Spark Streaming ā¢ A conservative approach, equivalent to batching ā¢ Can cause unnecessary latency (master coordination) ā¢ Processing is no longer continuous ā¢ Forces many tasks to be idle ā¢ Instead, in Apache Flink snapshots are pipelined Synchronous Snapshots 10
- 33. Pipelined Snapshots Snapshot Store async state copy 11
- 34. Pipelined Snapshots Snapshot Store async state copy insert markers 11
- 35. Pipelined Snapshots Snapshot Store async state copy insert markers A B C D E 11
- 36. Pipelined Snapshots Snapshot Store async state copy A B C D E 11
- 37. Pipelined Snapshots Snapshot Store async state copy A B C D E B 11
- 38. Pipelined Snapshots Snapshot Store async state copy epoch alignment A B C D E B 11
- 39. Pipelined Snapshots Snapshot Store async state copy epoch alignment A B C D E B A 11
- 40. Pipelined Snapshots Snapshot Store async state copy A B C D E B A C 11
- 41. Pipelined Snapshots Snapshot Store async state copy A B C D E B A C D E 11
- 42. Pipelined Snapshots Snapshot Store async state copy snapshot completes A B C D E B A C D E 11
- 44. Pipelined Snapshots (cycles) Problem: we cannot wait indeļ¬nitely for records in cycles 12
- 45. Pipelined Snapshots (cycles) Problem: we cannot wait indeļ¬nitely for records in cycles Solution: log in snapshot inļ¬ight records within a cycle Replay upon recovery. 12
- 46. ā¢ Offers exactly-once processing guarantees ā¢ Issued periodically/externally by the user ā¢ Naturally respects ļ¬ow control mechanisms ā¢ Channel state logging limited to cycles only ā¢ Multiple epoch snapshots can be pipelined ā¢ Can offer weaker at-least-once processing guarantees by simply dropping aligning vs no alignment cost Technique Highlights 13
- 47. 1. End-to-End Guarantees Snapshots 2. Reconļ¬guration 3. Version Control 4. Isolation Snapshots Usages 14
- 48. Exactly-Once: Input and Processing Important Assumptions ā¢ Input streams are persisted with offset indexes (e.g., Kafka, Kinesis) ā¢ Data Channels are FIFO and reliable (no loss) Each epoch either completes or repeats 15
- 49. ā¢ Idempontency ~ repeated operations can be tolerated after recovery/rollback (works for mutable stores). ā¢ Transactional Processing ~ Requires a two-phase coordination. A snapshot completion eventually leads to external commit (e.g., Flinkās HDFS RollingSink*) in-progress committedpendingpending epoch n-1 epoch n-2 epoch n-3epoch n Exactly-Once Output 16
- 50. Snapshots 2. Reconļ¬guration 3. Version Control 4. Isolation Snapshots Usages 1. End-to-End Guarantees 17
- 55. Dataļ¬ow Reconļ¬guration stop snap-1 snap-2 snap-3 ā¦ change parallelism Problem: How is state repartitioned from a snapshot? 18
- 57. Reconļ¬guration: The Issue 0x100: bob ā¦ ā¦ ā¦ ā¦ 0x449: alice reconļ¬gure case I full scan Scan Remote Storage for Responsible Keys 19
- 58. Reconļ¬guration: The Issue 0x100: bob ā¦ ā¦ ā¦ ā¦ 0x449: alice reconļ¬gure case I full scan Scan Remote Storage for Responsible Keys too slow 19
- 59. Reconļ¬guration: The Issue case II 0x100: bob ā¦ ā¦ ā¦ ā¦ 0x449: alice reconļ¬gure Include Key Locations in Snapshot Metadata bob: 0x100 carol: 0x344 ā¦ alice: 0x449 chuck: 0x630 ā¦ 0x100: bob ā¦ ā¦ ā¦ ā¦ 0x449: alice reconļ¬gure case I full scan Scan Remote Storage for Responsible Keys too slow 19
- 60. Reconļ¬guration: The Issue case II 0x100: bob ā¦ ā¦ ā¦ ā¦ 0x449: alice reconļ¬gure Include Key Locations in Snapshot Metadata bob: 0x100 carol: 0x344 ā¦ alice: 0x449 chuck: 0x630 ā¦ 0x100: bob ā¦ ā¦ ā¦ ā¦ 0x449: alice reconļ¬gure case I full scan Scan Remote Storage for Responsible Keys too slow too much 19
- 61. Reconļ¬guration: Key Groups Pre-partition state in hash(K) space, into key-groups bobā¦ ā¦ ā¦ ā¦ ā¦ ā¦ alice 20
- 62. Reconļ¬guration: Key Groups Pre-partition state in hash(K) space, into key-groups bobā¦ ā¦ ā¦ ā¦ ā¦ ā¦ ā¢ Snapshot Metadata: Contains a reference per stored Key-Group (less metadata) ā¢ Reconļ¬guration: Contiguous key-group allocation to available tasks (less IO) alice 20
- 63. Reconļ¬guration: Key Groups Pre-partition state in hash(K) space, into key-groups bobā¦ ā¦ ā¦ ā¦ ā¦ ā¦ ā¢ Snapshot Metadata: Contains a reference per stored Key-Group (less metadata) ā¢ Reconļ¬guration: Contiguous key-group allocation to available tasks (less IO) alice Note: number of key groups controls trade-off between metadata to keep and reconļ¬guration speed 20
- 64. Snapshots 2. Reconļ¬guration 3. Version Control 4. Isolation Snapshots Usages 1. End-to-End Guarantees 21
- 67. Version Control fork and update Pipeline v.1 Pipeline v.2 22
- 68. Version Control fork and update Pipeline v.1 Pipeline v.2 22
- 69. Version Control fork and update Pipeline v.1 Pipeline v.3 Pipeline v.2 22
- 70. Version Control fork and update Pipeline v.1 Pipeline v.3 Pipeline v.2 22
- 71. Snapshots 2. Reconļ¬guration 3. Version Control 4. Isolation Snapshots Usages 1. End-to-End Guarantees 23
- 73. Isolation Levels select from facebook.userID, clients.name ā¦ inner join clients on ā¦ read-committed (snapshot) read-uncommitted (dirty read on latest state) external query 24
- 74. Large Scale Deployment at King 25
- 75. Large Scale Deployment at King100 200 300 400 500 Global State Size (GB) 0 50 100 150 200 250 TotalSnapshottingTime(sec) total time / snapshot (alignment + async copies) 25
- 76. Large Scale Deployment at King100 200 300 400 500 Global State Size (GB) 0 50 100 150 200 250 TotalSnapshottingTime(sec) total time / snapshot (alignment + async copies) ~runtime overhead 25
- 77. Large Scale Deployment at King 30 50 70 Parallelism 0 200 400 600 800 1000 1200 1400 TotalAlignmentTime(msec) PROC WIN OUT alignment cost 100 200 300 400 500 Global State Size (GB) 0 50 100 150 200 250 TotalSnapshottingTime(sec) total time / snapshot (alignment + async copies) ~runtime overhead 25
- 78. Large Scale Deployment at King 30 50 70 Parallelism 0 200 400 600 800 1000 1200 1400 TotalAlignmentTime(msec) PROC WIN OUT alignment cost 100 200 300 400 500 Global State Size (GB) 0 50 100 150 200 250 TotalSnapshottingTime(sec) total time / snapshot (alignment + async copies) ~runtime overhead 25
- 79. Large Scale Deployment at King 30 50 70 Parallelism 0 200 400 600 800 1000 1200 1400 TotalAlignmentTime(msec) PROC WIN OUT alignment cost 100 200 300 400 500 Global State Size (GB) 0 50 100 150 200 250 TotalSnapshottingTime(sec) total time / snapshot (alignment + async copies) ~runtime overhead ā¢ #shuffles (keyby) ā¢ parallelism 25
- 80. Teaser: More paper highlights ā¢ We can use the same technique to coordinate externally managed state with snapshots. ā¢ Epoch markers can act as on-the-ļ¬y reconļ¬guration points. ā¢ Internals of asynchronous and incremental snapshots. 26
- 81. Paris Carbone<parisc@kth.se> - KTH Royal Institute of Technology Stephan Ewen<stephan@data-artisans.com> - data Artisans Gyula FĆ³ra<gyula.fora@king.com> - King Digital Entertainment Ltd Seif Haridi<haridi@kth.se> - KTH Royal Institute of Technology Stefan Richter<s.richter@data-artisans.com> - data Artisans Kostas Tzoumas<kostas@data-artisans.com> - data Artisans 27 State Management in Apache FlinkĀ® Consistent Stateful Distributed Stream Processing @vldb17