Till Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
5 likes7,420 views
Flink provides fault tolerance guarantees through checkpointing and recovery mechanisms. Checkpoints take consistent snapshots of distributed state and data, while barriers mark checkpoints in the data flow. This allows Flink to recover jobs from failures and resume processing from the last completed checkpoint. Flink also implements high availability by persisting metadata like the execution graph and checkpoints to Apache Zookeeper, enabling a standby JobManager to take over if the active one fails.
Till Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
- 1. Fault Tolerance and Job Recovery in Apache Flink™ Till Rohrmann trohrmann@apache.org @stsffap
- 2. 2
- 3. Better be safe than sorry Failures will happen EMC estimated $1.7 billion costs due to data loss and system downtime Recovery will save you time and costs Switch between algorithms Live upgrade of your system 3
- 5. Fault tolerance guarantees At most once • No guarantees at all At least once • For many applications sufficient Exactly once Flink provides all guarantees 5
- 6. Checkpoints Consistent snapshots of distributed data stream and operator state 6
- 7. Barriers Markers for checkpoints Injected in the data flow 7
- 8. 8 Alignment for multi-input operators
- 9. Operator State Stateless operators System state User defined state 9 ds.filter(_ != 0) ds.keyBy(0).window(TumblingTimeWindows.of(5, TimeUnit.SECONDS)) public class CounterSum implements RichReduceFunction<Long> { private OperatorState<Long> counter; @Override public Long reduce(Long v1, Long v2) throws Exception { counter.update(counter.value() + 1); return v1 + v2; } @Override public void open(Configuration config) { counter = getRuntimeContext().getOperatorState(“counter”, 0L, false); } }
- 10. 10
- 11. 11
- 12. 12
- 13. 13
- 14. Advantages Separation of app logic from recovery • Checkpointing interval is just a config parameter High throughput • Controllable checkpointing overhead Low impact on latency 14
- 15. 15
- 18. With high availability 18 JobManager TaskManager Stand-by JobManager Apache Zookeeper™ KEEP GOING
- 20. Persisting jobs 20 JobManager Client TaskManagers Apache Zookeeper™ 1. Submit job 2. Persist execution graph
- 21. Persisting jobs 21 JobManager Client TaskManagers Apache Zookeeper™ 1. Submit job 2. Persist execution graph 3. Write handle to ZooKeeper
- 22. Persisting jobs 22 JobManager Client TaskManagers Apache Zookeeper™ 1. Submit job 2. Persist execution graph 3. Write handle to ZooKeeper 4. Deploy tasks
- 23. Handling checkpoints 23 JobManager Client TaskManagers Apache Zookeeper™ 1. Take snapshots
- 24. Handling checkpoints 24 JobManager Client TaskManagers Apache Zookeeper™ 1. Take snapshots 2. Persist snapshots 3. Send handles to JM
- 25. Handling checkpoints 25 JobManager Client TaskManagers Apache Zookeeper™ 1. Take snapshots 2. Persist snapshots 3. Send handles to JM 4. Create global checkpoint
- 26. Handling checkpoints 26 JobManager Client TaskManagers Apache Zookeeper™ 1. Take snapshots 2. Persist snapshots 3. Send handles to JM 4. Create global checkpoint 5. Persist global checkpoint
- 27. Handling checkpoints 27 JobManager Client TaskManagers Apache Zookeeper™ 1. Take snapshots 2. Persist snapshots 3. Send handles to JM 4. Create global checkpoint 5. Persist global checkpoint 6. Write handle to ZooKeeper
- 28. Conclusion 28
- 29. 29
- 30. 30
- 31. TL;DL Job recovery mechanism with low latency and high throughput Exactly one processing semantics No single point of failure Flink will always keep processing your data 31
Editor's Notes
- #16: 30 nodes, 4 cores, 15 GB Flink 720,000 events per second per core 690,000 with checkpointing activated Storm With at-least-once: 2,600 events per second per core
- #34: GCE 30 instances with 4 cores and 15 GB of memory each. Flink master from July, 24th, Storm 0.9.3. All the code used for the evaluation can be found here. Flink 1.5 million elements per second per core Aggregate Throughput in cluster 182 million elements per second. Storm 82,000 elements per second per core Aggregate 0.57 million elements per second Storm with Acknowledge 4,700 elements per second per core, Latency 30-120 milliseconds Trident: 75,000 elements per second per core
- #35: Flink 0 Buffer timeout: latency median 0 msec, 99 %tile 20 msec 24,500 events per second per core