Message reliability in Kafka
- 1. Message Reliability in Kafka Mihai Ghita Data Engineer
- 2. ● Delivery guarantees provided by Kafka: ○ At most once ○ At least once ○ Exactly once ● Kafka features: ○ Kafka Idempotent Producer ○ Kafka Transactional Consumer and Producer Outline
- 3. Kafka Producer - Example
- 4. Kafka Producer - Example
- 5. Kafka Consumer - Example
- 6. Kafka Consumer - Example
- 7. Delivery Guarantees At most once At least once Exactly once Message Delivery Guarantees
- 8. “At Most Once” Guarantee
- 9. Kafka Consumer - “At Most Once” Guarantee Consumer 1 2 3 4 5 6 7 8 9 Batch Size 1.Read records 2.Commit unprocessed records Kafka 3. Data Processing
- 10. Kafka Consumer - “At Most Once” Guarantee Consumer 1 2 3 4 5 6 7 8 9 Batch Size 1.Read records 2.Commit unprocessed records Kafka Consumer 4.Read next records 5.Commit records Batch Size 3. Data Processing 6. Data Processing
- 11. Kafka Producer - “At Most Once” Guarantee (x,y) Producer Broker (x,y) Producer Broker 1. Send(x,y) 2. Append(x,y) 1. Send(a,b) 2. Append(a,b)
- 12. “At Least Once” Guarantee
- 13. Kafka Consumer - “At Least Once” Guarantee Consumer 1 2 3 4 5 6 7 8 9 Batch Size 1.Read records 3.Commit records Kafka 2. Data Processing
- 14. Kafka Consumer - “At Least Once” Guarantee Consumer 1 2 3 4 5 6 7 8 9 Batch Size 1.Read records 3.Commit records Kafka Consumer 4.Read next records 6.Commit records Batch Size 2. Data Processing 5. Data Processing
- 15. Kafka Producer - “At Least Once” Guarantee (x,y) Producer Broker (x,y) (a,b) Producer (a,b) Broker 1. Send(x,y) 2. Append(x,y) 3. Ack 1. Send(a,b) 2. Append(a,b) 3. Ack 5. Append(a,b) 4. Resend(a,b) 6. Ack
- 18. Acknowledgement failure leads to message duplication (x,y) Producer Broker (x,y) (a,b) Producer (a,b) Broker 1. Send(x,y) 2. Append(x,y) 3. Ack 1. Send(a,b) 2. Append(a,b) 3. Ack 5. Append(a,b) 4. Resend(a,b) 6. Ack
- 19. The idempotent producer will eliminate message duplication from producer side (x,y) Seq = 0 PID = 100 Producer Broker 1. Send(x,y) Seq = 0 PID = 100 2. Append(x,y) Seq = 0 PID = 100 3. Ack (x,y) Seq = 0 PID = 100 Producer Broker 4. Send(a,b) Seq = 1 PID = 100 2. Append(a,b) Seq = 1 PID = 100 3. Ack (a,b) Seq = 1 PID = 100 5. Ack-Duplicate 1. Send(a,b) Seq = 1 PID = 100
- 21. Producer Topic A, Partition 0 Topic B, Partition 0 Topic B, Partition 1 Transactional Producer
- 22. m1 m5 m3 m4 m6 m2 Producer Topic A, Partition 0 Topic B, Partition 0 Topic B, Partition 1 Transactional Producer
- 23. Transactional Producer m1 m5 C m3 m4 m6 C m2 C Producer Topic A, Partition 0 Topic B, Partition 0 Topic B, Partition 1 atomic commit
- 24. m1 m5 A m3 m4 m6 A m2 A Producer Topic A, Partition 0 Topic B, Partition 0 Topic B, Partition 1 atomic abort Transactional Producer
- 25. m1 m5 C m3 m4 m6 C m2 C Consumer Topic A, Partition 0 Topic B, Partition 0 Topic B, Partition 1 read committed Transactional Consumer
- 26. m11 m12 C m3 A m8 C A m2 m9 C Consumer Topic A, Partition 0 Topic B, Partition 0 Topic B, Partition 1 read committed Transactional Consumer
- 27. Why are the Transactions Important? Read Write Process Input Output
- 28. Write all results Mark all messages as consumed Write to the output topic Write offsets to the commit topic Why are the Transactions Important? All or nothing!
- 30. Kafka Transactions - Demo
- 31. ● Delivery guarantees provided by Kafka: ○ At most once ○ At least once ○ Exactly once ● Depending on your needs, choose the right delivery guarantee Putting it all Together
- 33. Kafka Transactional - Internal Implementation
- 34. Kafka Consumer - Internal Implementation