Exactly Once Delivery with Kafka - Kafka Tel-Aviv Meetup
- 1. natans@wix.com twitter @NSilnitsky linkedin/natansilnitsky github.com/natansil Exactly Once Delivery is a harsh mistress Natan Silnitsky Backend Infra Developer, Wix.com
- 2. A Scala/Java high-level SDK for Apache Kafka. Greyhound
- 3. As it happens, we have a distributed system at Wix. ~1400 micro-services @NSilnitsky
- 4. As it happens, we have a distributed system at Wix. Edit MySite ~1400 micro-services ~1015M messages every day @NSilnitsky
- 5. PurchaseCompleted Classic ecommerce flow UpdateInventory(ItemN) updates UpdateInventory(Item1) UpdateInventory(Item2) @NSilnitsky Payments Checkout Inventory
- 6. Classic ecommerce flow - hard to achieve PurchaseCompleted UpdateInventory(ItemN) updates UpdateInventory(Item1) UpdateInventory(Item2) 😳 But... how do we update all items exactly once? on failures/restarts too... Payments Checkout Inventory
- 7. PurchaseCompleted updates UpdateInventory(Item1) UpdateInventory(Item2) Classic ecommerce flow - hard to achieve Inventory: Item1 9 → 8 Item2 5 → 4 @NSilnitsky
- 8. PurchaseCompleted updates UpdateInventory(Item1) UpdateInventory(Item2) Classic ecommerce flow - hard to achieve Inventory: Item1 9 → 7 Item2 5 → 3 Payments * use DB
- 9. Achieving Exactly-Once delivery in distributed systems is NOT easy.
- 10. Message delivery over the network is unreliable. @NSilnitsky
- 11. Message delivery over the network is unreliable. Message Consumer Message Producer Broker @NSilnitsky
- 12. Kafka Broker Topic-1 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 4 Topic-2 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 4 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 (append-only logs) Partitions @NSilnitsky
- 13. Kafka Consumer Kafka Producer Kafka Broker Topic-1 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 4 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 @NSilnitsky
- 14. The options for message delivery Why Exactly-Once is difficult Kafka’s solution for Exactly-Once delivery
- 15. Kafka Broker Kafka Producer 0 1 2 3 4 5 😕 Services need to address message duplicates at-most-onceat-least-once exactly-once Producer retries on every failure
- 16. Kafka Broker 0 1 2 3 4 5 😕 Services need to handle messages exactly once consumerRecords = Consumer.poll process(consumerRecords) consumer.commit at-most-onceat-least-once exactly-once Consumer retries on every failure Kafka Consumer
- 17. Kafka Broker 0 1 2 3 4 5 😕 Messages may be lost Consumer commits before processing (AutoCommit - Default) consumerRecords = Consumer.poll Consumer.commit process(consumerRecords) at-most-onceat-least-once exactly-once Kafka Consumer
- 18. Kafka Broker 0 1 2 3 4 5 Kafka Consumer 😕 Really really really hard to do Messages are read and processed exactly once by the consumer ? at-least-once exactly-onceat-most-once
- 19. The options for message delivery Why Exactly-Once is difficult Kafka’s solution for Exactly-Once delivery
- 20. The “Two Generals Problem" thought experiment Alice Bob
- 21. Tomorrow 7 AM The “Two Generals Problem" thought experiment Alice Bob
- 22. The “Two Generals Problem" thought experiment Alice Bob
- 23. The “Two Generals Problem" thought experiment Alice Bob Tomorrow 7 AM
- 24. OK! Tomorrow 7 AM The “Two Generals Problem" thought experiment Alice Bob
- 25. ? The “Two Generals Problem" thought experiment Alice Bob OK!
- 26. The “Two Generals Problem" thought experiment AliceService BobService
- 27. The options for message delivery Why Exactly-Once is difficult Kafka’s solution for Exactly-Once delivery
- 28. Kafka Broker Processor Observer Consumer -Producer Consumer Topic A Topic B @NSilnitsky
- 29. Kafka Broker Processor Observer Consumer -Producer Consumer Topic A Topic B @NSilnitsky
- 30. Kafka Broker Kafka Idempotent Producer 0 1 2 3 4 5 0 1 1 Attaches offset to message Enable.idempotence = true @NSilnitsky
- 31. Kafka Broker Kafka Idempotent Producer 0 1 2 3 4 5 0 1 1 Attaches offset to message Enable.idempotence = true duplicate! @NSilnitsky
- 32. Kafka Broker Topic A Topic B Processor Observer Consumer -Producer Consumer @NSilnitsky
- 33. Kafka Broker consumer.poll Attaches offset to message + marks transaction ABCDEF Processor Observer AB 01 Topic A Topic B @NSilnitsky isolation.level = "read_committed"
- 34. Kafka Broker consumer.poll producer.beginTransaction Producer.send Attaches offset to message + marks transaction ABCDEF Processor Observer ABC 01 Topic A Topic B @NSilnitsky
- 35. Kafka Broker consumer.poll producer.beginTransaction Producer.send producer.sendOffsets Attaches offset to message + marks transaction ABCDEF Processor Observer ABC 012 Topic A Topic B @NSilnitsky
- 36. Kafka Broker consumer.poll producer.beginTransaction Producer.send producer.sendOffsets producer.commitTransaction Attaches offset to message + marks transaction ABCDEF Processor Observer ABC 012 Topic A Topic B @NSilnitsky
- 37. Kafka Broker consumer.poll producer.beginTransaction Producer.send producer.sendOffsets Attaches offset to message + marks transaction ABCDEF Processor Observer ABC 01 Topic A Topic B @NSilnitsky
- 38. Kafka Broker Throughput impact is between 3% to 25% worse. Processor Observer @NSilnitsky
- 39. Kafka Broker Throughput impact is between 2% to 25% worse. The bigger the batch the better the throughput and longer the latency. Processor Observer @NSilnitsky
- 40. Kafka Broker Processor Observer No end-to-end exactly-once guarantees out of the box. @NSilnitsky
- 41. Kafka Broker Deduplicate using offsets from Kafka Transaction @NSilnitsky ABC 012 Topic B Observer DB Table Partition Offset Value 0 2 C ... ... ...
- 42. In reality, this is more complex than how I describe it. Out of scope: ▪ Two-phase-commit with transaction coordinator ▪ Transaction log ▪ Additional “fencing” data ▪ 1 processor-producer - 1 partition (up to Kafka 2.4.x) @NSilnitsky
- 43. Exactly Once in Kafka Streams StreamsConfig: processing.guarantee = "exactly_once" consumers will be configured with: ● isolation.level = "read_committed" producers will be configured with: ● enable.idempotence = true @NSilnitsky
- 44. Purchase Completed UpdateInventory(Item1) Exactly Once (Classic ecommerce) flow at Wix UpdateInventory(Item2) UpdateInventory(ItemN) ABCDEF 1 iNB 1 I3I8 I1I2I5I5 01 updates @NSilnitsky
- 45. Kafka Broker Exactly Once (Classic ecommerce) flow at Wix - Dedup @NSilnitsky Topic B Observer Inventory Partition Offset Item Value 0 0 I1 9->8 0 1 I2 5->401 I1I2 UPDATE offset 0, Item1 -1 UPDATE offset 1 Item2 - 1
- 46. Kafka Broker Exactly Once (Classic ecommerce) flow at Wix - Dedup @NSilnitsky Topic B Observer Inventory Partition Offset Item Value 0 0 I1 8 0 1 I2 401 I1I2 UPDATE offset 0, Item1 -1 UPDATE offset 1 Item2 - 1 UPDATE offset 1 Item2 - 1
- 47. Exactly-Once delivery is... like the holy grail of message delivery over the network. It’s a tough nut to crack.
- 48. Exactly-Once delivery is... complex to implement, to use, and it requires fine-tuning.
- 49. Exactly-Once delivery is... crucial for achieving atomic actions in distributed systems.
- 50. Thank You natans@wix.com twitter @NSilnitsky linkedin/natansilnitsky github.com/natansil
- 51. Resources EoS in Kafka talk by Jason Gustafson Exactly-once Semantics are Possible by Neha Narkhede (Performance) Transactions in Apache Kafka Revisiting Exactly One Semantics by Jason Gustafson, Confluent Proposal to improve EOS Produce scalability - Kafka 2.5 How akka works with exactly once message delivery by Hugh McKee
- 52. A Scala/Java high-level SDK for Apache Kafka. github.com/wix/greyhound