Exactly once delivery is a harsh mistress - DevOps Days TLV
- 1. natans@wix.com twitter @NSilnitsky linkedin/natansilnitsky github.com/natansil Exactly Once Delivery is a harsh mistress Natan Silnitsky Backend Infra Developer, Wix.com
- 2. As it happens, we have a distributed system at Wix. ~1400 micro-services @NSilnitsky
- 3. As it happens, we have a distributed system at Wix. Edit MySite ~1400 micro-services ~850M messages every day @NSilnitsky
- 5. Classic ecommerce flow - hard to achieve PurchaseCompleted UpdateInventory(ItemN) updates UpdateInventory(Item1) UpdateInventory(Item2) 😳 But... how do we update all items exactly once? on failures/restarts too...
- 6. PurchaseCompleted updates UpdateInventory(Item1) UpdateInventory(Item2) Classic ecommerce flow - hard to achieve Inventory: Item1 9 → 8 Item2 5 → 4 @NSilnitsky
- 7. PurchaseCompleted updates UpdateInventory(Item1) UpdateInventory(Item2) Classic ecommerce flow - hard to achieve Inventory: Item1 9 → 8 Item2 5 → 3 @NSilnitsky
- 8. Achieving Exactly-Once delivery in distributed systems is NOT easy.
- 9. Message delivery over the network is unreliable. @NSilnitsky
- 10. Message delivery over the network is unreliable. Message Consumer Message Producer Broker @NSilnitsky
- 11. Kafka Broker Topic-1 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 4 Topic-2 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 4 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 (append-only logs) Partitions @NSilnitsky
- 12. Kafka Consumer Kafka Producer Kafka Broker Topic-1 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 45 0 1 2 3 4 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 @NSilnitsky
- 13. The options for message delivery Why Exactly-Once is difficult Kafka’s solution for Exactly-Once delivery
- 14. Kafka Broker Kafka Producer 0 1 2 3 4 5 😕 Services need to address message duplicates at-most-onceat-least-once exactly-once Producer retries on every failure
- 15. Kafka Broker 0 1 2 3 4 5 😕 Services need to handle messages exactly once consumerRecords = Consumer.poll process(consumerRecords) consumer.commit at-most-onceat-least-once exactly-once Consumer retries on every failure Kafka Consumer
- 16. Kafka Broker 0 1 2 3 4 5 😕 Messages may be lost Consumer commits before processing consumerRecords = Consumer.poll Consumer.commit process(consumerRecords) at-most-onceat-least-once exactly-once Kafka Consumer
- 17. Kafka Broker 0 1 2 3 4 5 Kafka Consumer 😕 Really really really hard to do Messages are read and processed exactly once by the consumer ? at-least-once exactly-onceat-most-once
- 18. The options for message delivery Why Exactly-Once is difficult Kafka’s solution for Exactly-Once delivery
- 19. The “Two Generals Problem" thought experiment Alice Bob
- 20. Tomorrow 7 AM The “Two Generals Problem" thought experiment Alice Bob
- 21. The “Two Generals Problem" thought experiment Alice Bob
- 22. The “Two Generals Problem" thought experiment Alice Bob Tomorrow 7 AM
- 23. OK! Tomorrow 7 AM The “Two Generals Problem" thought experiment Alice Bob
- 24. ? The “Two Generals Problem" thought experiment Alice Bob OK!
- 25. The “Two Generals Problem" thought experiment AliceService BobService
- 26. The options for message delivery Why Exactly-Once is difficult Kafka’s solution for Exactly-Once delivery
- 27. Kafka Broker Processor Observer Consumer -Producer Consumer Topic A Topic B @NSilnitsky
- 28. Kafka Broker Processor Observer Consumer -Producer Consumer Topic A Topic B @NSilnitsky
- 29. Kafka Broker Kafka Idempotent Producer 0 1 2 3 4 5 0 1 1 Attaches offset to message Enable.idempotence = true @NSilnitsky
- 30. Kafka Broker Kafka Idempotent Producer 0 1 2 3 4 5 0 1 1 Attaches offset to message Enable.idempotence = true duplicate! @NSilnitsky
- 31. Kafka Broker Topic A Topic B Processor Observer Consumer -Producer Consumer @NSilnitsky
- 32. Kafka Broker consumer.poll Attaches offset to message + marks transaction ABCDEF Processor Observer AB 01 Topic A Topic B @NSilnitsky
- 33. Kafka Broker consumer.poll producer.beginTransaction Producer.send Attaches offset to message + marks transaction ABCDEF Processor Observer ABC 01 Topic A Topic B @NSilnitsky
- 34. Kafka Broker consumer.poll producer.beginTransaction Producer.send producer.sendOffsets Attaches offset to message + marks transaction ABCDEF Processor Observer ABC 012 Topic A Topic B @NSilnitsky
- 35. Kafka Broker consumer.poll producer.beginTransaction Producer.send producer.sendOffsets producer.commitTransaction Attaches offset to message + marks transaction ABCDEF Processor Observer ABC 012 Topic A Topic B @NSilnitsky
- 36. Kafka Broker consumer.poll producer.beginTransaction Producer.send producer.sendOffsets Attaches offset to message + marks transaction ABCDEF Processor Observer ABC 01 Topic A Topic B @NSilnitsky
- 37. Kafka Broker The bigger the batch the longer the latency. Processor Observer @NSilnitsky
- 38. Kafka Broker Processor Observer No end-to-end exactly-once guarantees out of the box. @NSilnitsky
- 39. In reality, this is more complex than how I describe it. Out of scope: ▪ Two-phase-commit with transaction coordinator ▪ Transaction log ▪ Additional “fencing” data ▪ 1 processor-producer - 1 partition @NSilnitsky
- 40. Purchase Completed UpdateInventory(Item1) Exactly Once (Classic ecommerce) flow at Wix UpdateInventory(Item2) UpdateInventory(ItemN) ABCDEF 1 iNB 1 I3I8 I1I2I5I5 01 updates @NSilnitsky
- 41. Exactly-Once delivery is... like the holy grail of message delivery over the network. It’s a tough nut to crack.
- 42. Exactly-Once delivery is... complex to implement, to use, and it requires fine-tuning.
- 43. Exactly-Once delivery is... crucial for achieving atomic actions in distributed systems.
- 44. Thank You natans@wix.com twitter @NSilnitsky linkedin/natansilnitsky github.com/natansil
- 45. Resources EoS in Kafka talk by Jason Gustafson Exactly-once Semantics are Possible by Neha Narkhede Transactions in Apache Kafka Revisiting Exactly One Semantics by Jason Gustafson, Confluent Proposal to improve EOS Produce scalability How akka works with exactly once message delivery by Hugh McKee