Exactly-Once, Again: Adding EOS Support for Kafka Connect Source Connectors with Chris Egerton
0 likes1,220 views
The document discusses the implementation of exactly-once semantics (EOS) in Kafka Connect source connectors, detailing the importance of ensuring reliable event handling while integrating with external systems. It outlines challenges, strategies for achieving EOS, and concepts like zombie fencing to manage tasks effectively during connector operations. The content emphasizes the goal of minimal interruption and enhanced reliability in data processing using Kafka Connect.
![Zombie fencing: actually pretty easy?
reddit-source-0
reddit-source-1
reddit-source-2
reddit-source-3
ò#00Г–ƿ
„)} +Ρ#ƿ
;)Ю
0ƿ
–/ „Ч„<
reddit-source-0
reddit-source-1
reddit-source-2
reddit-source-3
ò#00Г–ƿ
„)} +Ρ#ƿ
;& #$ ƿ
–/ „Ч„<
Fences out
;"%Ρ"„ƿ
)} –
;"%Ρ"„ƿ
)} –
Fences out
ɀ q)Y ƿ
0)#„ƿ
–φГ„ƿ
$ )+Ч
ɀ ])Ю
ƿ
@
ƿ
$ Г„φƿ
🥲](https://image.slidesharecdn.com/bs44-20230517ksl23chrisegerton-230602122041-c1adba6b/85/Exactly-Once-Again-Adding-EOS-Support-for-Kafka-Connect-Source-Connectors-with-Chris-Egerton-19-320.jpg)
![Leadership change (with new tasks)
my-connector (3 tasks)
Config topic
my-connector (2 tasks)
q–/ +–Г& όƿ
„–/ –#
Write by leader
]#/ 0#+ƿ
μ/ Ю
Ю
„ƿ
)} –ƿ
)μƿ
ΡЮ
} „–#+ƿ
;Λ#μ)+#ƿ
+#Λ/ Ю
/ & Ρ#<
ò#Λ/ Ю
/ & Ρ#ƿ
;g ƿ
& #$ ƿ
Ю
#/ 0#+<
Ś¤"%–
ɀ H& ƿ
+#Λ/ Ю
/ & Ρ#P
ƿ
& #$ ƿ
Ю
#/ 0#+ƿ
φ/ „ƿ
–)ƿ
μ#& Ρ#ƿ
)} –ƿ
)Ю
0ƿ
–/ „Ч„
● But how can it tell?
ɀ ſ φ#ƿ
Ρ)& μГόƿ
–)ъГΡƿ
Ю
))Ч„ƿ
–φ#ƿ
„/ 8 #ƿ
Г& ƿ
Λ)–φƿ
„Ρ#& / +Г)„
ɀ ò#8 #8 Λ#+P
ƿ
$ #ƿ
$ / & –ƿ
–)ƿ
/¤)Г0ƿ
} & & #Ρ#„„/ +№
ƿ
Г& –#++} ъ–Г)& „](https://image.slidesharecdn.com/bs44-20230517ksl23chrisegerton-230602122041-c1adba6b/85/Exactly-Once-Again-Adding-EOS-Support-for-Kafka-Connect-Source-Connectors-with-Chris-Egerton-34-320.jpg)
![Zombie fencing: durable task counts
my-connector (3 tasks)
Ņ)%μГόƿ
–)ъГΡ
my-connector (2 tasks)
my-connector-task-count (2)
q–/ +–Г& όƿ
„–/ –#
ff #$ ƿ
–/ „Чƿ
Ρ)& μГό„
Rebalance (+ zombie fencing)
Ś¤"%–
ɀ H& ƿ
+#Λ/ Ю
/ & Ρ#T
ɂ ]#/ 0#+ƿ
μ#& Ρ#„ƿ
–φ+##ƿ
–/ „Ч„ƿ
Λ/ „#0ƿ
)& ƿ
Ю
/ –#„–ƿ
–/ „Чƿ
Ρ)} & –ƿ
+#Ρ)+0
ɂ ]#/ 0#+ƿ
$ +Г–#„ƿ
& #$ ƿ
–/ „Чƿ
Ρ)} & –ƿ
)μƿ
–$ )ƿ
–/ „Ч„ƿ
Λ/ „#0ƿ
)& ƿ
Ю
/ –#„–ƿ
–/ „Чƿ
Ρ)& μГό„
my-connector-task-count (3)](https://image.slidesharecdn.com/bs44-20230517ksl23chrisegerton-230602122041-c1adba6b/85/Exactly-Once-Again-Adding-EOS-Support-for-Kafka-Connect-Source-Connectors-with-Chris-Egerton-38-320.jpg)
Exactly-Once, Again: Adding EOS Support for Kafka Connect Source Connectors with Chris Egerton
- 1. Exactly-Once, Again: Adding EOS Support for Kafka Connect Source Connectors Chris Egerton
- 2. Nice to meet you! Chris Egerton Open Source Program Office @Aiven Apache Kafka committer and PMC member (Official bio) (Unofficial bio)
- 3. Exactly-Once, Again: Adding EOS Support for Kafka Connect Source Connectors
- 4. ● “Exactly-once semantics” ● “Semantics” instead of “delivery”, “guarantees”, “delivery guarantees”, etc. (see Two Generals’ Problem) ● Levels: ○ Probably-once ○ At-least-once ○ At-most-once ○ Exactly-once ● With all else equal, exactly-once is best ● But of course, it’s the hardest to implement EOS
- 5. Exactly-Once, Again: Adding EOS Support for Kafka Connect Source Connectors
- 6. Source Connectors ● Kafka stores and transmits events. Where do these events come from, and where do they go? ● DYI producer/consumer application? Nah 👎 ● Connectors: no-code (or low-code) applications to integrate Kafka with other systems ● Sink connectors write data from Kafka to the external system ● Source connectors read data from the external system into Kafka
- 7. Exactly-Once, Again: Adding EOS Support for Kafka Connect Source Connectors
- 8. Kafka Connect ● Distributed, horizontally-scalable, fault- tolerant ingest/export tool for Kafka ● Developers implement connectors against the Kafka Connect API ● Cluster administrators install connectors onto one or more Kafka Connect workers, which combine to form a cluster ● Users can then create and manage connectors on that cluster by submitting JSON configurations via a REST API ● (For users) No code required! { "name": "local-file-source", "connector.class": "FileStreamSink", "tasks.max": "1", "file": "test.txt", "topic": "connect-test" }
- 9. We’re going to talk about designing support for exactly-once semantics (EOS) with source connectors developed for Kafka Connect. In summary…
- 10. Challenges 1⃣ƿ Ø#μГ& Г& όƿ „)} +Ρ#ƿ )μμ„#–„ 2⃣ƿ q–)+Г& όƿ / & 0ƿ +#–+Г#¤Г& όƿ „)} +Ρ#ƿ )μμ„#–„ 3⃣ƿ ò#–+№ Г& όƿ 0#Ю Г¤#+№ ƿ –)ƿ ǽ/ μЧ/ 4⃣ƿ̀ )8 ΛГ#„ƿ 🧟🧟🧟🧟🧟
- 11. Source offsets, in detail ● Source connectors provide source records ● Source records come with source offsets (partition + offset) ● On startup, connectors use source offsets to know where to resume from ● Source offsets are stored in an offsets topic by Kafka Connect ✅🎉 // TODO ɀ Ø"μГ%Г%όƿ „)} +Ρ#ƿ )μμ„#–„ƿ ;1⃣<ƿ Г„ƿ –φ#ƿ +#„ъ)& „ГΛГЮ Г–№ ƿ )μƿ –φ#ƿ Ρ)& & #Ρ–)+ ɀ q–)*Г%όƿ , %-ƿ *"–*Г"¤Г%όƿ „)} +Ρ#ƿ )μμ„#–„ƿ ;2⃣<ƿ Г„ƿ –φ#ƿ +#„ъ)& „ГΛГЮ Г–№ ƿ )μƿ ǽ/μЧ/ƿ Ņ)& & #Ρ–
- 12. Exactly-once for Kafka clients ɀ ǽ@ nBCDEƿ ŚG/ Ρ–Ю № ƿ H& Ρ#ƿ Ø#Ю Г¤#+№ ƿ / & 0ƿ ſ +/ & „/ Ρ–Г)& / Ю ƿ ë #„„/ όГ& ό ɂ ò#Ю #/ „#0ƿ Г& ƿ LMNNMLML ɂ O #& Ρ#P ƿ Q ό/ Г& Sƿ Г& ƿ –φ#ƿ –Г–Ю # ɀ / -"0 ъ)–"%–ƿ ъ*)-} Ρ"*Tƿ +#–+Г#„ƿ $ Г–φ)} –ƿ 0} ъЮ ГΡ/ –#„ƿ ;3⃣< ɀ ſ *, %„, Ρ–Г)%, Ю ƿ ъ*)-} Ρ"*Tƿ / –)8 ГΡƿ Ρ+)„„V –)ъГΡƿ $ +Г–#„ ɀ ſ *, %„, Ρ–Г)%, Ю ƿ / Ø„Tƿ „Г& όЮ #ƿ $ +Г–#+ƿ ъ#+ƿ @ Ø ɂ / %Г–Г, Ю Г7"ƿ –*, %„, Ρ–Г)%„ƿ $ Г–φƿ / ƿ ъ+)0} Ρ#+ƿ –)ƿ μ"%Ρ"ƿ )} –ƿ )–φ#+ƿ ъ+)0} Ρ#+„ƿ $ Г–φƿ –φ#ƿ „/ 8 #ƿ –+/ & „/ Ρ–Г)& / Ю ƿ @ Ø ✅🎉
- 13. Tracking source offsets (2⃣) Ļ #μ)+#T ɀ n)Ю Ю ƿ Ρ)& & #Ρ–)+ƿ μ)+ƿ „)} +Ρ#ƿ +#Ρ)+0„ ɀ ỳ +Г–#ƿ +#Ρ)+0„ƿ –)ƿ ǽ/ μЧ/ ɀ n#+Г)0ГΡ/ Ю Ю № ƿ $ +Г–#ƿ ;Ρ)8 8 Г–<ƿ „)} +Ρ#ƿ )μμ„#–„ƿ –)ƿ ǽ/ μЧ/ ɀ n+)¤Г0#„ƿ / –V Ю #/ „–V )& Ρ#ƿ „} ъъ)+– μ–#+T ɀ Ļ #όГ& ƿ –+/ & „/ Ρ–Г)& ɀ n)Ю Ю ƿ Ρ)& & #Ρ–)+ƿ μ)+ƿ „)} +Ρ#ƿ +#Ρ)+0„ ɀ ỳ +Г–#ƿ +#Ρ)+0„ƿ Г8 8 #0Г/ –#Ю № ƿ –)ƿ ǽ/ μЧ/ ƿ ;„–ГЮ Ю < ɀ ỳ +Г–#ƿ „)} +Ρ#ƿ )μμ„#–„ƿ –)ƿ ǽ/ μЧ/ ƿ ɀ Ņ)8 8 Г–ƿ –+/ & „/ Ρ–Г)& ɀ n+)¤Г0#„ƿ #G/ Ρ–Ю № V )& Ρ#ƿ „} ъъ)+– ✅🎉
- 14. 75% done! 1⃣ƿ Ø#μГ& Г& όƿ „)} +Ρ#ƿ )μμ„#–„ 2⃣ƿ q–)+Г& όƿ / & 0ƿ +#–+Г#¤Г& όƿ „)} +Ρ#ƿ )μμ„#–„ 3⃣ƿ ò#–+№ Г& όƿ 0#Ю Г¤#+№ ƿ –)ƿ ǽ/ μЧ/ 🧟🧟🧟🧟🧟🧟🧟🧟🧟🧟🧟🧟🧟 🧟🧟🧟🧟🧟🧟🧟🧟🧟🧟🧟🧟🧟 🧟🧟🧟🧟🧟🧟🧟🧟🧟🧟🧟🧟🧟 ✅ ✅ ✅ 4⃣ƿ ſ φ/ –ƿ Ю #/¤#„Y
- 15. “Chekhov’s gun cabinet” Kafka Connect: diving deeper ɀ Ņ)%%"Ρ–)*„ƿ ό#& #+/–#ƿ Ρ)& μГό} +/–Г)& „ƿ μ)+ƿ )& #ƿ )+ƿ 8 )+#ƿ –, „Ч„ ɀ ſ / „Ч„ƿ Ρ/ & ƿ Λ#ƿ „ъ+#/ 0ƿ / Ρ+)„„ƿ 8 } Ю –ГъЮ #ƿ ǽ/ μЧ/ ƿ Ņ)& & #Ρ–ƿ $ )+Ч#+„ ɀ ſ / „Ч„ƿ / +#ƿ / „„Гό& #0ƿ –)ƿ $ )+Ч#+„ƿ 0} +Г& όƿ / ƿ *"Λ, Ю , %Ρ" ɀ ƿ „Г& όЮ #ƿ $ )+Ч#+P ƿ –φ#ƿ Ю ", -"*P ƿ 0#–#+8 Г& #„ƿ –φ/ –ƿ / „„Гό& 8 #& – ɀ ſ / „Чƿ Ρ)& μГό} +/ –Г)& „ƿ / +#ƿ „–)+#0ƿ Г& ƿ / ƿ „Г& όЮ #V ъ/+–Г–Г)& ƿ ǽ/μЧ/ƿ –)ъГΡƿ Ρ/ Ю Ю #0ƿ –φ#ƿ Ρ)%μГόƿ –)ъГΡ ɀ Z)+ƿ „)} +Ρ#ƿ Ρ)& & #Ρ–)+„P ƿ ǽ/ μЧ/ ƿ Ņ)& & #Ρ–ƿ } „#„ƿ )& #ƿ ъ+)0} Ρ#+ƿ ъ#+ƿ –/ „Чƿ –)ƿ „#& 0ƿ „)} +Ρ#ƿ +#Ρ)+0„ƿ –)ƿ ǽ/μЧ/
- 16. Zombie fencing: correctness goals ſ )Ю #+/ –#0ƿ „Ρ#& / +Г)„T ɀ ;[ & <ό+/ Ρ#μ} Ю ƿ $ )+Ч#+ƿ „φ} –0)$ & ɀ +ΛГ–+/ +№ ƿ $ )+Ч#+ƿ „–/ +–} ъ„ ɀ +ΛГ–+/ +ГЮ № V Ю )& όP ƿ / +ΛГ–+/ +ГЮ № V ъЮ /Ρ#0P ƿ / & 0ƿ /+ΛГ–+/+ГЮ № V –Г8 #0ƿ ъ/ } „#„
- 17. Zombie fencing: UX goals ● Minimal unnecessary interruptions (keep processing data) ● Minimal changes to connector code ● Minimal connector management API changes
- 18. Zombie fencing: actually pretty easy? ● Give each task a transactional ID derived from the name of the connector and the task ID ○ E.g., “reddit-source-0” or “chris-ksl-3” ● Let tasks fence out older instances on startup ○ Fencing: disabling a producer from writing to Kafka
- 19. Zombie fencing: actually pretty easy? reddit-source-0 reddit-source-1 reddit-source-2 reddit-source-3 ò#00Г–ƿ „)} +Ρ#ƿ ;)Ю 0ƿ –/ „Ч„< reddit-source-0 reddit-source-1 reddit-source-2 reddit-source-3 ò#00Г–ƿ „)} +Ρ#ƿ ;& #$ ƿ –/ „Ч„< Fences out ;"%Ρ"„ƿ )} – ;"%Ρ"„ƿ )} – Fences out ɀ q)Y ƿ 0)#„ƿ –φГ„ƿ $ )+Ч ɀ ])Ю ƿ @ ƿ $ Г„φƿ 🥲
- 20. Source partition reshuffling problem ɀ q)} +Ρ#ƿ Ρ)& & #Ρ–)+„ƿ +#/ 0ƿ μ+)8 ƿ „)} *Ρ"ƿ ъ, *–Г–Г)%„ ɂ Ø/ –/ Λ/ „#ƿ –/ ΛЮ #„P ƿ ǽ/ μЧ/ ƿ –)ъГΡ„P ƿ #–ΡM ɀ ſ φ#„#ƿ / +#ƿ / Ю Ю )Ρ/ –#0ƿ / Ρ+)„„ƿ –/ „Ч„ ɀ ſ φ/ –ƿ / Ю Ю )Ρ/ –Г)& ƿ Ρ/ & ƿ Ρφ/ & ό#ƿ )¤#+ƿ –Г8 #
- 21. Source partition reshuffling problem P0 Task 0 P1 ſ / „Чƿ N @ & Г–Г/ Ю ƿ / Ю Ю )Ρ/ –Г)& ƿ ^Ţƿ –/ „Ч„P ƿ Ţƿ ъ/ +–Г–Г)& „< P0 Task 0 P2 ƿ & #$ ƿ ъ/ +–Г–Г)& ƿ ^nŢ`ƿ Г„ƿ Ρ+#/ –#0 P1 ſ / „Чƿ N & ƿ #GГ„–Г& όƿ ъ/ +–Г–Г)& ƿ ^nN`ƿ Г„ƿ 0#Ю #–#0 P0 ſ / „Чƿ L P2 ſ / „Чƿ N P1 P2
- 22. Source partition reshuffling problem P0 ſ / „Чƿ L P2 HЮ 0ƿ –/ „Ч„ P1 Task 1 ff #$ ƿ –/ „Ч„ P2 ſ / „Чƿ N ;"%Ρ"„ƿ )} – ● New task T1 starts before old task T0 stops ɀ Ļ )–φƿ / +#ƿ / „„Гό& #0ƿ ъ/+–Г–Г)& ƿ nŢ ɀ Ø} ъЮ ГΡ/ –#„ƿ / Λ)} & 0b
- 23. Zombie fencing: second try ɀ ỳ φ#& #¤#+ƿ & #$ ƿ –/„Чƿ Ρ)& μГό} +/–Г)& „ƿ /+#ƿ +#/0ƿ μ+)8 ƿ –φ#ƿ Ρ)& μГόƿ –)ъГΡP ƿ ъ#+μ)+8 ƿ /ƿ +)} & 0ƿ )μƿ 7)0 ΛГ"ƿ μ"%ΡГ%ό ɂ Z)+ƿ #¤#+№ ƿ & #$ ƿ –/ „ЧP ƿ –/ Ч#ƿ Г–„ƿ –+/ & „/ Ρ–Г)& / Ю ƿ @ Øƿ / & 0ƿ ъ+##8 ъ–Г¤#Ю № ƿ Г& Г–Г/Ю Гc#ƿ –+/& „/Ρ–Г)& „ ɀ ff )$ ƿ Г–d„ƿ Г8 ъ)„„ГΛЮ #ƿ μ)+ƿ / &№ ƿ )Ю 0#+ƿ Г& „–/ & Ρ#„ƿ )μƿ & #$ Ю № V Ρ+#/ –#0ƿ –/ „Ч„ƿ –)ƿ Λ#ƿ +} & & Г& όƿ Λ#μ)+#ƿ / &№ ƿ & #$ #+ƿ Г& „–/ & Ρ#„ƿ / +#ƿ „–/ +–#0 ɀ Ø)ƿ –φГ„ƿ 0} +Г& όƿ +#Λ/ Ю / & Ρ#
- 24. Zombie fencing: first try reddit-source-0 reddit-source-1 reddit-source-2 reddit-source-3 ò#00Г–ƿ „)} +Ρ#ƿ ;)Ю 0ƿ –/ „Ч„< reddit-source-0 reddit-source-1 reddit-source-2 reddit-source-3 ò#00Г–ƿ „)} +Ρ#ƿ ;& #$ ƿ –/ „Ч„< ;"%Ρ"„ƿ )} – Fences out ;"%Ρ"„ƿ )} – ;"%Ρ"„ƿ )} –
- 25. Zombie fencing: second try reddit-source-0 reddit-source-1 reddit-source-2 reddit-source-3 ò#00Г–ƿ „)} +Ρ#ƿ ;)Ю 0ƿ –/ „Ч„< reddit-source-0 reddit-source-1 reddit-source-2 reddit-source-3 ò#00Г–ƿ „)} +Ρ#ƿ ;& #$ ƿ –/ „Ч„< ` )0 ΛГ"ƿ μ"%ΡГ%ό *)} %- ● Woohoo, we did it! ● Just kidding, put your hand back down
- 26. Zombie fencing: second try ɀ ỳ φ#& #¤#+ƿ & #$ ƿ –/„Чƿ Ρ)& μГό} +/–Г)& „ƿ /+#ƿ +#/0ƿ μ+)8 ƿ –φ#ƿ Ρ)& μГόƿ –)ъГΡP ƿ ъ#+μ)+8 ƿ /ƿ +)} & 0ƿ )μƿ 7)0 ΛГ"ƿ μ"%ΡГ%ό ɂ Z)+ƿ #¤#+№ ƿ & #$ ƿ –/ „ЧP ƿ –/ Ч#ƿ Г–„ƿ –+/ & „/ Ρ–Г)& / Ю ƿ @ Øƿ / & 0ƿ ъ+##8 ъ–Г¤#Ю № ƿ Г& Г–Г/Ю Гc#ƿ –+/& „/Ρ–Г)& „ ɀ ff )$ ƿ Г–d„ƿ Г8 ъ)„„ГΛЮ #ƿ μ)+ƿ / &№ ƿ )Ю 0#+ƿ Г& „–/ & Ρ#„ƿ )μƿ & #$ Ю № V Ρ+#/ –#0ƿ –/ „Ч„ƿ –)ƿ Λ#ƿ +} & & Г& όƿ Λ#μ)+#ƿ / &№ ƿ & #$ #+ƿ Г& „–/ & Ρ#„ƿ / +#ƿ „–/ +–#0 ɀ Ø)ƿ –φГ„ƿ 0} +Г& όƿ +#Λ/ Ю / & Ρ#
- 27. Connector resizing T0 T1 T2 T3 Z)} +ƿ –/ „Ч„ T0 T1 T2 ſ φ+##ƿ –/ „Ч„ ` )0 ΛГ"ƿ μ"%ΡГ%ό *)} %-= * - based on new tasks ɀ HЮ 0ƿ –/ „Чƿ +} & & Г& όƿ / μ–#+ƿ & #$ #+ƿ –/ „Ч„ƿ / +#ƿ „–/ +–#0 ɀ @ μƿ / ƿ ъ/ +–Г–Г)& ƿ Г„ƿ +#/„„Гό& #0ƿ μ+)8 ƿ )Ю 0ƿ ſ e ƿ –)ƿ & #$ ƿ ſ Lfſ Nfſ ŢY ɀ Ø} ъЮ ГΡ/ –#„ƿ / Λ)} & 0
- 28. Zombie fencing: third time’s the charm ɀ @ & „–#/ 0ƿ )μƿ } „Г& όƿ –φ#ƿ & #$ ƿ „#–ƿ )μƿ –/ „Чƿ Ρ)& μГό„ƿ μ)+ƿ )} +ƿ +)} & 0ƿ )μƿ c)8 ΛГ#ƿ μ#& ΡГ& όP ƿ } „#ƿ –φ#ƿ )Ю 0ƿ „#– ɀ ό/ Г& P ƿ 0)ƿ –φГ„ƿ 0} +Г& όƿ +#Λ/ Ю / & Ρ# ɀ Ю +Гόφ–P ƿ –φГ„ƿ φ/ „ƿ –)ƿ Λ#ƿ #& )} όφP ƿ +Гόφ– ɀ ỳ #Ю Ю Y ƿ φ)$ ƿ 0)ƿ $ #ƿ Ч& )$ ƿ / Λ)} –ƿ –φ#ƿ )Ю 0ƿ „#–ƿ )μƿ –/ „Чƿ Ρ)& μГό„
- 29. Zombie leaders ɀ ſ φ#ƿ Ю #/ 0#+ƿ )μƿ –φ#ƿ ΡЮ } „–#+ƿ Г„ƿ –φ#ƿ )& Ю № ƿ )& #ƿ / Ю Ю )$ #0ƿ –)ƿ ъ} ΛЮ Г„φƿ –/ „Чƿ Ρ)& μГό„ƿ –)ƿ –φ#ƿ Ρ)& μГόƿ –)ъГΡ ɀ O )$ #¤#+P ƿ $ #ƿ 0)& d–ƿ #& μ)+Ρ#ƿ –φГ„ƿ ¤#+№ ƿ „–+)& όЮ № ɀ q)8 #ƿ $ )+Ч#+„ƿ 8 /№ ƿ 8 Г„–/ Ч#& Ю № ƿ Λ#Ю Г#¤#ƿ –φ#№ ƿ / +#ƿ –φ#ƿ Ю #/ 0#+ ɀ @ & / ΡΡ} +/ –#ƿ –/ „Чƿ Ρ)& μГό„ƿ 8 /№ ƿ Λ#ƿ ъ} ΛЮ Г„φ#0ƿ Г& ƿ +/ ъГ0ƿ „} ΡΡ#„„Г)& P ƿ )¤#+$ +Г–Г& όƿ ¤/ Ю Г0ƿ –/ „Чƿ Ρ)& μГό„
- 30. ● Leader should fence out three tasks ● But leader only fences out two Zombie leaders my-connector (3 tasks) Ņ)%μГόƿ –)ъГΡ my-connector (2 tasks) my-connector (1 task) q–/ +–Г& όƿ „–/ –# Write by zombie leader ỳ +Г–#ƿ Λ№ ƿ / Ρ–} / Ю ƿ Ю #/ 0#+ ò#Λ/ Ю / & Ρ# Ś¤"%– ɀ H& ƿ +#Λ/ Ю / & Ρ#P ƿ Ю #/ 0#+ƿ „##„T ○ One new task ɂ ſ $ )ƿ )Ю 0ƿ –/ „Ч„
- 31. Zombie fencing: guarded config topic ɀ O )$ ƿ 0)ƿ $ #ƿ ъ+#¤#& –ƿ c)8 ΛГ#ƿ Ю #/ 0#+„ƿ μ+)8 ƿ / ΡΡ#„„Г& όƿ –φ#ƿ Ρ)& μГόƿ –)ъГΡ ɀ ſ +/ & „/ Ρ–Г)& / Ю ƿ ъ+)0} Ρ#+ƿ –)ƿ –φ#ƿ +#„Ρ} #ƿ ;/ ό/ Г& <b ɀ @ „ƿ –φГ„ƿ #& )} όφ ɀ +#ƿ $ #ƿ ό} / +/ & –##0ƿ –φ/ –ƿ –φ#ƿ Ρ)& μГόƿ –)ъГΡƿ Ρ/ & ƿ Λ#ƿ } „#0ƿ / „ƿ / ƿ „)} +Ρ#ƿ )μƿ –+} –φƿ μ)+ƿ c)8 ΛГ#ƿ μ#& ΡГ& όƿ & )$ ɀ 🙃🙃🙃
- 32. Leadership change ɀ @ μƿ / ƿ Ю #/ 0#+ƿ μ/ Ю Ю „ƿ )} –ƿ )μƿ –φ#ƿ ΡЮ } „–#+P ƿ / ƿ & #$ ƿ )& #ƿ Г„ƿ Ρφ)„#& ɀ O )$ ƿ 0)#„ƿ –φ#ƿ & #$ ƿ Ю #/ 0#+ƿ Ч& )$ ƿ $ φ/ –ƿ & ##0„ƿ μ#& ΡГ& ό
- 33. Leadership change (with new tasks) my-connector (3 tasks) Config topic my-connector (2 tasks) q–/ +–Г& όƿ „–/ –# ỳ +Г–#ƿ Λ№ ƿ Ю #/ 0#+ ò#Λ/ Ю / & Ρ#ƿ ;g ƿ c)8 ΛГ#ƿ μ#& ΡГ& ό< Leader falls out of cluster Ś¤"%– ɀ H& ƿ μГ+„–ƿ +#Λ/ Ю / & Ρ#P ƿ )Ю 0ƿ –/ „Ч„ƿ / +#ƿ μ#& Ρ#0ƿ )} –ƿ „} ΡΡ#„„μ} Ю Ю № Rebalance (+ new leader) ɀ H& ƿ „#Ρ)& 0ƿ +#Λ/ Ю / & Ρ#P ƿ & #$ ƿ Ю #/ 0#+ƿ 0)#„& d–ƿ φ/¤#ƿ –)ƿ 0)ƿ / &№ ƿ μ#& ΡГ& ό
- 34. Leadership change (with new tasks) my-connector (3 tasks) Config topic my-connector (2 tasks) q–/ +–Г& όƿ „–/ –# Write by leader ]#/ 0#+ƿ μ/ Ю Ю „ƿ )} –ƿ )μƿ ΡЮ } „–#+ƿ ;Λ#μ)+#ƿ +#Λ/ Ю / & Ρ#< ò#Λ/ Ю / & Ρ#ƿ ;g ƿ & #$ ƿ Ю #/ 0#+< Ś¤"%– ɀ H& ƿ +#Λ/ Ю / & Ρ#P ƿ & #$ ƿ Ю #/ 0#+ƿ φ/ „ƿ –)ƿ μ#& Ρ#ƿ )} –ƿ )Ю 0ƿ –/ „Ч„ ● But how can it tell? ɀ ſ φ#ƿ Ρ)& μГόƿ –)ъГΡƿ Ю ))Ч„ƿ –φ#ƿ „/ 8 #ƿ Г& ƿ Λ)–φƿ „Ρ#& / +Г)„ ɀ ò#8 #8 Λ#+P ƿ $ #ƿ $ / & –ƿ –)ƿ /¤)Г0ƿ } & & #Ρ#„„/ +№ ƿ Г& –#++} ъ–Г)& „
- 35. Zombie fencing: fence, then write ɀ Ņ} ++#& –ƿ „#я } #& Ρ#ƿ )μƿ #¤#& –„T ɂ n} ΛЮ Г„φƿ & #$ ƿ –/ „Чƿ Ρ)& μГό„ ɂ ò#Λ/ Ю / & Ρ# ɂ i ƿ̀ )8 ΛГ#ƿ μ#& ΡГ& ό ɂ q–/ +–ƿ & #$ ƿ –/ „Ч„ ɀ ff #$ ƿ )+0#+T ɂ ` )8 ΛГ#ƿ μ#& ΡГ& ό ɂ n} ΛЮ Г„φƿ & #$ ƿ –/ „Чƿ Ρ)& μГό„ ɂ ò#Λ/ Ю / & Ρ# ɂ q–/ +–ƿ & #$ ƿ –/ „Ч„ ɀ ſ φГ„ƿ φ/ „ƿ –)ƿ Λ#ƿ Г–P ƿ +Гόφ– ɀ j )} ƿ $ Г„φƿ 😈
- 36. That was not a good idea ● Poor UX ○ Causes tasks to fail in between zombie fencing and end of rebalance ○ Forcibly kills them, no chance to commit pending offsets ○ Looks like a bug to users ● Correctness issue ○ Users can manually restart failed tasks ○ Even in between zombie fencing and publishing new task configs ○ Uh oh, a zombie task made it to the other end of the rebalance!
- 37. Zombie fencing: durable task counts ● Forget the “fence then write” logic ● Instead, we explicitly track the number of to-be-fenced tasks in the config topic with a task count record ● These serve two purposes: ○ Explicitly: if fencing is necessary, how many tasks have to be fenced out ○ Implicitly: determine whether zombie fencing is necessary
- 38. Zombie fencing: durable task counts my-connector (3 tasks) Ņ)%μГόƿ –)ъГΡ my-connector (2 tasks) my-connector-task-count (2) q–/ +–Г& όƿ „–/ –# ff #$ ƿ –/ „Чƿ Ρ)& μГό„ Rebalance (+ zombie fencing) Ś¤"%– ɀ H& ƿ +#Λ/ Ю / & Ρ#T ɂ ]#/ 0#+ƿ μ#& Ρ#„ƿ –φ+##ƿ –/ „Ч„ƿ Λ/ „#0ƿ )& ƿ Ю / –#„–ƿ –/ „Чƿ Ρ)} & –ƿ +#Ρ)+0 ɂ ]#/ 0#+ƿ $ +Г–#„ƿ & #$ ƿ –/ „Чƿ Ρ)} & –ƿ )μƿ –$ )ƿ –/ „Ч„ƿ Λ/ „#0ƿ )& ƿ Ю / –#„–ƿ –/ „Чƿ Ρ)& μГό„ my-connector-task-count (3)
- 39. Zombie fencing: durable task counts my-connector (3 tasks) Config topic my-connector (2 tasks) my-connector-task-count (2) q–/ +–Г& όƿ „–/ –# ff #$ ƿ –/ „Чƿ Ρ)& μГό„ Rebalance (+ zombie fencing) Ś¤"%– my-connector-task-count (3) Safe to run bring up tasks? ✅ ❌ ✅
- 40. ɀ ỳ φ/–ƿ Ρ)} Ю 0ƿ ъ)„„ГΛЮ № ƿ Λ+#/Чƿ –φГ„ƿ & )$ ɀ Hφƿ № #/ φP ƿ $ φ/ –ƿ $ / „ƿ –φ/ –ƿ / Λ)} –ƿ –/ „Чƿ +#„–/ +–„ Zombie fencing: durable task counts
- 41. Laggy task startup ● Zombie fencing disables all initialized task producers from writing to Kafka ● What if a zombie task lags and hasn’t initialized its producer by the time zombie fencing for a new generation of tasks takes place? ● Or, what if a task is restarted on a zombie worker after zombie fencing takes place?
- 42. Laggy task startup reddit-source-0 reddit-source-1 reddit-source-2 reddit-source-3 ò#00Г–ƿ „)} +Ρ#ƿ ^k ƿ )Ю 0ƿ –/ „Ч„< reddit-source-0 reddit-source-1 reddit-source-2 ò#00Г–ƿ „)} +Ρ#ƿ ^e ƿ & #$ ƿ –/ „Ч„< ` )0 ΛГ"ƿ μ"%ΡГ%ό *)} %- reddit-source-3 ^ſ / „Чƿ Г„ƿ Ю / όόГ& όƿ 0} +Г& όƿ „–/ +–} ъ< ^ſ / „Чƿ φ/ „ƿ μГ& Г„φ#0ƿ „–/ +–} ъ<
- 43. Zombie fencing: check your work ɀ μ–#+ƿ Г& Г–Г/ Ю ГcГ& όƿ –+/ & „/ Ρ–Г)& „ƿ μ)+ƿ / ƿ –/ „Чƿ ъ+)0} Ρ#+P ƿ φ/¤#ƿ –)ƿ 8 / Ч#ƿ „} +#ƿ Г–d„ƿ „–ГЮ Ю ƿ „/ μ#ƿ –)ƿ +} & ƿ –φ#ƿ –/ „Ч ɀ ff #$ ƿ „#я } #& Ρ#ƿ )μƿ #¤#& –„T ɂ Ø#ΡГ0#ƿ –)ƿ ;+#<„–/ +–ƿ –/ „Ч ɂ Ņ+#/ –#ƿ ъ+)0} Ρ#+ƿ μ)+ƿ –/ „Чƿ / & 0ƿ Г& Г–Г/ Ю Гc#ƿ –+/ & „/ Ρ–Г)& „ ɂ ò#/ 0ƿ –)ƿ #& 0ƿ )μƿ Ρ)& μГόƿ –)ъГΡ ɂ @ μƿ & #$ ƿ –/ „Чƿ Ρ)& μГό„ƿ μ)} & 0P ƿ / Λ)+–ƿ „–/ +–} ъƿ / & 0ƿ / Λ/ & 0)& ƿ –φ#ƿ –/ „Ч ɂ H–φ#+$ Г„#P ƿ „/ μ#ƿ –)ƿ „–/ +–ƿ ъ+)Ρ#„„Г& όƿ 0/ –/ ɀ O /¤#ƿ $ #ƿ μГ& / Ю Ю № ƿ 0)& #ƿ Г–
- 45. Caveats ● Fencing during rebalancing is not a good idea ○ Makes rebalances more brittle ○ Requires a new rebalance any time we want to restart a task that failed due to failed zombie fencing ● Instead, we fence outside of rebalances ○ During task startup, workers issue a REST request to the leader to perform zombie fencing for the connector ○ The leader will perform that round (if necessary), then send back a 2XX response ○ If a non-2XX response is received, the task is marked failed ○ Tasks can easily be restarted
- 46. Caveats ɀ ſ φ+)$ / $ /№ ƿ ъ+)0} Ρ#+„ƿ μ)+ƿ Г& Г–Г/ Ю ГcГ& όƿ –+/ & „/ Ρ–Г)& „ƿ Г„ƿ $ / „–#μ} Ю ɀ ỳ #ƿ / 00#0ƿ / ƿ & #$ ƿ / 08 Г& ƿ ΡЮ Г#& –ƿ n@ ƿ Г& ƿ e Me ƿ –)ƿ 0)ƿ –φГ„ƿ Г& „–#/ 0 ɀ ỳ #ƿ φ/¤#ƿ –)ƿ Λ#ƿ Ρ/ +#μ} Ю ƿ / Λ)} –ƿ φ)$ ƿ –φ#ƿ Ю #/ 0#+ƿ } „#„ƿ –φ#ƿ –+/ & „/ Ρ–Г)& / Ю ƿ ъ+)0} Ρ#+ ɀ qГ8 ГЮ / +ƿ QΡЮ / Г8 V –φ#& V Ρφ#ΡЧSƿ Ю )όГΡƿ –)ƿ „)} +Ρ#ƿ –/ „Ч„
- 47. In practice @ 8 ъЮ #8 #& –/ –Г)& ƿ 0#–/ ГЮ „ƿ / +#ƿ Λ)+Г& όP ƿ φ)$ ƿ 0)ƿ $ #ƿ / Ρ–} / Ю Ю № ƿ } „#ƿ –φГ„ƿ μ#/ –} +#
- 48. In practice (cluster administrators) ɀ ff #$ ƿ ΡЮ } „–#+„T ɂ [ „#ƿ ¤#+„Г)& ƿ e Me MLƿ )+ƿ Ю / –#+ ɂ Ņ)& μГό} +#ƿ #¤#+№ ƿ $ )+Ч#+ƿ $ Г–φƿ exactly.once.source.support = enabled ɀ ŚGГ„–Г& όƿ ΡЮ } „–#+„T ɂ ò)Ю Ю Г& όƿ } ъό+/ 0#ƿ NE ɂ Ļ +Г& όƿ / Ю Ю ƿ $ )+Ч#+„ƿ –)ƿ e Me MLƿ )+ƿ Ю / –#+ ɂ Ņ)& μГό} +#ƿ #¤#+№ ƿ $ )+Ч#+ƿ $ Г–φƿ exactly.once.source.support = preparing ○ ò)Ю Ю Г& όƿ } ъό+/ 0#ƿ ŢE ɂ Ņ)& μГό} +#ƿ #¤#+№ ƿ $ )+Ч#+ƿ $ Г–φƿ exactly.once.source.support = enabled
- 49. In practice (downstream readers) ● Have to filter out records from aborted transactions ● If using the Java consumer, configure with isolation.level = read_committed ● For sink connectors, do at least one of the following: ○ Configure worker with consumer.isolation.level = read_committed ○ Configure connector with consumer.override.isolation.level = read_committed with (3.0.0 or later, with default worker configuration)
- 50. In practice (writing connectors) Have to define source offsets correctly public abstract class SourceTask { public abstract List<SourceRecord> poll(); } public class SourceRecord { public SourceRecord(Map<String, ?> sourcePartition, Map<String, ?> sourceOffset, ...) }
- 51. In practice (writing connectors) O /¤#ƿ –)ƿ } „"ƿ „)} +Ρ#ƿ )μμ„#–„ƿ Ρ)++#Ρ–Ю № public abstract class SourceTask { protected SourceTaskContext context; public abstract void start(Map<String, String> props); } public interface SourceTaskContext { OffsetStorageReader offsetStorageReader(); } public interface OffsetStorageReader { <T> Map<Map<String, T>, Map<String, Object>> offsets(Collection<Map<String, T>> partitions); }
- 52. In summary ɀ ŚG/ Ρ–Ю № V )& Ρ#ƿ Г„ƿ φ/ +0ƿ –)ƿ Г8 ъЮ #8 #& – ɂ Ś„ъ#ΡГ/ Ю Ю № ƿ φ/ & 0Ю Г& όƿ c)8 ΛГ#ƿ $ )+Ч#+„fc)8 ΛГ#ƿ –/ „Ч„ƿ / Ρ+)„„ƿ –/ „Чƿ +#Ρ)& μГό} +/ –Г)& „ ɀ ŚG/ Ρ–Ю № V )& Ρ#ƿ Г„ƿ ;φ)ъ#μ} Ю Ю № <ƿ #/ „№ ƿ –)ƿ } „# ɂ @ μƿ Г–d„ƿ & )–P ƿ φ/ +/ „„ƿ ъГ& όƿ 8 #ƿ )& ƿ ăГ+/b ɂ φ––ъ„TffГ„„} #„M/ ъ/ Ρφ#M)+όfТ Г+/ fъ+)Т #Ρ–„fǽ Zǽ fГ„„} #„ ɀ Z)+ƿ / Ю Ю ƿ –φ#ƿ 0#–/ ГЮ „P ƿ Ρφ#ΡЧƿ )} –ƿ ǽ@ nBnND ɂ φ––ъ„TffΡ$ ГЧГM/ ъ/ Ρφ#M)+όfΡ)& μЮ } #& Ρ#f0Г„ъЮ /№ fǽ Zǽ fǽ@ n BnNDṊ e i ŚG/ Ρ–Ю № V H& Ρ#g q} ъъ)+–g μ)+g q)} +Ρ#g Ņ)& & #Ρ–)+„
- 53. Thank you! ƿ Open Source Program Office @Aiven Ρφ#Г„&' ( Г¤&* +Г, Chris Egerton -Г* -Ρφ#Г„. &ό&#–,* . 12 33456 3 @C0urante