Exactly-once Stream Processing with Arroyo and Kafka
0 likes266 views
The document discusses Arroyo, a SQL stream processing engine built on Apache Arrow, designed for high-speed stateful computations in modern cloud environments. It focuses on achieving exactly-once stream processing through idempotent writes and a two-phase commit protocol, specifically with integration into Kafka. The presentation was delivered by Micah Wylde at the Kafka Summit in London on March 20, 2024.
Exactly-once Stream Processing with Arroyo and Kafka
- 1. Exactly Once Processing with Arroyo and Kafka
- 2. Exactly Once Stream Processing with Arroyo and Kafka Kafka Summit London / March 20, 2024 Micah Wylde co-founder, Arroyo @mwylde
- 3. What is Arroyo ● SQL stream processing engine built on Apache Arrow ● Stateful computations at millions of events/sec ● Designed for modern cloud environments with fast rescaling and recovery ● 10-20x faster than Flink
- 4. What is Arroyo https://github.com/ArroyoSystems/arroyo Fully open-source (apache 2)
- 6. X Retry the write → at least once Don’t retry → at most once
- 9. Solution 1: Idempotent writes enable.idempotence=true acks=all max.inflight.requests.per.connection= <=5 Producer config * default enabled since Kafka 3.0.0
- 11. A digression to stateful dataflows SELECT user_id, count(*) as count FROM transactions WHERE status = 'FAILED' GROUP BY user_id, hop(interval '5 seconds', interval '24 hours'); kafka source Filter status = ‘FAILED’ Sliding Window kafka sink s3 partition → last read offset s3 24-hour window state
- 12. To make this work, we need to introduce a two-phase commit protocol (2PC)
- 14. KIP-939: Support Participation in 2PC ● Producers can call InitProducerId without causing existing transactions to be cancelled ● Then, we can choose to commit or abort the ongoing transaction during recovery