TLV Data Plumbers: Exactly once processing
1 like951 views
The document discusses the challenges and solutions related to 'exactly once' processing in data systems, particularly focusing on the integration of Storm and Trident for real-time analytics. It emphasizes the significance of idempotency to overcome issues of transactional failures and the importance of maintaining clean data plumbing for effective insights. Various examples and common myths surrounding the concept of transactional states and log compaction in Kafka are presented as potential solutions to ensure reliable data processing.
TLV Data Plumbers: Exactly once processing
- 1. Exactly Once Processing The Sad Truth Yair Weinberger alooma | CTO @yairwein
- 2. #TLVDataPlumbers ● Create a community of data plumbers ● "The best minds of my generation are deleting commas from log files, and that makes me sad." @medriscoll, http://adage.com/... ● “For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights” @mrogati, http://www.nytimes.com/2014/08/18/... ● “It’s time to take care of our clogged data plumbing”, Tom Davenport, http://venturebeat.com/2014/09/18/...
- 3. A real-time platform that abstracts the data layer Mobile Servers Sensors Devices tons of plumbing to make it work Unscalable Rigid Leaky Slow Analytics Personalization Monitoring And more…
- 4. A real-time platform that abstracts the data layer Scalable Flexible Reliable Fast Servers Sensors Devices Mobile Analytics Personalization Monitoring And more…
- 5. Exactly Once Semantics Same goes for exactly-once semantics
- 6. Maybe exists Does not exist
- 12. storm + trident ● What is storm? ● What is trident? (what is transaction)
- 13. Exactly once processing - storm + trident
- 14. Common myths ● We use trident, so we guarantee “exactly once” ● If a tuple in a transaction failed, the whole transaction will be repeated, and the computation done on the transaction so far will be discarded ● Someone already put up a working transactional state
- 15. Transactional state is hard (impossible?) ● Trident States
- 16. Even if the backing store is transactional! Theory: - begin commit => begin transaction in the backing store - update state => write to the backing store - commit => commit in the backing store Practice: - This only works for single-threaded states! - commit is called once per thread Experience: - Even in single threaded state, the thread can crash between commit to the backing store and ack
- 17. Fake it ‘till you make it Idempotency to the rescue
- 18. Fake it ‘till you make it Idempotency to the rescue Simple (non distributed) example: TCP sequence numbers
- 19. Fake it ‘till you make it Idempotency to the rescue Simple (non distributed) example: TCP sequence numbers
- 20. Fake it ‘till you make it Idempotency to the rescue Simple (non distributed) example: TCP sequence numbers
- 21. Fake it ‘till you make it Idempotency to the rescue Simple (non distributed) example: TCP sequence numbers
- 22. Fake it ‘till you make it Idempotent distributed examples - Database with primary key (INSERT … IGNORE) - Kafka with log compaction
- 23. Kafka Log Compaction ● Stores only the most recent message per key, thus idempotent. ● We can achieve exactly once even without transactional state!
- 24. Exactly once - with idempotence
- 25. Maybe exists Does not exist
- 26. Questions?
- 27. https://storm.apache.org/documentation/Trident-state https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/ Storm Real-Time Processing Cookbook, Quinton Anderson https://github.com/quintona/trident-kafka- push/blob/master/src/main/java/com/github/quintona/KafkaState.java Resources