Kafka Streams at Scale (Deepak Goyal, Walmart Labs) Kafka Summit London 2019
3 likes1,343 views
The document discusses the implementation of Kafka Streams at scale by Walmart Labs, focusing on stateful processing of millions of real-time events. It outlines the architecture, challenges, and enhancements related to fault recovery, horizontal scalability, and cloud readiness, utilizing technologies like RocksDB. Future plans include features such as model inferencing and multi-tenancy support.
Kafka Streams at Scale (Deepak Goyal, Walmart Labs) Kafka Summit London 2019
- 1. Kafka Streams at Scale Deepak Goyal Customer Backbone (CBB) Walmart Labs
- 2. Stateful processing of multi- million events in real time Walmart
- 3. CBB Internal Kafka customer state updates, feature extraction, model-inferencing etc. event- filtering Interested Consumers CBB Data Pipeline Front Kafka 400 KS instances >1M eps
- 4. Kafka Streams •Ease of development •Library, not a framework •High processing throughput •Distributed-persistent storage •Exactly-once processing semantics @walmartlabs
- 6. Event Flow kafka cluster kafka streams cluster task-0 task-0’ stand-by task-0’’ stand-by change-log topic partition 0’’ partition 0’ partition 0 input topic partition 0’’ partition 0’ partition 0 rocks db rocks db active rocks db Event Flow @walmartlabs
- 7. Kafka Streams’ Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. RocksDB 5. Large Clusters @walmartlabs
- 8. Challenge 1. Fault Recovery Default bootstrap vs Cold bootstrap @walmartlabs
- 9. Default Bootstrap ChangeLog topic as a source of truth • Slow stand-by recovery • Log-Compacted change-log topics • Inefficient disk usage @walmartlabs
- 10. app cluster task-0’ stand-by task-0 kafka cluster task-0’ new stand-by change-log topic partition 0’’ partition 0’ partition 0 input topic partition 0’’ partition 0’ partition 0 active rocks db rocks db the source of truth stand-by recovery Default Bootstrap empty rocks db @walmartlabs
- 11. Cold Bootstrap @walmartlabs Active task as a source of truth • Lightning stand-by recovery • Efficient disk usage • Cross cloud cold-bootstraps
- 12. app cluster task-0’ stand-by task-0 task-0’ new stand-by kafka cluster change-log topic partition 0’’ partition 0’ partition 0 input topic partition 0’’ partition 0’ partition 0 active rocks db rocks db the source of truth Cold Bootstrap empty rocks db rocks db enhanced stand-by recovery the new source of truth @walmartlabs
- 13. Challenge 2. Horizontal Scalability •Dynamic Repartitioning •Scaling Lookups @walmartlabs
- 14. partition-0 0,4,8 partition-1 1,5,9 partition-2 2,6,10 partition-3 3,7,11 Repartitioning Logic partition-0 0,2,4,6,8,10 partition-1 1,3,5,7,9,11 partition-0 0,2,4,6,8,10 partition-1 1,3,5,7,9,11 partition-2 0,2,4,6,8,10 partition-3 1,3,5,7,9,11 partition-2 0,2,4,6,8,10 partition-3 1,3,5,7,9,11 @walmartlabs
- 15. Dynamic Repartitioning app cluster task-0 active rocks db task-0’ stand-by rocks db task-0’’ or 2 future stand-by rocks db task-2 becomes active rocks db task-2’ new stand-by empty rocks db rocks db @walmartlabs Scaling up from 2 to 4 partitions
- 16. Scaling Lookups •Queryable Stand-by •AKKA Server (Non Blocking IO) •Partition Specific Lookups @walmartlabs
- 17. Challenge 3. Cloud Readiness •Rack/AZ Aware Task Assignment •Partial Partition Assignment revocation @walmartlabs
- 18. Rack/AZ Aware Task Assignment AZ2 AZ3AZ1 task-0 active rocks db task-0’’ stand-by rocks db task-1 active rocks db task-1’ stand-by rocks db task-0’ stand-by rocks db task-1’’ stand-by rocks db StickyTaskAssignor using new config RACK_ID_CONFIG = “rack.id”; @walmartlabs https://github.com/apache/kafka/pull/4785
- 20. Enhancements to RocksDB Store •Column Family support •Eliminated Synchronized GETs •Queryable in suspended and restoration state @walmartlabs Challenge 4. RocksDB @walmartlabs
- 21. Challenge 5. Large Clusters •Rebalance time •Overriding broker defaults •Overriding stream defaults @walmartlabs
- 22. Rebalance Time Reduction •Bottleneck: Group Leader Broker •Partition Assignment Info • Compression •Better Encoding •24x smaller in terms of size @walmartlabs https://github.com/apache/kafka/pull/6162
- 23. Broker Configs Overriding Broker Defaults • message.max.bytes • replica.fetch.max.bytes • socket.request.max.bytes • offsets.load.buffer.size • min.insync.replicas @walmartlabs
- 24. Streams Configs Overriding Streams Defaults • acks (producer) • linger.ms (producer) • auto.offset.reset (consumer) • state.cleanup.delay.ms (streams) @walmartlabs
- 25. Benchmark @walmartlabs Kafka Cluster 17 (8-core) instances Streams Cluster 100 (2-core) instances Processing Rate 2.3M events per second
- 26. Up Next • Feature Extraction and Model Inferencing ! • Cold Bootstrap from other stand-by " • Cold-Bootstrap and Repartitioning for DSL# • TTL support for State Stores $ • Merge Operator for RocksJava % • Multi Tenancy & ' ( ) @walmartlabs
- 27. keep-streaming. . . . . . . . . . . . . . . . . . . @deepak-iiit Walmart We are hiring!