How Much Kafka?
6 likes944 views
This document discusses factors to consider when planning Apache Kafka capacity, including performance requirements, availability requirements, stability concerns, organizational concerns, operations comfort levels, Kafka limitations, costs, and upgrade rates. It recommends starting by choosing between ease of operation vs cost and fast vs stable. The document also addresses planning for multiple clusters, broker and Zookeeper nodes, topics, partitions, memory, clients, throughput, and benchmarking and tuning the Kafka deployment.
How Much Kafka?
- 1. 1 How Much Kafka? The Art and Science of Capacity Planning
- 2. 2 About Me ● Gwen Shapira ● Principal Data Architect ● At Confluent ● Committer to Apache Kafka ● Tweets. A lot. @gwenshap
- 3. 3 In Which We Answer Numeric Questions About Kafka
- 4. 4 Factors ● Performance Requirements ● Availability Requirements ● Stability concerns ● Organizational concerns ● Operations “comfort zone” ● Kafka limitations ● Costs ● Rate of upgrades
- 6. 6 Choose 2:
- 7. 7 It is tradeoffs all the way down ● Retention - disk size ● Throughput - network, CPU ● Producer performance - disk IO ● Consumer performance - CPU, memory Just performance requirements!
- 9. 9 Separate clusters for... ● DR ● Geographical distribution ● Writing vs Reading ● Real-time vs Batch ● Dev / Test ● High throughput ● Highly reliable ● Security
- 10. 10 You will have many clusters. Plan for it.
- 12. 12 Kafka is built to scale horizontally ● Largest cluster: 200+ nodes ● Lots of work on improving controller in 1.1, 2.0 ● Larger / more loaded brokers mean longer restarts and recovery. ● Larger brokers require tuning to take full advantage.
- 13. 13 Disks depends on version - Before 1.1: RAID 10 recommended - 1.1 and up: - KIP 112 - Broker will survive loss of disk - KIP 113 - Can assign replicas to specific disks
- 15. 15 Right now: 3 or 5
- 16. 16 Future: 0
- 18. 18 From capacity perspective Topics don’t exist.
- 20. 20 The 25K question 1. Read Jun Rao blog on topic 2. More partitions == more scale 3. More partitions == more throughput 4. More partitions != more speed 5. Controller improvements in 1.1 mean more partitions per broker
- 21. 21 Do not design for partition per user or device!
- 23. 23 # of partitions * Max Fetch Size + Compaction Buffer + “Extra”
- 25. 25 Lots. 50K has been done.
- 26. 26 Consumer groups have been tested at 50+ consumers per group
- 27. 27 Not all clients are same 1. Producers have very high throughput 2. Especially when tuned 3. EOS / Order require single-writer-for-entity 4. Many consumer groups is where Kafka shines
- 29. 29 Benchmarks ● Kafka ships with performance tools ● And your fav language has tools ● Your own workload (or similar) ● Your own configuration ● Your own failure scenarios
- 30. 30 Tuning ● Don’t fly blind ● Why is it slow? ● Where is the bottleneck? ● Version control for all configuration ● Automate the “change->test->observe” loop
- 31. 31 My broker is slow 101 ● Are all brokers working? ● Did you saturate network capacity? ● Is CPU utilization high? ● Are you running an old version? ● Do you have HUGE messages? ● Is it really the broker?
- 32. 32 Capture as many metrics as possible Alert / Dashboard select few
- 33. 33 Do you enjoy tuning, capacity planning, monitoring and all that? If yes - we are hiring. If not - check out Confluent Cloud
- 36. 36