From Message to Cluster: A Realworld Introduction to Kafka Capacity Planning
1 like629 views
The document discusses Kafka capacity planning, focusing on essential factors such as average message size, estimated daily quantity, replication factors, and network and disk throughput. It emphasizes the importance of stress testing, monitoring, and understanding trade-offs in Kafka's ecosystem for effective resource management. Additionally, it provides insights into compression options and the impact of topic partitions on performance and resource usage.
![{
"text": "RT @PostGradProblem: In preparation for the NFL lockout, I will be spending twice as much time analyzing my fantasy baseball
team during ...",
"truncated": true,
"in_reply_to_user_id": null,
"in_reply_to_status_id": null,
"favorited": false,
"source": "<a href="http://twitter.com/" rel="nofollow">Twitter for iPhone</a>",
"in_reply_to_screen_name": null,
"in_reply_to_status_id_str": null,
"id_str": "54691802283900928",
"entities": {
"user_mentions": [
{
"indices": [
3,
19
],
"screen_name": "PostGradProblem",
"id_str": "271572434",
"name": "PostGradProblems",
"id": 271572434
}
],
"urls": [ ],
"hashtags": [ ]
},
"contributors": null,
"retweeted": false,
"in_reply_to_user_id_str": null,
"place": null,
"retweet_count": 4,
"created_at": "Sun Apr 03 23:48:36 +0000 2011",
"retweeted_status": {
"text": "In preparation for the NFL lockout, I will be spending twice as much time analyzing my fantasy baseball team during
company time. #PGP",
"truncated": false,
"in_reply_to_user_id": null,
"in_reply_to_status_id": null,
"favorited": false,
"source": "<a href="http://www.hootsuite.com" rel="nofollow">HootSuite</a>",
"in_reply_to_screen_name": null,
"in_reply_to_status_id_str": null,
"id_str": "54640519019642881",
"entities": {
"user_mentions": [ ],
"urls": [ ],
"hashtags": [
Twitter JSON Payload ~6kb](https://image.slidesharecdn.com/clevelandkafka2020-11-02-201103190038/85/From-Message-to-Cluster-A-Realworld-Introduction-to-Kafka-Capacity-Planning-24-320.jpg)
From Message to Cluster: A Realworld Introduction to Kafka Capacity Planning
- 1. From Message to Cluster A Realworld Introduction to Kafka Capacity Planning. Jason “Jase” Bell - @jasonbelldata
- 4. MeetupCat is my spirit animal.
- 5. Flight Mode is ON! You may…… • Heckle. • Ask Questions. • Heckle More. • Talk about steak. • Heckle again.
- 6. What I’m Going To Cover
- 7. What I’m Going To Cover
- 8. • The Old Days. • The Now Times. • The Stuff We Don’t Talk About • The Message • What I Usually Ask For • Retention • Estimated Capacity • Compression • Stress Testing • Network and Disk Throughput • Topic Partitions • Kafka Connect • KSQL • Replicator • Parting Thoughts….. • ——————————————————— • Rapturous Applause • Encore (Probably Eye of the Tiger……)
- 9. The Old Days
- 11. The Now Times
- 13. The Stuff We Don’t Talk About
- 14. We think we know what we need from our Kafka Cluster
- 22. The Message
- 24. { "text": "RT @PostGradProblem: In preparation for the NFL lockout, I will be spending twice as much time analyzing my fantasy baseball team during ...", "truncated": true, "in_reply_to_user_id": null, "in_reply_to_status_id": null, "favorited": false, "source": "<a href="http://twitter.com/" rel="nofollow">Twitter for iPhone</a>", "in_reply_to_screen_name": null, "in_reply_to_status_id_str": null, "id_str": "54691802283900928", "entities": { "user_mentions": [ { "indices": [ 3, 19 ], "screen_name": "PostGradProblem", "id_str": "271572434", "name": "PostGradProblems", "id": 271572434 } ], "urls": [ ], "hashtags": [ ] }, "contributors": null, "retweeted": false, "in_reply_to_user_id_str": null, "place": null, "retweet_count": 4, "created_at": "Sun Apr 03 23:48:36 +0000 2011", "retweeted_status": { "text": "In preparation for the NFL lockout, I will be spending twice as much time analyzing my fantasy baseball team during company time. #PGP", "truncated": false, "in_reply_to_user_id": null, "in_reply_to_status_id": null, "favorited": false, "source": "<a href="http://www.hootsuite.com" rel="nofollow">HootSuite</a>", "in_reply_to_screen_name": null, "in_reply_to_status_id_str": null, "id_str": "54640519019642881", "entities": { "user_mentions": [ ], "urls": [ ], "hashtags": [ Twitter JSON Payload ~6kb
- 25. What I Usually Ask For
- 26. •Average Message Size What I’ll Ask Team For…
- 27. •Average Message Size •Estimated Daily Quantity What I’ll Ask Team For…
- 28. •Average Message Size •Estimated Daily Quantity •Any Peak Per Hour Quantity What I’ll Ask Team For…
- 29. •Average Message Size •Estimated Daily Quantity •Any Peak Per Hour Quantity •Desired Replication Factor What I’ll Ask Team For…
- 30. •Average Message Size •Estimated Daily Quantity •Any Peak Per Hour Quantity •Desired Replication Factor •Desired Partitions What I’ll Ask Team For…
- 31. •Average Message Size •Estimated Daily Quantity •Any Peak Per Hour Quantity •Desired Replication Factor •Desired Partitions •Minimum In-sync Replicas What I’ll Ask Team For…
- 32. •Average Message Size - (6 KB) •Estimated Daily Quantity - (10,000,000/d) •Any Peak Per Hour Quantity - (1,250,000) •Desired Replication Factor - (4) •Desired Partitions - (10) •Minimum In-sync Replicas - (2) What I’ll Ask Team For…
- 34. Estimated Capacity (Message size x 3) x Daily Qty x 1.4 (add 40%) = Volume per replicated broker.
- 35. Estimated Capacity (6KB x 3) x 10,0000,000 = 184,320,000 KB x 1.4 (add 40%) = 258,048,000 KB = 248.09 GB Roughly translates to 2.940 MB/sec
- 36. Estimated Capacity The x3 gives me a payload size with key, header, timestamp and the value. It’s just a rough calculation.
- 37. Estimated Capacity The x3 gives me a payload size with key, header, timestamp and the value. It’s just a rough calculation. Adding 40% overhead will give you some breathing space when someone does a stress test and doesn’t tell you…..
- 38. Retention (6KB x 3) x 10,0000,000 = 184,320,000 KB x 1.4 (add 40%) = 258,048,000 KB = 248.09 GB 248.09 GB/day x 14 days retention = 3.4 TB per broker.
- 39. Estimated Capacity df -hIs your friend…..
- 40. Estimated Capacity du -H .Is also your friend…..
- 42. Compression
- 43. Producer configuration compression.type defaults to “none”. Options are gzip, snappy, lz4 and zstd. Expect ~20%-40% message compression depending on the algorithm used.
- 44. Stress Testing
- 45. kafka-producer-perf-test --topic TOPIC --record-size SIZE_IN_BYTES
- 46. $ bin/kafka-producer-perf-test --topic testtopic --record-size 1000 --num- records 10000 --throughput 1000 --producer-props bootstrap.servers=localhost:9092 5003 records sent, 1000.4 records/sec (0.95 MB/sec), 1.6 ms avg latency, 182.0 ms max latency. 10000 records sent, 998.801438 records/sec (0.95 MB/sec), 1.12 ms avg latency, 182.00 ms max latency, 1 ms 50th, 2 ms 95th, 19 ms 99th, 23 ms 99.9th.
- 47. kafka-consumer-perf-test --broker-list host1:port1,host2:port2 --topic TOPIC
- 50. • D - Data to be written (MB/sec) • R - Replication Factor • C - Number of Consumer Groups (readers for each write)
- 51. The Volume of Writes: (D * R)
- 52. The Volume of Reads within Replication: ((R-1) * D) Reads happen internally by the replicas, this gives us:
- 53. The Volume of Reads within Replication: ((R - 1) * D) Reads happen internally by the replicas, this gives us: Adding the consumers we end up with: The Volume of Reads within Replication: (((R + C) - 1) * D)
- 54. We have memory! We have Caching! M/(D * R) = seconds of writes cached.
- 55. We have memory! We have Caching! M/(D * R) = seconds of writes cached. We have to assume that consumers might drop from the cache, consumers are running slower than expected or even that replicas might restart due to failure, patching or rolling restarts. Lagging Readers L = R + C - 1
- 56. Disk Throughput: D * R + L * D Network (reads) Throughput: ((R + C -1) * D) Network (writes) Throughput: D * R
- 57. Topic Partitions
- 62. You can set partitions either creating the topic (—partitions n) or afterwards.
- 63. Having a large number of partitions will have effects on Zookeeper znodes. • More network requests • If leader or broker goes down it may affect startup time as the broker returns to the cluster.
- 65. If you need to reduce partitions create a new topic and reduce the partition count.
- 66. Kafka Connect
- 69. Think about second and third order consequences if a connector would fail. What is the impact?
- 71. KSQL
- 72. ksqlDB •4 CPU Cores •32GB RAM •100GB SSD Disk •1Gbit Network Baseline Server Requirements
- 73. ksqlDB •Partition Count of 4 •Replication Factor of 1 Default Outbound Topic Assumptions (These settings can be modified within your CREATE query)
- 74. ksqlDB Some queries will require repartitioning and intermediate topics for certain operations, taking all available records. Default Outbound Topic Assumptions
- 75. ksqlDB Processing Small Message/Many Columns = CPU Saturation Default Outbound Topic Assumptions
- 76. ksqlDB Processing Large Message/Small Columns = Network Saturation Default Outbound Topic Assumptions
- 77. Replicator
- 78. Data Centre to Data Centre is going to lead to increased network latency.
- 79. On producers and consumers, use send.buffer.bytes and receive.buffer.bytes. On brokers, use socket.send.buffer.bytes and socket.receive.buffer.bytes.
- 81. Parting Thoughts
- 82. Consumer Group Lag Reports are your guiding light. (If you have Rundeck setup a scheduled job to email you the log output) 1
- 83. kafka-consumer-groups --bootstrap-server BROKER_ADDRESS --describe —group CONSUMER_GROUP --new-consumer
- 84. Kafka is about trade offs, from the producer right the way through to the consumer (and beyond). There’s no right or wrong answer, just experimentation, monitoring and learning. 2
- 85. While securing Kafka is important there is also a cost as certificates are verified and take up CPU resources. Your throughput will be affected. 3
- 86. The Kafka Ecosystem has increased in features over the last few years. This has lead to increased topic and disk space usages that need to be factored in to capacity planning calculations. 4
- 87. "Can you create me a topic please?”
- 89. Thank you. Many thanks to Shay and David for organising, everyone who attended and sent kind wishes. Lastly, a huge thank you to MeetupCat. Photo supplied by @jbfletch_