Balance Your Data Across Apache Kafka Partitions With Olena Kutsenko | Current 2022
0 likes404 views
This document provides a beginner's guide to balancing data across Apache Kafka partitions. It discusses optimizing producers by configuring parameters like acknowledgements, batch size, linger time, compression and record ordering. It also discusses optimizing consumers by configuring fetch size and number of partitions/consumers. The document recommends planning data partitioning in advance based on cardinality and ordering, and using custom partitioners. It suggests monitoring metrics like consumer lag and latency to identify issues.
Balance Your Data Across Apache Kafka Partitions With Olena Kutsenko | Current 2022
- 1. Beginners guide to balance your data across Apache Kafka partitions Olena Kutsenko Developer Advocate at Aiven 🦀
- 2. @OlenaKutsenko | @aiven_io 🥦 🧅 🫑🥕 🥖 🥔 🍎 🦞 🧀 🥫 🍅 Supermarket
- 3. @OlenaKutsenko | @aiven_io 🥦 🧅 🫑🥕 🥖 🥔 🍎 🦞 🧀 🥫 🍅 Supermarket
- 4. @OlenaKutsenko | @aiven_io 🍅 🥦 🧄 🧅 🫑🥕 🥖 🥔 🍎 🦞 🧀 🥫 💁 🍲 Supermarket 🍩 consumer producer
- 5. @OlenaKutsenko | @aiven_io topic 🔍 partition 1 partition 3 partition 2 consumer producer
- 6. @OlenaKutsenko | @aiven_io latency throughput and ⏳ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ ✉ Metrics
- 7. @OlenaKutsenko | @aiven_io Optimising producers acks = all 🦥 🤔 Sync or Async acks = 1 🏎 vs 1. Number of confirmations
- 8. @OlenaKutsenko | @aiven_io Optimising producers max batch size 📦 throughput latency high high low 1. Number of confirmations 2. Size of the batch
- 9. @OlenaKutsenko | @aiven_io Optimising producers when reaching linger.ms ⏱ 1. Number of confirmations 2. Size of the batch 3. Linger time max batch size 📦 Data sent on or 🔺 🔺
- 10. @OlenaKutsenko | @aiven_io Optimising producers 🗜 📦 1. Number of confirmations 2. Size of the batch 3. Linger time 4. Compression 📦📦 📦📦
- 11. @OlenaKutsenko | @aiven_io Optimising producers 1. Number of confirmations 2. Size of the batch 3. Linger time 4. Compression 5. Record ordering 🐘 🌾 🌳 ☀
- 12. @OlenaKutsenko | @aiven_io consumer p. 1 p. 2 p. 3 producer partitions 9 1 2 3 4 5 6 7 8 1 0 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 3 0 3 1 3 2 3 3 3 4 3 5 3 6 9 1 2 3 4 5 6 7 8 1 0 1 1 1 3 1 2 3 4 5 6 7 8 1
- 21. with most critical ordering. Key and data distribution 🦁 🐷 🐻 Instead of customer use shopping trip 🔺 🛒 🛍 🧾 💳 🔎 🔺 🔺 🔺
- 22. @OlenaKutsenko | @aiven_io 1.Find the highest cardinality, with most critical ordering. 2.Use custom partitioner key value Key and data distribution ✂
- 23. @OlenaKutsenko | @aiven_io Optimising consumers 1. Fetch size
- 24. @OlenaKutsenko | @aiven_io Optimising consumers 1. Fetch size 2. Number of partitions number of consumers in the group can be no bigger than number of partitions
- 25. @OlenaKutsenko | @aiven_io 1.Plan in advance 2.Observe and measure Action plan - consumer lag - fetch-latency-avg and fetch-latency-max - under replicated partitions
- 26. @OlenaKutsenko | @aiven_io Aiven’s “Rolling challenge” G10 go.aiven.io/aiven-challenge-current-na-22 Up for a challenge with Aiven for Apache Kafka? Participate and win a fun prize 🎁 Find us at