Balance Kafka Cluster with Zero Data Movement with Haochen Li & Yaodong Yang
0 likes888 views
The document discusses strategies for balancing Kafka clusters while minimizing data movement, emphasizing the importance of partition placement and load balancing for performance and cost efficiency. It outlines challenges such as heterogeneous partitions and storage retention requirements, and suggests implementing a continuous rebalancing mechanism based on load metrics. Additionally, it highlights the potential for improving load balance through a refined partition placement strategy and the plan to contribute to the Apache Kafka project.
Balance Kafka Cluster with Zero Data Movement with Haochen Li & Yaodong Yang
- 1. Balance Kafka Cluster with Zero Data Movement Yaodong Yang (Apple), Haochen Li (Apple)
- 2. Yaodong Yang, Apple Inc. May, 2023 Haochen Li, Apple Inc. NOT A CONTRIBUTION Balance Kafka Cluster with Zero Data Movement
- 3. Kafka Cluster Load Balancing • Bene fi ts • High Performance • Cost E ffi ciency • Determining Factors • Kafka Partition Placement • Kafka Partition Access Pattern • Challenges • Kafka Partitions are Heterogenous • Storage Retention Requirement • Produce & Consume Tra ffi c Pattern
- 4. Current Solution • Continuously rebalance Kafka cluster based on Load Metrics • collect the load metrics from Kafka • generate the cluster load model • compute the optimization proposal • execute the proposal • Overhead • data movement between di ff erent brokers • negative impact for producers and consumers • long time to fi nish (hours or even days) • infra cost
- 5. Data Ingestion Use Case • Workload Pattern • Data events are randomly assigned to partitions from the kafka topic • All partitions from one topic are consumed evenly • Kafka producers and consumers don’t have strict requirement for Kafka Partition Count • Kafka Partitions from the same topic • Same data volumes produced, consumed and retained
- 6. Kafka Partition Replica Placement • Partition Replica Placement Strategy • Partition Count • scale_number: Number of Leader Replica per broker for a topic • partition_count = scale_number * broker_count • Partition Replica Placement • For every Kafka Topic, the number of Replicas in each broker should be the same. • For every Kafka Topic, the number of Leader Replicas in each broker should be the same. • Same load on individual Kafka Brokers. • Same hardware utilization on individual Kafka brokers • CPU • Storage Volume • Network
- 7. Scenarios • New Topic Creation • Generate the Replica Assignment for the new topic • Create the topic in the Kafka cluster with the above Replica Assignment
- 8. Scenarios • Increase Partition Count: scale_number increase • Generate the Replica Assignment for the new partitions • Create partitions in the Kafka cluster with the above Replica Assignment
- 9. Scenarios • Add more brokers • Generate the Replica Assignment for partitions in new brokers • Create partitions in the Kafka cluster with the above Replica Assignment
- 10. Scenarios • Ingestion tra ffi c volume and retention changes • no impact on the load balance of Kafka Cluster • Remove some brokers • data movement is unavoidable • avoid it with cluster migration if possible • Cluster Migration & Merge • rebalance the cluster: • partition reassignment • scale_number increase
- 11. Implementation • Current • Implemented as a Topic Operator • Deployed in production • Plan • Open a KIP in Apache Kafka Project • Contribute back to upstream
- 12. Take Away • Partition Placement Strategy can greatly improve the Load Balance of Kafka Clusters
- 13. Thank you!