Keep Your Kafka Cloud Costs in Check with Showbacks
0 likes150 views
The document discusses managing Kafka cloud costs through a method called Kafka Showbacks, which assigns a proportional cost of cluster usage to different teams. It outlines strategies for identifying cluster costs, optimizing client configurations, and correlating client and cluster metrics to calculate costs effectively. Additionally, it emphasizes the importance of standardizing practices and suggests future steps for expanding the approach to other platforms.
Keep Your Kafka Cloud Costs in Check with Showbacks
- 1. © 2023 New Relic, Inc. All rights reserved Irene Rodríguez Antón Rodríguez Keep Your Kafka Cloud Costs in Check with Showbacks
- 2. © 2023 New Relic, Inc. All rights reserved New Relic Ingests Telemetry Data
- 3. © 2023 New Relic, Inc. All rights reserved Some production ingest numbers: 3.5TB/min 7.3B dpm 35 Kafka clusters 3 seconds
- 4. © 2023 New Relic, Inc. All rights reserved Ingest Pipelines @ New Relic
- 5. © 2023 New Relic, Inc. All rights reserved What’s the problem? Kafka client configurations impact the cluster sizing and reliability
- 6. © 2023 New Relic, Inc. All rights reserved How do we solve it? Kafka showbacks: a proportional part of the cost of the cluster is assigned to teams using it
- 7. © 2023 New Relic, Inc. All rights reserved Kafka cluster costs The first step is to identify your Kafka cluster costs and how to access them programmatically
- 8. © 2023 New Relic, Inc. All rights reserved How to assign Kafka cluster costs to teams? 1. Identify the cluster bottlenecks 2. Identify the client optimizations minimizing those bottlenecks 3. Identify the Client metrics and how they correlate with the Cluster metrics 4. Assign the costs based on the Client and Cluster metrics
- 9. © 2023 New Relic, Inc. All rights reserved Algorithm for brokers / CKUs ● Unused CPU cost assigned to Kafka Platform ● The remaining CPU cost assigned to every team based on the requests/s metric ● “linger.ms” client optimization to decrease the cost
- 10. © 2023 New Relic, Inc. All rights reserved Algorithm for network throughput ● The cluster cost is assigned proportionally to the client network throughput ● Be careful! There are many hidden costs ● Producer compression can make a big difference
- 11. © 2023 New Relic, Inc. All rights reserved Algorithm for storage ● Unused disk cost is assigned to Kafka Platform team ● We calculate the size of every topic and the consumers of each topic. ● We assign the used disk cost to the consumers proportionally to the size of the topic ● The goal is to choose the right value for topic configuration retention.ms
- 12. © 2023 New Relic, Inc. All rights reserved KafkaShowBackDemo: Architecture
- 13. © 2023 New Relic, Inc. All rights reserved KafkaShowBackDemo: Data generated Kafka Showback repository
- 14. © 2023 New Relic, Inc. All rights reserved KafkaShowBackDemo: Dashboard cost
- 15. © 2023 New Relic, Inc. All rights reserved KafkaShowBackDemo: CKUs cost
- 16. © 2023 New Relic, Inc. All rights reserved KafkaShowBackDemo: CKUs cost
- 17. © 2023 New Relic, Inc. All rights reserved KafkaShowBackDemo: CKUs cost
- 18. © 2023 New Relic, Inc. All rights reserved KafkaShowBackDemo: Network write cost
- 19. © 2023 New Relic, Inc. All rights reserved KafkaShowBackDemo: Request bytes
- 20. © 2023 New Relic, Inc. All rights reserved KafkaShowBackDemo: Compression
- 21. © 2023 New Relic, Inc. All rights reserved KafkaShowBackDemo: Network read cost
- 22. © 2023 New Relic, Inc. All rights reserved KafkaShowBackDemo: Response bytes
- 23. © 2023 New Relic, Inc. All rights reserved KafkaShowBackDemo: Storage
- 24. © 2023 New Relic, Inc. All rights reserved Challenges Standardize clientId Avoid bad patterns Identify non related Kafka costs
- 25. © 2023 New Relic, Inc. All rights reserved Next steps ● More Kafka flavours and Cloud providers ● Apply the same pattern to other platforms: routing, CDC, etc. ● Auto-dynamically optimize configuration for clients ● KIP-714: Client metrics and observability
- 26. © 2023 New Relic, Inc. All rights reserved Thank you. Irene: https://www.linkedin.com/in/irodriguezclaveria/ Anton: https://twitter.com/antonmry https://github.com/antonmry/kafka-showbacks-demo