Kafka Summit SF 2017 - Running Kafka as a Service at Scale
0 likes1,609 views
The document discusses the challenges and strategies involved in running Apache Kafka as a service at scale, emphasizing issues such as availability, latency, and durability in distributed systems. It highlights the importance of observability, testing practices, and improvements made to leader election and controller functionalities to enhance performance and reliability. The document also outlines measures to manage latency and ensures predictable performance amidst varying client demands.
Kafka Summit SF 2017 - Running Kafka as a Service at Scale
- 1. 1 Running Apache Kafka as a Service at Scale Sriram Subramanian, Director of Engineering
- 2. 2 52% of Kafka users in the cloud
- 4. 4 Managing Distributed Systems is really really hard
- 5. 5 Failure is inevitable – Cloud makes it worse
- 6. 6 Need to adapt to change
- 7. 7 Observability is a dark art You can’t fix what you can’t see
- 8. 8 Observability is a dark art Observability is a dark art perfected only with time and experience!
- 9. 9 Kafka is no different
- 10. 10 Takes years of iteration = v1.0
- 11. 11 Takes years of iteration &
- 13. 13 A stitch in time saves nine test data
- 14. 14 Types of testing we do today
- 16. 16
- 17. 17 Broker 1 Broker 2 Broker N Producer Producer Producer Consumer Consumer Kafka Cluster Topic partition REPLICATIONREPLICATION Failed tests involved correlated failures
- 18. 18 m1 m3 m4 m5 m1 m2 m4 m5 Replica A Replica B 0 1 2 3 0 1 2 3 State of the logs
- 19. 19 m1 m1 Replica A Replica B 0 0 HW HW HW = High Water Mark – Tracks the committed messages L How Replication Works?
- 20. 20 m1 m1 Replica A Replica B 0 0 HW HW m2 L How Replication Works?
- 21. 21 m1 m1 Replica A Replica B 0 0 HW HW L m21 How Replication Works?
- 22. 22 m1 m1 Replica A Replica B 0 0 HW HW L m2 m2 1 How Replication Works?
- 23. 23 m1 m1 Replica A Replica B 0 0 HW HW L m2m2 11 How Replication Works?
- 24. 24 m1 m1 Replica A Replica B 0 0 HW HW L m2m2 ACK 11 How Replication Works?
- 25. 25 m1 m1 Replica A Replica B 0 0 HW HW L m2m21 1 How Replication Works?
- 26. 26 m1 m1 Replica A Replica B 0 0 HW HW L m2m21 1 Updated HW to 1 How Replication Works?
- 27. 27 m1 m1 Replica A Replica B 0 0 HW HW L m2m21 1 How Replication Works?
- 28. 28
- 29. 29 m1 m1 Replica A Replica B 0 0 HW HW L m2m21 1 Root Cause
- 30. 30 m1 m1 Replica A Replica B 0 0 HW HW L m2m21 1 Root Cause
- 31. 31 m1 m1 Replica A Replica B 0 0 HW HW L m2Truncate 1 Root Cause
- 32. 32 m1 m1 Replica A Replica B 0 0 HW HW L m21 Root Cause
- 33. 33 m1 m1 Replica A Replica B 0 0 HW HW L m21 Root Cause Get m2
- 34. 34 m1 m1 Replica A Replica B 0 0 HW HW L m21 Root Cause
- 35. 35 m1 m1 Replica A Replica B 0 0 HW HW m21 Root Cause L Message M2 is Lost
- 37. 37 m1 m1 Replica A Replica B 0 0 HW HW L m2m21 1 Zero Data Loss LG0 0 LEADER GENERATION MAP LG0 LG0 LG0 LG0
- 38. 38 m1 m1 Replica A Replica B 0 0 HW HW L m2m21 1 Zero Data Loss LG0 0 LEADER GENERATION MAP LG0 LG0 LG0 LG0
- 39. 39 m1 m1 Replica A Replica B 0 0 HW HW L m2m21 1 LG0 0 LEADER GENERATION MAP LG0 LG0 LG0 LG0 Leader Generation Request LG0 Zero Data Loss
- 40. 40 m1 m1 Replica A Replica B 0 0 HW HW L m2m21 1 LG0 0 LEADER GENERATION MAP LG0 LG0 LG0 LG0 Offset = 1 Zero Data Loss
- 41. 41 m1 m1 Replica A Replica B 0 0 HW HW L m2m21 1 LG0 0 LEADER GENERATION MAP LG0 LG0 LG0 LG0 Zero Data Loss
- 42. 42 m1 m1 Replica A Replica B 0 0 HW HW L m2m21 1 LG0 0 LEADER GENERATION MAP LG0 LG0 LG0 LG0 Zero Data Loss
- 43. 43 m1 m1 Replica A Replica B 0 0 HW HW m2m21 1 LG0 0 LEADER GENERATION MAP LG0 LG0 LG0 LG0 L Zero Data Loss
- 44. 44 When availability became more important than ability
- 45. 45 Broker 1 Broker 2 Broker N Producer Producer Producer Consumer Consumer Kafka Cluster Topic partition REPLICATIONREPLICATION Controller Where did my topic partitions go?
- 46. 46 ● Leader Election ● Replica Reassignment ● Create Topic ● Delete Topic ● Add Partitions ● Broker start and shutdown Responsibilities of the Controller No Controller - No Cluster
- 47. 47 ● Controller state - topic creation, topic deletion etc ● Time taken to perform an operation ● Rate at which an admin operation is performed ● Queue sizes within the controller Lack of Metrics
- 48. 48 Root Cause Partition 1 Partition 2 Partition 3 Partition 4
- 49. 49 Root Cause Upgrades! ● Synchronous per-partition zookeeper writes ● Sequential per-partition controller-to-broker requests ● Complicated concurrency semantics ● No separation of control plane from data plane
- 50. 50 How did we fix it? Partition 1,2,3,4
- 51. 51 Zero controller downtime! ● Highly available cluster ● 10x faster leader elections ● More number of topic partitions per cluster ● Faster broker shutdown and upgrades
- 52. 52 When everything is INFINITE nothing is ever ENOUGH
- 53. 53 Broker 1 Broker 2 Broker N Producer Producer Producer Consumer Consumer Kafka Cluster Topic partition REPLICATIONREPLICATION Can I get my latency?
- 54. 54 But I have bytes quota set ● Throttle byte rate per second on the broker ● Response delayed on exceeding threshold ● Avoids bad clients from consuming all the bandwidth
- 55. 55 Root Cause Byte Rate Quotas is useful but not sufficient
- 56. 56 Root Cause ● Too many small sized requests ● DDos attack from client ● Decompression on the server takes a long time ● With more consumer instances, more requests
- 57. 57 bin/kafka-configs --zookeeper localhost:2181 --alter --add-config 'request_percentage=50' --entity-name user1 --entity-type users Request Quota - percentage of time a client can spend on request handler (I/O) threads and network threads within each quota window Predictable Latency
- 58. 58 + + Apache Kafka Tools Automation
- 59. 59 Apache Kafka as a Service
- 60. 60 confluent.io/confluent-cloud How do you get on Confluent Cloud?
- 61. 61 Thank You!