Apache kafka
- 1. Apache Kafka A distributed publish-subscribe messaging system Neha Narkhede, LinkedIn nnarkhede@linkedin.com, 11/11/11
- 3. Introduction to pub-sub Kafka at LinkedIn Hadoop and Kafka Design Performance Outline
- 4. What is pub sub ? Producer Consumer Producer Consumer Topic 1 Topic 2 Topic 3 subscribe publish(topic, msg) Publish subscribe system msg msg
- 5. Introduction to pub-sub Kafka at LinkedIn Hadoop and Kafka Design Performance Outline
- 6. Motivation Activity tracking Operational metrics
- 7. Kafka Distributed Persistent High throughput
- 8. Kafka at LinkedIn Frontend Real time monitoring BrokerBrokerBroker Hadoop DWH Security systems News feed Kafka Frontend ServiceService
- 9. Introduction to pub-sub Kafka at LinkedIn Hadoop and Kafka Design Performance Outline
- 10. Hadoop Data Load for Kafka Live data center Offline data center HadoopHadoopDev Hadoop FrontendFrontendReal time consumers KafkaKafkaKafka KafkaKafkaKafka HadoopHadoopPROD Hadoop
- 11. Multi DC data deployments Live data centers Offline data centers Kafka Real time consumers Real time consumers Real time consumers Real time consumers Real time consumers Real time consumers Kafka Hadoop HadoopHadoopHadoop Hadoop HadoopHadoopDWH
- 12. Volume 20B events/day 3 terabytes/day 150K events/sec
- 13. Message queues •ActiveMQ •TIBCO Log aggregators •Flume •Scribe •Low throughput •Secondary indexes •Tuned for low latency •Focus on HDFS •Push model •No rewindable consumption KAFKA
- 14. What Kafka offers Very high performance Elastically scalable Low operational overhead Durable, highly available (coming soon)
- 15. Architecture Producer Consumer Producer Broker Broker Broker Broker Consumer ZK
- 16. Introduction to pub-sub Kafka at LinkedIn Hadoop and Kafka Design Performance Outline
- 17. Efficiency #1: simple storage Each topic has an ever-growing log A log == a list of files A message is addressed by a log offset
- 18. Efficiency #2: careful transfer Batch send and receive No message caching in JVM Rely on file system buffering Zero-copy transfer: file -> socket
- 19. Multi subscribers 1 file system operation per request Consumption is cheap SLA based message retention Rewindable consumption
- 20. Guarantees Data integrity checks At least once delivery In order delivery, per partition
- 21. Automatic load balancing Consumer Producer Broker Broker Consumer Producer
- 22. Auditing # events published = # events consumed
- 23. Introduction to pub-sub Kafka at LinkedIn Hadoop and Kafka Design Performance Outline
- 24. Performance 2 Linux boxes • 16 2.0 GHz cores • 6 7200 rpm SATA drive RAID 10 • 24GB memory • 1Gb network link 200 byte messages Producer batch size 200 messages
- 25. Basic performance metrics • Producer batch size = 40K • Consumer batch size = 1MB • 100 topics, broker flush interval = 100K – Producer throughput = 90 MB/sec – Consumer throughput = 60 MB/sec – Consumer latency = 220 ms
- 26. Latency vs throughput 0 50 100 150 200 250 0 20 40 60 80 100 Producer throughput in MB/sec Consumerlatencyinms (100 topics, 1 producer, 1 broker)
- 27. Scalability 101 190 293 381 0 50 100 150 200 250 300 350 400 1 broker 2 brokers 3 brokers 4 brokers ThroughputinMB/s (10 topics, broker flush interval 100K)
- 28. Throughput vs Unconsumed data 0 40000 80000 120000 160000 200000 10 105 199 294 388 473 567 662 756 851 945 1039 (1 topic, broker flush interval 10K) Throughputinmsg/s Unconsumed data in GB
- 29. State of the system 4 clusters per colo, 4 servers each 850 socket connections per server 20 TB 430 topics Batched frontend to offline datacenter latency => 6-10 secs Frontend to Hadoop latency => 5 min
- 30. State of the system Successfully deployed in production at LinkedIn and other startups Apache incubator inclusion 0.7 Release • Compression • Cluster mirroring
- 31. Replication
- 32. Some project ideas Security Long poll More compression codecs Locality of consumption
- 33. Team • Jay Kreps • Jun Rao • Neha Narkhede • Joel Koshy • Chris Burroughs
- 37. API
Editor's Notes
- #5: Multiple subscribers, decoupling Some examples : email notification on adding connection etc
- #9: Every service in LinkedIn uses Kafka. 10-15 of real time consumers using Kafka
- #11: Give some background on – basically generating business reports Tracking data is crucial to measure user engagement – unique member visits per day, ad metrics CTR to ad publishers Also data analytics – new models for PYMK Collapse multple boxes into one. Show multiple Hadoop clusters If you track some new data, it will automatically reach Hadoop in a few minutes.
- #14: Messaging systems tuned for very low latency, but not high volume
- #18: One topic is enough
- #26: s
- #29: Add compression line here instead
- #30: Batching on frontends, batching on both kafka clusters, tuned for throughput
- #35: Kafka webpage here
- #36: single producer to send 10 million messages, 200 bytes each. single consumer thread Flush interval 10K ActiveMQ – syncOnWrite=false, KahaDB RabbitMQ – mandatory=true, immediate=false, file persistence the producer throughput in messages/secondKafka produces at the rate of 50K / second with batch size of 1 400K / second with batch size of 50. These numbers are orders of magnitude higher than that of ActiveMQ and at least twice better than RabbitMQ. No producer ACKS more compact storage format 9 bytes/message overhead in Kafka 144 bytes/message overhead in ActiveMQ Cost of maintaining heavy index structures One thread in ActiveMQ spent most of its time accessing a B-Tree to maintain message metadata and state
- #37: Kafka consumes at 22K messages / sec, more than 4 times that of ActiveMQ and RabbitMQ More efficient storage format - fewer bytes transferred from server to consumer The broker in ActiveMQ/RabbitMQ had to maintain delivery status for each message. One of the threads in ActiveMQ kept writing KahaDB pages to disk during the test. Kafka uses sendFile API to reduce the transmission overhead