Apache kafka
Download as PPTX, PDF32 likes24,233 views
This document discusses Apache Kafka, an open-source distributed event streaming platform. It provides an introduction to Kafka's design and capabilities including: 1) Kafka is a distributed publish-subscribe messaging system that can handle high throughput workloads with low latency. 2) It is designed for real-time data pipelines and activity streaming and can be used for transporting logs, metrics collection, and building real-time applications. 3) Kafka supports distributed, scalable, fault-tolerant storage and processing of streaming data across multiple producers and consumers.
Apache kafka
- 2. Why Kafka Introduction Design Q&A
- 3. *Apache ActiveMQ, JBoss HornetQ, Zero MQ, RabbitMQ are respective brands of Apache Software Foundation, JBoss Inc, iMatix Corporation and Vmware Inc.
- 5. Transportation of logs Activity Stream in Real time. Collection of Performance Metrics ◦ CPU/IO/Memory usage ◦ Application Specific Time taken to load a web-page. Time taken by Multiple Services while building a web-page. No of requests. No of hits on a particular page/url.
- 6. Scalable: Need to be Highly Scalable. A lot of Data. It can be billions of message. Reliability of messages, What If, I loose a small no. of messages. Is it fine with me ?. Distributed : Multiple Producers, Multiple Consumers High-throughput: Does not require to have JMS Standards, as it may be overkill for some use-cases like transportation of logs. ◦ As per JMS, each message has to be acknowledged back. ◦ Exactly one delivery guarantee requires two-phase commit.
- 7. An Apache Project, initially developed by LinkedIn's SNA team. A High-throughput distributed Publish- Subscribe based messaging system. A Kind of Data Pipeline Written in Scala. Does not follow JMS Standards, neither uses JMS APIs. Supports both queue and topic semantics.
- 9. Handshake Producer Zookeeper Consumer Coordination Producer Kafka Broker . . Producer . Consumer . . Producer Kafka Broker
- 10. Coordination Handshake Store Consumed Offset Producer Zookeeper and Watch for Cluster Consumer 1 event (groupId1) Producer Kafka Broker (Partition 1) Consumer 2 . (groupId1) . Producer . . . Producer Kafka Broker (Partition 2) * Consumer 3 (groupId1) * Consumer 3 would not receive any data, as number of consumers are more than number of partitions.
- 11. Filesystem Cache Zero-copy transfer of messages Batching of Messages Batch Compression Automatic Producer Load balancing. Broker does not Push messages to Consumer, Consumer Polls messages from Broker. And Some others. Cluster formation of Broker/Consumer using Zookeeper, So on the fly more consumer, broker can be introduced. The new cluster rebalancing will be taken care by Zookeeper Data is persisted in broker and is not removed on consumption (till retention period), so if one consumer fails while consuming, same message can be re-consume again later from broker. Simplified storage mechanism for message, not for each message per consumer.
- 12. Producer Performance Consumer Performance Credit : http://research.microsoft.com/en-us/UM/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf