Apache kafka
- 1. Apache Kafka Degendra Sivakoti MSc IT | Islington College
- 2. Apache Kafka Apache Kafka® is an open sourced distributed streaming platform.
- 3. History - Developed by LinkedIn / Microsoft - Open sourced in early 2011 - Donated to Apache Software Foundation on 2012 - Latest release and stable version 2.5.0 - Written in Scala/ Java
- 4. Why Kafka?
- 8. Major Features - Kafka as Messaging System - Kafka for Stream Processing
- 9. Core Components - Zookeeper - Core dependency as of version 2.5.0 - Broker - Kafka server - Producer - Produces message to topic partition - Consumer - Listens to topic partition - Topic - Partition
- 10. Topic - A topic can have one or more partitions - Topics are always multi-subscriber - Partitions (logs): - One leader and zero or more followers - Distributed
- 11. Producer - Produce message to a topic - Select topic partition - Normally, uses round-robin technique to balance load
- 12. Consumer - Consumer listens to one or more topics - Consumer belongs to a consumer group - Only on consumer from a consumer group can consume message from a topic partition. - Rebalancing - More consumers in a group than partition means idle consumers
- 13. Example
- 14. Kafka and BigData - Process existing big data - Sitting on HDFS - Sitting in a database - How does new data get into your cluster? - New log from web servers - New sensor data from IoT systems - New stock trades - Use for data ingestion in Hadoop system
- 15. Kafka and Hadoop Ecosystem
- 16. Questions?
- 17. Thank you!