Kafka 101 - Meetup Kafka BR - Oracle
1 like111 views
The document discusses Kafka, an open-source distributed event streaming platform. It provides an overview of Kafka concepts including producers that write data, consumers that read data, topics to which data is published, brokers that manage the cluster, and Zookeeper which manages the cluster metadata. It also discusses partitions for parallelism and consumer groups for distributing consumption across consumers.
Kafka 101 - Meetup Kafka BR - Oracle
- 1. Kafka
- 2. Software Engineer CNCF Serverless WG CloudEvents SDK Maintainer Hello! I am Fábio José I am here because kafka is awesome! https://github.com/fabiojose 2
- 4. Kafka is . . . ▪ High throughput and fault tolerant messaging platform ▪ The best of both worlds: topics and queues ▪ Platform that delivers the messages at least once ▪ Reliable & a persistent storage 4
- 5. Kafka isn’t . . . ▪ Product that concerns about who (concerns about groups) ▪ A queue or a topic queue ▪ Smart! Kafka is dumb (on its best meaning) ▪ Slow or heavyweight 5
- 6. Main Concepts Producer Who writes data Consumer Who reads data Topics Where data is persisted 6
- 7. Main Concepts Broker Who... Cluster Who... Zookeeper Who manages the cluster and the Kafka metadata 7
- 8. Main Concepts Consumer Group A way to spread out the data consumption, even if consumers are geo separated Partition Parallelism units 8
- 9. 9 Kafka Broker topic a partition #0 partition #1 Producer topic a topic b partition #0 partition #1 partition #2 Producer topic a Producer topic b Producer topic b topic a Consumer #3 Consumer #1 Consumer #2 Consumer Group Y Consumer #3 Consumer #1 Consumer #2 Consumer Group Z a b topic b
- 10. Kafka Broker topic a partition #0 partition #1 Producer topic a topic b partition #0 partition #1 partition #2 Producer topic a Producer topic b Producer topic b topic a topic b Consumer #3 Consumer #1 Consumer #2 Consumer Group Y Consumer #3 Consumer #1 Consumer #2 Consumer Group Z a b
- 11. Kafka Broker topic a partition #0 partition #1 Producer topic a topic b partition #0 partition #1 partition #2 Producer topic a Producer topic b Producer topic b topic a topic b Consumer #3 Consumer #1 Consumer #2 Consumer Group Y Consumer #3 Consumer #1 Consumer #2 Consumer Group Z a b
- 12. Kafka Broker topic a partition #0 partition #1 Producer topic a topic b partition #0 partition #1 partition #2 Producer topic a Producer topic b Producer topic b topic a topic b Consumer #3 Consumer #1 Consumer #2 Consumer Group Y Consumer #3 Consumer #1 Consumer #2 Consumer Group Z a b
- 13. Kafka Broker topic a partition #0 partition #1 Producer topic a topic b partition #0 partition #1 partition #2 Producer topic a Producer topic b Producer topic b topic a topic b Consumer #3 Consumer #1 Consumer #2 Consumer Group Y Consumer #3 Consumer #1 Consumer #2 Consumer Group Z a b
- 14. Kafka Broker topic a partition #0 partition #1 Producer topic a topic b partition #0 partition #1 partition #2 Producer topic a Producer topic b Producer topic b topic a topic b Consumer #3 Consumer #1 Consumer #2 Consumer Group Y Consumer #3 Consumer #1 Consumer #2 Consumer Group Z a b
- 15. Kafka Broker topic a partition #0 partition #1 Producer topic a topic b partition #0 partition #1 partition #2 Producer topic a Producer topic b Producer topic b topic a Consumer #3 Consumer #1 Consumer #2 Consumer Group Y Consumer #3 Consumer #1 Consumer #2 Consumer Group Z a b topic b
- 16. Kafka Broker topic a partition #0 partition #1 Producer topic a topic b partition #0 partition #1 partition #2 Producer topic a Producer topic b Producer topic b topic a Consumer #3 Consumer #1 Consumer #2 Consumer Group Y Consumer #3 Consumer #1 Consumer #2 Consumer Group Z a b 💤 topic b #0 #1
- 17. Kafka Broker topic a partition #0 partition #1 Producer topic a topic b partition #0 partition #1 partition #2 Producer topic a Producer topic b Producer topic b topic a Consumer #3 Consumer #1 Consumer #2 Consumer Group Y Consumer #3 Consumer #1 Consumer #2 Consumer Group Z a b 💤 topic b #0 #1 #0 #2#1
- 18. Kafka Broker topic a partition #0 partition #1 Producer topic a topic b partition #0 partition #1 partition #2 Producer topic a Producer topic b Producer topic b topic a Consumer #3 Consumer #1 Consumer #2 Consumer Group Y Consumer #3 Consumer #1 Consumer #2 Consumer Group Z a b 💤 topic b #0 #1 #0 #2#1
- 19. Kafka Broker topic a partition #0 partition #1 Producer topic a topic b partition #0 partition #1 partition #2 Producer topic a Producer topic b Producer topic b topic a Consumer #3 Consumer #1 Consumer #2 Consumer Group Y Consumer #3 Consumer #1 Consumer #2 Consumer Group Z a b 💤 topic b #0 #1 #0 #2#1
- 20. Kafka BR
- 21. Thanks! Any questions? You can find me at ▪ @fabioj0se ▪ fabiojose@gmail.com 21
- 22. Credits Special thanks to all the people who made and released these awesome resources for free: ▪ Presentation template by SlidesCarnival ▪ Photographs by Unsplash 22