Stream Processing with Big Data: Knowledgent Big Data Palooza Meet-Up
Download as PPTX, PDF0 likes917 views
The document presents an overview of Apache Kafka, a distributed publish-subscribe messaging system used for real-time data processing. It covers key concepts such as topics, partitions, producers, and consumers, along with their roles within the Kafka ecosystem. Additionally, it discusses practical use cases like trading systems and online gaming for leveraging Kafka in big data environments.
Stream Processing with Big Data: Knowledgent Big Data Palooza Meet-Up
- 1. Stream Processing with Big Data Learn Apache Kafka Kishore Veleti Big Data Engineer ©2014 Knowledgent Group Inc. All Rights Reserved
- 2. • Big Data Engineer at Knowledgent • Background in enterprise application development using Hadoop stack, Java, PHP • Worked in Healthcare, Banking, and Social Media Applications • Passionate in sharing knowledge ©2014 Knowledgent 2 Group Inc. All Rights Reserved About Me
- 3. Tutorial ©2014 Knowledgent 3 Group Inc. All Rights Reserved
- 4. ©2014 Knowledgent 4 Group Inc. All Rights Reserved We will discuss: •What is Apache Kafka? • Apache Kafka Terminology • Apache Kafka – about Topic & Partition • Apache Kafka hands-on
- 5. What is Apache Kafka? • Apache Kafka is a publish-subscribe messaging system implemented as a distributed commit log ©2014 Knowledgent 5 Group Inc. All Rights Reserved • It is written in Java/Scala • Built by LinkedIn to process activity stream data from their website
- 6. • All the messages in Kafka are real-time • There are many subscribers to a message • Kafka persists messages to the disk • Messages are retained for a specific time period • Subscribers/clients store the state of their reads • Easy to replay messages ©2014 Knowledgent 6 Group Inc. All Rights Reserved What is Apache Kafka?
- 7. Apache Kafka Terminology ©2014 Knowledgent 7 Group Inc. All Rights Reserved • Message: A datum to send • Topic: Kafka maintains messages in categories called “topics” • Partition: A logical division of a topic • Producer: An API to publish messages to Kafka topic • Broker: A server • Cluster: Kafka cluster comprises one or more brokers • Consumer: API to consume published messages and process further • Replication: Kafka replicates log for each partition across servers
- 8. Apache Kafka Terminology & Big Picture Message Topic Partition Producer Broker Consumer At a high level, producers send messages over the network to the Kafka cluster. Kafka cluster in turn serves them up to consumers. ©2014 Knowledgent 8 Group Inc. All Rights Reserved
- 9. Apache Kafka Terminology & Big Picture Message Topic Partition Producer Broker Consumer Let’s do a hands-on exercise of Kafka with knowledge we’ve learned until now ©2014 Knowledgent 9 Group Inc. All Rights Reserved
- 10. Apache Kafka: About Topic and Partition Message Topic Partition Producer Broker Consumer In Kafka for each topic a partition log is maintained. Each partition is an ordered, immutable sequence of messages that is appended to Each message in the partition is assigned a sequential id number called the offset ©2014 Knowledgent 10 Group Inc. All Rights Reserved Partition 1 Writes Partition 2 Partition 3
- 11. Apache Kafka: About Topic and Partition Message Topic Partition Producer Broker Consumer In Kafka, a Producer is an API to publish messages to topic ©2014 Knowledgent 11 Group Inc. All Rights Reserved
- 12. Apache Kafka: About Topic and Partition Message Topic Partition Producer Broker Consumer In Kafka, a Consumer is an API to consume messages from topics ©2014 Knowledgent 12 Group Inc. All Rights Reserved
- 13. Apache Kafka Terminology & Big Picture Message Topic Partition Producer Broker Consumer Let’s do a hands-on exercise of Kafka with knowledge we’ve learned until now ©2014 Knowledgent 13 Group Inc. All Rights Reserved
- 14. Apache Kafka Use Cases ©2014 Knowledgent 14 Group Inc. All Rights Reserved • Trading Systems - Risk Identification in real-time • Change Data Capture - Capturing the changed data into data lake environment • Online Gaming - Identifying top scorers of a game
- 15. Thank you! Questions? ©2014 Knowledgent 15 Group Inc. All Rights Reserved