Kafka and Kafka Streams Intro at iwomm in London
1 like159 views
Kafka is a distributed streaming platform that allows publishing and subscribing to streams of records. It uses a broker-based architecture with replication and partitioning to provide scalability and fault tolerance. Kafka Streams is a library for building stream processing applications that run as lightweight microservices embedded in the applications. It allows stateless and stateful transformations on streams of records using concepts like streams and tables. The demo will showcase Kafka Streams for stream processing use cases like enrichment and aggregation.
Kafka and Kafka Streams Intro at iwomm in London
- 1. Kafka, Brokers, Streaming IWOMM London, 25/04/2018 @eschmiegelow
- 2. Agenda I. Kafka - Brokers II. Kafka Streaming III. Demo
- 4. I. Kafka – message broking at scale Kafka – what is it? A publish-subscribe messaging system built as a distributed commit log Some facts: - http://kafka.apache.org - Developed at LinkedIn and open sourced in 2011 - Top Level project since 2012 and backed by Confluent - Users: LinkedIn (obviously), Yahoo, Netflix, Spotify and many others
- 5. I. Kafka – overview Kafka is persistent and distributed, offers replication and runs on a cluster of brokers coordinated by Zookeeper. Publishers send messages to the brokers. The brokers persist messages to disk in a log based queue. Each message is indexed by its offset Consumers request the messages using an offset/length based API
- 6. I. Kafka - Architecture • Everything is distributed: • Producers • Consumers • Brokers • Queues • Messages are persistent with TTL • Queues are append-only • Consumers maintain their own state • Throughput with partitioning and replication
- 7. I. Kafka Message Brokering
- 8. I. Kafka Topics Topics are queues – organized in partitions. A topic can have multiple partitions, these are replicated to multiple brokers.
- 9. I. Kafka Message Routing Messages are keyed, the hash is used to route messages to partitions
- 10. I. Kafka Compacted Topics Compacted Topics are specialised topics that retain the latest version of a key in memory
- 11. I. Kafka Consumers Two different APIs: • Low Level (Simple Consumer) with fine grained control over message offset and commits • Consumer Group API, which allows group coordinated consumption
- 12. I. Use Cases Some typical use cases: • Metrics − Kafka is often used for operational monitoring data. This involves aggregating statistics from distributed applications to produce centralized feeds of operational data. • Event Sourcing - Kafka is an excellent backend for Events thanks to its preservation of order, multiple consumption and exactly once semantics. • Stream Processing − Popular frameworks such as Akka Streams and Spark Streaming read data from a topic, processes it, and write processed data to a new topic where it becomes available for users and applications.
- 14. II. Kafka Streams - Overview Kafka Streams – what is it? Kafka Streams is Confluent’s lightweight approach to stream processing and is: ● an add on-library to any application ● Built on a the simple concept of pipes and sinks with functional transformations in between ● Runs within your applications, e.g. docker ● Is built for the JVM, i.e. Java, Scala and Kotlin
- 15. II. Kafka Streams - Benefits Why another stream processing framework? ● Kafka Streams is not a replacement for Flink, Spark and others ● It’s super lightweight, has almost no dependencies other than kafka ● Offers minimal latency, one record at a time processing ➔ It’s the ideal conversion, enrichment, and transformation toolkit for topic to topic processing
- 16. II. Kafka Streams - Apps Kafka Streams components live in your apps
- 17. II. Kafka Streams - Apps … but communicate with the kafka cluster and coordinate their work in managed threads. Internally, a Streams App has a topology of processors, each of which are responsible for a step in the flow, be it consuming from a source, transforming a record or writing to a sink.
- 18. II. Kafka Streams - Apps Distribution and parallelism: Consumers and threads
- 19. II. Kafka Streams - API Concepts Kafka Streams has two core concepts - streams and tables ● Streams are simply a sequence of records and correspond to a non compacted topic in Kafka ● Tables represent a key-centric view on topics and are usually backed by comcated topics In Kafka Streams, these interchangeable, i.e. a table can be converted to a stream representation and vice-versa
- 20. II. Kafka Streams - API Concepts In addition, Kafka streams allow stateless and stateful transformations: ● Stateless transformations are record by record based and cover the typical functions such as map, filter, groupBy etc.. ● Stateful transformations require a store and cover functions over collections of records such as window operations, reduce, count and aggregate
- 21. III. Demo