Apache Kafka Streams Use Case
Download as PPTX, PDF0 likes2,023 views
The document outlines Kafka Streams as a distributed and fault-tolerant stream processing framework, emphasizing the concepts of ktable and kstream, which represent data records in a changelog and support various transformations. It covers time notions like event time and processing time, aggregation operations, and join functionalities between streams and tables. Additionally, it presents a use case for implementing a price alert feature using Kafka Streams, highlighting its event-driven capabilities.
Apache Kafka Streams Use Case
- 1. Kafka Streams Distributed, fault tolerant stream processing
- 2. Kafka Streams: Stream-table duality ● Almost every stream processing needs database ● Stream of key-value messages can be considered as changelog of virtual two- column table ● Also changelog of this table can produce the same original stream
- 3. Kafka Streams Concepts: Ktable and Kstream ● In KStream each data record represents a self-contained datum in unbounded dataset ("alice", 1) --> ("alice", 3): sum for key “alice” will return 4 ● In KTable each data record represents an update in changelog stream ("alice", 1) --> ("alice", 3): sum for key “alice” will return 3 ● KTable provides an ability to look up current values of data records by keys, which is available through join operations.
- 4. Kafka Streams: Stateless transformations ● Both KStream and KTable are created from Kafka topic ● Following standard stateless transformations are available: - Branch (split) - Filter - Map - Flatmap - GroupBy - Peek etc…. ● All transformation methods can be chained together to compose a complex processor topology
- 5. Kafka Streams: Time Kafka Streams supports the following notions of time: ● Event time: The point in time when an event or data record occurred, created “by the source” ● Processing time: The point in time when the event or data record happens to be processed by the stream processing application ● Ingestion-time: The point in time when an event or data record is stored in a topic partition by a Kafka broker.
- 6. Kafka Streams: Aggregation and Join ● An aggregation operation takes one input stream or table, and yields a new table by combining multiple input records into a single output record. Examples of aggregations are computing counts or sum. ● Aggregation API has standard implementations ready like count and reduce. ● A join operation merges two input streams and/or tables based on the keys of their data records, and yields a new stream/table ● Records can be joined only by key. ● Logic depends on type of sources, so KStream-to-KStream, KStream-to-KTable and KTable-to-KTable join are slightly different operations.
- 8. Usecase: Price alert feature
- 9. Usecase: Price alert feature
- 10. Price alert feature: Classic ● Save user requests for price alerts to database ● Pay attention to flight offers add/update date ● Create complicated database queries to join user requests and new offers according to date, source and destination ● Use scheduler for publishing price alerts
- 11. Price alert feature: Event driven!
- 12. Price alert feature: Kafka Streams solution
- 13. Anatoly Tikhonov anatoly.tihonov@mentory.io https://www.linkedin.com/in/anatoly-tikhonov- 8617a483/ THANK YOU!