Kafka Streams VS Spark Structured Streaming - Modern Stream Processing Engines Compared
2 likes236 views
This document provides a comparison between Kafka Streams and Spark Structured Streaming, highlighting key features of both stream processing engines. Kafka Streams focuses on fault tolerance and high-availability with a high-level DSL, while Spark Structured Streaming supports ETL, machine learning, and offers a more extensive data source variety. The author, Jacek Laskowski, emphasizes the different programming capabilities and functionalities of each engine for developers and data scientists.
Kafka Streams VS Spark Structured Streaming - Modern Stream Processing Engines Compared
- 1. Kafka Streams vs Spark Structured Streaming Modern Stream Processing Engines Compared © Jacek Laskowski / @JacekLaskowski / jacek@japila.pl / DataMass Summit 2019
- 2. ● A freelance IT consultant ● Specializing in Spark, Kafka, Kafka Streams, Scala ● Development | Consulting | Training ● "The Internals Of" online books ● Among contributors to Apache Spark ● Among Confluent Community Catalyst (Class of 2019 - 2020) ● Contact me at jacek@japila.pl ● Follow @JacekLaskowski on twitter for more #ApacheSpark #ApacheKafka #KafkaStreams Jacek Laskowski
- 3. Friendly reminder Pictures...take a lot of pictures! 📷 © Jacek Laskowski / @JacekLaskowski / jacek@japila.pl
- 4. The Features of Both 1. Stream Processing Engines 2. High-Level DSL for defining processing flow (logic) a. Topology b. Dataflow 3. Low-Level API for custom flows 4. Logical and physical plans a. Logical “what” and executable “how” © Jacek Laskowski / @JacekLaskowski / jacek@japila.pl
- 5. Kafka Streams (1 of 2) 1. Kafka Streams 2.3.0 2. Java and Scala APIs 3. Yet Another Command-Line Application (“YACA”) a. High-availability and fault tolerance OOTB b. Creating consumer groups OOTB 4. Support for Apache Kafka only a. Use Kafka Connect to go beyond Kafka 5. Data Abstractions a. High-level Streams DSL (KStream, KTable, KGlobalTable) b. Low-level Processor API 6. One record at a time © Jacek Laskowski / @JacekLaskowski / jacek@japila.pl
- 6. Kafka Streams (2 of 2) 1. ETL only a. No support for SQL or Machine Learning b. KSQL 2. Java 11 supported 3. Scala 2.12 4. No interactive shell / REPL for learning and prototyping 5. Rich join support (stream-stream, stream-table, stream-global table) 6. Reading from and writing to a single Kafka cluster 7. Uses RocksDB for persistent state storage © Jacek Laskowski / @JacekLaskowski / jacek@japila.pl
- 7. Kafka Streams Code / Topology (1 of 2) © Jacek Laskowski / @JacekLaskowski / jacek@japila.pl
- 8. Kafka Streams Code / Execution Env (1 of 2) © Jacek Laskowski / @JacekLaskowski / jacek@japila.pl
- 9. Spark Structured Streaming (1 of 2) 1. Apache Spark 2.4.4 2. Stream Processing API for Scala, Java, Python, SQL a. Useful for software developers and data scientists 3. Requires cluster manager a. Apache Hadoop’s YARN / Apache Mesos / DC/OS / Spark Standalone 4. Lots of data sources a. Kafka, JSON, parquet, CSV, Avro, ORC, socket b. Data Source API 5. Data abstraction: streaming Dataset © Jacek Laskowski / @JacekLaskowski / jacek@japila.pl
- 10. Spark Structured Streaming (2 of 2) 1. ETL + Machine Learning a. Spark MLlib supports streaming Datasets 2. Java 8 only 3. Scala 2.12 4. spark-shell for learning and prototyping 5. Streaming joins and aggregations 6. Reading from one Kafka cluster and writing to another Kafka cluster 7. Uses Hadoop DFS (HDFS) for checkpointing and persistent state storage © Jacek Laskowski / @JacekLaskowski / jacek@japila.pl
- 11. Spark “Streams” Code / Loading Data (1 of 3) © Jacek Laskowski / @JacekLaskowski / jacek@japila.pl
- 12. Spark “Streams” Code / Processing (2 of 3) © Jacek Laskowski / @JacekLaskowski / jacek@japila.pl
- 13. Spark “Streams” Code / Saving Data (3 of 3) © Jacek Laskowski / @JacekLaskowski / jacek@japila.pl
- 14. “The Internals Of” Online Books 1. The Internals of Kafka Streams 2. The Internals of Spark Structured Streaming
- 15. Questions? 1. Follow @jaceklaskowski on twitter (DMs open) 2. Upvote my questions and answers on StackOverflow 3. Contact me at jacek@japila.pl 4. Connect with me at LinkedIn © Jacek Laskowski / @JacekLaskowski / jacek@japila.pl