Hands On With Spark: Creating A Fast Data Pipeline With Structured Streaming And Spark Streaming
3 likes18,889 views
The document discusses creating a fast data pipeline using Apache Spark's Structured Streaming and Spark Streaming. It presents a sensor anomaly detection pipeline that uses Structured Streaming for data exploration, preparation, and anomaly detection, and Spark Streaming for online model creation and training. It compares the execution and abstraction models of Structured Streaming and Spark Streaming, and demonstrates how to build the sensor anomaly detection pipeline using Kafka sources and sinks with SQL operations, event time windows, and watermarks.
![@maasg
Structured Streaming
val ctx= new StreamingContext(conf,Seconds(1))
val lines = ssc.socketTextStream ("localhost" , 9999)
val words = lines.flatMap(_.split(" "))
val pairs = words.map(word => (word, 1))
val wordCounts = pairs.reduceByKey(_ + _)
wordCounts.print()
ctx.start()
https://spark.apache.org/docs/latest/streaming-programming-guide.html
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
val lines = spark.readStream
.format("socket")
.option("host", "localhost")
.option("port", 9999)
.load()
val words= lines.as[String].flatMap(_.split(" "))
val wordCounts = words.groupBy("value").count()
val query = wordCounts.writeStream
.outputMode("complete")
.format("console")
.start()](https://image.slidesharecdn.com/webinar-streaming-with-spark1-181207125041/85/Hands-On-With-Spark-Creating-A-Fast-Data-Pipeline-With-Structured-Streaming-And-Spark-Streaming-10-320.jpg)
![@maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]](https://image.slidesharecdn.com/webinar-streaming-with-spark1-181207125041/85/Hands-On-With-Spark-Creating-A-Fast-Data-Pipeline-With-Structured-Streaming-And-Spark-Streaming-13-320.jpg)
![@maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]
Kafka Source
Memory Sink
SQL Operations](https://image.slidesharecdn.com/webinar-streaming-with-spark1-181207125041/85/Hands-On-With-Spark-Creating-A-Fast-Data-Pipeline-With-Structured-Streaming-And-Spark-Streaming-17-320.jpg)
![@maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]
Kafka Source
Memory Sink
SQL Operations
Event Time
Windows
Watermark
Kafka Sink](https://image.slidesharecdn.com/webinar-streaming-with-spark1-181207125041/85/Hands-On-With-Spark-Creating-A-Fast-Data-Pipeline-With-Structured-Streaming-And-Spark-Streaming-18-320.jpg)
![@maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]
Kafka Source
Memory Sink
SQL Operations
Event Time
Windows
Watermark
Kafka Sink
RDD Programming
Local vs Distributed
Use Spark SQL
Kafka Source + Sink](https://image.slidesharecdn.com/webinar-streaming-with-spark1-181207125041/85/Hands-On-With-Spark-Creating-A-Fast-Data-Pipeline-With-Structured-Streaming-And-Spark-Streaming-19-320.jpg)
![@maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]
Kafka Source
Memory Sink
SQL Operations
Event Time
Windows
Watermark
Kafka Sink
RDD Programming
Local vs Distributed
Use Spark SQL
Kafka Source + Sink
Controlled Streams:
Spark Streaming +
Structured Streaming](https://image.slidesharecdn.com/webinar-streaming-with-spark1-181207125041/85/Hands-On-With-Spark-Creating-A-Fast-Data-Pipeline-With-Structured-Streaming-And-Spark-Streaming-20-320.jpg)
Hands On With Spark: Creating A Fast Data Pipeline With Structured Streaming And Spark Streaming
- 1. Hands-on with Apache Spark: Creating a Fast Data Pipeline with Structured Streaming and Spark Streaming Gerard Maas Senior SW Engineer, Lightbend, Inc.
- 2. Gerard Maas Señor SW Engineer @maasg https://github.com/maasg https://www.linkedin.com/ in/gerardmaas/ https://stackoverflow.com /users/764040/maasg
- 3. @maasg Agenda Creating a Fast Data Pipeline with Structured Streaming and Spark Streaming
- 6. @maasg Data Pipelines ● Create Composable Streaming Applications ● Using the Best Tool for the Job ● Generating a Network Effect
- 7. @maasg Agenda Creating a Fast Data Pipeline with Structured Streaming and Spark Streaming
- 8. @maasg Apache Spark Core Spark SQL SparkMLLib SparkStreaming Structured Streaming Datasets/Frames GraphFrames Data Sources
- 9. @maasg Apache Spark Core Spark SQL SparkMLLib SparkStreaming StructuredStreaming Datasets/Frames GraphFrames Data Sources RDD DStream Query
- 10. @maasg Structured Streaming val ctx= new StreamingContext(conf,Seconds(1)) val lines = ssc.socketTextStream ("localhost" , 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ctx.start() https://spark.apache.org/docs/latest/streaming-programming-guide.html https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words= lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count() val query = wordCounts.writeStream .outputMode("complete") .format("console") .start()
- 11. @maasg Time Execution Abstraction Structured Streaming Spark Streaming Abstract (Processing Time, Event Time) Fixed to microbatch Streaming Interval Fixed Micro batch, Best Effort MB, Continuous (NRT) Fixed Micro batch DataFrames/Dataset DStream, RDD Access to the scheduler
- 12. @maasg Agenda Hands On with Spark: Creating a Fast Data Pipeline with Structured Streaming and Spark Streaming
- 13. @maasg Sensor Data Multiplexer Data Exploration [Structured Streaming] Sensor Anomaly Detection Pipeline Data Preparation [Structured Streaming] Online Model Creation + Training [Spark Streaming] Anomaly Detection [Structured Streaming]
- 14. @maasg Sensor Data Multiplexer Structured StreamingLocal Process Sensor Anomaly Detection Structured Streaming
- 15. @maasg Sensor Data Multiplexer Structured StreamingLocal Process Sensor Anomaly Detection Structured Streaming
- 16. @maasg Live
- 17. @maasg Sensor Data Multiplexer Data Exploration [Structured Streaming] Sensor Anomaly Detection Pipeline Data Preparation [Structured Streaming] Online Model Creation + Training [Spark Streaming] Anomaly Detection [Structured Streaming] Kafka Source Memory Sink SQL Operations
- 18. @maasg Sensor Data Multiplexer Data Exploration [Structured Streaming] Sensor Anomaly Detection Pipeline Data Preparation [Structured Streaming] Online Model Creation + Training [Spark Streaming] Anomaly Detection [Structured Streaming] Kafka Source Memory Sink SQL Operations Event Time Windows Watermark Kafka Sink
- 19. @maasg Sensor Data Multiplexer Data Exploration [Structured Streaming] Sensor Anomaly Detection Pipeline Data Preparation [Structured Streaming] Online Model Creation + Training [Spark Streaming] Anomaly Detection [Structured Streaming] Kafka Source Memory Sink SQL Operations Event Time Windows Watermark Kafka Sink RDD Programming Local vs Distributed Use Spark SQL Kafka Source + Sink
- 20. @maasg Sensor Data Multiplexer Data Exploration [Structured Streaming] Sensor Anomaly Detection Pipeline Data Preparation [Structured Streaming] Online Model Creation + Training [Spark Streaming] Anomaly Detection [Structured Streaming] Kafka Source Memory Sink SQL Operations Event Time Windows Watermark Kafka Sink RDD Programming Local vs Distributed Use Spark SQL Kafka Source + Sink Controlled Streams: Spark Streaming + Structured Streaming
- 21. @maasg Time Execution Abstraction Structured Streaming Spark Streaming Abstract (Processing Time, Event Time) Fixed to microbatch Streaming Interval Fixed Micro batch, Best Effort MB, Continuous (NRT) Fixed Micro batch DataFrames/Dataset DStream, RDD Access to the scheduler
- 22. Resources Notebooks used today: https://github.com/maasg/spark-notebooks/tree/master/streaming-anomaly-detection Pipelines: https://www.reactivesummit.org/2018/schedule/taking-the-pain-out-of-deploying-streaming-applic ations Structured Streaming + Spark Streaming: https://www.reactivesummit.org/2018/schedule/processing-fast-data-with-apache-spark-the-tale-of -two-streaming-apis Fast Data: https://www.lightbend.com/products/fast-data-platform
- 23. Gerard Maas Señor SW Engineer @maasg https://github.com/maasg https://www.linkedin.com/ in/gerardmaas/ https://stackoverflow.com /users/764040/maasg
- 25. Thank You!