Spark Streaming: Pushing the throughput limits by Francois Garillot and Gerard Maas
17 likes4,204 views
This document discusses Spark Streaming and how it can push throughput limits in a reactive way. It describes how Spark Streaming works by breaking streams into micro-batches and processing them through Spark. It also discusses how Spark Streaming can be made more reactive by incorporating principles from Reactive Streams, including composable back pressure. The document concludes by discussing challenges like data locality and providing resources for further information.
![@maasg @huitseeker
Spark Streaming (Refresher)
DStream[T]
RDD[T] RDD[T] RDD[T] RDD[T] RDD[T]
t0 t1 t2 t3 ti ti+1](https://image.slidesharecdn.com/06gmaasfgarillot-151103190447-lva1-app6892/85/Spark-Streaming-Pushing-the-throughput-limits-by-Francois-Garillot-and-Gerard-Maas-4-320.jpg)
![@maasg @huitseeker
Spark Streaming (Refresher)
DStream[T]
RDD[T] RDD[T] RDD[T] RDD[T] RDD[T]
t0 t1 t2 t3 ti ti+1
RDD[U] RDD[U] RDD[U] RDD[U] RDD[U]
Transformations](https://image.slidesharecdn.com/06gmaasfgarillot-151103190447-lva1-app6892/85/Spark-Streaming-Pushing-the-throughput-limits-by-Francois-Garillot-and-Gerard-Maas-5-320.jpg)
![@maasg @huitseeker
Spark Streaming (Refresher)
DStream[T]
RDD[T] RDD[T] RDD[T] RDD[T] RDD[T]
t0 t1 t2 t3 ti ti+1
RDD[U] RDD[U] RDD[U] RDD[U] RDD[U]
Actions
Transformations](https://image.slidesharecdn.com/06gmaasfgarillot-151103190447-lva1-app6892/85/Spark-Streaming-Pushing-the-throughput-limits-by-Francois-Garillot-and-Gerard-Maas-6-320.jpg)
![Kafka:The Receiver-less model
Simplified Parallelism
Efficiency
Exactly-once semantics
Less degrees of freedom
val directKafkaStream = KafkaUtils.
createDirectStream[
[key class],
[value class],
[key decoder class],
[value decoder class] ](
streamingContext, [map of Kafka parameters], [set
of topics to consume]
)
spark.streaming.kafka.maxRatePerPartition](https://image.slidesharecdn.com/06gmaasfgarillot-151103190447-lva1-app6892/85/Spark-Streaming-Pushing-the-throughput-limits-by-Francois-Garillot-and-Gerard-Maas-24-320.jpg)
Spark Streaming: Pushing the throughput limits by Francois Garillot and Gerard Maas
- 1. SPARK STREAMING: PUSHING THE THROUGHPUT LIMITS, THE REACTIVE WAY François Garillot, Gerard Maas
- 2. Who Are We ? Gerard Maas Data Processing Team Lead François Garillot work done at Spark Streaming at
- 3. @maasg @huitseeker Spark Streaming (Refresher)
- 4. @maasg @huitseeker Spark Streaming (Refresher) DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1
- 5. @maasg @huitseeker Spark Streaming (Refresher) DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 RDD[U] RDD[U] RDD[U] RDD[U] RDD[U] Transformations
- 6. @maasg @huitseeker Spark Streaming (Refresher) DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 RDD[U] RDD[U] RDD[U] RDD[U] RDD[U] Actions Transformations
- 7. @maasg @huitseeker Spark Streaming (Refresher) Spark API for Streams Fault-tolerant High Throughput Scalable
- 8. @maasg @huitseeker Streaming Spark t0 t1 t2 #0 Consumer Consumer Consumer Scheduling
- 9. @maasg @huitseeker Streaming Spark t0 t1 t2 #1 Consumer Consumer Consumer #0 Scheduling Process Time < Batch Interval
- 10. @maasg @huitseeker Streaming Spark t0 t1 t2 #2 Consumer Consumer Consumer #0 #1 #3 Scheduling Scheduling Delay
- 11. @maasg @huitseeker From Streams to μbatches Consumer #0 #1 batchInterval blockInterval Spark Streaming Spark #partitions = receivers x batchInterval / blockInterval
- 12. @maasg @huitseeker From Streams to μbatches #0 RDD Partitions Spark Spark Executors Spark Streaming
- 13. @maasg @huitseeker From Streams to μbatches #0 RDD Spark Spark Executors Spark Streaming
- 14. @maasg @huitseeker From Streams to μbatches #0 RDD Spark Spark Executors Spark Streaming
- 15. @maasg @huitseeker From Streams to μbatches #0 RDD Spark Spark Executors Spark Streaming
- 18. @maasg @huitseeker From Streams to μbatches Consumer #0 #1 batchInterval blockInterval Spark Streaming Spark #partitions = receivers x batchInterval / blockInterval
- 19. @maasg @huitseeker From Streams to μbatches Consumer #0 #1 batchInterval blockInterval Spark Streaming Spark spark.streaming.blockInterval = batchInterval x receivers / (partitionFactor x sparkCores)
- 20. @maasg @huitseeker The Importance of Caching dstream.foreachRDD { rdd => rdd.cache() // cache the RDD before iterating! keys.foreach{ key => rdd.filter(elem=> key(elem) == key).saveAsFooBar(...) } rdd.unpersist() }
- 21. @maasg @huitseeker Intervals (Read TD’s Adaptive Stream Processing using Dynamic Batch Sizing before drawing any conclusions !) O(n²) O(n²) O(n) O(n)
- 22. @maasg @huitseeker The Receiver model spark.streaming.receiver.maxRate Fault tolerance ? WAL
- 23. @maasg @huitseeker Direct Kafka Stream compute(offsets)
- 24. Kafka:The Receiver-less model Simplified Parallelism Efficiency Exactly-once semantics Less degrees of freedom val directKafkaStream = KafkaUtils. createDirectStream[ [key class], [value class], [key decoder class], [value decoder class] ]( streamingContext, [map of Kafka parameters], [set of topics to consume] ) spark.streaming.kafka.maxRatePerPartition
- 26. @maasg @huitseeker Reactive Principles Reactive Streams : composable back-pressure
- 27. @maasg @huitseeker Spark Streaming made Reactive
- 28. @maasg @huitseeker Spark Streaming made Reactive
- 29. @maasg @huitseeker Spark Streaming made Reactive
- 30. @maasg @huitseeker Spark Streaming Made Reactive
- 33. @maasg @huitseeker Pain point : Data Locality - Where is your job getting executed ? spark.locality.wait & spark.streaming.blockInterval - On Mesos, it’s worse (SPARK-4940)
- 34. @maasg @huitseeker Resources Backpressure in Spark Streaming: http://blog.garillot.net/post/121183250481/a-quick-update-on-spark-streaming-work-since-i The Virdata’s Spark Streaming tuning guide: http://www.virdata.com/tuning-spark/ TD’s paper on dynamic batch sizing : http://dl.acm.org/citation.cfm?id=2670995 Diving into Spark Streaming Execution Model: https://databricks.com/blog/2015/07/30/diving-into-spark-streamings-execution-model.html Spark Streaming / Storm Trident numbered comparison: https://www.cs.utoronto.ca/~patricio/docs/Analysis_of_Real_Time_Stream_Processing_Systems_Considering_Latency.pdf Kafka direct approach: https://github.com/koeninger/kafka-exactly-once/blob/master/blogpost.md