SPARK STREAMING Spark Hadoop User Group Munich Meetup 2016
1 like377 views
Spark Streaming is a framework for scalable, fault-tolerant stream processing of data in motion. It allows processing of live data streams using the same programming model as regular batch processing. Spark Streaming receives live input data streams, discretizes them into batches called DStreams, and processes the data using transformations like map, reduce, join and windowing.
SPARK STREAMING Spark Hadoop User Group Munich Meetup 2016
- 1. SPARK STREAMING ... IN 10 MINUTES sven.tessmann@comsysto.com @setema
- 3. FROM "DATA AT REST" TOWARDS "DATA IN MOTION"
- 4. LinkedIn processes 1.1 Trillion Messages a day (2015, using Kafka) Microsoft processes 10TB of O ce 365 Event Data a day (2015, using Spark)
- 5. "FAST DATA"
- 8. "scalable, high-throughput, fault-tolerant stream processing framework"
- 10. DISCRETIZED STREAM (DSTREAM) a sequence of distributed datasets (RDDs) representing a continuous stream of data
- 13. Execution Model
- 14. LET'S SEE SOME CODE
- 15. THANK YOU!