Real Time Data Processing With Spark Streaming, Node.js and Redis with Visualization
Download as PPTX, PDF1 like2,668 views
The document provides an overview of Spark Streaming, covering its concepts, execution environment, and parallelism. It includes a demonstration with data sources such as Redis and technologies like Node.js and AngularJS integrated into the dashboard. Key aspects of Spark Streaming such as the driver program, RDDs, dstreams, and receiver types are discussed in detail.
Real Time Data Processing With Spark Streaming, Node.js and Redis with Visualization
- 1. RealTime DataProcessing with Spark Streaming Brandon O’Brien Oct 26th, 2016
- 2. Spark Streaming: Intro 1. Intro 2. Demo + highlevelwalkthrough 3. Sparkin detail 4. Detaileddemowalkthroughand/or workshop
- 3. Spark Streaming Sparkexperiencelevel? Selectone: Beginner Intermediate Expert
- 5. Spark Streaming: Demo Info • Data Source: • Data Producer Thread • Redis • Data Consumer • Spark as Stream Consumer • Redis Publish • Dashboard: • Node.js/Redis Integration • Socket.io Publish • AngularJS + JavaScript
- 6. Spark Streaming: Spark in detail SPARK IN DETAIL
- 7. Spark Streaming: Concepts Application: • Driver program • RDD • Partition • Elements • DStream • InputReceiver • 1 JVM for driver program • 1 JVM per executor Cluster: • Master • Executors • Resources • Cores • Gigs RAM • Cluster Types: • Standalone • Mesos • YARN
- 8. Spark Streaming: Lazy execution //Allocate resources on cluster val conf = new SparkConf().setAppName(appName).setMaster(master) val sc = new SparkContext(conf) //Lazy definition of logical processing (transformations) val textFile = sc.textFile("README.md") .filter(line=> {line.length> 10}) //foreachPartition() triggers execution (actions) textFile.foreachPartition(partition=> { partition.foreach(line => { println(line) }) }) • Use rdd.persist() when multiple actions are called on the same RDD
- 9. Spark Streaming: Execution Env • Distributed data, distributed code • RDD partitions are distributed across executors • Actions trigger execution and return results to the driver program • Code is executed on either the driver or executors • Be careful of function closures! //Function arguments to transformations executed on executors val textFile = sc.textFile("README.md") .filter(line=> {line.length> 10}) //collect() triggers execution (actions) //executed on driver. foreachPartition executed on executors textFile.collect().foreach(line => { println(line) })
- 10. Spark Streaming: Execution Env
- 11. Spark Streaming: Parallelism • RDD partitions are processed in parallel • Elements in a single partition are processed serially • You control the number of partitions in an RDD • If you need to guarantee any particular ordering of processing, use groupByKey() to force all elements with the same key onto the same partitions • Be careful of shuffles val textFile = sc.textFile("README.md”) val singlePartitionRDD = textFile.repartition(1) val linesByKey = shopResultsEnriched .map(line => (getPartitionKey(line), line)) .groupByKey()
- 12. Spark Streaming: DStreams • Receiver Types • Kafka (Receiver + Direct) • Flume • Kinesis • TCP Socket • Custom (Ex: redis.receiver.RedisReceiver.scala) • Note: Kafka receiver will consume an entire core (no context switch)
- 13. RealTime DataProcessing with Spark Streaming Brandon O’Brien Oct 26th, 2016
Editor's Notes
- #3: Streaming data, streaming customer behavior, up to thousands per second, Kafka cluster, new apps produce/consume streams, Spark higher level than Storm.