Spark Summit EU talk by Kaarthik Sivashanmugam
Download as PPTX, PDF1 like551 views
This document introduces Mobius, a C# API for building Apache Spark applications in .NET. Mobius allows organizations invested in .NET to develop Spark jobs using C#. It provides bindings between the Scala/Java Spark API and C#, enabling reuse of existing .NET libraries in Spark and development of Spark applications using C# and F#. The document demonstrates running and debugging Mobius applications, and discusses performance considerations and internals of the Mobius driver and worker architecture.
![CSharpRDD
• C# operations use CSharpRDD which needs CLR to execute
– If no C# transformation or UDF, CLR is not needed ~ execution is
entirely JVM-based
• RDD<byte[]>
– Data is stored as serialized objects and sent to C# worker process
• Transformations are pipelined when possible
– Avoids unnecessary serialization & deserialization within a stage](https://image.slidesharecdn.com/9ckaarthiksivashanmugam-161103201414/85/Spark-Summit-EU-talk-by-Kaarthik-Sivashanmugam-20-320.jpg)
Spark Summit EU talk by Kaarthik Sivashanmugam
- 1. Build Your Next Apache Spark Job in .NET Using Mobius Kaarthik Sivashanmugam @kaarthikss
- 2. Mobius C# API for building Apache Spark applications in .NET
- 3. Motivation • Enable organizations invested deeply in .NET to build Apache Spark applications in C# • Reuse of existing .NET libraries in Spark applications
- 4. Yet Another Language Binding? Popularity of C# • StackOverflow.com Developer Survey • RedMonk Programming Language Rankings .NET ecosystem ~ enabling languages like F# Spark Survey Results MOST IMPORTANT ASPECTS OF SPARK 2015 2016 FASTEST GROWING AREAS FROM 2014 TO 2015 2015 2016
- 5. Mobius & Spark Scala/Java API SparkR PySpark C# API Apache Spark Spark Apps in C#
- 7. Develop & Launch Mobius Applications Spark Client A Get Mobius release B Get Mobius driver and dependencies 1 Add Reference to Mobius package in NuGet 2 Develop, debug, test Mobius driver application 3 Build Mobius driver Run sparkclr-submit.cmd or sparkclr-submit.sh C Runs Spark job Example: sparkclr-submit.cmd --master spark://IP:PORT --total-executor-cores 200 --executor-memory 12g --exe Pi.exe D:MobiusexamplesPi
- 8. Mobius & Spark C# Worker CLR IPC Sockets C# Worker CLR IPC Sockets C# Worker CLR IPC Sockets C# Driver CLR IPC Sockets SparkExecutor SparkExecutor SparkExecutor SparkContext JVM JVM JVM JVM Workers Driver Mobius can be used with any existing Spark cluster (Standalone, YARN) in Windows & Linux
- 9. DEMO – RUNNING MOBIUS APP
- 11. DEMO – DEBUGGING MOBIUS APP
- 12. Debug Mode > sparkclr-submit.cmd debug Debug Mobius Word Count Example in VS (set CSharpWorkerPath in config)
- 13. DEMO – USING MOBIUS C# SHELL
- 14. sparkclr-shell.cmd > var rdd = sc.Parallelize(Enumerable.Range(0, 100), 2); > rdd.Reduce((x,y) => x+y) //prints sum of the values;
- 15. DEMO – USING F# SHELL
- 16. fsi.exe > sparkclr-submit.cmd debug > fsi --use:C:tempmobius-init.fsx let dataframe = session.Read().Json(@"C:tempdata.json");; dataframe.Show();; dataframe.ShowSchema();; dataframe.Count();;
- 17. Kafka Streaming Example Initialize StreamingContext & Checkpoint Create Kafka DStream Use DStream transformations to count logs by loglevel within a time window Save log count Start stream processing
- 18. Mobius in Linux • Mono is used for using Mobius with Spark in Linux • Mobius project CI (build, unit & functional tests) in Ubuntu • Mobius validated in Ubuntu, CentOS, OSX • Mobius validated with Spark clusters in Azure HDInsight and Amazon Web Services EMR • More info at linux-instructions.md @ GitHub
- 19. Project Info • https://github.com/Microsoft/Mobius Contributions welcome! • MIT license • Discussions – StackOverflow: tag “SparkCLR” – Gitter: https://gitter.im/Microsoft/Mobius – Twitter: @MobiusForSpark
- 20. CSharpRDD • C# operations use CSharpRDD which needs CLR to execute – If no C# transformation or UDF, CLR is not needed ~ execution is entirely JVM-based • RDD<byte[]> – Data is stored as serialized objects and sent to C# worker process • Transformations are pipelined when possible – Avoids unnecessary serialization & deserialization within a stage
- 21. Performance Considerations • Map & Filter RDD operations in C# require serialization & deserialization of data ~ impacts performance – C# operations are pipelined when possible ~ minimizes Ser/De – Persistence is handled by JVM ~ checkpoint/cache on a RDD impacts pipelining for CLR operations • DataFrame operations without C# UDFs do not require Ser/De – Perf will be same as native Scala-based Spark application – Execution plan optimization & code generation perf improvements in Spark leveraged
- 22. INTERNALS OF DRIVER & WORKER
- 23. Driver-side Interop CSharpRunner JVM 1 Launch sparkclr-submit.cmd or sparkclr-submit.sh CSharpBackendLaunch Netty server creating proxy for JVM calls 2 C# Driver Launch C# process using port number from CSharpBackend 3 CLR SparkConf SparkContext Create and manage Proxies for JVM objects SparkConf SparkContext Interop Components Mirror C#-side operations Invoke JVM methods RDD DataFrame DStream …CSharpRDD RDD DataFrame DStream PipelinedRDD …
- 24. 1 Compute 2 CLR CSharpWorker.exe Launch Worker-side Interop JVM CSharpRDD Executor Spark Worker 3 Read bytes 5 Write bytes 4 Execute C# operation 1 Compute
- 25. THANK YOU. • Mobius is production-ready & Cloud-ready • Use Mobius to build Apache Spark jobs in .NET • Contribute to github.com/Microsoft/Mobius • @MobiusForSpark
Editor's Notes
- #11: using System.Linq; var rdd = sc.Parallelize(Enumerable.Range(0, 100), 2); rdd.Reduce((x,y) => x+y) //prints sum of the values; rdd.Count() rdd.Map(n => n+1).Reduce((x,y) => x+y)
- #13: using System.Linq; var rdd = sc.Parallelize(Enumerable.Range(0, 100), 2); rdd.Reduce((x,y) => x+y) //prints sum of the values; rdd.Count() rdd.Map(n => n+1).Reduce((x,y) => x+y)
- #14: using System.Linq; var rdd = sc.Parallelize(Enumerable.Range(0, 100), 2)
- #15: using System.Linq; var rdd = sc.Parallelize(Enumerable.Range(0, 100), 2); rdd.Reduce((x,y) => x+y) //prints sum of the values; rdd.Count() rdd.Map(n => n+1).Reduce((x,y) => x+y)
- #17: using System.Linq; var rdd = sc.Parallelize(Enumerable.Range(0, 100), 2); rdd.Reduce((x,y) => x+y) //prints sum of the values; rdd.Count() rdd.Map(n => n+1).Reduce((x,y) => x+y)