Apache® Spark™ 1.6 presented by Databricks co-founder Patrick Wendell
52 likes8,908 views
The document provides an overview of Apache Spark 1.6, highlighting key features such as unified memory management, performance improvements, and enhancements to the Spark SQL and MLlib. It details the various APIs, new algorithms, and improvements in data processing, as well as the pipeline persistence for machine learning models. The release is set to ship through the Apache Foundation in December, with a focus on collaboration and community engagement.
![Unified Memory Management in Spark 1.6
• Can cross between execution and storage memory
• When execution memory exceedsits own region,it can borrow as much of the
storage space as isfree and vice versa
• Borrowed storage memory can be evicted at any time
• Significantly reducesconfiguration
• Can define low water mark for storage (below which we won’tevict)
• Reference:[SPARK-10000]](https://image.slidesharecdn.com/spark-1-151203050842-lva1-app6891/85/Apache-Spark-1-6-presented-by-Databricks-co-founder-Patrick-Wendell-12-320.jpg)
![Dataset
“Encoder”converts from JVM Objectinto
a Dataset Row
Checkout[SPARK-9999]
JVM Object
Dataset
Row
encoder](https://image.slidesharecdn.com/spark-1-151203050842-lva1-app6891/85/Apache-Spark-1-6-presented-by-Databricks-co-founder-Patrick-Wendell-14-320.jpg)
![Dataset API in Spark 1.6
Typed interface over DataFrames / Tungsten
case class Person(name: String, age: Long)
val dataframe = read.json(“people.json”)
val ds: Dataset[Person] = dataframe.as[Person]
ds.filter(p => p.name.startsWith(“M”))
.groupBy($“name”)
.avg(“age”)](https://image.slidesharecdn.com/spark-1-151203050842-lva1-app6891/85/Apache-Spark-1-6-presented-by-Databricks-co-founder-Patrick-Wendell-15-320.jpg)
![Advanced Layout of Cached Data
Storing partitioning and orderingschemes in In-memory table scan
allowsfor performance improvements: e.g. in Joins, an extra partition step can
be saved based on thisinformation
Adding distributeBy and localSort to DF API
Similar to HiveQL’s DISTRIBUTE BY
allowsthe userto control the partitioning and ordering of a data set
Checkout [SPARK-4849]](https://image.slidesharecdn.com/spark-1-151203050842-lva1-app6891/85/Apache-Spark-1-6-presented-by-Databricks-co-founder-Patrick-Wendell-19-320.jpg)
![[Streaming] New improved state management
Introducing a DStream transformation for stateful streamprocessing
Does notscan every key
Easier to implementcommon use cases
timeout of idle data
returning items otherthan state
SupercedesupdateStateByKey in functionality and performance.
trackStateByKey (note, this name may change)](https://image.slidesharecdn.com/spark-1-151203050842-lva1-app6891/85/Apache-Spark-1-6-presented-by-Databricks-co-founder-Patrick-Wendell-20-320.jpg)
![[Streaming] trackStateByKey example
(name may change)
// Initial RDD input
val initialRDD = ssc.sparkContext.parallelize(...)
// ReceiverInputDStream
val lines = ssc.socketTextStream(...)
val words = lines.flatMap(...)
val wordDStream = words.map(x => (x, 1))
// stateDStream using trackStateByKey
val trackStateFunc = (...) { ... }
val stateDStream =
wordDStream.trackStateByKey(StateSpec.function(trackStateFunc).initialSta
te(initialRDD))](https://image.slidesharecdn.com/spark-1-151203050842-lva1-app6891/85/Apache-Spark-1-6-presented-by-Databricks-co-founder-Patrick-Wendell-21-320.jpg)
![[Streaming] Display the failed output op in Streaming
Checkout:
[SPARK-10885] PR#8950](https://image.slidesharecdn.com/spark-1-151203050842-lva1-app6891/85/Apache-Spark-1-6-presented-by-Databricks-co-founder-Patrick-Wendell-22-320.jpg)
![[MLlib]: Pipeline persistence
Persist ML Pipelinesto:
Save models in the spark.ml API
Re-run workflows in a reproducible manner
Export modelsto non-Sparkapps (e.g., model server)
This is more complexthan ML model persistence because:
Must persist Transformers and Estimators, not just Models.
We need a standard way to persist Params.
Pipelines and other meta-algorithms can contain other Transformers and Estimators,
including as Params.
We should save feature metadata with Models](https://image.slidesharecdn.com/spark-1-151203050842-lva1-app6891/85/Apache-Spark-1-6-presented-by-Databricks-co-founder-Patrick-Wendell-23-320.jpg)
![[MLlib]: Pipeline persistence
Reference[SPARK-6725]
Adding model export/import to
the spark.ml API.
Adding the internal
Saveable/Loadable API and
Parquet-based format](https://image.slidesharecdn.com/spark-1-151203050842-lva1-app6891/85/Apache-Spark-1-6-presented-by-Databricks-co-founder-Patrick-Wendell-24-320.jpg)
![R-like statistics for GLMs
Provide R-like summary
statistics for ordinary least
squares via normal equation
solver
Check out[SPARK-9836]](https://image.slidesharecdn.com/spark-1-151203050842-lva1-app6891/85/Apache-Spark-1-6-presented-by-Databricks-co-founder-Patrick-Wendell-25-320.jpg)
Apache® Spark™ 1.6 presented by Databricks co-founder Patrick Wendell
- 1. Apache Spark Release 1.6 Patrick Wendell
- 2. About Me @pwendell U.C. BerkeleyPhD, left to co-found Databricks Coordinate community roadmap Frequentreleasemanager for Spark
- 3. About Databricks Founded by Spark team, donated Spark to Apachein 2013 and lead developmenttoday. Collaborative, cloud-hosted data platform powered by Spark Free trial to check it out https://databricks.com/ We’re hiring!
- 4. … Apache Spark Engine Spark Core Spark Streaming Spark SQL MLlib GraphX Unified engineacross diverse workloads & environments Scale out, fault tolerant Python, Java, Scala, and R APIs Standard libraries
- 6. Users Distributors & Apps
- 7. Spark’s 3 Month Release Cycle For production jobs, use the latest release To try out unreleasedfeaturesor fixes, use nightly builds people.apache.org/~pwendell/spark-nightly/ master branch-1.6 V1.6.0 V1.6.1
- 8. Spark 1.6
- 9. Spark 1.6 Release Will ship upstreamthrough Apachefoundation in December (likely) Key themes Out of the box performance Previews of key new API’s Follow along with me at http://bit.ly/1OBkjMM
- 11. Memory Management in Spark: <= 1.5 • Two separate memory managers: • Execution memory: computation of shuffles, joins,sorts, aggregations • Storage memory: caching and propagating internal data sources across cluster • Challengeswith this: • Manual intervention to avoid unnecessary spilling • No good defaultsfor all workloads – meaning lost efficiency • Goal: Allowmemory regionsto shrink/growdynamically
- 12. Unified Memory Management in Spark 1.6 • Can cross between execution and storage memory • When execution memory exceedsits own region,it can borrow as much of the storage space as isfree and vice versa • Borrowed storage memory can be evicted at any time • Significantly reducesconfiguration • Can define low water mark for storage (below which we won’tevict) • Reference:[SPARK-10000]
- 13. History of Spark API’s RDD API (2011) Distribute collection of JVM objects Functional operators (map, filter, etc) DataFrame API (2013) Distribute collection of Row objects Expression-based operations and UDF’s Logical plans and optimizer Fast/efficient
- 14. Dataset “Encoder”converts from JVM Objectinto a Dataset Row Checkout[SPARK-9999] JVM Object Dataset Row encoder
- 15. Dataset API in Spark 1.6 Typed interface over DataFrames / Tungsten case class Person(name: String, age: Long) val dataframe = read.json(“people.json”) val ds: Dataset[Person] = dataframe.as[Person] ds.filter(p => p.name.startsWith(“M”)) .groupBy($“name”) .avg(“age”)
- 16. Tungsten Execution PythonSQL R Streaming DataFrame (& Dataset) Advanced Analytics
- 17. Other Notable Core Engine Features SQL directly over files Advanced JSON parsing Better instrumentation for SQL operators
- 18. Demos of What We Learned So Far
- 19. Advanced Layout of Cached Data Storing partitioning and orderingschemes in In-memory table scan allowsfor performance improvements: e.g. in Joins, an extra partition step can be saved based on thisinformation Adding distributeBy and localSort to DF API Similar to HiveQL’s DISTRIBUTE BY allowsthe userto control the partitioning and ordering of a data set Checkout [SPARK-4849]
- 20. [Streaming] New improved state management Introducing a DStream transformation for stateful streamprocessing Does notscan every key Easier to implementcommon use cases timeout of idle data returning items otherthan state SupercedesupdateStateByKey in functionality and performance. trackStateByKey (note, this name may change)
- 21. [Streaming] trackStateByKey example (name may change) // Initial RDD input val initialRDD = ssc.sparkContext.parallelize(...) // ReceiverInputDStream val lines = ssc.socketTextStream(...) val words = lines.flatMap(...) val wordDStream = words.map(x => (x, 1)) // stateDStream using trackStateByKey val trackStateFunc = (...) { ... } val stateDStream = wordDStream.trackStateByKey(StateSpec.function(trackStateFunc).initialSta te(initialRDD))
- 22. [Streaming] Display the failed output op in Streaming Checkout: [SPARK-10885] PR#8950
- 23. [MLlib]: Pipeline persistence Persist ML Pipelinesto: Save models in the spark.ml API Re-run workflows in a reproducible manner Export modelsto non-Sparkapps (e.g., model server) This is more complexthan ML model persistence because: Must persist Transformers and Estimators, not just Models. We need a standard way to persist Params. Pipelines and other meta-algorithms can contain other Transformers and Estimators, including as Params. We should save feature metadata with Models
- 24. [MLlib]: Pipeline persistence Reference[SPARK-6725] Adding model export/import to the spark.ml API. Adding the internal Saveable/Loadable API and Parquet-based format
- 25. R-like statistics for GLMs Provide R-like summary statistics for ordinary least squares via normal equation solver Check out[SPARK-9836]
- 26. Performance SPARK-10000 Unified Memory Management- Shared memory for execution and caching instead of exclusive division ofthe regions. SPARK-10917,SPARK-11149 In-memory ColumnarCache Performance - Significant (up to 14x) speed up when caching data that containscomplextypes in DataFrames or SQL. SPARK-11389 SQL Execution Using Off-Heap Memory - Support for configuring query execution to occur using off-heap memory to avoid GC overhead
- 27. Performance (continued) SPARK-4849 Advanced Layoutof Cached Data - storing partitioning and ordering schemesin In-memory table scan, and adding distributeBy and localSort to DF API SPARK-9858 Adaptive query execution - Initial supportfor automatically selecting the number of reducersfor joinsand aggregations.
- 28. Spark SQL SPARK-9999 Dataset API SPARK-11197 SQL Querieson Files SPARK-11745 Reading non-standard JSON files SPARK-10412 Per-operator Metrics for SQL Execution SPARK-11329 Star (*) expansion forStructTypes SPARK-11111 Fastnull-safe joins SPARK-10978 Datasource API Avoid Double Filter
- 29. Spark Streaming API Updates SPARK-2629 New improved state management SPARK-11198 Kinesisrecord deaggregation SPARK-10891 Kinesismessage handlerfunction SPARK-6328 Python Streaming ListenerAPI UI Improvements Made failuresvisible in the streaming tab, in the timelines,batch list, and batch detailspage. Made outputoperationsvisible in the streaming tab as progress bars
- 30. MLlib: New algorithms / models SPARK-8518 Survival analysis- Log-linearmodel for survival analysis SPARK-9834 Normal equation forleast squares - Normal equation solver,providing R-like model summary statistics SPARK-3147 Online hypothesistesting - A/B testing in the Spark Streaming framework SPARK-9930 New feature transformers - ChiSqSelector, QuantileDiscretizer,SQL transformer SPARK-6517 Bisecting K-Meansclustering - Fast top-down clustering variantof K- Means
- 31. MLlib: API Improvements ML Pipelines SPARK-6725 Pipeline persistence - Save/load for ML Pipelines, with partial coverage of spark.ml algorithms SPARK-5565 LDA in ML Pipelines- API for Latent Dirichlet Allocation in ML Pipelines R API SPARK-9836 R-like statistics for GLMs - (Partial) R-like stats for ordinary least squares via summary(model) SPARK-9681 Feature interactions in R formula -Interaction operator ":"in R formula Python API - Many improvementsto Python API to approach feature parity
- 32. MLlib: Miscellaneous Improvements SPARK-7685 , SPARK-9642 Instance weightsfor GLMs - Logistic and Linear Regression can take instance weights SPARK-10384,SPARK-10385 Univariate and bivariate statistics in DataFrames - Variance, stddev,correlations, etc. SPARK-10117 LIBSVM data source - LIBSVM as a SQL data source
- 33. For More Information Apache Spark 1.6.0 Release Preview:http://apache-spark-developers- list.1001551.n3.nabble.com/ANNOUNCE-Spark-1-6-0-Release-Preview- td15314.html Spark 1.6 Preview available in Databricks: https://databricks.com/blog/2015/11/20/announcing-spark-1-6-preview-in- databricks.html
- 34. Notebooks Spark 1.6 ImprovementsNotebook: http://cdn2.hubspot.net/hubfs/438089/notebooks/Spark_1.6_Improvements.ht ml?t=1448929686268 Spark 1.6 R ImprovementsNotebook: http://cdn2.hubspot.net/hubfs/438089/notebooks/Spark_1.6_R_Improvements. html?t=1448946977231
- 35. Join us at Spark Summit East February16-18, 2016 | New York City
- 36. Thanks!