Advanced Data Science on Spark-(Reza Zadeh, Stanford)
15 likes4,769 views
The document provides an overview of Spark and its machine learning library MLlib. It discusses how Spark uses resilient distributed datasets (RDDs) to perform distributed computing tasks across clusters in a fault-tolerant manner. It summarizes the key capabilities of MLlib, including its support for common machine learning algorithms and how MLlib can be used together with other Spark components like Spark Streaming, GraphX, and SQL. The document also briefly discusses future directions for MLlib, such as tighter integration with DataFrames and new optimization methods.
![Resilient Distributed Datasets (RDDs)
Main idea: Resilient Distributed Datasets
» Immutable collections of objects, spread across cluster
» Statically typed: RDD[T] has objects of type T
val sc = new SparkContext()!
val lines = sc.textFile("log.txt") // RDD[String]!
!
// Transform using standard collection operations!
val errors = lines.filter(_.startsWith("ERROR"))!
val messages = errors.map(_.split(‘t’)(2))!
!
messages.saveAsTextFile("errors.txt")!
lazily evaluated
kicks off a computation](https://image.slidesharecdn.com/qggqmvmqtow7a3c7xmes-signature-3cf9294a7bfebfdfcce57e9cc1bcdd5388b43a4e16ef2b29e8c2cab313ecbd3d-poli-150629181152-lva1-app6892/85/Advanced-Data-Science-on-Spark-Reza-Zadeh-Stanford-14-320.jpg)
Advanced Data Science on Spark-(Reza Zadeh, Stanford)
- 1. Reza Zadeh Advanced Data Science on Spark @Reza_Zadeh | http://reza-zadeh.com
- 2. Data Science Problem Data growing faster than processing speeds Only solution is to parallelize on large clusters » Wide use in both enterprises and web industry How do we program these things?
- 3. Use a Cluster Convex Optimization Matrix Factorization Machine Learning Numerical Linear Algebra Large Graph analysis Streaming and online algorithms Following lectures on http://stanford.edu/~rezab/dao Slides at http://stanford.edu/~rezab/slides/sparksummit2015
- 4. Outline Data Flow Engines and Spark The Three Dimensions of Machine Learning Built-in Libraries MLlib + {Streaming, GraphX, SQL} Future of MLlib
- 5. Traditional Network Programming Message-passing between nodes (e.g. MPI) Very difficult to do at scale: » How to split problem across nodes? • Must consider network & data locality » How to deal with failures? (inevitable at scale) » Even worse: stragglers (node not failed, but slow) » Ethernet networking not fast » Have to write programs for each machine Rarely used in commodity datacenters
- 6. Disk vs Memory L1 cache reference: 0.5 ns L2 cache reference: 7 ns Mutex lock/unlock: 100 ns Main memory reference: 100 ns Disk seek: 10,000,000 ns
- 7. Network vs Local Send 2K bytes over 1 Gbps network: 20,000 ns Read 1 MB sequentially from memory: 250,000 ns Round trip within same datacenter: 500,000 ns Read 1 MB sequentially from network: 10,000,000 ns Read 1 MB sequentially from disk: 30,000,000 ns Send packet CA->Netherlands->CA: 150,000,000 ns
- 8. Data Flow Models Restrict the programming interface so that the system can do more automatically Express jobs as graphs of high-level operators » System picks how to split each operator into tasks and where to run each task » Run parts twice fault recovery Biggest example: MapReduce Map Map Map Reduce Reduce
- 9. iter. 1 iter. 2 . . . Input file system" read file system" write file system" read file system" write Input query 1 query 2 query 3 result 1 result 2 result 3 . . . file system" read Commonly spend 90% of time doing I/O Example: Iterative Apps
- 10. MapReduce evolved MapReduce is great at one-pass computation, but inefficient for multi-pass algorithms No efficient primitives for data sharing » State between steps goes to distributed file system » Slow due to replication & disk storage
- 11. Verdict MapReduce algorithms research doesn’t go to waste, it just gets sped up and easier to use Still useful to study as an algorithmic framework, silly to use directly
- 12. Spark Computing Engine Extends a programming language with a distributed collection data-structure » “Resilient distributed datasets” (RDD) Open source at Apache » Most active community in big data, with 50+ companies contributing Clean APIs in Java, Scala, Python Community: SparkR, being released in 1.4!
- 13. Key Idea Resilient Distributed Datasets (RDDs) » Collections of objects across a cluster with user controlled partitioning & storage (memory, disk, ...) » Built via parallel transformations (map, filter, …) » The world only lets you make make RDDs such that they can be: Automatically rebuilt on failure
- 14. Resilient Distributed Datasets (RDDs) Main idea: Resilient Distributed Datasets » Immutable collections of objects, spread across cluster » Statically typed: RDD[T] has objects of type T val sc = new SparkContext()! val lines = sc.textFile("log.txt") // RDD[String]! ! // Transform using standard collection operations! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split(‘t’)(2))! ! messages.saveAsTextFile("errors.txt")! lazily evaluated kicks off a computation
- 15. Fault Tolerance file.map(lambda rec: (rec.type, 1)) .reduceByKey(lambda x, y: x + y) .filter(lambda (type, count): count > 10) filter reduce map Inputfile RDDs track lineage info to rebuild lost data
- 16. filter reduce map Inputfile Fault Tolerance file.map(lambda rec: (rec.type, 1)) .reduceByKey(lambda x, y: x + y) .filter(lambda (type, count): count > 10) RDDs track lineage info to rebuild lost data
- 17. Partitioning file.map(lambda rec: (rec.type, 1)) .reduceByKey(lambda x, y: x + y) .filter(lambda (type, count): count > 10) filter reduce map Inputfile RDDs know their partitioning functions Known to be" hash-partitioned Also known
- 18. MLlib: Available algorithms classification: logistic regression, linear SVM," naïve Bayes, least squares, classification tree regression: generalized linear models (GLMs), regression tree collaborative filtering: alternating least squares (ALS), non-negative matrix factorization (NMF) clustering: k-means|| decomposition: SVD, PCA optimization: stochastic gradient descent, L-BFGS
- 20. ML Objectives Almost all machine learning objectives are optimized using this update w is a vector of dimension d" we’re trying to find the best w via optimization
- 21. Scaling 1) Data size 2) Number of models 3) Model size
- 22. Logistic Regression Goal: find best line separating two sets of points + – + + + + + + + + – – – – – – – – + target – random initial line
- 23. Data Scaling data = spark.textFile(...).map(readPoint).cache() w = numpy.random.rand(D) for i in range(iterations): gradient = data.map(lambda p: (1 / (1 + exp(-‐p.y * w.dot(p.x)))) * p.y * p.x ).reduce(lambda a, b: a + b) w -‐= gradient print “Final w: %s” % w
- 24. Separable Updates Can be generalized for » Unconstrained optimization » Smooth or non-smooth » LBFGS, Conjugate Gradient, Accelerated Gradient methods, …
- 25. Logistic Regression Results 0 500 1000 1500 2000 2500 3000 3500 4000 1 5 10 20 30 RunningTime(s) Number of Iterations Hadoop Spark 110 s / iteration first iteration 80 s further iterations 1 s 100 GB of data on 50 m1.xlarge EC2 machines
- 26. Behavior with Less RAM 68.8 58.1 40.7 29.7 11.5 0 20 40 60 80 100 0% 25% 50% 75% 100% Iterationtime(s) % of working set in memory
- 27. Lots of little models Is embarrassingly parallel Most of the work should be handled by data flow paradigm ML pipelines does this
- 29. Model Scaling Linear models only need to compute the dot product of each example with model Use a BlockMatrix to store data, use joins to compute dot products Coming in 1.5
- 30. Model Scaling Data joined with model (weight):
- 32. A General Platform Spark Core Spark Streaming" real-time Spark SQL structured GraphX graph MLlib machine learning … Standard libraries included with Spark
- 33. Benefit for Users Same engine performs data extraction, model training and interactive queries … DFS read DFS write parse DFS read DFS write train DFS read DFS write query DFS DFS read parse train query Separate engines Spark
- 34. Machine Learning Library (MLlib) 70+ contributors in past year points = context.sql(“select latitude, longitude from tweets”)! model = KMeans.train(points, 10)! !
- 35. MLlib algorithms classification: logistic regression, linear SVM," naïve Bayes, classification tree regression: generalized linear models (GLMs), regression tree collaborative filtering: alternating least squares (ALS), non-negative matrix factorization (NMF) clustering: k-means|| decomposition: SVD, PCA optimization: stochastic gradient descent, L-BFGS
- 36. GraphX
- 37. General graph processing library Build graph using RDDs of nodes and edges Run standard algorithms such as PageRank GraphX
- 38. Spark Streaming Run a streaming computation as a series of very small, deterministic batch jobs Spark Spark Streaming batches of X seconds live data stream processed results • Chop up the live stream into batches of X seconds • Spark treats each batch of data as RDDs and processes them using RDD opera;ons • Finally, the processed results of the RDD opera;ons are returned in batches
- 39. Spark Streaming Run a streaming computation as a series of very small, deterministic batch jobs Spark Spark Streaming batches of X seconds live data stream processed results • Batch sizes as low as ½ second, latency ~ 1 second • Poten;al for combining batch processing and streaming processing in the same system
- 40. Spark SQL // Run SQL statements! val teenagers = context.sql(! "SELECT name FROM people WHERE age >= 13 AND age <= 19")! ! // The results of SQL queries are RDDs of Row objects! val names = teenagers.map(t => "Name: " + t(0)).collect()!
- 41. MLlib + {Streaming, GraphX, SQL}
- 42. A General Platform Spark Core Spark Streaming" real-time Spark SQL structured GraphX graph MLlib machine learning … Standard libraries included with Spark
- 43. MLlib + Streaming As of Spark 1.1, you can train linear models in a streaming fashion, k-means as of 1.2 Model weights are updated via SGD, thus amenable to streaming More work needed for decision trees
- 44. MLlib + SQL df = context.sql(“select latitude, longitude from tweets”)! model = pipeline.fit(df)! DataFrames in Spark 1.3! (March 2015) Powerful coupled with new pipeline API
- 45. MLlib + GraphX
- 46. Future of MLlib
- 47. Goals Tighter integration with DataFrame and spark.ml API Accelerated gradient methods & Optimization interface Model export: PMML (current export exists in Spark 1.3, but not PMML, which lacks distributed models) Scaling: Model scaling