Scrap Your MapReduce - Apache Spark
Download as PPTX, PDF2 likes1,021 views
This document discusses the limitations of traditional MapReduce and highlights Apache Spark as an advanced cluster computing engine that addresses issues such as latency and complex workflows. It emphasizes Spark's advantages, including in-memory data caching, a user-friendly programming model, and improved performance due to reduced I/O operations. Additionally, it covers key concepts like Resilient Distributed Datasets (RDDs) and their operations, showcasing Spark's functionality and ease of testing.
![24
RDD operations
Transformations
map(f : T ⇒ U) : RDD[T] ⇒ RDD[U]
filter(f : T ⇒ Bool) : RDD[T] ⇒ RDD[T]
flatMap(f : T ⇒ Seq[U]) : RDD[T] ⇒ RDD[U]
sample(fraction : Float) : RDD[T] ⇒ RDD[T] (Deterministic sampling)
union() : (RDD[T],RDD[T]) ⇒ RDD[T]
join() : (RDD[(K, V)],RDD[(K, W)]) ⇒ RDD[(K, (V, W))]
groupByKey() : RDD[(K, V)] ⇒ RDD[(K, Seq[V])]
reduceByKey(f : (V,V) ⇒ V) : RDD[(K, V)] ⇒ RDD[(K, V)]
partitionBy(p : Partitioner[K]) : RDD[(K, V)] ⇒ RDD[(K, V)]](https://image.slidesharecdn.com/mapreduceapachesparkindicthreadsunmeshrahul-150304010847-conversion-gate01/85/Scrap-Your-MapReduce-Apache-Spark-24-320.jpg)
![25
Actions
count() : RDD[T] ⇒ Long
collect() : RDD[T] ⇒ Seq[T]
reduce(f : (T,T) ⇒ T) : RDD[T] ⇒ T
lookup(k : K) : RDD[(K, V)] ⇒ Seq[V] (On hash/range partitioned
RDDs)
save(path : String) : Outputs RDD to a storage system, e.g., HDFS](https://image.slidesharecdn.com/mapreduceapachesparkindicthreadsunmeshrahul-150304010847-conversion-gate01/85/Scrap-Your-MapReduce-Apache-Spark-25-320.jpg)
Scrap Your MapReduce - Apache Spark
- 1. Lightning-fast cluster computing Rahul Kavale(rahulkav@thoughtworks.com) Unmesh Joshi(uvjoshi@thoughtworks.com)
- 2. 2 Some properties of “Big Data” •Big data is inherently immutable, meaning it is not supposed to updated once generated. •Mostly the operations are coarse grained when it comes to write •Commodity hardware makes more sense for storage/computation of such enormous data,hence the data is distributed across cluster of many such machines • The distributed nature makes the programming complicated.
- 3. 3 Brush up for Hadoop concepts Distributed Storage => HDFS Cluster Manager => YARN Fault tolerance => achieved via replication Job scheduling => Scheduler in YARN Mapper Reducer Combiner
- 5. 5 Map Reduce Programming Model
- 9. 9 MapReduce pain points • considerable latency • only Map and Reduce phases • Non trivial to test • results into complex workflow • Not suitable for Iterative processing
- 10. 10 Immutability and MapReduce model • HDFS storage is immutable or append-only. • The MapReduce model lacks to exploit the immutable nature of the data. • The intermediate results are persisted resulting in huge of IO, causing a serious performance hit.
- 11. 11 Wouldn’t it be very nice if we could have• Low latency • Programmer friendly programming model • Unified ecosystem • Fault tolerance and other typical distributed system properties • Easily testable code • Of course open source :)
- 12. 12 What is Apache Spark • Cluster computing Engine • Abstracts the storage and cluster management • Unified interfaces to data • API in Scala, Python, Java, R*
- 13. 13 Where does it fit in existing Bigdata ecosystem http://www.kdnuggets.com/2014/06/yarn-all-rage-hadoop-summit.html
- 14. 14 Why should you care about Apache Spark • Abstracts underlying storage, • Abstracts cluster management • Easy programming model • Very easy to test the code • Highly performant
- 15. 15 • Petabyte sort record https://databricks.com/blog/2014/10/10/spark-petabyte-sort.html
- 16. 16 • Offers in memory caching of data • Specialized Applications • GraphX for graph processing • Spark Streaming • MLib for Machine learning • Spark SQL • Data exploration via Spark-Shell
- 18. 18 Word Count example val file = spark.textFile("input path") val counts = file.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey((a, b) => a + b) counts.saveAsTextFile("destination path")
- 19. 19 Comparing example with MapReduce
- 20. 20 Spark Shell Demo • SparkContext • RDD • RDD operations
- 21. 21 RDD • RDD stands for Resilient Distributed Dataset. • basic abstraction for Spark
- 22. 22 • Equivalent of Distributed collections. • The interface makes distributed nature of underlying data transparent. • RDD is immutable • Can be created via, • parallelising a collection, • transforming an existing RDD by applying a transformation function, • reading from a persistent data store like HDFS.
- 23. 23 RDD is lazily evaluated RDD has two type of operations • Transformations Create a DAG of transformations to be applied on the RDD Does not evaluating anything • Actions Evaluate the DAG of transformations
- 24. 24 RDD operations Transformations map(f : T ⇒ U) : RDD[T] ⇒ RDD[U] filter(f : T ⇒ Bool) : RDD[T] ⇒ RDD[T] flatMap(f : T ⇒ Seq[U]) : RDD[T] ⇒ RDD[U] sample(fraction : Float) : RDD[T] ⇒ RDD[T] (Deterministic sampling) union() : (RDD[T],RDD[T]) ⇒ RDD[T] join() : (RDD[(K, V)],RDD[(K, W)]) ⇒ RDD[(K, (V, W))] groupByKey() : RDD[(K, V)] ⇒ RDD[(K, Seq[V])] reduceByKey(f : (V,V) ⇒ V) : RDD[(K, V)] ⇒ RDD[(K, V)] partitionBy(p : Partitioner[K]) : RDD[(K, V)] ⇒ RDD[(K, V)]
- 25. 25 Actions count() : RDD[T] ⇒ Long collect() : RDD[T] ⇒ Seq[T] reduce(f : (T,T) ⇒ T) : RDD[T] ⇒ T lookup(k : K) : RDD[(K, V)] ⇒ Seq[V] (On hash/range partitioned RDDs) save(path : String) : Outputs RDD to a storage system, e.g., HDFS
- 26. 26 Job Execution
- 27. 27 Spark Execution in Context of YARN http://kb.cnblogs.com/page/198414/
- 28. 28 Fault tolerance via lineage MappedRDD FilteredRDD FlatMappedRDD MappedRDD HadoopRDD
- 29. 29 Testing
- 30. 30 Why is Spark more performant than MapReduce
- 31. 31 Reduced IO • No disk IO between phases since phases themselves are pipelined • No network IO involved unless a shuffle is required
- 32. 32 No Mandatory Shuffle • Programs not bounded by map and reduce phases • No mandatory Shuffle and sort required
- 33. 33 In memory caching of data • Optional In memory caching • DAG engine can apply certain optimisations since when an action is called, it knows what all transformations as to be applied
- 34. 34 Questions?
- 35. 35 Thank You!