«Почему Spark отнюдь не так хорош»
Download as PPTX, PDF2 likes1,898 views
The document provides an overview of Apache Spark, emphasizing its ability to run programs significantly faster than MapReduce, with speeds up to 100x in memory and 10x on disk. It highlights Spark's functionalities, including resilient distributed datasets (RDDs), Spark Streaming for real-time data processing, and related technologies such as Google Cloud Dataflow. Additionally, the document touches on performance optimization techniques and mentions various resources and APIs associated with Spark.
![Memory
val value = task.run(taskId, attemptNumber)
val valueBytes = resultSer.serialize(value)
val directResult = new DirectTaskResult(valueBytes,
accumUpdates, task.metrics.orNull)
val serializedDirectResult = ser.serialize(directResult)
Default JavaSerializer
public synchronized byte toByteArray()[] {
return Arrays.copyOf(buf, count);
}](https://image.slidesharecdn.com/spark-150326065657-conversion-gate01/85/Spark-42-320.jpg)
«Почему Spark отнюдь не так хорош»
- 2. Intro
- 4. RDD Resilient Distributed Dataset
- 5. RDD Resilient Distributed Dataset SchemaRDD
- 6. DAG
- 7. DAG
- 8. DAG
- 9. Mythology Spark is not MapReduce
- 10. Mythology Spark is not MapReduce Run programs up to 100x faster than MapReduce in memory, or 10x faster on disk
- 11. Mythology Spark is not MapReduce Run programs up to 100x faster than MapReduce in memory, or 10x faster on disk InMemory processing
- 12. Mythology Spark is not MapReduce Run programs up to 100x faster than MapReduce in memory, or 10x faster on disk InMemory processing Spark Streaming is real-time streaming
- 13. Mythology Spark is not MapReduce Run programs up to 100x faster than MapReduce in memory, or 10x faster on disk InMemory processing Spark Streaming is real-time streaming Lightning-fast cluster computing
- 14. MapReduce
- 15. MapReduce
- 16. MapReduce
- 17. Not MapReduce
- 18. Spark Run programs up to 100x faster than MapReduce in memory, or 10x faster on disk
- 19. Spark Run programs up to 100x faster than Hadoop MapReduce* in memory, or 10x faster on disk *Hadoop without Tez
- 21. InMemory
- 22. InMemory The MapReduce and Spark shuffles use a “pull” model. Every map task writes out data to local disk, and then the reduce tasks make remote requests to fetch that data http://blog.cloudera.com/blog/2015/01/improving-sort-performance-in-apache-spark-its-a-double/
- 23. Spark Streaming RDD DAG
- 24. Spark Streaming
- 26. Spark Streaming
- 27. Google Cloud Dataflow One of the most compelling aspects of Cloud Dataflow is its approach to one of the most difficult problems facing data engineers: how to develop pipeline logic that can execute in both batch and streaming contexts. http://blog.cloudera.com/blog/2015/01/new-in-cloudera-labs-google-cloud- dataflow-on-apache-spark/
- 33. Spark Logging Pipeline Indexes Job progress Effective Memory Network
- 34. Example
- 38. Job Progress Accumulators Broadcast
- 39. Memory val value = task.run(taskId, attemptNumber)
- 40. Memory val value = task.run(taskId, attemptNumber) val valueBytes = resultSer.serialize(value)
- 41. Memory val value = task.run(taskId, attemptNumber) val valueBytes = resultSer.serialize(value) val directResult = new DirectTaskResult(valueBytes, accumUpdates, task.metrics.orNull) val serializedDirectResult = ser.serialize(directResult)
- 42. Memory val value = task.run(taskId, attemptNumber) val valueBytes = resultSer.serialize(value) val directResult = new DirectTaskResult(valueBytes, accumUpdates, task.metrics.orNull) val serializedDirectResult = ser.serialize(directResult) Default JavaSerializer public synchronized byte toByteArray()[] { return Arrays.copyOf(buf, count); }
- 43. Network
- 44. Network
- 45. Network Problem with firewall/nat/multiple ip/etc.
- 46. SQL Shark (dead) Spark SQL Spark on Hive
- 47. SparkR
- 48. SparkR Unstable API Minimum docs
- 49. SparkR Unstable API Minimum docs Rstudio Server
- 50. Links Spark http://spark.apache.org/ Flink http://flink.apache.org/ Tez http://tez.apache.org/
Editor's Notes
- #9: mapValues