![Dataset[Row]
Partition = set of Row's](https://image.slidesharecdn.com/apachecon-startingwithapachesparkbestpractices-170520185427/85/Apache-Big-Data-Starting-with-Apache-Spark-Best-Practices-25-320.jpg)
![Pattern matching
Mix pattern with SQL syntax
motifs = g.find("(a)-[e]->(b); (b)-
[e2]->(a); !(c)-[]->(a)").filter("a.id
= 'MIA'")](https://image.slidesharecdn.com/apachecon-startingwithapachesparkbestpractices-170520185427/85/Apache-Big-Data-Starting-with-Apache-Spark-Best-Practices-51-320.jpg)
![Understanding Queries
explain() is your friend
but it could be hard to understand at
times == Parsed Logical Plan ==
Aggregate [count(1) AS count#79L]
+- Sort [speed_y#49 ASC], true
+- Join Inner, (speed_x#48 = speed_y#49)
:- Project [speed#2 AS speed_x#48, dist#3]
: +- LogicalRDD [speed#2, dist#3]
+- Project [speed#18 AS speed_y#49, dist#19]
+- LogicalRDD [speed#18, dist#19]](https://image.slidesharecdn.com/apachecon-startingwithapachesparkbestpractices-170520185427/85/Apache-Big-Data-Starting-with-Apache-Spark-Best-Practices-76-320.jpg)
![== Physical Plan ==
*HashAggregate(keys=[], functions=[count(1)], output=[count#79L])
+- Exchange SinglePartition
+- *HashAggregate(keys=[], functions=[partial_count(1)],
output=[count#83L])
+- *Project
+- *Sort [speed_y#49 ASC], true, 0
+- Exchange rangepartitioning(speed_y#49 ASC, 200)
+- *Project [speed_y#49]
+- *SortMergeJoin [speed_x#48], [speed_y#49], Inne
:- *Sort [speed_x#48 ASC], false, 0
: +- Exchange hashpartitioning(speed_x#48, 200
: +- *Project [speed#2 AS speed_x#48]
: +- *Filter isnotnull(speed#2)
: +- Scan ExistingRDD[speed#2,dist#3]
+- *Sort [speed_y#49 ASC], false, 0
+- Exchange hashpartitioning(speed_y#49, 200](https://image.slidesharecdn.com/apachecon-startingwithapachesparkbestpractices-170520185427/85/Apache-Big-Data-Starting-with-Apache-Spark-Best-Practices-78-320.jpg)
![UDF vs Builtin Example
Remember Predicate Pushdown?
val isSeattle = udf { (s: String) => s == "Seattle" }
cities.where(isSeattle('name))
*Filter UDF(name#2)
+- *FileScan parquet [id#128L,name#2] Batched: true, Format:
ParquetFormat, InputPaths: file:/Users/b/cities.parquet,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:bigint,name:string>](https://image.slidesharecdn.com/apachecon-startingwithapachesparkbestpractices-170520185427/85/Apache-Big-Data-Starting-with-Apache-Spark-Best-Practices-80-320.jpg)
![UDF vs Builtin Example
cities.where('name === "Seattle")
*Project [id#128L, name#2]
+- *Filter (isnotnull(name#2) && (name#2 = Seattle))
+- *FileScan parquet [id#128L,name#2] Batched: true, Format:
ParquetFormat, InputPaths: file:/Users/b/cities.parquet,
PartitionFilters: [], PushedFilters: [IsNotNull(name),
EqualTo(name,Seattle)], ReadSchema:
struct<id:bigint,name:string>](https://image.slidesharecdn.com/apachecon-startingwithapachesparkbestpractices-170520185427/85/Apache-Big-Data-Starting-with-Apache-Spark-Best-Practices-81-320.jpg)
![BroadcastHashJoin
left.join(right, Seq("id"), "leftanti").explain
== Physical Plan ==
*BroadcastHashJoin [id#50], [id#60], LeftAnti,
BuildRight
:- LocalTableScan [id#50, left#51]
+- BroadcastExchange
HashedRelationBroadcastMode(List(cast(input[0, int,
false] as bigint)))
+- LocalTableScan [id#60]](https://image.slidesharecdn.com/apachecon-startingwithapachesparkbestpractices-170520185427/85/Apache-Big-Data-Starting-with-Apache-Spark-Best-Practices-87-320.jpg)
Apache: Big Data - Starting with Apache Spark, Best Practices
- 2. Starting with Apache Spark, Best Practices and Learning from the Field Felix Cheung, Principal Engineer + Spark Committer Spark@Microsoft
- 7. Resilient - Fault tolerant
- 10. Tungsten AMPLab becoming RISELab • Drizzle – low latency execution, 3.5x lower than Spark Streaming • Ernest – performance prediction, automatically choose the optimal resource config on the cloud
- 12. Deployment Scheduler Resource Manager (aka Cluster Manager) - Spark History Server, Spark UI Spark Core
- 15. Doing multiple things at the same time
- 16. A unit of parallelization
- 17. Manipulating data - immutable "Narrow" "Wide"
- 20. Processing: sorting, serialize/deserialize, compression Transfer: disk IO, network bandwidth/latency Take up memory, or spill to disk for intermediate results ("shuffle file")
- 21. Materialize results Execute the chain of transformations that leads to output – lazy evaluation count collect -> take write
- 22. DataFrame Dataset Data source Execution engine - Catalyst SQL
- 25. Dataset[Row] Partition = set of Row's
- 26. "format" - Parquet, CSV, JSON, or Cassandra, HBase
- 28. Ability to process expressions as early in the plan as possible
- 29. spark.read.jdbc(jdbcUrl, "food", connectionProperties) // with pushdown spark.read.jdbc(jdbcUrl, "food", connectionProperties).select("hotdog", "pizza", "sushi")
- 30. Discretized Streams (DStreams) Receiver DStream Direct DStream Basic and Advanced Sources Streaming
- 31. Source Reliability Receiver + Write Ahead Log (WAL) Checkpointing
- 36. Only for reliable messaging sources that supports read from position Stronger fault-tolerance, exactly-once* No receiver/WAL – less resource, lower overhead
- 37. Saving to reliable storage to recover from failure 1. Metadata checkpointing StreamingContext.checkpoint() 2. Data checkpointing dstream.checkpoint()
- 39. DataFrame-based - leverage optimizations and support transformations a sequence of algorithms - PipelineStages
- 41. Feature transformer - take a DataFrame and its Column and append one or more new Column
- 43. Estimators An algorithm DataFrame -> Model A Model is a Transformer LinearRegression KMeans
- 44. Evaluator Metric to measure Model performance on held-out test data
- 46. MLWriter/MLReader Pipeline persistence Include transformers, estimators, Params
- 48. Directed multigraph with user properties on edges and vertices SEA NYC LAX
- 50. DataFrame-based Simplify loading graph data, wrangling Support Graph Queries
- 51. Pattern matching Mix pattern with SQL syntax motifs = g.find("(a)-[e]->(b); (b)- [e2]->(a); !(c)-[]->(a)").filter("a.id = 'MIA'")
- 53. Extending same DataFrame to include incremental execution of unbounded input Reliability, correctness / exactly-once - checkpointing (2.1 JSON format)
- 54. Stream as Unbounded Input https://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html
- 55. Watermark (2.1) - handling of late data Streaming ETL, joining static data, partitioning, windowing
- 56. FileStreamSource KafkaSource MemoryStream (not for production) TextSocketSource MQTT
- 57. FileStreamSink (new formats in 2.1) ConsoleSink ForeachSink (Scala only) MemorySink – as Temp View
- 59. streamingDF = ( spark .readStream .schema(jsonSchema) .option("maxFilesPerTrigger", 1) .json(inputPath) ) # Take a list of files as a stream
- 60. streamingCountsDF = ( streamingDF .groupBy( streamingDF.word, window( streamingDF.time, "1 hour")) .count() )
- 61. query = ( streamingCountsDF .writeStream .format("memory") .queryName("word_counts") .outputMode("complete") .start() ) spark.sql("select count from word_counts order by time")
- 63. How much going in affects how much work it's going to take
- 64. Size does matter! CSV or JSON is "simple" but also tend to be big JSON-> Parquet (compressed) - 7x faster
- 65. Format also does matter Recommended format - Parquet Default data source/format • VectorizedReader • Better dictionary decoding
- 66. Parquet Columnar Format Column chunk co-located Metadata and headers for skipping
- 68. Compression is a factor gzip <100MB/s vs snappy 500MB/s Tradeoffs: faster or smaller? Spark 2.0+ defaults to snappy
- 69. Sidenote: Table Partitioning Storage data into groups of partitioning columns Encoded path structure matches Hive table/event_date=2017-02-01
- 73. Executor tab
- 74. SQL tab
- 76. Understanding Queries explain() is your friend but it could be hard to understand at times == Parsed Logical Plan == Aggregate [count(1) AS count#79L] +- Sort [speed_y#49 ASC], true +- Join Inner, (speed_x#48 = speed_y#49) :- Project [speed#2 AS speed_x#48, dist#3] : +- LogicalRDD [speed#2, dist#3] +- Project [speed#18 AS speed_y#49, dist#19] +- LogicalRDD [speed#18, dist#19]
- 78. == Physical Plan == *HashAggregate(keys=[], functions=[count(1)], output=[count#79L]) +- Exchange SinglePartition +- *HashAggregate(keys=[], functions=[partial_count(1)], output=[count#83L]) +- *Project +- *Sort [speed_y#49 ASC], true, 0 +- Exchange rangepartitioning(speed_y#49 ASC, 200) +- *Project [speed_y#49] +- *SortMergeJoin [speed_x#48], [speed_y#49], Inne :- *Sort [speed_x#48 ASC], false, 0 : +- Exchange hashpartitioning(speed_x#48, 200 : +- *Project [speed#2 AS speed_x#48] : +- *Filter isnotnull(speed#2) : +- Scan ExistingRDD[speed#2,dist#3] +- *Sort [speed_y#49 ASC], false, 0 +- Exchange hashpartitioning(speed_y#49, 200
- 79. UDF Write you own custom transforms But... Catalyst can't see through it (yet?!) Always prefer to use builtin transforms as much as possible
- 80. UDF vs Builtin Example Remember Predicate Pushdown? val isSeattle = udf { (s: String) => s == "Seattle" } cities.where(isSeattle('name)) *Filter UDF(name#2) +- *FileScan parquet [id#128L,name#2] Batched: true, Format: ParquetFormat, InputPaths: file:/Users/b/cities.parquet, PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:bigint,name:string>
- 81. UDF vs Builtin Example cities.where('name === "Seattle") *Project [id#128L, name#2] +- *Filter (isnotnull(name#2) && (name#2 = Seattle)) +- *FileScan parquet [id#128L,name#2] Batched: true, Format: ParquetFormat, InputPaths: file:/Users/b/cities.parquet, PartitionFilters: [], PushedFilters: [IsNotNull(name), EqualTo(name,Seattle)], ReadSchema: struct<id:bigint,name:string>
- 82. UDF in Python Avoid! Why? Pickling, transfer, extra memory to run Python interpreter - Hard to debug errors! from pyspark.sql.types import IntegerType sqlContext.udf.register("stringLengthInt", lambda x: len(x), IntegerType()) sqlContext.sql("SELECT stringLengthInt('test')").take(1)
- 83. Going for Performance Stored in compressed Parquet Partitioned table Predicate Pushdown Avoid UDF
- 84. Shuffling for Join Can be very expensive
- 85. Optimizing for Join Partition! Narrow transform if left and right partitioned with same scheme
- 86. Optimizing for Join Broadcast Join (aka Map-Side Join in Hadoop) Smaller table against large table - avoid shuffling large table Default 10MB auto broadcast
- 87. BroadcastHashJoin left.join(right, Seq("id"), "leftanti").explain == Physical Plan == *BroadcastHashJoin [id#50], [id#60], LeftAnti, BuildRight :- LocalTableScan [id#50, left#51] +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint))) +- LocalTableScan [id#60]
- 88. Repartition To numPartitions or by Columns Increase parallelism – will shuffle coalesce() – combine partitions in place
- 89. Cache cache() or persist() Flush least-recently-used (LRU) - Make sure there is enough memory! MEMORY_AND_DISK to avoid expensive recompute (but spill to disk is slow)
- 90. Streaming Use Structured Streaming (2.1+) If not... If reliable messaging (Kafka) use Direct DStream
- 91. Metadata - Config Position from streaming source (aka offset) - could get duplicates! (at-least-once) Pending batches
- 92. Persist stateful transformations - data lost if not saved Cut short execution that could grow indefinitely
- 93. Direct DStream Checkpoint also store offset Turn off auto commit - do when in good state for exactly- once
- 94. Checkpointing Stream/ML/Graph/SQL - more efficient indefinite/iterative - recovery Generally not versioning-safe Use reliable distributed file system (caution on “object store”)
- 97. Hadoop WebLog Spark SQL External Data Source BI Tools HDFS Hive Hive Metastore FrontEnd Hourly
- 102. Spark Streaming Message Bus Spark ML Kafka Storage Spark SQL Visualization External Data Source BI Tools Data Lake Hive Metastore Spark ML
- 106. Spark Streaming Message Bus Storage SQL Spark SQL Spark SQL Data Factory
- 107. Moving