Correctness and Performance of Apache Spark SQL
0 likes155 views
1) The document discusses ensuring correctness and performance of Apache Spark SQL through extensive testing that goes beyond unit and integration testing to include techniques like random query generation, failure testing, and micro and macro benchmarks. 2) It describes Databricks' continuous integration pipeline for Spark that classifies issues, analyzes regressions, and alerts on problems to minimize impacts. 3) As an example, it details an investigation that found a 18% regression in query performance between Spark 2.3 and 2.4 due to changes in hashing functions, and the steps taken like profiling and bisection to root cause and address the issue.
![Lead function (1/3)
SELECT (t1.decimal0803_col_3) / (t1.decimal0803_col_3) AS decimal_col,
CAST(696 AS STRING) AS char_col, t1.decimal0803_col_3,
(COALESCE(CAST('0.02' AS DECIMAL(10,10)),
CAST('0.47' AS DECIMAL(10,10)),
CAST('-0.53' AS DECIMAL(10,10)))) +
(LEAD(-65, 4) OVER (ORDER BY (t1.decimal0803_col_3) / (t1.decimal0803_col_3),
CAST(696 AS STRING))) AS decimal_col_1,
CAST(-349 AS STRING) AS char_col_1
FROM table_16 t1
WHERE (943) > (889)
Error: Column 4 in row 10 does not match:
[1.0, 696, -871.81, <<-64.98>>, -349] SPARK row
[1.0, 696, -871.81, <<None>>, -349] POSTGRESQL row](https://image.slidesharecdn.com/sparksummiteucorrectnessandperformanceofapachesparksql-190123083132/85/Correctness-and-Performance-of-Apache-Spark-SQL-19-320.jpg)
Correctness and Performance of Apache Spark SQL
- 1. Correctness and Performance of Apache Spark SQL Spark + AI Summit, London October 4, 2018 1
- 2. 2 NICOLAS POGGI Databricks, Performance Engineer • Spark benchmarking Barcelona Supercomputing - Microsoft Research Centre • Lead researcher ALOJA project • New architectures for Big Data BarcelonaTech (UPC), PhD in Computer Architecture • Autonomic resource manager for the cloud • Web customer modeling About us BOGDAN GHIT Databricks, Software Engineer • SQL performance optimizations IBM T.J. Watson, Research Intern • Bid advisor for cloud spot markets Delft University of Technology, PhD in Computer Science • Resource management in datacenters • Performance of Spark, Hadoop
- 3. Databricks ecosystem 3 ToolsDevelopers DBR Cluster Manager Infrastructure Customers
- 4. DBR 5.0 DBR 4.3-LTS DBR 4.3 Feb’18 Beta Full Support Marked for deprecation Deprecated Databricks runtime (DBR) releases Our goal is to make releases automatic and frequent Jun’18 Oct’18 Feb’19 Jun’19 Oct’19 Feb’20 Spark 2.4 Spark 2.3 Spark 2.3
- 5. Apache Spark contributions 5 Hundreds of commits monthly to the Apache Spark project Numberofcommits At this pace of development, mistakes are bound to happen
- 6. Where do these contributions go? 6 Scope of the testing Developers put a significant engineering effort in testing Query Input data Configuration Over 200 built-in functions
- 7. Yet another brick in the wall Unit testing is not enough to guarantee correctness and performance Unit testing Integration E2E Micro Benchmarks Plan stability Fuzz testing Macro benchmarks Stress testing Customer workloads Failure testing
- 8. 8 Continuous Integration pipeline New artifacts Metrics - Correctness - Performance Test Alerts - Merge - Build Dev - Rules - Policies Analyze
- 9. 9 Classification and alerting - Impact - Scope - Correlation - Confirm? Failure Regression - Minimize - Drill-down - Profile - Compare - Validate Events Re-test Alert Classify Root-cause Correctness Performance
- 10. 10 Failure Regression Events Re-test Alert Classify Root-cause Correctness Performance Correctness
- 11. Random query generation 11 Query profile Model translator Spark Query Postgres Query vs vs
- 12. ... ... DDL and datagen 12 ... ... BigIntBoolean Timestamp Decimal FloatInteger SmallInt String Choose a data type Random number of rows Random number of columns Random number of tables Random partition columns
- 13. Recursive query model 13 SQL Query WITH FROMUNION SELECT Functions Constant GROUP BY ORDER BY Table Column Alias Query Clause Expression JOIN WHERE
- 14. Probabilistic query profile Independent weights • Optional query clauses Inter-dependent weights • Join types • Select functions ORDER BY UNION GROUP BY WHERE 10% 10% 50% 10%
- 15. Coalesce flattening (1/4) SELECT COALESCE(t2.smallint_col_3, t1.smallint_col_3, t2.smallint_col_3) AS int_col, IF(NULL, VARIANCE(COALESCE(t2.smallint_col_3, t1.smallint_col_3, t2.smallint_col_3)), COALESCE(t2.smallint_col_3, t1.smallint_col_3, t2.smallint_col_3)) AS int_col_1, STDDEV(t2.double_col_2) AS float_col, COALESCE(MIN((t1.smallint_col_3) - (COALESCE(t2.smallint_col_3, t1.smallint_col_3, t2.smallint_col_3))), COALESCE(t2.smallint_col_3, t1.smallint_col_3, t2.smallint_col_3), COALESCE(t2.smallint_col_3, t1.smallint_col_3, t2.smallint_col_3)) AS int_col_2 FROM table_4 t1 INNER JOIN table_4 t2 ON (t2.timestamp_col_7) = (t1.timestamp_col_7) WHERE (t1.smallint_col_3) IN (CAST('0.04' AS DECIMAL(10,10)), t1.smallint_col_3) GROUP BY COALESCE(t2.smallint_col_3, t1.smallint_col_3, t2.smallint_col_3) Small dataset with 2 tables of 5x5 size Within 10 randomly generated queries Error: Operation is in ERROR_STATE
- 16. Coalesce flattening (2/3) Aggregate Project Join FILTERSCAN foo SCAN bar foo.id IN (CAST(‘0.04’ AS DECIMAL(10, 10)), foo.id) foo.ts = bar.ts COALESCE(COALESCE(foo.id, foo.val), 88) GROUP BY COALESCE(foo.id, foo.val)
- 17. Coalesce flattening (3/4) Aggregate Project Join FILTERSCAN t1 SCAN t2 foo.id IN (CAST(‘0.04’ AS DECIMAL(10, 10)), foo.id) foo.ts = bar.ts COALESCE(COALESCE(foo.id, foo.val), 88) COALESCE(foo.id, foo.val)
- 18. Coalesce flattening (4/4) Aggregate Project SCAN foo Minimized query: SELECT COALESCE(COALESCE(foo.id, foo.val), 88) FROM foo GROUP BY COALESCE(foo.id, foo.val) Analyzing the error ● The optimizer flattens the nested coalesce calls ● The SELECT clause doesn’t contain the GROUP BY expression ● Possibly a problem with any GROUP BY expression that can be optimized
- 19. Lead function (1/3) SELECT (t1.decimal0803_col_3) / (t1.decimal0803_col_3) AS decimal_col, CAST(696 AS STRING) AS char_col, t1.decimal0803_col_3, (COALESCE(CAST('0.02' AS DECIMAL(10,10)), CAST('0.47' AS DECIMAL(10,10)), CAST('-0.53' AS DECIMAL(10,10)))) + (LEAD(-65, 4) OVER (ORDER BY (t1.decimal0803_col_3) / (t1.decimal0803_col_3), CAST(696 AS STRING))) AS decimal_col_1, CAST(-349 AS STRING) AS char_col_1 FROM table_16 t1 WHERE (943) > (889) Error: Column 4 in row 10 does not match: [1.0, 696, -871.81, <<-64.98>>, -349] SPARK row [1.0, 696, -871.81, <<None>>, -349] POSTGRESQL row
- 20. Lead function (2/3) Project FILTER SCAN foo WHERE expr COALESCE(expr) + LEAD(-65, 4) OVER ORDER BY expr
- 21. Lead function (3/3) Project FILTER WHERE expr COALESCE(expr) + LEAD(-65, 4) OVER ORDER BY expr Analyzing the error ● Using constant input values breaks the behaviour of the LEAD function ● SC-16633: https://github.com/apache/spark/pull/14284 SCAN foo
- 23. Benchmarking tools •We use spark-sql-perf public library for TPC workloads • Provides datagen and import scripts • local, cluster, S3 • Dashboards for analyzing results •The Spark micro benchmarks •And the async-profiler • to produce flamegraphs 23 https://github.com/databricks/spark-sql-perf Source: http://www.brendangregg.com/flamegraphs.html CPU Flame Graph
- 25. Per query drill-down: 67 First, scope and validate • in 2.4-master (dev) compared • to 2.3 in DBR 4.3 (prod) Query 67: 18% regression From 320s to 390s
- 26. Q67 executor profile for Spark 2.4-master
- 27. Side-by-side 2.3 vs 2.4: find the differences Spark 2.3 Spark 2.4
- 28. Framegraph diff zoom Red slower White new unsafe/Platform.copyMemory() unsafe/BytesToBytesMap.safeLookup New: hash/Murmur3_x86_32.hashUTF8String() Murmur3_x86_32.hashUnsafeBytesBlock() Look for hints: - Mem mgmt - Hashing - unsafe
- 29. Root-causing Results: • Spark 2.3: hashUnsafeBytes() -> 40µs • Spark 2.4 hashUnsafeBytesBlock() -> 140µs • also slower UTF8String.getBytes() Microbenchmark for UTF8String GIT BISECT 1.) 2.) 3.)
- 30. It is a journey to get a release out DBR and Spark testing and performance are a continuous effort • Over a month effort to bring performance to improving TPC-DS 2.4-master vs. 2.3 at SF 1000 15% 5% < 0%
- 31. Conclusion Spark in production is not just the framework Unit and integration testing are not enough We need Spark specific tools to automate the process to ensure both correctness and performance
- 32. Thanks! Correctness and Performance of Apache Spark SQL October 2018 32 Test AnalyzeDev