"Lessons learned using Apache Spark for self-service data prep in SaaS world"
0 likes100 views
This presentation discusses Workday's use of Apache Spark for self-service data preparation and analytics within its SaaS platform. It covers Workday's unified analytics platform powered by Spark, how Prism uses Spark for interactive data prep and publishing, and lessons learned in areas like nested SQL optimization, plan deduplication, broadcast join tuning, and case-insensitive string grouping. The presentation aims to share Workday's production experiences leveraging Spark for analytics in a multi-tenant SaaS environment.
![Lesson #1: Example Dependency Graph
[first.name], [last.name], [income],
concat([first.name],”.”, [last.name]) as [full.name],
[income] * 0.28 as [federal.tax],
[income] *0.10 as [state.tax],
concat([full.name],”@workday.com”) as [email]
first.name last.name income
full.name federal.tax
email
state.tax
2nd level
1st level
30#UnifiedAnalytics #SparkAISummit](https://image.slidesharecdn.com/sais2019-lessonslearnedusingapachesparkforself-servicedataprepinsaasworld-190429174751/85/Lessons-learned-using-Apache-Spark-for-self-service-data-prep-in-SaaS-world-30-320.jpg)
![select [income] * 0.10 as [state_tax], *
from (select [income] * 0.28 as [federal_tax], *
from (select concat([full.name],”@workday.com”) as [email], *
from (select concat([first.name],”.”, [last.name]) as [full.name], *
from (select [first.name], [last.name], [income] from base_table))))
Lesson #1: SQL Before Optimization
4 levels of nested SQL
31#UnifiedAnalytics #SparkAISummit](https://image.slidesharecdn.com/sais2019-lessonslearnedusingapachesparkforself-servicedataprepinsaasworld-190429174751/85/Lessons-learned-using-Apache-Spark-for-self-service-data-prep-in-SaaS-world-31-320.jpg)
![Lesson #1: SQL After Optimization
2 levels of nested SQL
32
select concat([full.name],”@workday.com”) as [email], *
from (select concat([first.name],”.”, [last.name]) as [full.name],
[income] * 0.28 as [federal_tax],
[income] * 0.10 as [state_tax], *
from (select [first.name], [last.name], [income] from base_table)))
#UnifiedAnalytics #SparkAISummit](https://image.slidesharecdn.com/sais2019-lessonslearnedusingapachesparkforself-servicedataprepinsaasworld-190429174751/85/Lessons-learned-using-Apache-Spark-for-self-service-data-prep-in-SaaS-world-32-320.jpg)
"Lessons learned using Apache Spark for self-service data prep in SaaS world"
- 1. Pavel Hardak (Product Manager, Workday) Jianneng Li (Software Engineer, Workday) Lessons Learned Using Apache Spark for Self-Service Data Prep (and More) in SaaS World #UnifiedAnalytics #SparkAISummit
- 2. This presentation may contain forward-looking statements for which there are risks, uncertainties, and assumptions. If the risks materialize or assumptions prove incorrect, Workday’s business results and directions could differ materially from results implied by the forward-looking statements. Forward-looking statements include any statements regarding strategies or plans for future operations; any statements concerning new features, enhancements or upgrades to our existing applications or plans for future applications; and any statements of belief. Further information on risks that could affect Workday’s results is included in our filings with the Securities and Exchange Commission which are available on the Workday investor relations webpage: www.workday.com/company/investor_relations.php Workday assumes no obligation for and does not intend to update any forward-looking statements. Any unreleased services, features, functionality or enhancements referenced in any Workday document, roadmap, blog, our website, press release or public statement that are not currently available are subject to change at Workday’s discretion and may not be delivered as planned or at all. Customers who purchase Workday, Inc. services should make their purchase decisions upon services, features, and functions that are currently available. Safe Harbor Statement #UnifiedAnalytics #SparkAISummit 2
- 3. Agenda ● Workday - Finance and HCM in the cloud ● Workday Platform - “Power of One” ● Prism Analytics - Powered by Apache Spark ● Production Stories & Lessons Learned ● Questions 3#UnifiedAnalytics #SparkAISummit 3
- 4. #UnifiedAnalytics #SparkAISummit 4 ● “Pure” SaaS apps suite ○ Finance and HCM ● Customers: 2,500+ ○ 200+ of Fortune 500 ● Revenue: $2.82B ○ Growth: 32% YoY Plan Execute Analyze Planning Financial Management Human Capital Management Prism Analytics and Reporting
- 6. 6 Business Process Framework Object Data Model Reporting and Analytics Security Integration Cloud One Source for Data | One Security Model | One Experience | One Community Machine Learning One Platform #UnifiedAnalytics #SparkAISummit
- 7. #UnifiedAnalytics #SparkAISummit 7 Durable Business Process Framework Object Data Model Reporting and Analytics Security Integration Cloud One Source for Data | One Security Model | One Experience | One Community Machine Learning One Platform Object Data Model MetadataExtensible
- 8. #UnifiedAnalytics #SparkAISummit 8 Business Process Framework Object Data Model Reporting and Analytics Security Integration Cloud One Source for Data | One Security Model | One Experience | One Community Machine Learning One Platform Security Encryption Privacy and Compliance Trust
- 9. #UnifiedAnalytics #SparkAISummit 9 Business Process Framework Object Data Model Reporting and Analytics Security Integration Cloud One Source for Data | One Security Model | One Experience | One Community Machine Learning One Platform Reporting and Analytics Dashboards CollaborationDistribution
- 10. #UnifiedAnalytics #SparkAISummit 10 Plan Execute Analyze Planning Financial Management Human Capital Management Prism Analytics and Reporting
- 11. Workday Planning Workday Financial Management Workday Human Capital Management Workday Prism Analytics and Reporting Prism Analytics Integrate 3rd Party Data Data Management Data Preparation Data Discovery Report Publishing 11#UnifiedAnalytics #SparkAISummit Plan Execute Analyze Planning Financial Management Human Capital Management Prism Analytics and Reporting
- 12. Workday Prism Analytics The full spectrum of Finance and HCM insights, all within Workday. Workday Data + Non-Workday Data #UnifiedAnalytics #SparkAISummit 12
- 13. Finance, HCM Operational Industry systems Legacy systems More… CRM Service ticketing Surveys Point of Sale Stock grants Map Ingest Preparation AnalysisAcquisition Reporting Worksheets Data Discovery Cleanse and Transform Blend Datasets Apply Security Permissions Publish Data Source Prism Analytics Workflow 13#UnifiedAnalytics #SparkAISummit
- 14. Prism Prism Prism HDFS / S3 Query Engine Spark Driver Spark Executor Interactive Data Prep Spark Driver Spark Executor Spark Driver Data Prep Publishing YARN Spark Executor Spark Executor Spark in Prism Analytics #UnifiedAnalytics #SparkAISummit 14
- 15. Interactive Data Prep in Prism Transform Stages Number of samples Examples and statistics 15#UnifiedAnalytics #SparkAISummit
- 16. Interactive Data Prep in Prism 16#UnifiedAnalytics #SparkAISummit
- 17. Interactive Data Prep in Prism Powered by Spark Edit Transform 17#UnifiedAnalytics #SparkAISummit
- 18. Data Prep Publishing in Prism Also powered by Spark 18#UnifiedAnalytics #SparkAISummit
- 19. 19#UnifiedAnalytics #SparkAISummit Interactive Publishing Data size 100 - 100K rows Billions of rows Sampling Yes No Caching Yes No Latency Seconds Minutes to hours Result Returned in memory Written to disk SLA Best effort Consistent performance Data Prep: Interactive vs. Publishing
- 20. 20#UnifiedAnalytics #SparkAISummit Data Prep: Interactive vs. Publishing Same plan!
- 21. Prism Logical Model 21#UnifiedAnalytics #SparkAISummit
- 22. Prism Logical Model • Superset of SQL operators • Compiles to Spark plans through Spark SQL • Implements custom Catalyst rules and strategies 22#UnifiedAnalytics #SparkAISummit#UnifiedAnalytics #SparkAISummit
- 23. Example: Interactive Data Prep Operators 23#UnifiedAnalytics #SparkAISummit#UnifiedAnalytics #SparkAISummit IngestSampler LogicalIngestSampler IngestSamplerExec IngestSamplerRDD Prism Logical Plan RDD Spark Physical Plan Spark Logical Plan
- 24. Prism Data Types 24#UnifiedAnalytics #SparkAISummit
- 25. Implementing Additional Data Types • Prism has a richer type system than Catalyst • Uses StructType and StructField to implement additional data types 25#UnifiedAnalytics #SparkAISummit
- 26. Example: Prism Currency Type object CurrencyType extends StructType( Array( StructField(“amount”,DecimalType(26, 6)), StructField(“code”, StringType))) >> { “amount”: 1000.000000, “code”: “USD” } >> { “amount”: -999.000000, “code”: “YEN” } 26#UnifiedAnalytics #SparkAISummit
- 28. Lessons #1: Nested SQL 28#UnifiedAnalytics #SparkAISummit
- 29. Lesson #1: Nested SQL 29#UnifiedAnalytics #SparkAISummit • SQL requires computed columns to be nested – SELECT 1 as c1, c1 + 1 as c2; /* ✗ */ – SELECT c1 + 1 as c2 FROM (SELECT 1 as c1); /* ✓ */ • First version: one nesting per computed column – Does not scale to 100s of columns – Takes a long time to compile and optimize
- 30. Lesson #1: Example Dependency Graph [first.name], [last.name], [income], concat([first.name],”.”, [last.name]) as [full.name], [income] * 0.28 as [federal.tax], [income] *0.10 as [state.tax], concat([full.name],”@workday.com”) as [email] first.name last.name income full.name federal.tax email state.tax 2nd level 1st level 30#UnifiedAnalytics #SparkAISummit
- 31. select [income] * 0.10 as [state_tax], * from (select [income] * 0.28 as [federal_tax], * from (select concat([full.name],”@workday.com”) as [email], * from (select concat([first.name],”.”, [last.name]) as [full.name], * from (select [first.name], [last.name], [income] from base_table)))) Lesson #1: SQL Before Optimization 4 levels of nested SQL 31#UnifiedAnalytics #SparkAISummit
- 32. Lesson #1: SQL After Optimization 2 levels of nested SQL 32 select concat([full.name],”@workday.com”) as [email], * from (select concat([first.name],”.”, [last.name]) as [full.name], [income] * 0.28 as [federal_tax], [income] * 0.10 as [state_tax], * from (select [first.name], [last.name], [income] from base_table))) #UnifiedAnalytics #SparkAISummit
- 33. Lesson #2: Plan Blowup 33#UnifiedAnalytics #SparkAISummit
- 34. Lesson #2: Plan Blowup 34#UnifiedAnalytics #SparkAISummit • Generated plans can have duplicate operators • E.g. self joins and self unions • Need to de-duplicate to improve performance
- 35. Lesson #2: Deduping Prism Logical Plan 35#UnifiedAnalytics #SparkAISummit Union( Sample(k=100, Parse(“Dataset A”)), Join( Sample(k=100, Parse(“Dataset A”)), Parse(“Dataset B”)), Join( Sample(k=100, Parse(“Dataset A”)), Parse(“Dataset B”)) )
- 36. Union( Cache(ID=1, Sample(k=100, Parse(“Dataset A”))), Join( Cache(ID=1, ∅), Parse(“Dataset B”)), Join( Sample(k=100, Parse(“Dataset A”)), Parse(“Dataset B”)) ) Lesson #2: Deduping Prism Logical Plan 36#UnifiedAnalytics #SparkAISummit Union( Sample(k=100, Parse(“Dataset A”)), Join( Sample(k=100, Parse(“Dataset A”)), Parse(“Dataset B”)), Join( Sample(k=100, Parse(“Dataset A”)), Parse(“Dataset B”)) )
- 37. Union( Cache(ID=1, Sample(k=100, Parse(“Dataset A”))), Cache(ID=2, Join( Cache(ID=1, ∅), Parse(“Dataset B”))), Cache(ID=2, ∅) ) Lesson #2: Deduping Prism Logical Plan 37#UnifiedAnalytics #SparkAISummit Union( Sample(k=100, Parse(“Dataset A”)), Join( Sample(k=100, Parse(“Dataset A”)), Parse(“Dataset B”)), Join( Sample(k=100, Parse(“Dataset A”)), Parse(“Dataset B”)) )
- 38. Lesson #2: Deduping Spark Tree String 38#UnifiedAnalytics #SparkAISummit (1) Project #2 +- (2) Join #2 :- (3) Project #1 : +- (4) Join #1 : :- (5) Scan #1 : +- (6) Scan #1 +- (7) Project #1 +- (8) Join #1 :- (9) Scan #1 +- (10) Scan #1
- 39. Lesson #2: Deduping Spark Tree String 39#UnifiedAnalytics #SparkAISummit (1) Project #2 +- (2) Join #2 :- (3) Project #1 : +- (4) Join #1 : :- (5) Scan #1 : +- (6) Scan #1 +- (7) Project #1 +- (8) Join #1 :- (9) Scan #1 +- (10) Scan #1 (1) Project #2 +- (2) Join #2 :- (3) Project #1 : +- (4) Join #1 : :- (5) Scan #1 : +- (6) Lines 5-5 +- (7) Lines 3-6
- 40. Lesson #3: Broadcast Join Tuning 40#UnifiedAnalytics #SparkAISummit
- 41. Node 1 A 1 B 3 C 6 D 7 AA 2 BB 5 CC 9 Node 2 E 2 F 4 G 5 H 8 EE 3 FF 8 Node 1 Node 2 A 1 B 3 C 6 D 7 AA 2 BB 5 CC 9 EE 3 FF 8 E 2 F 4 G 5 H 8 AA 2 BB 5 CC 9 EE 3 FF 8 Broadcast Join #UnifiedAnalytics #SparkAISummit 41 Lesson #3: Broadcast Join Review
- 42. • Spark’s broadcasting mechanism is inefficient – Broadcasted data goes through the driver – No global limit on broadcasted data – Complex jobs can make driver run out of memory Lesson #3: Spark Broadcast 42#UnifiedAnalytics #SparkAISummit Driver Executor 1 Executor 2 (1) Driver collects broadcasted data from executors (2) Driver sends broadcasted data to executors
- 43. • Initially disabled broadcast joins for stability • Expectation: small number of joins, all large joins Lesson #3: Disabling Broadcast Joins 43#UnifiedAnalytics #SparkAISummit spark.sql.autoBroadcastJoinThreshold = -1
- 44. Lesson #3: Re-Enabling Broadcast Joins 44 • Reality: large number of joins, many are small • Re-enabled broadcast join with a low threshold • 2-10x runtime improvement #UnifiedAnalytics #SparkAISummit spark.sql.autoBroadcastJoinThreshold = 1000000
- 45. Lesson #4: Case-Insensitive Grouping 45#UnifiedAnalytics #SparkAISummit
- 46. Prism Prism Prism HDFS / S3 Query Engine Spark Driver Spark Executor Interactive Data Prep Spark Driver Spark Executor Spark Driver Data Prep Publishing YARN Spark Executor Spark Executor Lesson #4: Spark in Query Engine #UnifiedAnalytics #SparkAISummit 46
- 47. 47#UnifiedAnalytics #SparkAISummit Lesson #4: Spark in Query Engine
- 48. Sum of Billing Amount per Billing Location BillingLocation BillingAmount CALIFORNIA 100000 california 50000 california 40000 Illinois 25000 Texas 15000 TeXas 60000 texas 5000 BillingLocation TotalBillingAmount CALIFORNIA 100000 california 90000 TeXas 60000 Illinois 25000 texas 15000 Texas 5000 SELECT BillingLocation, SUM(BillingAmount) AS TotalBillingAmount FROM InsuranceClaims GROUP BY BillingLocation ORDER BY TotalBillingAmount 48#UnifiedAnalytics #SparkAISummit Lesson #4: Grouping on String Columns
- 49. Sum of Billing Amount per Billing Location SELECT MIN(BillingLocation) AS BillingLocation, SUM(BillingAmount) AS TotalBillingAmount FROM InsuranceClaims GROUP BY UPPER(BillingLocation) ORDER BY TotalBillingAmount BillingLocation TotalBillingAmount CALIFORNIA 190000 TeXas 80000 Illinois 25000 In Workday, grouping on strings columns is case insensitive 49 BillingLocation BillingAmount CALIFORNIA 100000 california 50000 california 40000 Illinois 25000 Texas 15000 TeXas 60000 texas 5000 #UnifiedAnalytics #SparkAISummit Lesson #4: Grouping on String Columns
- 50. GROUP BY stringField GROUP BY UPPER(stringField) + MIN(stringField) ~7x regression 50#UnifiedAnalytics #SparkAISummit Lesson #4: Case-Insensitive Grouping is Costly
- 51. Aggregation on strings uses Spark uses SortAggregate operator ➔ Modified Spark’s HashAggregate to support strings Regression reduced to ~3x SortAggregate HashAggregate 51#UnifiedAnalytics #SparkAISummit Lesson #4: Aggregation on String Columns
- 52. In Spark’s HashAggregate operator, functions used in GROUPING operator were getting evaluated twice Regression reduced to ~2x UPPER evaluated twice UPPER evaluated only once 52#UnifiedAnalytics #SparkAISummit Lesson #4: Reducing Function Evaluations
- 53. Precompute uppercase for all characters ➔ replace toUpperCase() on each char by a simple array lookup Regression reduced to ~1.5x (and want to decrease more...) UPPER Optimized UPPER 53#UnifiedAnalytics #SparkAISummit Lesson #4: Optimizing Spark’s UPPER Function
- 54. And one more thing... 54#UnifiedAnalytics #SparkAISummit
- 55. HDFS / S3 Prism 1 Tenant 1 Prism 2 Tenant 2 Prism 3 Tenant 3 Prism 4 Tenant 4 Spark Cluster Spark Cluster Spark Cluster Spark Cluster Current – Single-Tenanted Spark Clusters 55#UnifiedAnalytics #SparkAISummit
- 56. HDFS / S3 Spark Cluster Spark Cluster Tenant 2 Tenant 4Tenant 3 Tenant 6Tenant 5 Tenant 7Tenant 1 Tenant 8 Prism 1 Prism 2 Prism 3 Spark Cluster Future – Multi-Tenanted Spark Clusters 56#UnifiedAnalytics #SparkAISummit