Spark Summit EU talk by Oscar Castaneda
4 likes1,110 views
The document discusses transforming Excel spreadsheets into Spark DataFrames by automatically translating Excel formulas into Spark code. It presents a program transformation pipeline that takes Excel formulas, parses them using a grammar and parser to generate a parse tree, and then generates Spark code from the parse tree. Key aspects covered include using an existing grammar and parser called XLParser to parse Excel formulas, treating Excel as a domain-specific language, and generating code by writing a pretty printer for the target Spark language. The talk concludes with a demonstration of the code generation approach.
Spark Summit EU talk by Oscar Castaneda
- 1. Sparksheet - Transforming Spreadsheets into Spark Data Frames Oscar Castañeda-Villagrán Universidad del Valle de Guatemala
- 2. About • Researcher at Universidad del Valle de Guatemala. • Research Interests: • Program Transformation, • Programming Education Research, • Online Learning to Rank.
- 4. Prototyping Spark programs with … http://bit.ly/2e5GmyY http://bit.ly/2edYfMs
- 5. http://bit.ly/2e5GmyY http://bit.ly/2edYfMshttp://bit.ly/2e0TZA8 Prototyping Spark programs with Excel
- 6. Agenda • Problem Statement and Motivation • Architecture • Program Transformation • Pipeline • Code-to-Code Transformation • Parsing Excel Formulas • Grammar • Parse Tree • XLParser • Excel as a DSL • Generating Code • Demo • Q&A
- 7. Disclaimer(s) • Ongoing research … • We will focus on how to create a Program Transformation Pipeline.
- 8. Problem Statement Spark programs can be prototyped in Excel but manually translating Excel formulas to Spark programs is tedious and error-prone.
- 9. Motivation • “Straight path” between column-oriented Excel “programs” and Spark programs that make use of the DataFrame API. • But, manually translating Excel formulas to Spark is tedious and error-prone. • What if: Excel compiler?
- 10. Problem Statement Given that column-oriented Excel applications can be manually translated to Spark programs, … … find a way to automate translation of Excel formulas so that data pipelines can be prototyped in Excel … … and Scala/Python code generated to run in Spark.
- 11. Motivation Automatically translate Excel Formulas to … Spark. Data! http://bit.ly/2expmoF
- 12. Architecture Excel Formulas Columnar Data Take ES snapshot Restore ES snapshot Black Box Data! Refine Code http://bit.ly/2em6RUK http://bit.ly/2e5H1jL
- 13. Excel Formulas Columnar Data Take ES snapshot Restore ES snapshot Black Box Data! Refine Code Architecture http://bit.ly/2em6RUK http://bit.ly/2e5H1jL
- 14. Excel Formulas Columnar Data Take ES snapshot Restore ES snapshot Black Box Data! Refine Code Tomorrow: Spark Cluster with Elasticsearch InsideArchitecture http://bit.ly/2em6RUK http://bit.ly/2e5H1jL
- 16. Program Transformation “A program transformation is any operation that takes a computer program and generates another program.” https://en.wikipedia.org/wiki/Program_transformation
- 17. Architecture Excel Formulas Columnar Data Take ES snapshot Restore ES snapshot Black Box Data! Refine Code
- 19. Code-to-Code Transformation “The input to the code generator typically consists of a parse tree or an abstract syntax tree.” https://en.wikipedia.org/wiki/Code_generation_(compiler)
- 20. http://bit.ly/2dH0ybF We need a Grammar!
- 21. We need a Parse Tree! Parse Excel Formulas
- 22. We need a Parse Tree! Parse Excel Formulas http://bit.ly/2e0TZA8
- 23. We need a Parse Tree! Generate Scala CodeParse Excel Formulas http://bit.ly/2e0TZA8
- 24. We need a Grammar! http://bit.ly/2dH0ybF
- 25. XLParser "If I have seen further, it is by standing upon the shoulders of giants" — Sir Isaac Newton A Grammar for Spreadsheet Formulas Evaluated on Two Large Datasets – Efthimia Aivaloglou, David Hoepelman & Felienne Hermans, Proceedings of SCAM ’15
- 26. XLParser "If I have seen further, it is by standing upon the shoulders of giants" — Sir Isaac Newton A Grammar for Spreadsheet Formulas Evaluated on Two Large Datasets – Efthimia Aivaloglou, David Hoepelman & Felienne Hermans, Proceedings of SCAM ’15
- 27. XLParser "If I have seen further, it is by standing upon the shoulders of giants" — Sir Isaac Newton Excel Formula SUM(A,C) http://xlparser.perfectxl.nl/demo
- 28. XLParser "If I have seen further, it is by standing upon the shoulders of giants" — Sir Isaac Newton Parse Tree! SUM(A,C) Excel Formula http://xlparser.perfectxl.nl/demo
- 29. Excel as a DSL • External DSL: parsed independently. • XLParser gives us a Parse Tree from an Excel Formula. • Given the Parse Tree, generate code!
- 30. How do you generate code from parsed Excel Formulas? ?
- 31. Generating Code “An elegant way to generate code from an AST is to write a class for each non-terminal node in the tree, and then each node in the tree simply generates the piece of code that it is responsible for.” http://www.codeproject.com/Articles/26975/Writing-Your-First-Domain-Specific-Language-Part
- 32. Generating Code A practical way to generate code is to take a Parse Tree and write a pretty printer for the target language. http://bit.ly/2em73DM
- 33. Generating Code from an AST SUM(A,C)
- 34. Generating Code from an AST
- 35. Generating Code from an AST
- 36. Demo!
- 37. What have we seen? • Column-Oriented Excel Applications as Prototypes for Spark programs • Program Transformation. • How to model as a Pipeline. • Why considered a Code-to-Code Transformation. • How to Parse Excel Formulas. • Grammar • Parse Tree • XLParser • Excel as a DSL. • How can we Generate Code? • Demo.
- 38. Next Steps • Translate ~500 Excel Formulas. • Modeling Machine Learning in Excel. • Prototype D|’s and ML|’s in Excel.
- 39. Q&A
- 40. Q&A Tomorrow: Spark Cluster with Elasticsearch Inside
- 41. THANK YOU. Email: ofcastaneda@uvg.edu.gt Twitter: @oscar_castaneda