Optimizing Apache Spark UDFs
0 likes1,051 views
The document covers the optimization of User-Defined Functions (UDFs) in Spark SQL, highlighting their benefits, performance concerns, and alternative approaches for reimplementation as native functions. It details the importance of string handling, performance comparisons, and practical implementation advice, showcasing significant performance improvements from 20% to 200% for various functions. Feedback is encouraged to enhance future sessions.
![Reimplementing - SQL
▪ Functions require editing of FunctionsRegistry.scala if sql support is
needed.
▪ sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
▪ Register your function
▪ expression[PlusOneNative]("plusonenative“)
▪ Recompile spark code to generate spark-catalyst_*.jar
▪ Edit the pom to add dependency to your previously created jar.
▪ Replace under spark/jars location](https://image.slidesharecdn.com/37shivangisrivastava-201124203042/85/Optimizing-Apache-Spark-UDFs-20-320.jpg)
Optimizing Apache Spark UDFs
- 1. Optimizing Spark UDFs Shivangi Srivastava Senior Engineering Manager
- 2. Presenter Introduction ▪ Shivangi Srivastava ▪ Senior Engineering Leader at Informatica ▪ 10 + years of experience with Distributed computing, database systems ▪ Linkedin - https://www.linkedin.com/in/shivangi-srivastava1
- 3. Informatica • Leading provider of Data Engineering solutions • Informatica offerings –
- 4. Agenda ▪ Introduction to UDF ▪ Example and benefits of UDF ▪ Performance considerations ▪ Suggested Alternatives ▪ Performance results ▪ Questions/Feedback
- 5. Introduction to UDFs ▪ User-Defined Functions (aka UDF) is a feature of Spark SQL to define new Column-based functions that extend the vocabulary of Spark SQL’s DSL for transforming Datasets. ▪ UDFs are key features of most SQL environment and extend the system’s in-built functionality. ▪ Custom functions can be defined and registered as UDFs in Spark SQL with an associated alias that is made available to SQL queries.
- 6. Typical Example ▪ Define a UDF in scala val plusOne = udf((x: Int) => x + 1) ▪ Register the UDF spark.udf.register("plusOne", plusOne) ▪ Usage spark.sql("SELECT plusOne(5)").show() // +------+ // |UDF(5)| // +------+ // | 6| // +------+
- 7. Benefits of UDFs ▪ Extends the in-built capabilities of Spark SQL. ▪ Simple and straightforward to implement. ▪ Plug and play architecture. ▪ Define once and use across multiple dataframes. ▪ Backward compatible. Stable API not impacted by version upgrades.
- 8. Performance concerns with UDFs ▪ UDFs are black-box to Spark optimizations. ▪ UDFs block many spark optimizations like ▪ WholeStageCodegen ▪ Null Optimizations ▪ Predicate Pushdown ▪ More optimizations from Catalyst Optimizer ▪ String Handling within UDFs ▪ UTF-8 to UTF-16 conversion. Spark maintains string in UTF-8 encoding versus Java runtime encodes in UTF-16. ▪ Any String input to UDF requires UTF-8 to UTF-16 conversion. ▪ Conversely, a String output requires a UTF-16 to UTF-8 conversion. age codegen
- 9. Analyzing physical plan X+1 versus PlusOne
- 10. Analyzing physical plan X+1 versus plusOne
- 11. Analyzing physical plan – Predicate Pushdown X+1 versus plusOne
- 12. Analyzing physical plan – Predicate Pushdown X+1 versus plusOne
- 13. Redesign UDF implementation ▪ Implement UDFs as Spark native functions. ▪ Design Goals for reimplementing. ▪ Extend Spark’s capabilities with minimal changes to the existing Spark code. ▪ Ability to upgrade to later Spark versions without significant engineering effort
- 14. Reimplementing Spark UDFs as Spark native ▪ Create a new project structure like spark.
- 15. Reimplementing Spark UDFs as Spark native ▪ Extend from Spark’s Expression class ▪ UnaryExpression – Single argument expressions ▪ TernaryExpression – Multi argument expressions ▪ Satisfy expression contract
- 16. Reimplementing Spark UDFs as Spark native ▪ Examples for existing Spark functions can be found at – ▪ sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/
- 17. Reimplementing Spark UDFs ▪ Define the function implementation
- 18. Reimplementing Spark UDFs as Spark native functions ▪ Add new function definition file to package org.apache.spark.sql ▪ Compile and add the jar to spark/jars folder.
- 19. Reimplementing - Usage ▪ Select as a normal function available within spark library.
- 20. Reimplementing - SQL ▪ Functions require editing of FunctionsRegistry.scala if sql support is needed. ▪ sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala ▪ Register your function ▪ expression[PlusOneNative]("plusonenative“) ▪ Recompile spark code to generate spark-catalyst_*.jar ▪ Edit the pom to add dependency to your previously created jar. ▪ Replace under spark/jars location
- 21. Reimplementing - Usage ▪ Exercise using spark-sql
- 22. Analyzing physical plan – Predicate Pushdown plusOneNative
- 23. More Tips and Tricks ▪ Make conscious effort to avoid temporary object allocation. ▪ Use Scala’s while construct over for. ▪ For causes creation of temporary object creation. ▪ Consider imperative style over functional style. ▪ Consider using thread static variable to allocate temporary buffer ▪ Sometimes UTF-8 to UTF-16 conversion is required, consider lazy conversion based on presence of UTF-16 characters. ▪ Inspect the string for presence of UTF-16 characters
- 24. Performance Comparison – String function ▪ Performance improvement of about 20% to 200% ▪ Overhead of UTF-8 to UTF-16 conversion avoided. SF1 SF10 SF50 optimization disabled 0:02:57 0:04:13 0:18:29 optimization enabled 0:02:38 0:03:26 0:06:21 0:00:00 0:02:53 0:05:46 0:08:38 0:11:31 0:14:24 0:17:17 0:20:10 Time(hh:mm:ss) Scale Factor String Function optimization disabled optimization enabled
- 25. Performance Comparison – Date function ▪ Performance improvement of about 20% to 100% ▪ Avoided creating temporary objects ▪ Used imperative style programming ▪ Used while instead of for SF1 SF10 SF50 optimization disabled 0:03:10 0:04:28 0:10:09 optimization enabled 0:02:38 0:02:41 0:05:39 0:00:00 0:01:26 0:02:53 0:04:19 0:05:46 0:07:12 0:08:38 0:10:05 0:11:31 Time(hh:mm:ss) Scale Factor Date Function optimization disabled optimization enabled
- 26. Performance Comparison – Numeric function ▪ Performance improvement of about 15% to 50% SF1 SF10 SF50 optimization disabled 0:04:15 0:06:07 0:10:41 optimization enabled 0:03:36 0:04:04 0:09:15 0:00:00 0:01:26 0:02:53 0:04:19 0:05:46 0:07:12 0:08:38 0:10:05 0:11:31 Time(hh:mm:ss) Scale Factor Numeric Function optimization disabled optimization enabled
- 27. Performance comparison - Summary ▪ 200% faster for certain String functions with large datasets (50GB). ▪ 50%-100% faster for date and numeric functions. ▪ Performance difference becomes noticeable with larger datasets. ▪ Conversion and optimization cost goes up. ▪ Garbage collection overhead becomes a significant contributor to the overall execution time.
- 28. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.