Adi Polak - Light up the Spark in Catalyst by avoiding UDFs - Codemotion Berlin 2018
0 likes37 views
Adie Polak discusses the drawbacks of using custom UDFs in Apache Spark, emphasizing that they are not optimized by Spark and can lead to lost optimizations like constant folding and predicate pushdown. The document advocates for using standard column-based functions and dataset operators before resorting to custom UDFs, which should be treated as a last resort. Ultimately, it suggests utilizing Spark's built-in functions and exploring available options through the DataFrame API.
Adi Polak - Light up the Spark in Catalyst by avoiding UDFs - Codemotion Berlin 2018
- 1. Adi Polak Spark UDFs are EviL, Catalyst to the rEsCue!
- 2. • Adi Polak • Sr. Cloud Relation developer • Previous Security researcher • Majored in Machine Learning • Tel Avivian • BGU alumni • Co-founderr of FLIP • Spark & Scala enthusiast • Foodie Who am I @adipolak @adipolak
- 4. Real-time analytics on Big Data
- 5. • Apache Spark with Scala • Spark 2.3 • Catalyst optimization • Spark custom UDFs ..OK
- 8. CATALYST
- 9. Fundamentals of Catalyst Optimizer SUB Attribute(x) SUB some_func(1) some_func(2) Tree Rules SUB Attribute(x) some_func(-1)
- 10. Spark SQL Execution Plan Logical optimization –> Optimization rules • Constant folding • Predicate pushdown • Projection pruning • … Physical Planning –> Planning strategies Catalyst Frontend Backend
- 11. What is Spark Custom UDF
- 12. What is Spark Custom UDF "Use the higher-level standard Column-based functions with Dataset operators whenever possible before reverting to using your own custom UDF functions since UDFs are a blackbox for Spark and so it does not even try to optimize them."
- 13. What is Spark Custom UDF "Use the higher-level standard Column-based functions with Dataset operators whenever possible before reverting to using your own custom UDF functions since UDFs are a blackbox for Spark and so it does not even try to optimize them."
- 14. What is Spark Custom UDF "Use the higher-level standard Column-based functions with Dataset operators whenever possible before reverting to using your own custom UDF functions since UDFs are a blackbox for Spark and so it does not even try to optimize them."
- 15. What do we lose when using Custom UDF ? •Constant folding •Predicate pushdown
- 16. What can we do ?
- 17. Use queryExecution & explain(true) Catalyst Frontend Backend
- 19. Use queryExecution & explain(true) API My UDF Register
- 20. Use queryExecution & explain(true) API My UDFs Register
- 21. Lost Push Down filter
- 22. What can be done instead? sql functions DataFrame API: Aggregate functions Collection functions Date time functions Math functions Non-aggregate functions Sorting functions String functions Window functions sql functions Column API Expression operations..
- 23. How can I find what functions are available? arrayContains, minute, round, rand, spark_partition_id, isin … version
- 24. Can you show a complex example? Sure Meh…
- 25. Using column functions ... GREAT SUCCESS !
- 27. Takeaways • Use UDFs as a last resort • Always check yourself with dataFrame.explain(true)
- 28. Reference • www.kaggle.com • http://bit.ly/adiuserguide • http://bit.ly/whatisdatabricks • http://bit.ly/databrickstutorial • http://bit.ly/clitools