Performant data processing with PySpark, SparkR and DataFrame API
12 likes3,455 views
This document discusses using PySpark, SparkR and DataFrame APIs to perform efficient data processing with Apache Spark. It explains that while Python and R can be used with Spark, performance may be slower than Java and Scala since data needs to be transferred between the JVM and the non-JVM language runtime. DataFrame APIs allow working with data within the JVM, avoiding this overhead and providing near-native performance when using Python, R or other non-JVM languages. Examples demonstrate how to use DataFrames and SQL with filters to optimize performance before using user-defined functions that require data transfer. Ingesting data in a DataFrame-native format like Parquet is also recommended for efficiency.
![Executor
DataFrame APIs compared to
RDD APIs by Examples
JVM
DataFrame,
Cached
Python
lambda items:
items[0] == ‘abc’
transfer
DataFrame,
result
transfer
Driver](https://image.slidesharecdn.com/20151121rtcpyspark-151121133456-lva1-app6892/85/Performant-data-processing-with-PySpark-SparkR-and-DataFrame-API-18-320.jpg)
![Executor
DataFrame APIs compared to
RDD APIs by Examples
JVM
DataFrame,
Cached
filter(df[“_1”]
== “abc”)
transfer
DataFrame,
result
Driver](https://image.slidesharecdn.com/20151121rtcpyspark-151121133456-lva1-app6892/85/Performant-data-processing-with-PySpark-SparkR-and-DataFrame-API-19-320.jpg)
Performant data processing with PySpark, SparkR and DataFrame API
- 1. Performant data processing with PySpark, SparkR and DataFrame API Ryuji Tamagawa from Osaka Many Thanks to Holden Karau, for the discussion we had about this talk.
- 2. Agenda Who am I ? Spark Spark and non-JVM languages DataFrame APIs come to rescue Examples
- 3. Who am I ? Software engineer working for Sky, from architecture design to troubleshooting in the field Translator working with O’Reilly Japan ‘Learning Spark’ is the 27th book Prized Rakuten tech award Silver 2010 for translating ‘Hadoop the definitive guide’ A bed for 6 cats
- 4. Works of 2015 Available Jan, 2016 ?
- 6. Motivation for today’s talk I want to deal with my ‘Big’ data, WITH PYTHON !!
- 7. Apache Spark
- 8. Apache Spark You may already have heard a lot Fast, distributed data processing framework with high-level APIs Written in Scala, run in JVM OS HDFS Hive e.t.c. HBaseMapReduce YARN Impala e.t.c（in- memory SQL engine） Spark （Spark Streaming, MLlib, GraphX, Spark SQL)
- 9. Why it’s fast Do not need to write temporary data to storage every time Do not need to invoke JVM process every time map JVM Invocation I/0 HDFS reduce JVM Invocation I/0 map JVM Invocation I/0 reduce JVM Invocation I/0 f1（read data to RDD） Executor（JVM）Invocation HDFS I/O f2 f3 f4（persist to storage） f5（does shuffle） I/O f6 f7 Memory(RDDs) access access access access I/O access access MapReduce Spark
- 11. Spark supports non-JVM languages Shells PySpark, for Python users SparkR, for R users GUI Environment : Jupiter, RStudio You can write application code in these languages
- 12. The Web UI tells us a lot http://<address>:4040
- 13. Performance problems with those languages Data processing performance with those languages may be several times slower than JVM languages The reason lies in the architecture https://cwiki.apache.org/confluence/ display/SPARK/PySpark+Internals
- 14. The choices you have had Learn Scala Write (more lines of) code in Java Use non-JVM languages with more CPU cores to make up the performance gap
- 15. DataFrame APIs come to the rescue !
- 16. DataFrame Tabular data with schema based on RDD Successor of Schema RDD (Since 1.4) Has rich set of APIs for data operation Or, you can simply use SQL!
- 17. Do it within JVM When you call DataFrame APIs from non-JVM Languages, data will not be transferred between JVM and the language runtime Obviously, the performance is almost same compared to JVM languages Only code goes through
- 18. Executor DataFrame APIs compared to RDD APIs by Examples JVM DataFrame, Cached Python lambda items: items[0] == ‘abc’ transfer DataFrame, result transfer Driver
- 19. Executor DataFrame APIs compared to RDD APIs by Examples JVM DataFrame, Cached filter(df[“_1”] == “abc”) transfer DataFrame, result Driver
- 20. Watch out for UDFs You can write UDFs in Python You can use lambdas in Python, too Once you use them, data flows between the two worlds slen = udf( lambda s: len(s), IntegerType()) df.select( slen(df.name)) .collect()
- 21. Make it small first, then use UDFs Filter or sample your ‘big’ data with DataFrame APIs Then use UDFs SQL optimizer does not take it into account when making plans (so far) ‘BIG’ data in DataFrame filtering with ‘native APIs’ ‘Small’ data in DataFrame whatever operation with UDFs
- 22. Make it small first, then use UDFs Filter or sample your ‘big’ data with DataFrame APIs Then use UDFs SQL optimizer does not take it into account when making plans (so far) slen = udf( lambda s: len(s), IntegerType()) sqc.SQL( ‘select… from df where fname like “tama%” and slen(name)’ ).collect() processed first !
- 23. Ingesting Data It’s slow to Deal with files like CSVs by non-JVM driver Anyway, convert raw data to ‘Dataframe-native’ formats like Parquet at first You can process Such files directly from JVM processes (executors) even when using non-JVM languages Executor JVM DataFrameDriver Local Data Py4J Driver Machine HDFS (Parquet)
- 24. Driver Machine Ingesting Data Executor JVM DataFrameDriver Py4Jcode only HDFS (Parquet) code only It’s slow to Deal with files like CSVs by non-JVM driver Anyway, convert raw data to ‘Dataframe-native’ formats like Parquet at first You can process Such files directly from JVM processes (executors) even when using non-JVM languages
- 26. Parquet: general purpose file format for analytic workload Columnar storage : reduces I/O significantly High compression rate projection pushdown Today, workloads become CPU- intensive : very fast read, CPU-internal- aware