20161215 python pandas-spark四方山話
7 likes1,239 views
This document discusses using Python, Pandas, and Spark 2.0 for data analysis. It covers loading CSV and other data formats into Pandas and Spark DataFrames, using Parquet for efficient storage, and transferring data between Pandas and Spark for hybrid processing using CPUs and SSDs. The last part discusses new features in Spark 2.0 like SQL support on DataFrames and improved Python integration.
![Python + Spark
• Python serialize
• DataFrame API UDF
UDF Scala/Java
• http://www.slideshare.net/dragan10/performant-data-processing-with-pyspark-sparkr-
and-dataframe-api
Executor
JVM
DataFrame,
Cached
Python
lambda items:
items[0] == ‘abc’
transfer
DataFrame,
result
transfer
Driver](https://image.slidesharecdn.com/20161215python-pandas-spark-161221231309/85/20161215-python-pandas-spark-25-320.jpg)
20161215 python pandas-spark四方山話
- 1. Python, Pandas, Spark 2.0 Sky
- 3. • • Python 2000 (**) • db tech showcase MongoDB • • FB: Ryuji Tamagawa • Twitter : tamagawa_ryuji
- 5. 2017
- 7. • • Python / Pandas • Spark 2.0
- 8. Part 1 :
- 9. • • • csv
- 11. • Spark API RDD ~1.3 DataFrame / DataSet 1.4~ • DataFrame API RDD API Python Spark
- 12. DataFrame • RDB / • R Pandas Spark Spark R / Pandas Spark +
- 13. Part 2 :
- 15. • • CPU • • Pandas read_csv zip CSV Pandas
- 16. 2 • CSV CPU Pandas zip CSV CPU … • Parquet ! •
- 17. : Parquet I/O • • Spark Parquet • Python Parquet
- 18. HDFS / S3 Parquet Parquet
- 21. • • I/O Pandas • Spark • DataFrame Pandas → Spark Spark → Pandas Pandas → Spark • Apache Arrow
- 23. Apache Spark 2.0 • 1.x • 2.0 1.x • DataFrame API Python • databricks http://go.databricks.com/mastering-apache-spark-2.0 •
- 24. Spark 2.0 • CPU • CPU • SQL DataFrame • + SSD • CSV zip Pandas read_csv
- 25. Python + Spark • Python serialize • DataFrame API UDF UDF Scala/Java • http://www.slideshare.net/dragan10/performant-data-processing-with-pyspark-sparkr- and-dataframe-api Executor JVM DataFrame, Cached Python lambda items: items[0] == ‘abc’ transfer DataFrame, result transfer Driver