An Introduction to Apache Arrow for Python Programmers.pptx
- 1. Presented by: An Introduction to Apache Arrow for Python Programmers 03 June 2022 Marlene Mhangami
- 3. 3 database developers and dataframe library maintainers
- 4. 4 data represented in memory
- 5. 5 What is Arrow? Arrow is fundamentally about 3 things: Libraries Ecosystem Format 1. 2. 3. arrow.apache.org
- 6. 6 Problem: How can we take advantage of all the advancements in modern hardware?
- 7. 7 Multiple Cores -The average CPU has 4-8 cores -Modern CPUs have between 2-64 cores -GPUs can contain 1000’s of cores SIMD (single instruction multiple data)
- 8. 8 Benefits of Arrow - Take advantage of modern hardware - Store data in columns not rows (better for analytics work)
- 10. 10
- 12. 12 Problem: How can we move data efficiently between projects, languages or systems
- 13. 13 CSV FILES ● EXPENSIVE TO READ AND WRITE TO ● CONVERT FROM STRING FORMAT TO ● NO RICH SUPPORT FOR TYPES
- 14. 14 2. Libraries
- 17. 17 Arrow ● Columnar format ● Binary ● Language agnostic ● memory mapped ● Zero copy
- 18. 18 Problem: How can we make the libraries as fast as possible?
- 19. 19 3. Ecosystem
- 20. 20 Community - Large language agnostic community - Open Source - Continuous improvement
- 21. 21 In-memory analytics PyArrow `pip install pyarrow` `import pyarrow as pa` `pd.read_parquet(x.parquet, engine=’pyarrow’)`
- 24. Projects Using Arrow Ray Hugging Face Graphistry Apache Spark Dask https://arrow.apache.org/powered_by/
- 26. Thanks