Parallelize data processing - 2023-10-24
0 likes17 views
The document discusses the parallelization of data processing with a focus on performance challenges and results using various libraries like pandas, Dask, and Modin on different datasets. It compares execution times, highlights best practices for optimizing performance, and emphasizes the importance of vectorization and multiprocessing. Resources such as links to benchmarking tools and related works are provided to support the findings.
Parallelize data processing - 2023-10-24
- 1. Parallelize data processing @PyDataVenice #13 #Meetup #PyData
- 2. Alessandra Bilardi Data & Automation Specialist ● AWS User Group Venezia member ● Coderdojo member ● PyData Venezia member alessandra.bilardi@gmail.com @abilardi bilardi
- 4. Goal
- 5. Goal
- 6. Goal
- 7. Goal
- 8. Challenge
- 9. Challenge
- 10. Challenge
- 11. Challenge
- 12. Challenge
- 14. Challenge
- 17. Solution
- 19. Solution
- 20. Solution Goal: performance Challenge: on 4 CPU & 8GB RAM ● try.sh ○ for each dataset ■ run.py ● run.py ○ for each library ■ importlib ■ main() ● main() ○ for each method ■ timeit before/after ■ print timing ■ gc.collect()
- 21. Datasets - rows Goal: execution timing Challenge: resources ● 10 ● 50 ● 100 ● 500 ● 1.000 ● 5.000 ● 10.000 ● 50.000 ● 100.000 ● 500.000 ● 1.000.000 ● 5.000.000 ● 10.000.000
- 22. Libraries Goal: learning curve Challenge: the same behaviours ● concurrent ● dask ● joblib ○ joblib on dask ● modin ○ modin on dask ● multiprocessing ● multiprocesspandas ● pandarallel ● pandas ● parallelize ● pyspark ● swifter
- 23. run_libraries.py
- 24. run_libraries.py
- 25. run_libraries.py
- 26. main()
- 27. main()
- 28. Solution main()
- 29. Results
- 30. Results
- 31. Results
- 32. Results $ grep "hlp.get_pd_sample()" * concurrent_actions.py: sample = hlp.get_pd_sample() joblib_actions.py: sample = hlp.get_pd_sample() joblibdask_actions.py: sample = hlp.get_pd_sample() multiprocessing_actions.py: sample = hlp.get_pd_sample() multiprocesspandas_actions.py: sample = hlp.get_pd_sample() pandarallel_actions.py: sample = hlp.get_pd_sample() pandas_actions.py:sample = hlp.get_pd_sample() parallelize_actions.py: sample = hlp.get_pd_sample() swifter_actions.py: sample = hlp.get_pd_sample()
- 33. Results
- 34. Results
- 35. Results ✅ ❌
- 36. Results
- 37. Results
- 41. Results
- 42. Results
- 43. Results
- 44. Results
- 45. Solution main()
- 46. Results
- 47. Results
- 48. Segmenti casuali di codice o è qualcosa di più ..
- 49. Results
- 50. Results
- 51. Results
- 54. Oddities Goal: performance Challenge: on 4 CPU & 8GB RAM ● pandas takes longer ○ than libs who use it ● pandas generate more ○ pids & threads than pyspark ● modin generate more ○ pids & threads than others ● modin works better if ..
- 55. Best practies Goal: performance Challenge: on 4 CPU & 8GB RAM ● vectorialization ○ better than apply ● multiprocessing ● pay attention ○ df.copy() ○ groupby / sum / apply
- 56. Thanks for listening. @PyDataVenice #13 #Meetup #PyData