Time travel: Let’s learn from the history of Python packaging!
1 like375 views
This document provides a history of Python packaging and discusses lessons learned. It covers: 1. The evolution of Python packaging standards and tools from 1991 to present, including changes to metadata schemas, build tools, package publishing/indexing, installers, and support for multiple platforms/versions. 2. Challenges that existed before and after the community developed packaging solutions, such as companies initially developing their own internal tools and later adopting community standards. 3. How the community's open solutions grew faster than any individual company could achieve, and how problems like building/publishing packages are generally applicable rather than company-specific.
Time travel: Let’s learn from the history of Python packaging!
- 1. Kir Chou @ PyCon TW 2020 1 Time travel: Let’s learn from the history of Python packaging!
- 2. Kir Chou • PyCon TW Speaker 2017, 2018, 2019 • PyCon JP Speaker 2019, 2020 • SDE @ Amazon Search 2 @k1rch0u kir.chou note35
- 3. Agenda • Motivation • History of Python software distribution • Learn from history • Challenge before community’s python packaging solution • Challenge after community’s python packaging solution • Takeaway • Appendix 3
- 4. Motivation 4
- 5. 5 One day, our builder Tool team announced… Let’s migrate to idiomatic tools!
- 7. 7 @di_codes
- 8. History of Python software distribution 8
- 10. 10 How to give/take the work to/from others?
- 11. How to give/take the work to/from others? 11 Give Take
- 12. How to give/take the work to/from others? 12 Give Take
- 13. 13
- 14. RFC822 14 1991 20052003 Python 1.0 Python 2.0 2000 PEP241 PEP345 PEP314 disutils (egg) setuptools 2006 easy_install PEP405 tox virtualenv 20072008 2011 PyPI
- 15. 15 Metadata: PEP241 (2001), PEP314 (2003), PEP345 (2005) Build tool: distutils, setuptools
- 16. 16 setuptools: setup.py Metadata in EGG xxx.egg-info/PKG-INFO
- 17. 17 PyPI (Cheese Shop): PEP301 (2002/Nov) Package Installer: easy_install
- 18. 18 Isolated Environments (Build / Test / Release): tox, virtualenv (2007), venv@PEP405 (2011)
- 19. 19
- 20. 20 1991 2011 Python 1.0 2008 Python 3.0 RFC822 PEP566 PEP459 PEP426 PEP427,440PEP376 20052003 Python 2.0 2000 PEP241 PEP345 PEP314 disutils (egg) setuptools 2006 wheel easy_install bento/twine PyPI pip PEP405 tox virtualenv 2007 PEP394 PEP582 poetry pipenv 2012 2017 PyPI warehouse PEP262
- 21. 21 Deferred Metadata 2.0: PEP426 (2012), PEP459 (2013) Metadata 2.1: PEP566 (2017/Dec)
- 23. 23 Version & Dependency: PEP386 (2009/Superseded), pip-tools/pip-sync (2012) PEP440 (2013), PEP508 (2015)
- 24. 24 Package installer (2008): pip Publish Tool (2013): bento / twine / flit PyPI (2018): PyPI warehouse
- 25. 25 Database: PEP262 (2001/Deferred), PEP376 (2009) Wheel: PEP425 (2012) / PEP427 (2012)
- 26. 26 PEP376 Database of Installed Python Distributions xxx.dist-info/WHEEL
- 27. 27
- 28. 28 One For All: Conda / Pipenv / Poetry PEP517 (2015), PEP518 (2016)
- 30. 30 1. Metadata Schema Evolution
- 31. Evolution 31 2. Build Tools 1. Metadata Schema
- 32. 32 2. Build Tools 1. Metadata Schema 3. Publish Tools / Package Index Evolution
- 33. 33 2. Build Tools 1. Metadata Schema 3. Publish Tools / Package Index Evolution 4. Installer / Deployment Tools
- 34. 34 2. Build Tools 1. Metadata Schema 3. Publish Tools / Package Index 5. Multiple platform Evolution 4. Installer / Deployment Tools
- 35. 35 2. Build Tools 1. Metadata Schema 3. Publish Tools / Package Index 5. Multiple platform / version Evolution 4. Installer / Deployment Tools
- 36. 36 1991 Python 1.0 wheel pipenv poetry The pattern is repeated per generation 2008 Python 3.0 tox Python 2.0 2000 disutils (egg) setuptools virtualenv bento/twine easy_install pip PyPI warehouse PyPI
- 37. 1994 Amazon 1998 Google 37 Challenge BEFORE community’s python packaging solution
- 38. 38 1991 1998 Python 1.0 Google 1994 Amazon No Python packaging solution
- 39. 39 How to give/take the work to/from others?
- 40. Solve same problems in history • Build tool • build tool + dependency resolver • Support multiple Python version • Publish tool / Private Package Registry • Installer / Deployment tool • Support isolated environment • Support multiple platforms 40
- 41. 41 2. Internal Build Tools 1. Internal Package Metadata 3. Internal Package Registry 4. Internal Deployment Tools
- 42. 42 3p packages are all handled by internal tools
- 43. 43
- 44. 44 Company Community X growth rate X Community solutions grow faster now! ~2012
- 45. Packaging problem is mostly general! • Build System • What is a Python package? • How does build tool resolve Python’s dependencies? • How to build a Python package? • Publish System / Private Package Registry • How to publish your Python package? • How to download other Python’s dependencies? • Deployment System • How to isolate your Python environment? • Which platform does the Python package support? 45
- 46. Shift to community solution
- 47. 47 Challenge AFTER community’s python packaging solution
- 48. 1. Ecosystem 48 Community’s solution Company’s solution Community’s solution Company’s solution one way two way
- 49. 49
- 50. 50 Cloud Services Business Cloud Service Provider Developer Community Sponsorship Pay $ Invent New solution Adopt new community solution One for all, all for one
- 51. 51 2. More standards ∝ More tools
- 52. 52
- 53. 53
- 54. 54 Join PyCon! Join the community!
- 55. 55 Review the past to understand the present Takeaway History & Past Challenges Current Challenges Community
- 56. Thanks! 56
- 57. Appendix 57
- 58. History • Dustin Ingram @ SciPy 2018 Inside the Cheeseshop: How Python Packaging Works • History of packaging written by Martijn Faassen • Kenneth Reitz @ PyCon 2018 Pipenv: The Future of Python Dependency Management • Clinton Roy @ Kiwi PyCon X (2019) The Packaging Lifecycle with Poetry • History of wheel written by Daniel Holth 58
- 59. Packaging • Dave Forgac @ PyOhio 2015 Python Packaging from Init to Deploy • Elana Hashman @ PyCon 2019 The Black Magic of Python Wheel • Official Document: Packaging binary extensions (2013) 59
- 60. Runtime (Virtual) Environment • Carl Meyer @ PyCon 2011 Reverse-engineering Ian Bicking's brain: inside pip and virtualenv • Bernat Gabor @ EuroPython 2019 Status quo of virtual environments 60
- 61. 61 Why did 3 big cloud platforms provide Package Registry service JUST recently? 2019 May 2020 June 2020 July