Managing python at scale without breaking the bank
Download as PPTX, PDF2 likes3,398 views
The document discusses J.P. Morgan's use of Python for its Athena platform, which supports pricing, trading, risk management, and analytics with a focus on continuous delivery. It highlights the challenges and benefits of managing a large-scale Python environment, including robust production systems and developer productivity. Key takeaways include the importance of automated testing and the flexibility of Python, while also acknowledging the difficulties related to open source package upgrades and production stability.
Managing python at scale without breaking the bank
- 1. Managing Python at scale without breaking the bank Michael (Misha) Tselman PyData NY 2017
- 2. Agenda • J.P. Morgan and Athena • Objectives • Continuous delivery • Under the hood • Challenges • Conclusions • Q&A
- 3. J.P. Morgan • One of the world’s biggest banks • $2.5 trillion assets • $95 billion revenue • Processing $5 trillion payments every day • 230,000+ employees globally • One of the world’s biggest tech companies • 44,000+ employees in Technology • $9.5 billion annual investment in technology and innovation
- 4. Athena • Python-based Pricing, Trading, Risk Management, and Analytics platform with tools for Data Science and Machine Learning • Thousands of users across multiple business lines • 1500+ Python developers use and contribute to the platform • 150,000 python modules, 35 million lines of python code • 500+ Python packages from the Open Source. • Rapid development and deployment model that puts developers and quants at the heart of the business.
- 5. Athena Foundation • Hydra ( globally replicated object database ) • Reactive Athena ( C++/Python reactive dataflow framework ) • Pixie Graph ( directed acyclic dependency graph ) • Athena Application framework based on QT • Athena Web ( tornado, html5, websockets, javascript, web assembly ) • Job scheduling ( ~270,000 jobs daily which kick-off ~1M processes ) • Integration with Compute Grid ( tens of thousands of cores + GPUs )
- 6. Objectives • Keep end-users and clients happy • Ensure robustness and stability of our production systems • Keep developers productive and efficient • Provide quants and data scientists with the best research tools • Encourage sharing and global consistency across business lines
- 7. Approach • Conceptually: • Continuous delivery: • 10,000 – 15,000 production changes every week. • Full visibility of the entire code base. Anyone can contribute. • Instant global deployment • Under the hood: • Globally replicated object databases for code (and data) • Monorepo – Monolithic code base • Extensively automated testing
- 8. Continuous delivery Write code & tests Test Commit Ask for a bless Push Run PROS • Time to market • User satisfaction • Developer productivity CONS • Fear of change / stability • High reliance on automation • Tricky in distributed systems 10,000 - 15,000 modules pushed to production every week
- 9. Layering of changes / Effective runtime Developer’s layer B3 C2 Shared staging / UAT A2 Effective Runtime A2 B3 C2 D1 B2 Production A1 B1 C1 D1 E1 E1
- 10. Alternatives to filesystem based source DB-LDN DB-NYC DB-TKO “lib.foo” “ def hello(): print ‘world’ “lib.bar” def hello(): print ‘pydata’ “lib.bar @ 2017-10-01 12:33” “ def hello(): print ‘jpmorgan’ “lib.bar @ 2017-09-21 10:16” “...” • Use globally replicated database • Customize the importer • SourceMarkers - Take advantage of transactions & timestamps
- 11. Python and Binary Runtime prod old prod prod new Python Source C++ & 3rd party
- 12. Some Challenges • Open source package upgrades • API changes • Change of pickled/stored representation • Numerical changes • Runtime/binary dependencies • Limited branching • Streamlines production • Does not fit some research/experimental workflows • Full reproducibility requires “freezing” all code including the binary train
- 13. Conclusions • Python’s flexibility makes things easier • Good integration tests ensure compatibility and consistency • Modules don’t have to be loaded from a filesystem • Production stability does not imply slow delivery and deployment • Open source does not imply free • Shared platform does not imply shared knowledge
- 14. References • J.P. Morgan http://www.jpmorgan.com/techcareers • The motivation for a monolithic codebase http://cacm.acm.org/magazines/2016/7/204032-why-google-stores- billions-of-lines-of-code-in-a-single-repository/fulltext
- 15. Q&A
Editor's Notes
- #9: Ghost tests.
- #12: At the bottom, a more traditional release train of binaries and 3rd party packages. prod.new continuously changing until lockdown. Allows for seamless testing of new features against the python baseline. Globally distributed and instantly available for import in any region for any user. Not just a repo, but a fully deployed codebase at the same time.