Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
6 likes7,842 views
Spotify uses a service oriented architecture with over 50 services written in Java and Python running on over 7,000 servers. They were previously deploying Python applications via Debian packages but found this took too much time managing dependencies. They developed dh-virtualenv, which bundles application dependencies into a virtualenv deployed in the package. This isolates dependencies, allows rolling back packages, and ensures the same environment is used for development and deployment. They have seen benefits including no longer being constrained to Debian stable and easier code and dependency isolation.
![22
Example: Using Sentry
Define Dependencies in requirements.txt
sentry[postgres]==6.2.0
eventlet==0.13.0
hiredis==0.1.1
django-‐auth-‐ldap==1.1.4](https://image.slidesharecdn.com/virtualenvindebian-140130093050-phpapp02/85/Deployment-Tools-and-Techniques-at-Spotify-Virtualenv-in-debian-by-Chris-Angove-22-320.jpg)
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
- 1. 1 Spotify Tech Web/iOS/Android/Native clients Service oriented architecture > 50 services in Java and Python Primarily using Cassandra for persistent storage 7,000 Servers in production ~500 in Engineers worldwide
- 2. 2 What have we been doing? Lost in the forest of dependency building
- 3. 3 Deploy via Debian Packages Use Debian Stable repositories Package/Backport needed libraries as needed All Dependencies defined in Debian package Deployed via Puppet/Aptitude
- 4. 4 Mirroring P a c k a g e A p p s P y P i
- 5. 5 What’s wrong with that? Spending too much time managing dependencies and not enough building cool shit
- 6. 6 Using Debian Stable does not included latest and greatest
- 7. 7 Requires “Backporting” for newer libraries
- 8. 8 Does not used normal python packaging
- 9. 9 Limited use on developer systems (or bad/duplicated information)
- 11. 11 What you test is NOT what you deploy
- 12. 12 Dependency lock in due to sharing a common python environment
- 13. 13 There MUST be a better way Avoid duplication of effort
- 14. 14 There MUST be a better way Increase stability and repeatability
- 15. 15 There MUST be a better way Offer easy rollback
- 16. 16 There MUST be a better way Scale
- 17. 17 What about now? Deploying Python @ Scale
- 19. 19 Dh == Debian Helper • • • • • Registers itself as a Debian Helper Bundles the requirements in requirements.txt into virtual environment Deploys virtualenv into /usr/share/python/<package-name> Still requires native dependencies to be installed via packaging Just include the following in the debian/rules: %: dh $@ -‐-‐with python-‐virtualenv
- 20. 20 Mirroring V i r t u a l e n v Pip A p p s
- 21. 21 IT Works!
- 22. 22 Example: Using Sentry Define Dependencies in requirements.txt sentry[postgres]==6.2.0 eventlet==0.13.0 hiredis==0.1.1 django-‐auth-‐ldap==1.1.4
- 23. 23 Example: Using Sentry Define Debian dependencies in control file Build-Depends: python (>= 2.6.6-3~), debhelper (>= 8), dh-virtualenv, python-dev, libpq-dev, libldap2-dev, libsasl2-dev, Standards-Version: 3.9.3 X-Python-Version: >= 2.6
- 24. 24 Example: Using Sentry Use dh_virtualenv in Debian rules file %: dh $@ --with python-virtualenv override_dh_virtualenv: dh_virtualenv --index-url='http://localhost/simple'
- 25. 25 Have done this with RPMs as well (@ Amplify) Within RPM build virtaulenv and mark it as relocatable: /usr/local/bin/virtualenv --no-site-packages build_venv source build_venv/bin/activate env CFLAGS="$RPM_OPT_FLAGS" easy_install -Z . deactivate /usr/local/bin/virtualenv --relocatable build_venv
- 26. 26 Benefits No longer stuck on Debian stable
- 27. 27 Benefits No more backporting of new libraries or versions
- 28. 28 Benefits Code isolation/use own dependencies
- 29. 29 Benefits Single package rollback option
- 30. 30 Benefits Development and Deployed systems using same cross-platform setup
- 31. 31 What’s Next?
- 32. 32 Not Sure…
- 33. 33 Dh_virtualenv now available (ironically) in Debian Unstable repositories
- 34. 34 Still requires managing native dependencies locally
- 35. 35 But experimenting with containers!
- 36. 36 Stay Tuned!
- 37. 37 Thanks to Jyrki Pulliainen and Jim Whitehead @ Spotify Stockholm! Whitepaper: http://labs.spotify.com/2013/10/10/packagingin-your-packaging-dh-virtualenv/ Code: https://github.com/spotify/dh-virtualenv
- 38. 38 Questions? Chris Angove, Chapter Lead - Backend Email: cangove@spotify.com Blog: http://labs.spotify.com/ Listen: https://play.spotify.com/
- 39. Want to join the band? Check out spotify.com/jobs or @Spotifyjobs for more information. January 24, 2014