Buildout: How to maintain big app stacks without losing your mind
3 likes1,591 views
Buildout is a tool for managing Python application environments and dependencies. It allows users to define all of an application's dependencies, configurations, and installation instructions in a single configuration file. Buildout handles tasks like installing packages and their dependencies, compiling code, configuring applications, and deploying applications in a reproducible manner across different environments.
![Anatomy of a Buildout buildout.cfg: [buildout] parts = MyPart [MyPart] recipe = recipepackage arg1 = value1](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-18-320.jpg)
![Hello World [buildout] parts = helloworld chmod [helloworld] recipe = collective.recipe.template output = ${buildout:bin-directory}/hello input = inline: echo 'hello world' [chmod] recipe = plone.recipe.command command = chmod +x ${helloworld:output}](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-21-320.jpg)
![Dependencies and Substitutions [ buildout] parts = chmod [ helloworld] recipe = collective.recipe.template output = ${buildout:bin-directory}/hello input = inline: echo 'hello world' [ chmod] recipe = plone.recipe.command command = chmod +x ${helloworld:output}](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-22-320.jpg)
![Installing Packages (easy_install) [buildout] parts = helloworld [helloworld] recipe = zc.recipe.egg eggs = Fabulous Pillow Interpreter = python $ bin/python -m fabulous.text hello world](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-23-320.jpg)
![Installing Scripts [ buildout] parts = helloworld [ helloworld] recipe = zc.recipe.egg eggs = fabulous Pillow # if package has its own entry-point, you don't need the stuff below initialization = from fabulous import text; print text.Text("Hello World", shadow=True) go = lambda: True scripts = hello entry-points = hello=__main__:go](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-24-320.jpg)
![Inside bin/script (not rocket science) #!/Users/dylanjay/Projects/virtual/buildout/bin/python import sys sys.path[0:0] = [ '/../download-cache/eggs/elementtree-1.2.7_20070827_preview-py2.4.egg', '/../download-cache/eggs/archetypes.kss-1.4.3-py2.4.egg', … ] import plone.recipe.zope2instance.ctl if __name__ == '__main__': plone.recipe.zope2instance.ctl.main( ["-C", '/Users/dylanjay/Projects/gcio/parts/instance/etc/zope.conf'] + sys.argv[1:])](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-25-320.jpg)
![Inside Recipes class HelloRecipe(object): def __init__(self, buildout, name, options): self.input = options.get('input','world') self.output = options['output'] = 'hello %s' % self.input self.location = options['location'] = self.buildout['buildout']['bin-directory']+'hello' def install(self): # Return files that were created by the recipe. The buildout # will remove all returned files upon reinstall. f = open(self.location).write(self.output) ; f.close() return [self.location] def update(self): pass](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-26-320.jpg)
![mr.developer [buildout] parts = helloworld extensions = mr.developer auto-checkout= helloworld [sources] helloworld = git git://github.com/topher200/genetic-hello-world-python.git [ helloworld] recipe=zc.recipe.egg eggs = genetic-hello-world-python # WARNING: this example doesn't work](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-27-320.jpg)
![zc.recipe.cmmi [buildout] parts = instance varnish [instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone Http-address = 127.0.0.1:8080 [varnish-build] recipe = zc.recipe.cmmi url = http://downloads.sourceforge.net/varnish/varnish-2.1.3.tar.gz [varnish] recipe = plone.recipe.varnish daemon = ${varnish-build:location}/sbin/varnishd bind = 127.0.0.1:8080 backends = ${instance:http-address}](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-28-320.jpg)
![Versions [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone >= 4.1](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-29-320.jpg)
![Versions Pinning [ buildout] parts = instance [ instance] recipe = plone.recipe.zope2instance eggs = Plone [ versions] Plone= 4.1](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-30-320.jpg)
![Versions KGS [ buildout] extends = http://dist.plone.org/release/4.1/versions.cfg parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-31-320.jpg)
![dumppickedversions [buildout] extensions = buildout.dumppickedversions $ bin/buildout Getting distribution for 'buildout.dumppickedversions'. ... *************** PICKED VERSIONS **************** [versions] myegg = 1.1 setuptools = 2.1 zc.buildout = 1.5.3 zc.recipe.egg = 1.5.2 <BLANKLINE> *************** /PICKED VERSIONS ***************](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-34-320.jpg)
![Saving versions [ buildout] extensions = buildout.dumppickedversions dump-picked-versions-file = picked.cfg](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-35-320.jpg)
![Macros [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone http-address = 127.0.0.1:8080 [instance2] <=instance1 http-address = 127.0.0.1:8081 [ instance2] <= instance1 http-address = 127.0.0.1:8081 [ varnish-build] recipe = zc.recipe.cmmi url = http://downloads.sourceforge.net/varnish/varnish-2.1.3.tar.gz [ varnish] recipe = plone.recipe.varnish daemon = ${varnish-build:location}/sbin/varnishd bind = 127.0.0.1:80 backends = ${ instance:http-address} ${ instance1:http-address} ${ instance2:http-address}](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-36-320.jpg)
![mr.scripty [ ports_base] Instance1 = 80 Instance2 = 81 Instance3 = 83 [ ports] recipe=mr.scripty OFFSET = 1000 init= ... for key,value in self.buildout['ports_base'].items(): ... self.options[key] = str(int(value)+int(self.OFFSET))](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-37-320.jpg)
![mr.scripty [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone http-address = ${ports:instance1} [ instance2] <= instance1 http-address = ${ports:instance2} [ instance2] <= instance1 http-address = ${ports:instance3}](https://image.slidesharecdn.com/buildout-pyconau-110821003002-phpapp01/85/Buildout-How-to-maintain-big-app-stacks-without-losing-your-mind-38-320.jpg)
Buildout: How to maintain big app stacks without losing your mind
- 1. Buildout How to maintain big app stacks without losing your mind Dylan Jay Sypy.org
- 2. History Made by Jim Fulton. 2006
- 3. “It should be possible to check-in a buildout specification and reproduce the same software”
- 4. What problem does it solve?
- 5. Isolating Python Libraries Like virtualenv
- 6. Managing depencies/versions Bit like pip requires.txt
- 7. Compiling/Building stuff Bit like Make or Ant
- 8. Installing/Configuring/Deploying Bit like Chef or Puppet
- 9. Checkout Development Code Bit like svn externals
- 10. Simplifying Config Templates Bit like Paster Script
- 11. But wait there's more! We'll do all that in one .cfg file
- 12. So what is buildout? Gaffer tape is like the force. It has a light side, a dark side, and it holds the universe together.
- 13. App Stacks Nginix, Varinish, HAProxy, Zope, Plone, MySQL
- 14. Perl, Munin
- 15. Pyramid, Gevent, pyramid_socketio, Redis
- 16. Django, customapp1, customapp2, customapp3
- 17. Etc etc
- 18. Anatomy of a Buildout buildout.cfg: [buildout] parts = MyPart [MyPart] recipe = recipepackage arg1 = value1
- 19. Initialising $ easy_install zc.buildout … Finished processing dependencies for zc.buildout $ buildout init Creating directory '/Users/dylanjay/Projects/sandpit/buildout/bin'. Creating directory '/Users/dylanjay/Projects/sandpit/buildout/parts'. Creating directory '/Users/dylanjay/Projects/sandpit/buildout/develop-eggs'. Generated script '/Users/dylanjay/Projects/sandpit/buildout/bin/buildout'. $ bin/buildout
- 20. “What goes on in buildout, stays in buildout”
- 21. Hello World [buildout] parts = helloworld chmod [helloworld] recipe = collective.recipe.template output = ${buildout:bin-directory}/hello input = inline: echo 'hello world' [chmod] recipe = plone.recipe.command command = chmod +x ${helloworld:output}
- 22. Dependencies and Substitutions [ buildout] parts = chmod [ helloworld] recipe = collective.recipe.template output = ${buildout:bin-directory}/hello input = inline: echo 'hello world' [ chmod] recipe = plone.recipe.command command = chmod +x ${helloworld:output}
- 23. Installing Packages (easy_install) [buildout] parts = helloworld [helloworld] recipe = zc.recipe.egg eggs = Fabulous Pillow Interpreter = python $ bin/python -m fabulous.text hello world
- 24. Installing Scripts [ buildout] parts = helloworld [ helloworld] recipe = zc.recipe.egg eggs = fabulous Pillow # if package has its own entry-point, you don't need the stuff below initialization = from fabulous import text; print text.Text("Hello World", shadow=True) go = lambda: True scripts = hello entry-points = hello=__main__:go
- 25. Inside bin/script (not rocket science) #!/Users/dylanjay/Projects/virtual/buildout/bin/python import sys sys.path[0:0] = [ '/../download-cache/eggs/elementtree-1.2.7_20070827_preview-py2.4.egg', '/../download-cache/eggs/archetypes.kss-1.4.3-py2.4.egg', … ] import plone.recipe.zope2instance.ctl if __name__ == '__main__': plone.recipe.zope2instance.ctl.main( ["-C", '/Users/dylanjay/Projects/gcio/parts/instance/etc/zope.conf'] + sys.argv[1:])
- 26. Inside Recipes class HelloRecipe(object): def __init__(self, buildout, name, options): self.input = options.get('input','world') self.output = options['output'] = 'hello %s' % self.input self.location = options['location'] = self.buildout['buildout']['bin-directory']+'hello' def install(self): # Return files that were created by the recipe. The buildout # will remove all returned files upon reinstall. f = open(self.location).write(self.output) ; f.close() return [self.location] def update(self): pass
- 27. mr.developer [buildout] parts = helloworld extensions = mr.developer auto-checkout= helloworld [sources] helloworld = git git://github.com/topher200/genetic-hello-world-python.git [ helloworld] recipe=zc.recipe.egg eggs = genetic-hello-world-python # WARNING: this example doesn't work
- 28. zc.recipe.cmmi [buildout] parts = instance varnish [instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone Http-address = 127.0.0.1:8080 [varnish-build] recipe = zc.recipe.cmmi url = http://downloads.sourceforge.net/varnish/varnish-2.1.3.tar.gz [varnish] recipe = plone.recipe.varnish daemon = ${varnish-build:location}/sbin/varnishd bind = 127.0.0.1:8080 backends = ${instance:http-address}
- 29. Versions [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone >= 4.1
- 30. Versions Pinning [ buildout] parts = instance [ instance] recipe = plone.recipe.zope2instance eggs = Plone [ versions] Plone= 4.1
- 31. Versions KGS [ buildout] extends = http://dist.plone.org/release/4.1/versions.cfg parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone
- 32. Version Conflicts Installing. Getting section zeoserver. Initializing part zeoserver. Error: There is a version conflict. We already have: zope.component 3.8.0 but five.localsitemanager 1.1 requires 'zope.component<3.6dev'. Conflicts are caused by buildout having no way to know depenency specs until its too late.
- 33. Version Conflict Prevention $ bin/buildout -N (prevent auto upgrading packages) $ bin/buidout -v (discover where conflict arose) Or Pinning versions
- 34. dumppickedversions [buildout] extensions = buildout.dumppickedversions $ bin/buildout Getting distribution for 'buildout.dumppickedversions'. ... *************** PICKED VERSIONS **************** [versions] myegg = 1.1 setuptools = 2.1 zc.buildout = 1.5.3 zc.recipe.egg = 1.5.2 <BLANKLINE> *************** /PICKED VERSIONS ***************
- 35. Saving versions [ buildout] extensions = buildout.dumppickedversions dump-picked-versions-file = picked.cfg
- 36. Macros [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone http-address = 127.0.0.1:8080 [instance2] <=instance1 http-address = 127.0.0.1:8081 [ instance2] <= instance1 http-address = 127.0.0.1:8081 [ varnish-build] recipe = zc.recipe.cmmi url = http://downloads.sourceforge.net/varnish/varnish-2.1.3.tar.gz [ varnish] recipe = plone.recipe.varnish daemon = ${varnish-build:location}/sbin/varnishd bind = 127.0.0.1:80 backends = ${ instance:http-address} ${ instance1:http-address} ${ instance2:http-address}
- 37. mr.scripty [ ports_base] Instance1 = 80 Instance2 = 81 Instance3 = 83 [ ports] recipe=mr.scripty OFFSET = 1000 init= ... for key,value in self.buildout['ports_base'].items(): ... self.options[key] = str(int(value)+int(self.OFFSET))
- 38. mr.scripty [ buildout] parts = instance varnish [ instance] recipe = plone.recipe.zope2instance eggs = Plone HelloWorldPlone http-address = ${ports:instance1} [ instance2] <= instance1 http-address = ${ports:instance2} [ instance2] <= instance1 http-address = ${ports:instance3}
- 39. Extending #staging.cfg [buildout] extends = buildout.cfg [ports] OFFSET=8000
- 40. Annotation mode $ bin/buildout annotate Annotated sections ================== [bfg] dependent-scripts= true /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg eggs= repoze.bfg mobme /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg index= http://dist.repoze.org/bfg/current/simple /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg recipe= zc.recipe.egg /Users/dylanjay/Projects/sandpit/mobme/buildout.cfg [buildout] accept-buildout-test-releases= false DEFAULT_VALUE
- 41. Other Recipes
- 42. collective.hostout Deploys a buildout to a new location (host)
- 43. Uses Fabric under the hood
- 46. Dylan Jay
- 47. Twitter: djay75