SaltConf2015: SaltStack at Scale Automating Your Automation
Download as PPTX, PDF23 likes4,591 views
This document discusses Saltstack configuration management at scale for a large cloud provider. It outlines how the company standardized on Salt formulas to promote code reuse, independent testing of states, and consistent configuration across projects. Key aspects covered include using Map.jinja files, version control, continuous integration pipelines for testing and building packages, and workflows for merging formula changes from development to production.
SaltConf2015: SaltStack at Scale Automating Your Automation
- 1. 1 Saltstack at Scale: Automating the Automation Steven Gonzales | EMC | Principal Software Engineer #SaltConf15
- 2. 2 • We build and run BIG clouds • Object Storage • Remote teams • Smart people • SRE/DEV • We are hiring! EMC2 CLOUD SERVICES
- 3. 3 • Exabyte+ Object Storage • Datacenters around the world • 10,000+ nodes • 14+ Salt Masters SALT AT SCALE WHAT DO WE MEAN BY SCALE?
- 4. 4 STARTING OUT FIRST FLIGHT WITH SALT
- 5. 5 • Salt Version 0.17.x • Tool for remote command execution • Developed use of states • Monolithic Salt repo • Used in only one project STARTING OUT FIRST FLIGHT WITH SALT
- 6. 6 • Poor code reuse across projects • Divergent copies across projects • Difficult to deploy pillar/states • Hard to test • Lack of consistent patterns STARTING OUT FIRST FLIGHT WITH SALT
- 7. 7 • Version control • Design patterns • Code reuse • Testing • Deployment • Build system! INFRASTRUCTURE AS CODE TREAT IT LIKE A SOFTWARE PRODUCT
- 8. 8 ENTER THE FORMULA PATTERNS & PRACTICES
- 9. 9 • Promotes code reuse • End divergence • Project repos should only contain pillar data • Test states independently • Still needs standards SALT FORMULAS WHY FORMULAS?
- 10. 10 SALT FORMULA LAYOUT • _modules and _states • Debian packaging • The states folder (i.e.: docker) • Map.jinja • Test Directory • Makefile • Cookiecutter!
- 11. 11 • Map.jinja • Every setting is defined and has a default value • OS map, defaults, and pillar merged into settings dict • Pillar acts as an override • One place to see configuration options SALT FORMULAS SETTINGS & CONFIGURATION
- 12. 12 SALT FORMULA SETTINGS & CONFIGURATION • States and templates access all data though the settings dict
- 13. 13 • Provision VMs • Built-in Salt provisioner • Fast for local testing • Disposable environment • Private Networks • Vagrant up! SALT FORMULAS TESTING WITH VAGRANT
- 14. 14 • Write functional tests in python • Nosetests • Tox • Execute against the Vagrant environment SALT FORMULAS PYTHON TEST TOOLS
- 15. 15 • Install to /srv/formula • Support installing _modules and _states • Support versioning & dependencies • Easy rollback • Not every network has access to Git SALT FORMULAS HOW TO INSTALL?
- 16. 16 • Don’t reinvent the wheel! • `apt-get install salt-formula` SALT FORMULAS INSTALL WITH SYSTEM PACKAGES
- 17. 17 AUTOMATING THE AUTOMATION BUILD AND DEPLOYMENT
- 18. 18 • Salt-formula • Bootstraps salt masters/minions • Installs formulas/pillar from system pkgs • Salt masters highstate themselves SALT INCEPTION CORE TO AUTOMATION
- 19. 19 • Formulas have devel and master branches • Developers PR against devel • Merged to master after successful build/testing WORKFLOWS GIT BRANCHING
- 20. 20 • Build server Go.cd • Dashboard of build status • Pipeline Templates • Pipelines for devel and master • Notifications WORKFLOWS AUTOMATING THE BUILD
- 21. 21 • Vagrant up & execute functional tests • Build a package • Push package into repo • Devel to testing repo, master to prod repo WORKFLOWS BUILD STEPS
- 23. 23 • Developers PR to merge to devel • Build, package, & push to test repo • Auto-install to test environment • Merge devel to master • Build, package, & push to prod repo • Ready for Production WORKFLOW THE FULL PIPELINE
- 24. 24 • Still plenty to convert to formulas • Tests for pillar data • Improve tooling for formula dependencies • Expanded use of external pillar NEXT STEPS ALWAYS ROOM TO IMPROVE
- 25. 25 Thoughtworks Go.cd: Continuous build and delivery system http://www.go.cd/ Vagrant: Lightweight VM provisioning software https://www.vagrantup.com/ Tox Automation Project: Python test automation for running tests inside a virtual environment https://tox.readthedocs.org USEFUL LINKS Nosetests: Python Test Runner https://nose.readthedocs.org Cookiecutter: Project for rendering new projects from a jinja template https://cookiecutter.readthedocs.org