Automating Your Salt Tests
- 1. Automating Your Salt Tests Ryan Currah github@ryancurrah twitter@rcurrah
- 2. MY EXPERIENCE • Was lead Salt engineer for large Canadian bank • Develop formulas • Salt master architecture • Salt git development strategy • Salt testing and testing automation
- 3. TALK GOAL Help newer Salt users start off on the right foot
- 4. DON’T FORGET TO AUTOMATE TESTING If you don’t automate testing it will become a bottleneck If you do automate testing you will catch issues early and feel confident when you release
- 5. TESTING AS CODE Requirements • State testing defined as code • Jobs defined as code • Pipeline capabilities • Fast testing feedback • Tools were on premise
- 6. TESTING TOOLS SELECTED • Code analysis: shellcheck (bash), flake8 (python) • State testing: ServerSpec • Container: Docker • Test orchestration: Jenkins Pipeline (Formerly Workflow) • Versioning: Git tag + bumpversion
- 7. CODE ANALYSIS ON FORMULAS Not uncommon to see scripts in formulas Code analysis enables • Good coding styles • Consistency • Express how we want code to be written • No jinja templated scripts (Use cmd.script w/ input arguments instead)
- 8. TEST STATE Two different views of state • For example in the issue github.com/saltstack/salt/issues/24647 • file.copy state with force=True did not work • Reported no changes when the file should have changed • ServerSpec tests caught this issue Audit formula changes • Formula changes will be captured by tests
- 9. ServerSpec What can you test? • Process is running • Service is running • Service is enabled on boot • Port is open • Firewall rule exists • Package is installed • User and group exists • File exists • File user and group ownership • File hashes • The list goes on see… http://serverspec.org/resource_types.html Reuseable spec files known a shared files • http://serverspec.org/advanced_tips.html • https://github.com/rubyisbeautiful/serverspec_examples
- 10. JOBS AS CODE Jenkins Pipeline enables jobs as code and a deilvery pipeline (without the spaghetti jobs) • Jenkinsfile (Groovy script) placed in the Salt formula repo • Pipeline stages (Build, QA, Production) • Pipeline visualization and metrics • Average build time, current build time, stage views, failed steps
- 11. JENKINS PLUGINS USED • Pipeline • Pipeline: Multibranch • Pipeline Remote Loader Plugin • Git Plugin • SSH Agent Plugin • AnsiColor Plugin
- 12. JENKINS SERVER SETUP • Docker installed • Docker allow Jenkins user to run it • Testing container with systemd, salt-minion, ruby, shellcheck and flake8 installed • Bumpversion installed • Git SSH credential setup for pushing tags
- 13. RUNNING SERVICES IN CONTAINERS • Install systemd in the testing container • Remove systemd links in /etc/systemd • Mount cgroups to container (ro) • Privileged container or –v /tmp:/run • Run command /usr/sbin/init Source: https://rhatdan.wordpress.com/2014/04/30/running-systemd-within-a-docker-container/
- 14. FORMULA REPOSITORIES • Create a repository for each formula • Repository has a tests folder • Release updates independent of other states • Use git backend for formulas
- 16. DEVELOPING LOCALLY • Provide salt formula developers a local development environment • Create a vagrant box or docker image (RHEL developer licenses are free now) • Provide an example formula with tests • Document how to get started • Jenkins feature testing
- 17. GOOD PULL REQUESTS Salt best practices • Less Jinja the better • No Jinja whitespace control characters in sls files • defaults.yaml • Pillars are for overiding defaults • Predictable and sane state id’s https://docs.saltstack.com/en/latest/topics/best_practices.html https://docs.saltstack.com/en/latest/topics/development/conventions/ Require tests pass Review code together • Potential bugs • Readability • Test coverage (ServerSpec tests) • Compliment / reinforce good practices http://kevinlondon.com/2015/05/05/code-review-best-practices.html
- 18. RELASING NEW SALT FORUMLAS Use semantic versioning • Example v0.2.0 • semver.org Tag releases • Tag new version v0.3.0 • Re-tag “latest” Bumpversion will automatically bump the forumula version Rollback “latest” to a previous version if things break Salt-masters use the gitfs backend docs.saltstack.com/en/latest/topics/tutorials/gitfs.html#per-remote-configuration-parameters
- 19. SALTSTACK GIT BACKENDS Pygit and Gitpython Pygit2 • Supports authentication • Cloning process tends to hang Gitpython • Easy to install • No authentication supported
- 20. DEMO Jenkins Pipeline • Pipeline script • Multibanch pipeline jobs • Testing feature branches • Testing master branches • ServerSpec output • Automatic promotion
- 21. STATE EVENTS What is a state event? Result of a state execution in a structured format that a can be parsed Why state events? • Analytics on state execution results • Alert operations on failed states • Easily compile issue reports for engineering More details: http://www.currah.ca/tech/2015/12/09/salt-state-events-splunk.html New splunk returner: https://github.com/saltstack/salt/blob/develop/salt/returners/splunk.py
- 22. EVENT RETURNER • Event returners only run on the master • Salt has a list of event returners or build your own • Create your own returner, place in _returners directory • Returner state events basically state.sls and state.highstate • Add minion id and jid to state return data for correlating state events to a specific host or action More details: http://www.currah.ca/tech/2015/12/09/salt-state-events-splunk.html Splunk returner: https://github.com/saltstack/salt/blob/develop/salt/returners/splunk.py List of Returners: https://docs.saltstack.com/en/latest/ref/returners/
- 23. EXAMPLE STATE EVENT Automatic Field Extraction
- 24. DEMO SALTSTACK STATE EVENTS • Generating events • Searching events • Event dashboard
- 25. CODE • https://github.com/ryancurrah/vagrant-salt-talk • https://github.com/ryancurrah/salt-minion-dockerfile • https://github.com/ryancurrah/salt-pipeline • https://github.com/ryancurrah/salt-apache-formula • https://github.com/ryancurrah/salt-lint-example • https://github.com/ryancurrah/salt-splunk-http-returner
- 26. Questions?
- 27. Please Provide Feedback Use the SaltConf16 event app to provide feedback for this presentation.