It Sounded Good on Paper - Lessons Learned with Puppet
2 likes290 views
The document outlines key tips for deploying and managing Puppet configurations effectively. It emphasizes the importance of minimizing application-specific configurations, maintaining a structured code base, and ensuring secure handling of secrets. Additionally, it provides practical advice on testing, server management, and using Hiera for better configuration management.
![Puppet File Server
[ssl_keys]
path /secure/ssl_keys
allow 10.160.10.100
deny *.testdomain.com
file { ‘/etc/httpd/keys/ssl.key’:
source => ‘puppet:///ssl_keys/ssl.key'
}](https://image.slidesharecdn.com/itsoundedgood-151024114750-lva1-app6891/85/It-Sounded-Good-on-Paper-Lessons-Learned-with-Puppet-21-320.jpg)
It Sounded Good on Paper - Lessons Learned with Puppet
- 1. It Sounded Good on Paper Jeffery Smith Manager, Site Reliability Engineering |GrubHub @DarkAndNerdy
- 2. Bio - Jeff Smith • Manager, Site Reliability Engineering at GrubHub • Yes, we are also hiring • Yes, there is free food. Yes it’s totally awesome to work here Email: jsmith@grubhub.com Twitter: @DarkAndNerdy Blog: www.allthingsdork.com
- 4. Reduce the Pain Through Sharing 10 11 12 Tips on What We Did Wrong
- 6. Tip #1 - Minimize App Specific Config • Drives up the number of roles to maintain • Can slow down developers if they don’t also write Puppet Code • Pushes deployment responsibilities into Configuration Management
- 7. Draw the line • Deployment process should handle deployment type tasks • Your roles can be more generic, like “Tomcat Server” or “Web Server” vs “Payroll_App_Server”
- 8. Tip #2 - Code Base Structure Matters • Avoid doing too much in a module. It’s OK if your module makes assumptions, just document them include apache::install include apache::config apache::vhost { ‘website’: ensure => installed } apache::vhost { ‘website’: ensure => installed, } Assumes nothing Assumes Apache is Installed
- 9. Tip #2 - Code Base Structure Matters • Use the modules/profiles/roles pattern - Gary Larizza • A single code base regardless of environments
- 10. Tip #3 - Everyone is watching! • Passwords in codebase prevents you from sharing • Will an auditor throw-up in their mouth? • That sense of shame you feel is probably a real issue
- 11. Tip #4 - A Repo Per Module • In a single Repository, versioning becomes complicated • Changes become inter-mingled • Automated testing requires some inspection of changes to save on test time.
- 12. Speaking of Testing • Not all testing is valuable. Basic Puppet operations aren’t worth testing • Keep tests fast or you’ll get burned at 2am during an emergency patch
- 13. Tip #5 - Never Hand Touch a Server Always build the server from scratch, using Puppet only. If you make a single change, destroy and rebuild.
- 14. Tip #6 - There are only 3 numbers When writing your manifests, you should think of your resources in these terms • 0 - This resource can never exist • 1 - There can only ever be one of these resources • Many - There will be 1 or more of these resources If you think there’s only 2 of a thing, someone is secretly planning the 3rd
- 15. Tip #7 - Exported Resources are Evil Not Entirely • Attempt to solve the problem with PuppetDB Query • Move the exported resource to a runtime config store • If necessary, develop clear patterns for tagging resources. It aids in collection
- 16. Tip #8 - Branching Strategies - Ugh
- 17. Gotchas with this approach • Need to blow away develop branch and clone with master semi-regularly • Bad change can break all of pre-prod (OOPS) • Merge conflicts in Develop that don’t exist in PROD • Do you do PRs for Develop? Slows things down
- 18. Alternatives • Separate Branch for each server environment? • Need to keep in sync with master • Makes knowledge of changes local to the SRE that made them • Always work off master • Added risk • Tons of trash commits
- 19. Tip #9 - Be realistic about your secrets • Use Hiera eyaml - https://github.com/TomPoulton/ hiera-eyaml • If you’re using JSON, get over it, convert to YAML and use Hiera eyaml. https://www.npmjs.com/package/ yamljs • Key distribution is still tricky Your Rube Goldberg solution is useless if it ends in a plain text file being loaded
- 20. Key Distribution • It sucks everywhere. You’re not alone • Just because it sucks, doesn’t mean you can store keys in Source Control • Puppet File Server has some options • Allow by IP address or hostname • Distribution to Puppet Master varies :-(
- 21. Puppet File Server [ssl_keys] path /secure/ssl_keys allow 10.160.10.100 deny *.testdomain.com file { ‘/etc/httpd/keys/ssl.key’: source => ‘puppet:///ssl_keys/ssl.key' }
- 22. Tip #10 - Build Monitoring in Puppet Find a Puppet module for Nagios or build one, but put your monitoring in Puppet • Weird things happen in ALL environments • One Sys Admins test environment is a developer’s production environment • Provisioning a new node and implementing monitoring shouldn’t be two separate steps
- 23. Tip #11 - Stop Using ‘exec’ • Execs take the elegance and readability out of Puppet Code, especially when executing shell scripts • You probably don’t do a good job of only triggering the exec when needed • Congrats! Your script is idempotent……..but…
- 24. Tip #12 - Use Hiera • CASE statements are often a code smell • Hiera allows you to use the same code for all environments • Don’t be afraid of Hiera structures, but whatever you do for the love of God don’t number them
- 25. Example Hiera Structure • Nodes • node/certname • Environment • <environment names> • DataCenter • <datacenters> • Performance Profile • local, integration, staging, prod • Globals
- 26. THANKS! Email: jsmith@grubhub.com Twitter: @DarkAndNerdy Blog: www.allthingsdork.com