Lessons I Learned While Scaling to 5000 Puppet Agents
20 likes15,150 views
The document outlines strategies for scaling infrastructure management to 5000 agents using Puppet, emphasizing the importance of consistency and automation in setup processes. It details best practices in environment configuration, version control, and module management, alongside the transition to effective tools like Hiera for data management. The evolution from a lengthy, error-prone deployment process to a streamlined approach is highlighted, showcasing significant improvements in efficiency and reliability.
Lessons I Learned While Scaling to 5000 Puppet Agents
- 1. Learning Lessons Scaling to 5000 Agents Russ Johnson rjohnson@stubhub.com @professoruss Confidential Slide 1
- 2. #whoami Started out in the mid 90’s Recovering Windows Admin Storage Guy Datacenter Monkey Once upon a time network guy At StubHub since December 2006 Working on puppet adoption in a crazy infrastructure Puppet certified April 9, 2013 2
- 3. 4 Puppet Masters DEV/QA PROD DR CORP April 9, 2013 3
- 4. The road to sanity April 9, 2013 4
- 5. Set up your master properly Apache/Passenger Tune Passenger PassengerMaxPoolSize 32 PassengerMinInstances 4 PassengerMaxRequests 10000 PassengerStatThrottleRate 30 16 cores, 32GB load average: 4.03, 3.71, 3.45 4000+ agents April 9, 2013 5
- 6. Thundering herds April 9, 2013 6
- 7. Build your hosts the same way! Old way: systemimager, vmware clones, manual installs Results: INCONSISTENCY! April 9, 2013 7
- 8. Build your hosts the same way! New way: Cobbler < 5m bare metal to on the network Results: Same results every time! No drift between base April 9, 2013 8
- 9. Set up your working environment properly Geppetto – eclipse based IDE http://cloudsmith.github.com/geppetto/index.html VIM • Pathogen – For autoloading vim plugins • Snipmate – Snippets • Tabular – Text filtering and alignment • Syntastic – Syntax checking • mv-vim-puppet – Make vim puppet friendly • puppet-lint – Syntax checker (gem) April 9, 2013 9
- 11. Set up your working environment properly + April 9, 2013 11
- 12. Version Control is not enough Ever do a 4 way diff across 60 modules to find most of them different? April 9, 2013 12
- 13. What to do? How do I stop manual edits? How do I deal with 80+ Dev/QA Environments? Versioning? Librarian? Pulp? Branching? Internal Forge? Dynamic Environments? Puppet Module Tool? What does PuppetLabs do? April 9, 2013 13
- 14. PuppetLabs seems to know what to do Let’s investigate puppet module tool http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html April 9, 2013 14
- 15. Generate a module http://docs.puppetlabs.com/puppet/2.7/reference/modules_publishing.html April 9, 2013 15
- 17. Document the manifest http://rdoc.sourceforge.net/ April 9, 2013 17
- 18. Write Documentation? April 9, 2013 18
- 19. Free Docs! puppet doc -a -o /var/www/html/puppetdocs --mode rdoc April 9, 2013 19
- 20. What’s actually installed? April 9, 2013 20
- 21. Catching live edits and preventing them Splunk -> puppet module changes -> alerting The NOC will hunt you down! April 9, 2013 21
- 22. Build and install the module tar –xzf /tmp/work/stubhub-puppetserver/pkg/stubhub- puppetserver-0.0.1.tar.gz –C /etc/puppet/environments/ staging/modules/puppetserver April 9, 2013 22
- 23. Releasing like that? April 9, 2013 23
- 24. Internal Forge mod_rewrite: Simulate the api – redirect to json metadata files $htmlroot/api/v1/releases.json?module=user/module ruby script: Generate metadata files for each module release and all modules. Similar to createrepo (yum) April 9, 2013 24
- 25. Internal Forge - Search April 9, 2013 25
- 26. Internal Forge - install April 9, 2013 26
- 27. Internal Forge - upgrade April 9, 2013 27
- 28. Case statements? How bout Hiera? April 9, 2013 28
- 29. Avoid case statement insanity case $::system_role { ‘browse’, ‘search’: { …do some stuff… } ‘db’: { …other stuff… } ‘otherrole’: { …please make it stop!!!! } } April 9, 2013 29
- 30. hieradata $hieradata/browse.yaml: --- module::parameter: ‘foo’ $hieradata/search.yaml: --- module::parameter: ‘bar’ $hieradata/defaults.yaml: --- module::parameter: ‘I want this everywhere unless there are overrides’ April 9, 2013 30
- 31. Case -> variables -> hiera § 9000 lines of case statements § 1000 lines with case/variables § ~20 lines with defined type Code compression FTW! April 9, 2013 31
- 32. Dynamic Environments § puppet.conf: modulepath = /etc/puppet/environments/$environment/modules manifest = /etc/puppet/environments/$environment/manifests/site.pp manifestdir = /etc/puppet/environments/$environment/manifests § hiera.yaml: :datadir: '/etc/puppet/environments/%{environment}/hieradata' April 9, 2013 32
- 33. Release process § Syntax check/validate § Test on VMs § Build module package § Release to internal forge § puppet module install to staging environment § Test again! § puppet module install to production environment April 9, 2013 33
- 34. The road to yesop § Staging § Process § Repeatability § Consistency § Document everything § Breaking things where it’s cheap § Test everything! April 9, 2013 34
- 35. Then VS now § Environment build time: – Then: 3+ weeks • It was wrong • It didn’t work • Nobody knew what to expect – Now: < 1 day • It’s the same every time • We know exactly what’s installed • Internal consumers get what they expect • Less outages from human error April 9, 2013 35
- 36. Questions? April 9, 2013 36