Puppet Camp Chicago 2014: Puppet at backstop another year of lessons
3 likes984 views
Puppet is used to manage around 350 machines for a financial services company. The company moved their Puppet code to GitHub in 2013, which led to a significant increase in contributors from outside the systems administration team. They use a Git workflow with master and office branches and pull requests. Jenkins is used to test and deploy Puppet changes. MCollective and CloudStack are also leveraged in their environment. While Puppet works well overall, the speaker notes some challenges around ActiveMQ stability and the need to consider alternatives to storing all configuration data in Puppet.
![‣ R.I.’s scripts [1]
‣ Erwan Ben Souiden, aka labynocle [2]
‣ A script I butchered to help
34
[1]: https://github.com/ripienaar/monitoring-scripts/tree/master/activemq
[2]: https://github.com/labynocle/nagios_plugins_by_labynocle/tree/master/check_activeMQ
[3]: https://github.com/BillWeiss/MiscScripts/blob/master/check_activemq.pl
I know, I know, bleh, Nagios](https://image.slidesharecdn.com/puppetatbackstop-anotheryearoflessons-2-140819133008-phpapp01/85/Puppet-Camp-Chicago-2014-Puppet-at-backstop-another-year-of-lessons-34-320.jpg)
Puppet Camp Chicago 2014: Puppet at backstop another year of lessons
- 1. Puppet at Backstop Another year of lessons 1
- 2. $ whoami ‣ Bill Weiss <bill@backstopsolutions.com> ‣ @BillWeiss ‣ Probably responsible for 90% of the badness in this talk, 25% of the good ‣ Still not a designer 2
- 3. What’s this talk about? aka, who cares? 3
- 4. 4 Image via MemeCrunch. Original owned by the History Channel.
- 5. No, wait, that’s not it (Come talk to me afterward about that) 5
- 6. ‣ I’m here to talk about things that work (and don’t) for us in our environment ‣ Hopefully most of it is useful to you ‣ Some might just be there so you can point and laugh 6
- 7. –You “What the heck is Backstop?” 7
- 8. Super helpful, I know 8
- 9. SaaS for institutional investment firms ‣ That’s it. ‣ You’re probably not our target audience ‣ If you are, awesome, let’s talk later 9
- 10. Some numbers ‣ ~195 Puppeted machines in production (US) ‣ ~350 total ‣ 2 datacenters, 2 AWS AZs, 1 office closet full of machines ‣ 50+ Engineering folks (IT, dev, QA, etc) Numbers of machines from MCollective! 10
- 11. Logo owned by the CentOS project Logo also owned by Oracle, of course Logo owned by Oracle Logo by David Heinemeier Hansson 11
- 12. ‣ Big application is J2EE on JBoss ‣ Lots of smaller bits (“services”) on Tomcat ‣ Second app is Ruby on Rails 12
- 13. How ‘bout that Puppet? ‣ Moved to GitHub 2013/11/18 ‣ ~4100 commits since then (~15 commits/day!) ‣ 2700 from people outside sysadmin team ‣ ~1350 pull requests from 21 contributors 13
- 14. The move to GitHub was awesome ‣ 4 years of Subversion before that had 29 authors and 6700 commits ‣ Other things at play here beside just s/svn/git/ , but it helped a lot ‣ How? 14
- 15. ! git workflow ‣ Org-owned repo, two branches ‣ master ➡ production ‣ office ➡ testing (pre-prod) ‣ Everyone forks from master and sends pull requests to office ‣ Pull changes from office to master as needed 15
- 16. ‣ Everyone has their own fork with work branches ‣ Branch names probably are Jira ticket numbers ‣ Everyone bases from master, not office, so we can pull changes independently 16
- 17. This is why we have so many contributors 17
- 18. ‣ If you’re a software shop, you already have a workflow. Do that ‣ Otherwise, this is working well for us 18
- 19. " git hooks ‣ We had a pretty good set of pre-commit hooks in svn to test changes ‣ Github doesn’t do that # ‣ Git pre-commit hooks exist, but it’s up to each person to set them up ‣ No awesome answer yet ‣ The answer might be… 19
- 20. ! Jenkins! ‣ We used to have this “deploy” job on the Puppet masters as a cron job: • (cd /etc/puppet && svn up) ‣ Nothing could go wrong there, right? ‣ For a while we just did it manually instead 20
- 21. ‣ Jenkins polls for changes to either branch ‣ Runs a deploy job that grabs repo, runs librarian-puppet, a “build” step, and then pushes it out to the masters ‣ Determines masters via mco call ‣ All the good (and bad) of capistrano for the push 21
- 22. 22 alias puppet-test='sudo puppet agent --test --noop -- environment' That deploys an environment titled UserBranch, camel-cased
- 23. ‣ Jenkins also runs a test suite in parallel ‣ Some day that’ll be a precursor to the deploy job. Some day… 23
- 24. " Sometimes you just want to do something to prod ‣ All that well-defined procedure means time for quick fixes goes up ‣ Make a branch, make some changes, commit, push, PR to office, merge, PR to prod, merge, run deploy… 24
- 25. ‣ Possible answers here: • Pull some things out into their own repos • Accept that it’s slower • Tool it up, make the machines do the slow parts 25
- 26. ! MCollective is still awesome 26 Original image: http://xkcd.com/353/ Cheapo editing courtesy OS X Preview “Annotate” Same graphic as last year’s talk. I love you, mco!
- 27. It’s like Frank’s Red Hot ‣ Our deploys (not IT, app teams) now use it to figure out which machines to deploy to ‣ We wrote an agent for our VM system (Proxmox) Which, uh, evidently we haven’t released. Will do. ‣ Stop/start the world for Oracle maintenance 27
- 28. " ActiveMQ is hard ‣ MCollective sits on top of ActiveMQ ‣ Easy, right? Just install it (with Puppet, even) and go to town ‣ Second datacenter? Yeah, just get them talking to each other and… 28
- 29. A sad example 29
- 30. Suddenly, one day… ‣ Pretend x is time ‣ y is memory usage ‣ That’s our activeMQ • At 2AM • Every day 30
- 31. 31 “Sudden Clarity Clarence” via memecreator.eu Original image? Who knows. Might be Clarence.
- 32. ‣ Long story short: make really sure your broker names are unique across your network ‣ Really really sure 32
- 33. That won’t happen to us ‣ Maybe not ‣ Just know you’re adding another service you rely on ‣ Monitor it ‣ Measure it ‣ Have a game plan for what to do when it goes horribly wrong 33
- 34. ‣ R.I.’s scripts [1] ‣ Erwan Ben Souiden, aka labynocle [2] ‣ A script I butchered to help 34 [1]: https://github.com/ripienaar/monitoring-scripts/tree/master/activemq [2]: https://github.com/labynocle/nagios_plugins_by_labynocle/tree/master/check_activeMQ [3]: https://github.com/BillWeiss/MiscScripts/blob/master/check_activemq.pl I know, I know, bleh, Nagios
- 35. ! Cloudstack! ‣ It’s great ‣ Setup was easy ‣ Only using it in test now (so we only kinda care if the machines burn down) 35 That’s the CloudStack monkey, owned by the Apache Software Foundation
- 36. ‣ Remember that puppet-test alias? ‣ That + CloudStack means people can build machines to test Puppet changes to ‣ I was surprised recently by a service that just showed up ready to run in prod, Puppet and all 36
- 37. You don’t have to use CloudStack, but you need something like it ‣ That could be OpenStack, Vagrant, even some AWS credit ‣ You just need fast to build disposable machines 37
- 38. " Would be better in prod ‣ We’re not yet running CloudStack in production. It’ll happen someday. ‣ So, while I’m singing its praises, I can’t say we’re all in. 38
- 39. ! create_resources ‣ I have to show off one chunk of code ‣ Let me quote from TFM: 39 ‣ That’s vague, right? The example in TFM didn’t inspire me either, so I glossed over it for years. Converts a hash into a set of resources and adds them to the catalog.
- 40. I already said something about Frank’s Red Hot, right? Almost everywhere I use it looks like this 40 $thing = loadYaml(“${yamlDir}/thing/${environment}.yaml”) create_resources( some::define, $thing ) That same YAML gets used in a bunch of those. backstopsolutions: lastOctet: 226 clientFqdn: 'www.backstopsolutions.com' seconddn: 'backstopsolutions.com' includeFile: 'true'
- 41. ‣ One define uses the lastOctet and some knowledge about our systems to set up an interface on the load balancers ‣ Another uses the hostnames to set up a couple of vhosts for the Rails app ‣ Yet another adds it into several DNS zones 41
- 42. ‣ Another benefit: it’s YAML. You don’t have to write it by hand ‣ Grab some external data source, frob as needed, dump out some YAML. As long as you meet the spec, you’ll be fine 42
- 43. Huge gotcha here ‣ Everything you pass that YAML to needs to do something with all the parameters named in the file ‣ Adding a field thus means changing all those defines ‣ We have a lot of ‘$foo = “dontcare”’ in there. Just come up with an idiom and stick to it ‣ You could probably do something clever to filter out params? 43
- 44. " Maybe not everything goes in Puppet ‣ Last year I had this slide (included verbatim) 44
- 45. Well, I can’t always be right 45
- 46. ‣ Think to yourself: do I ever want to do this thing without running Puppet? ‣ Does it make sense for this to go through the same test -> prod cycle? ‣ Do the same people need to see this content as the Puppet manifests? ‣ Can it be generated in one place, or do you need to gather data from several? 46
- 47. ‣ As we’ve grown to more sites and more Puppet masters, this has hurt all over the place ‣ The deploy concept (vs `svn up`) made a mess of the checkin of this derived data ‣ I still think it’s an OK approach, but it needs thought 47
- 48. Examples ‣ DNS data: there’s only one actual copy of your DNS, and that’s what the world can see. What does a beta machine do with it? ‣ Load balancer configs: multiple datacenters need to know about each other, but can’t see the other’s database to derive config from 48
- 50. Want to help out? We’re hiring 50 ‣ Sysadmins ‣ Developers (Rails or Java) ‣ Testers ‣ Lots more backstopsolutions.com/careers
- 51. DevOpsDays Chicago! ‣ October 7th and 8th @ the Sears Tower ‣ CFP open until the 22nd ‣ Registration open now! Use code PUPPET_CAMP for 10% off devopsdays.org/events/2014-chicago/registration/ ‣ I might be on the committee 51 I refuse to call it the Willis Tower, y’all
- 52. Puppet Users Group ‣ Hosted by this guy, downtown, ~monthly (coming soon!) ‣ meetup.com/Chicago-Puppet-users-group/ 52