The Art and Zen of Managing Nagios With Puppet
- 1. Michael Merideth - VictorOps The Art and Zen of Managing Nagios with Puppet
- 2. Puppet and Nagios • Why use config management? • Using Puppet with Nagios: • Client-generated config • Server-generated config • Data separation • Provisioning and de-provisioning • Monitoring Puppet • Demo Time!
- 3. Why Config Management? The age of virtualization • IT pros, even at startups, are managing hundreds of nodes per person • There are simply too many running OSes for most shops to manage manually • Config management solves the problems of drift and inconsistency • Config management becomes living documentation
- 4. DevOps and Config Management DevOps from an IT perspective: • Infrastructure as code • SCM as an IT workflow tool • Continuous integration • Unit testing • Integration testing • Letting the developers into the sandbox
- 5. Key Puppet Features • Exported Resources • Hiera • Inline Templates • Facter • Nagios Resource Types • Puppet Forge
- 6. Key Puppet Features Exported Resources: • A resource gets defined on one host, and implemented on another • Requires the used of PuppetDB, which means a Puppet Master • Saves having to use shared storage or insecure file transfers • Keeps the number of config files low
- 7. Key Puppet Features Hiera: • Separate code from data • Set defaults and provide overrides • Encrypted back-ends mean security, even if your source code is stolen
- 8. Key Puppet Features Inline Templates: • Not just for file content • Allows the use of native Ruby code within Puppet policy
- 9. Key Puppet Features Facter: • Variable values defined at run-time, on the client • Easily extensible
- 10. Key Puppet Features Nagios Resource Types: • Makes creating Nagios configs dead simple • Enforces correct syntax • Not suitable for every config in every file • Warning: these will probably get deprecated and moved to a PuppetForge module
- 11. Key Puppet Features Puppet Forge: • A public repository of Puppet modules • Modules can be libraries, defined resource types, or classes • OMG CAUTION: 3rd party submissions are not evaluated by Puppet Labs
- 12. Why Not Use an Existing Module? There are several user-submitted Nagios modules in the Forge, but I wanted one that would: • Automatically add new hosts • Provision some services within other modules • Automatically remove decommissioned hosts • Make use of host and service groups
- 13. Putting It All Together Hiera defines the variables • Sane defaults for most values • Environment-specific overrides (dev, staging, production) • Site-specific overrides for different datacenters
- 14. Putting It All Together Clients build their own config: • Each client figures out its own hostgroup memberships • Array is built with an inline template • Other facts integrate into the host definition • NRPE config built from a common template
- 15. Putting It All Together Special service definitions get embedded with the service module • A change to a service means a change to how you monitor it • Manage that all in one place
- 16. Putting It All Together Common service definitions are templatized • Not everything needs to get built dynamically • Some services are monitored on all hosts • Less dynamic config means shorter catalog compilation • Templates can be easier to debug
- 17. Putting It All Together Config gets checked before it goes live • Build a complete config in a non-live directory • Run a check with nagios -v after the build • Only if that succeeds, do we move the config into the live directory
- 18. Ugly Hack Alert Dynamic files get built every run, installed if there’s a diff • Allows keeping backup directories with previous versions of the configs • Allows automatic removal of decommissioned hosts • Prevents excessive Nagios restarts
- 19. Sidebar: Monitoring Puppet On the Puppet master: • Apache and passenger processes • PuppetDB processes • Optionally puppet-dashboard or Foreman processes On the client: • Watch file age on last_run_summary.yaml
- 20. The Demo Environment Vagrant • AWESOME way to test Puppet code as you develop Github repository • Contains Vagrantfile and complete Puppet policy • github.com/victorops/puppet-nagios
- 21. • Running low on slides, so it must be… Demo Time!
- 22. The Demo Environment github.com/victorops/puppet-nagios • Check it out! • Contribute! • Or fork it! I don’t mind! • Help work towards a Puppet Forge module: • Cleaner Foreman provisioning • More types of config resources • Cross platform support • Documentation