Jeff mc cune sf 2010
1 like1,699 views
This document discusses using Puppet and Splunk to audit change management policies and ensure compliance. It outlines how Puppet manages system configurations but the logs provide no link between configuration changes and commits. The document proposes enhancing Puppet reports to tag events with the Git commit, allowing Splunk to search for the commit that caused a specific event. This would help determine if systems were compliant by tying configuration changes to commits and answering questions about what happened and why.
![The Missing Link
• puppetmasterd -‐-‐config-‐version
/demo/get-‐config-‐version-‐script
• [root@puppet ~]# /demo/get-‐config-‐version
ref="refs/heads/jeff" commit="b585f7fe"
• Jeff’s processor, --reports=logversion
Should ship with puppet “soon”
Monday, October 11, 2010 13](https://image.slidesharecdn.com/jeffmccunesf2010-101013163805-phpapp01/85/Jeff-mc-cune-sf-2010-13-320.jpg)
![get-config-version
#! /bin/bash
set -‐u
set -‐e
cd /demo/puppet-‐demotools
ref="$(git symbolic-‐ref HEAD)"
if [[ -‐f .git/"${ref}" ]]; then
commit="$(cat .git/${ref})"
else
commit="UNKNOWN"
fi
echo "ref="${ref}" commit="${commit}""
Monday, October 11, 2010 14](https://image.slidesharecdn.com/jeffmccunesf2010-101013163805-phpapp01/85/Jeff-mc-cune-sf-2010-14-320.jpg)
Jeff mc cune sf 2010
- 1. Are we compliant? Auditing Change Management Policies with Splunk and Puppet http://bit.ly/puppetsplunkslides Jeff McCune jeff@puppetlabs.com Monday, October 11, 2010 1
- 2. Jeff McCune • Joined Puppet Labs in May, 2010 • Former SA at Netsmart Technologies • Solaris / RedHat Web App Infrastructure • Human Health Information Systems • HIPPA, SAS 70 Type II Compliance Monday, October 11, 2010 2
- 3. What’s this all about? • Audits are a fact of life • Systems drift • Puppet Master manifests change • The logs provide no link • Puppet and Git in synchrony with Splunk Monday, October 11, 2010 3
- 4. Fun with Regulations • Increased focus on compliance • SAS 70 • HIPPA • IPA • PCI DSS • etc, etc... Monday, October 11, 2010 4
- 5. Compliance is Easy Clones Golden VM Monday, October 11, 2010 5
- 6. Drifting in and out of Compliance Follow procedures Justify the change Firefighting Inevitable Constant drift Monday, October 11, 2010 6
- 7. The Trouble with Time • Are we compliant? • right now? • last week? • last year? • Why weren’t we? • Why is this difficult? Monday, October 11, 2010 7
- 8. Advanced Management • We have next-generation tools • Puppet • Git • Subversion • Splunk • Redmine Monday, October 11, 2010 8
- 9. Two major issues • Propagation • Time Monday, October 11, 2010 9
- 10. Change Propagation Many Nodes a872b46 Larry’s commit Monday, October 11, 2010 10
- 11. Time “Why did that one thing happen that one time?” Monday, October 11, 2010 11
- 12. Bridge the Gap Events Commits Monday, October 11, 2010 12
- 13. The Missing Link • puppetmasterd -‐-‐config-‐version /demo/get-‐config-‐version-‐script • [root@puppet ~]# /demo/get-‐config-‐version ref="refs/heads/jeff" commit="b585f7fe" • Jeff’s processor, --reports=logversion Should ship with puppet “soon” Monday, October 11, 2010 13
- 14. get-config-version #! /bin/bash set -‐u set -‐e cd /demo/puppet-‐demotools ref="$(git symbolic-‐ref HEAD)" if [[ -‐f .git/"${ref}" ]]; then commit="$(cat .git/${ref})" else commit="UNKNOWN" fi echo "ref="${ref}" commit="${commit}"" Monday, October 11, 2010 14
- 15. logversion.rb # Create logversion.rb by copying log.rb def process self.logs.each do |log| saved_message = "#{log.message}" log.message << " " << log.version Puppet::Util::Log.newmessage(log) log.message = saved_message end end Monday, October 11, 2010 15
- 16. Untagged Events Monday, October 11, 2010 16
- 17. Tagged Events Monday, October 11, 2010 17
- 18. Who to blame? Blame this guy The commit proves it Monday, October 11, 2010 18
- 19. Putting it all together • Demo time! Monday, October 11, 2010 19
- 20. Steps to Reproduce • Fork and clone puppet-‐demotools on github • logversion.rb goes into /usr/lib/ruby/site_ruby/1.8/puppet/reports • --config_version /path/to/your/script • --reports=logversion,store • Make sure syslog catches daemon.* and splunk is indexing syslog • Note: syslog outputs are off with -‐-‐verbose Monday, October 11, 2010 20
- 21. Future Work • Commit hooks into puppetmasterd activation and the ticketing system • Splunk URLs to redmine, trac, salesforce... • Closed loop from business case to system modification by puppet. Monday, October 11, 2010 21
- 22. Questions? • Google Moderator • http://bit.ly/arewecompliant? • http://bit.ly/puppetsplunkslides • Twitter: 0xEFF • Email: jeff@puppetlabs.com Monday, October 11, 2010 22