Oscar: Rapid Iteration with Vagrant and Puppet Enterprise - PuppetConf 2013
3 likes8,539 views
The document discusses the integration of Vagrant and Puppet Enterprise for efficient development and operations workflows at Puppet Labs. It highlights Vagrant's benefits for creating reproducible local VM environments and addresses its limitations in highly mutable configurations. Additionally, it introduces 'Oscar,' a compilation of Vagrant plugins designed to streamline deployment and configuration in mutable environments.
![A descent into madness
(I’m sorry.)
DONT. DO. THIS.
#vi:setft=ruby:
require'yaml'
begin
#Loadconfig
yaml_config=File.join(File.dirname(__FILE__),"config.yaml")
config=nil
File.open(yaml_config)do|fd|
config=YAML.load(fd.read)
end
nodes =config["nodes"]
profiles =config["profiles"]
nodes.eachdo|node|
#SetdefaultPEconfiguration,andallownodeoverridingofthesevalues
defaults={"pe"=>config['pe']}
node.merge!(defaults)do|key,oldval,newval|
ifoldval.is_a?Hash
newval.mergeoldval
else
warn"Triedtomergehashvalueswith#{key}=>[#{oldval},#{newval}],butwasnotahash.Using#{oldval}."
oldval
end
end
profile =node["profile"]
node.merge!profiles[profile]
end
rescue=>e
puts"Malformedormissingconfig.yaml:#{e}"
putse.backtrace
exit!(1)
end
#Thisisanextensionofthecommonnodedefinition,asitmakesprovisions
#forcustomizingthemasterformoreseamlessinteractionwiththebase
#system
defprovision_master(config,node,attributes)
#Manifestsandmodulesneedtobemountedonthemasterviasharedfolders,
#butthedefault/vagrantmounthaspermissionsandownershipthatconflicts
#withthemasterandpe-puppet.Wemountthesefoldersseparatelywith
#loosenedpermissions.
config.vm.share_folder'manifests','/manifests','./manifests',:extra=>'fmode=644,dmode=755,fmask=022,dmask=022'
config.vm.share_folder'modules','/modules','./modules', :extra=>'fmode=644,dmode=755,fmask=022,dmask=022'
#Updatepuppet.conftoaddthemanifestdirdirectivetopointtothe
#/manifestsmount,ifthedirectiveisn'talreadypresent.
node.vm.provision:shelldo|shell|
shell.inline=<<-EOT
sed-i'
2{
/manifest/!i
manifestdir=/manifests
}
'/etc/puppetlabs/puppet/puppet.conf
EOT
end
#Updatepuppet.conftoaddthemodulepathdirectivetopointtothe
#/modulemount,ifithasn'talreadybeenset.
node.vm.provision:shelldo|shell|
shell.inline=<<-EOT
sed-i'
/modulepath/{
/vagrant/!s,$,:/modules,
}
'/etc/puppetlabs/puppet/puppet.conf
EOT
end
#Rewritetheoldesite.ppconfigsinceit'snotused,andwarnpeople
#aboutthis.
node.vm.provision:shelldo|shell|
shell.inline=%{echo"#/etc/puppetlabs/puppet/manifestsisnotused;see/manifests.">/etc/puppetlabs/puppet/manifests/site.pp}
end
#BoostRAMforthemastersothatactivemqdoesn'tasplode
node.vm.customize(["modifyvm",:id,"--memory","1024"])
#Enableautosigningonthemaster
node.vm.provision:shelldo|shell|
shell.inline=%{echo'*'>/etc/puppetlabs/puppet/autosign.conf}
end
end
#Addsthevagrantconfigurationtweaks
defconfigure_node(config,node,attributes)
node.vm.box=attributes["boxname"]
#Applyallspecifiedportforwards
attributes["forwards"].eachdo|(src,dest)|
node.vm.forward_portsrc,dest
endifattributes["forwards"]#<--Iamamonster
#Addinoptionalper-nodeconfiguration
node.vm.box_url=attributes["boxurl"]ifattributes["boxurl"]
node.vm.network:hostonly,attributes["address"]ifattributes["address"]
node.vm.boot_mode=attributes[:gui]ifattributes[:gui]
end
#Provideprovisioningdetailsforthisnode
defprovision_node(config,node,attributes)
#HackinfauxDNS
#PuppetenterpriserequiressomethingresemblingfunctioningDNStobe
#installedcorrectly
attributes["hosts_entries"].eachdo|entry|
node.vm.provision:shelldo|shell|
shell.inline=%{grep"#{entry}"/etc/hosts||echo"#{entry}">>/etc/hosts}
end
end
#Setthemachinehostname
node.vm.provision:shelldo|shell|
shell.inline=%{hostname#{attributes["name"]}}
end
node.vm.provision:shelldo|shell|
shell.inline=%{domainnamepuppetlabs.test}
end
end
definstall_pe(config,node,attributes)
#Customizetheanswersfileforeachnode
node.vm.provision:shelldo|shell|
shell.inline=%{sed-e's/%%CERTNAME%%/#{attributes["name"]}/'</vagrant/answers/#{attributes["role"]}.txt>/tmp/answers.txt}
end
#IfthePEversionis0.0.0,bypasspuppetinstallation.
#Thiscouldeasilymakelaterprovisioningfail.
ifattributes['pe']['version'].match%r[^0.0]
warn"RequestedPEversionwassetto#{attributes['pe']['version']};willnotautomaticallyinstallPE"
return
end
#Assembletheinstallercommand
fragments=[]
fragments<<"2>&1"
fragments<<attributes['pe']['installer']['executable']
fragments<<'-a/tmp/answers.txt'
fragments<<attributes['pe']['installer']['args'].join('')
installer_cmd=fragments.join('').gsub(':version',attributes['pe']['version'])
#InstallPuppetEnterpriseifithasn'tbeeninstalledyet.Ifthemachine
#isrebootedthenthisprovisioningstepwillbeskipped.
node.vm.provision:shelldo|shell|
shell.inline=<<-EOT
if![-f/opt/pe_version];then
#{installer_cmd}
fi
EOT
end
end
Vagrant::Config.rundo|config|
#GeneratealistofnodeswithstaticIPaddresses
hosts_entries=nodes.select{|h|h["address"]}.map{|h|%{#{h["address"]}#{h["name"]}}}
#TweakeachhostforPuppetEnterprise,andtheninstallPEitself.
nodes.eachdo|attributes|
config.vm.defineattributes["name"]do|node|
attributes["hosts_entries"]=hosts_entries
configure_node(config,node,attributes)
provision_node(config,node,attributes)
install_pe(config,node,attributes)
ifattributes["role"].match/master/
provision_master(config,node,attributes)
end
end
end
end](https://image.slidesharecdn.com/oscar-rapiditerationwithvagrantandpuppetenterprise-130823182412-phpapp02/85/Oscar-Rapid-Iteration-with-Vagrant-and-Puppet-Enterprise-PuppetConf-2013-25-320.jpg)
![Before config builder
Vagrantfile
config.vm.define :puppetmaster do |box|
flavor = :centos_6
set_box box, S3_BOXES[:centos_64_nocm]
# NOTE: Hostonly _may_ mean no internets if this adapter gets found first!!!
# Check /etc/resolv.conf !
box.vm.network :hostonly, "192.168.23.20"
# NOTE: Share folders, such as Git checkouts of the Puppet source code
share_puppet_source box
box.vm.provision :shell, :inline => BOOTSTRAPS[flavor]
provision_box box, 'server.pp'
end](https://image.slidesharecdn.com/oscar-rapiditerationwithvagrantandpuppetenterprise-130823182412-phpapp02/85/Oscar-Rapid-Iteration-with-Vagrant-and-Puppet-Enterprise-PuppetConf-2013-37-320.jpg)
Oscar: Rapid Iteration with Vagrant and Puppet Enterprise - PuppetConf 2013
- 1. Oscar Rapid Iteration with Vagrant and Puppet Enterprise Adrien Thebo | Puppet Labs @nullfinch | Freenode: finch
- 2. Hi.
- 3. Introductions n’ stuff On/Off Operations/Development, ~8 years Devops before it was cool
- 4. Introductions n’ stuff Puppet Labs, 2.5 years Operations Engineer, 2 years Community Software Engineer, 6 months
- 5. Things I do
- 6. r10k Better Puppet deployment, powered by robots
- 8. So how about that Vagrant?
- 9. Mandatory audience fun time! Who has… heard of Vagrant? tried Vagrant? used Vagrant regularly? developed on Vagrant or plugins? created Vagrant?
- 10. A brief introduction to Vagrant
- 11. Good idea, bad idea Good idea: hand crafted, one of a kind artisan furniture Bad idea: hand crafted, one of a kind artisan VMs
- 12. Before Vagrant Hand rolled VMs No automation Unreproducible Full of forgotten hacks
- 13. Vagrant is… Local VM infrastructure without the pain Easy to configure Reproducible Portable
- 14. Vagrant Workflow generate provision use/break discard It’s your very own cloud!
- 15. Laying the Groundwork Operations @ Puppet Labs … Support @ Puppet Labs? Infrastructure & client support
- 17. Supporting PE 2 Need to reproduce customer problems Various supported platforms Various node configurations
- 18. Supporting PE 2 Speed/reproducibility paramount Have to meet SLA Also handle other operations work
- 19. Support ♥ Vagrant Receive ticket Duplicate client environment Build environment Reproduce & debug Help customer^W^Wprofit!!!
- 20. Limitations If this presentation was all roses and butterflies Then it would be very boring.
- 21. Choose your own adventure Like any tool, Vagrant makes tradeoffs Some use cases are easier than others
- 22. Vagrant excels at… Static/Stable environments Configure VM definition Set up provisioning git commit -m 'Add Vagrantfile'
- 23. Vagrant struggles with… Highly mutable environments Complex provisioning steps Configuration reuse Partial configuration Distributable configuration
- 24. The Vagrantfile DSL Pro - all the power of Ruby Con - all the power of Ruby
- 25. A descent into madness (I’m sorry.) DONT. DO. THIS. #vi:setft=ruby: require'yaml' begin #Loadconfig yaml_config=File.join(File.dirname(__FILE__),"config.yaml") config=nil File.open(yaml_config)do|fd| config=YAML.load(fd.read) end nodes =config["nodes"] profiles =config["profiles"] nodes.eachdo|node| #SetdefaultPEconfiguration,andallownodeoverridingofthesevalues defaults={"pe"=>config['pe']} node.merge!(defaults)do|key,oldval,newval| ifoldval.is_a?Hash newval.mergeoldval else warn"Triedtomergehashvalueswith#{key}=>[#{oldval},#{newval}],butwasnotahash.Using#{oldval}." oldval end end profile =node["profile"] node.merge!profiles[profile] end rescue=>e puts"Malformedormissingconfig.yaml:#{e}" putse.backtrace exit!(1) end #Thisisanextensionofthecommonnodedefinition,asitmakesprovisions #forcustomizingthemasterformoreseamlessinteractionwiththebase #system defprovision_master(config,node,attributes) #Manifestsandmodulesneedtobemountedonthemasterviasharedfolders, #butthedefault/vagrantmounthaspermissionsandownershipthatconflicts #withthemasterandpe-puppet.Wemountthesefoldersseparatelywith #loosenedpermissions. config.vm.share_folder'manifests','/manifests','./manifests',:extra=>'fmode=644,dmode=755,fmask=022,dmask=022' config.vm.share_folder'modules','/modules','./modules', :extra=>'fmode=644,dmode=755,fmask=022,dmask=022' #Updatepuppet.conftoaddthemanifestdirdirectivetopointtothe #/manifestsmount,ifthedirectiveisn'talreadypresent. node.vm.provision:shelldo|shell| shell.inline=<<-EOT sed-i' 2{ /manifest/!i manifestdir=/manifests } '/etc/puppetlabs/puppet/puppet.conf EOT end #Updatepuppet.conftoaddthemodulepathdirectivetopointtothe #/modulemount,ifithasn'talreadybeenset. node.vm.provision:shelldo|shell| shell.inline=<<-EOT sed-i' /modulepath/{ /vagrant/!s,$,:/modules, } '/etc/puppetlabs/puppet/puppet.conf EOT end #Rewritetheoldesite.ppconfigsinceit'snotused,andwarnpeople #aboutthis. node.vm.provision:shelldo|shell| shell.inline=%{echo"#/etc/puppetlabs/puppet/manifestsisnotused;see/manifests.">/etc/puppetlabs/puppet/manifests/site.pp} end #BoostRAMforthemastersothatactivemqdoesn'tasplode node.vm.customize(["modifyvm",:id,"--memory","1024"]) #Enableautosigningonthemaster node.vm.provision:shelldo|shell| shell.inline=%{echo'*'>/etc/puppetlabs/puppet/autosign.conf} end end #Addsthevagrantconfigurationtweaks defconfigure_node(config,node,attributes) node.vm.box=attributes["boxname"] #Applyallspecifiedportforwards attributes["forwards"].eachdo|(src,dest)| node.vm.forward_portsrc,dest endifattributes["forwards"]#<--Iamamonster #Addinoptionalper-nodeconfiguration node.vm.box_url=attributes["boxurl"]ifattributes["boxurl"] node.vm.network:hostonly,attributes["address"]ifattributes["address"] node.vm.boot_mode=attributes[:gui]ifattributes[:gui] end #Provideprovisioningdetailsforthisnode defprovision_node(config,node,attributes) #HackinfauxDNS #PuppetenterpriserequiressomethingresemblingfunctioningDNStobe #installedcorrectly attributes["hosts_entries"].eachdo|entry| node.vm.provision:shelldo|shell| shell.inline=%{grep"#{entry}"/etc/hosts||echo"#{entry}">>/etc/hosts} end end #Setthemachinehostname node.vm.provision:shelldo|shell| shell.inline=%{hostname#{attributes["name"]}} end node.vm.provision:shelldo|shell| shell.inline=%{domainnamepuppetlabs.test} end end definstall_pe(config,node,attributes) #Customizetheanswersfileforeachnode node.vm.provision:shelldo|shell| shell.inline=%{sed-e's/%%CERTNAME%%/#{attributes["name"]}/'</vagrant/answers/#{attributes["role"]}.txt>/tmp/answers.txt} end #IfthePEversionis0.0.0,bypasspuppetinstallation. #Thiscouldeasilymakelaterprovisioningfail. ifattributes['pe']['version'].match%r[^0.0] warn"RequestedPEversionwassetto#{attributes['pe']['version']};willnotautomaticallyinstallPE" return end #Assembletheinstallercommand fragments=[] fragments<<"2>&1" fragments<<attributes['pe']['installer']['executable'] fragments<<'-a/tmp/answers.txt' fragments<<attributes['pe']['installer']['args'].join('') installer_cmd=fragments.join('').gsub(':version',attributes['pe']['version']) #InstallPuppetEnterpriseifithasn'tbeeninstalledyet.Ifthemachine #isrebootedthenthisprovisioningstepwillbeskipped. node.vm.provision:shelldo|shell| shell.inline=<<-EOT if![-f/opt/pe_version];then #{installer_cmd} fi EOT end end Vagrant::Config.rundo|config| #GeneratealistofnodeswithstaticIPaddresses hosts_entries=nodes.select{|h|h["address"]}.map{|h|%{#{h["address"]}#{h["name"]}}} #TweakeachhostforPuppetEnterprise,andtheninstallPEitself. nodes.eachdo|attributes| config.vm.defineattributes["name"]do|node| attributes["hosts_entries"]=hosts_entries configure_node(config,node,attributes) provision_node(config,node,attributes) install_pe(config,node,attributes) ifattributes["role"].match/master/ provision_master(config,node,attributes) end end end end
- 26. Vagrant and networking No default internal networking Flaky DHCP DNS resolution
- 27. Vagrant and networking Solutions! Run your own infrastructure! DHCPD BIND
- 28. The reality Can’t support every configuration Shouldn’t support every configuration Batteries not included
- 30. Oscar in a nutshell Set of Vagrant plugins Built for mutable environments
- 32. Vagrant hosts DNS record types: starting with A: A, AAAA, AFSDB, APL starting with C: CAA, CERT, CNAME starting with D: DHCID, DLV, DNAME, DNSKEY, DS … We need: name -> ip address
- 33. Vagrant hosts Inside of a transitory environment Query private network addresses -> /etc/hosts Go do something you care about Or manage BIND on $platform
- 34. Vagrant Auto-network Have to add extra interfaces f$&k it. STATIC IP ADDRESSES FOR ALL config.vm.network :private_network, :auto_network => true
- 35. Vagrant PE Build PE configuration optimized for Vagrant Download installers on demand
- 36. Vagrant config builder Configuration as data
- 37. Before config builder Vagrantfile config.vm.define :puppetmaster do |box| flavor = :centos_6 set_box box, S3_BOXES[:centos_64_nocm] # NOTE: Hostonly _may_ mean no internets if this adapter gets found first!!! # Check /etc/resolv.conf ! box.vm.network :hostonly, "192.168.23.20" # NOTE: Share folders, such as Git checkouts of the Puppet source code share_puppet_source box box.vm.provision :shell, :inline => BOOTSTRAPS[flavor] provision_box box, 'server.pp' end
- 38. After config builder roles.yaml --- roles: puppetmaster: private_networks: - {auto_network: true} synced_folders: - {host_path: ~/Source/puppetlabs, guest_path: /puppetlabs} provisioners: - {type: hosts} - {type: pe_bootstrap, role: master} puppetagent: private_networks: - {auto_network: true} provisioners: - {type: hosts} - {type: pe_bootstrap}
- 39. After config builder vms.yaml --- vms: - name: master box: centos_64_nocom roles: puppetmaster - name: agent box: debian_64_nocm roles: puppetagent
- 40. Pick your data source YAML JSON XML interpretive dance
- 41. What does Oscar do? Dependencies! Everything is a standalone plugin Mix and match
- 42. What does Oscar do? Templates and defaults Sane defaults to get you started
- 43. PE stack in a box Configure your development environment like production Develop your modules in complete isolation Simulate app deployments before going live Pre-production in a box! Stable Puppet environment
- 44. What Oscar gets you All the perks of Vagrant Minimal user setup Complex config made easy
- 45. What Oscar gets you Goal - from zero to PE in vagrant up
- 46. Who uses it? Commercial Support Open Source Support Sales Engineering People… and stuff…
- 47. Demo time! Basic spin up Sales eng env
- 48. Questions?
- 49. Try it yourself Code on Github Under active development contributions accepted!
- 50. Thanks Tom Linkin! Chris Barker! Charlie Sharpsteen! Hunter Haugen! Adam Crews! The fabulous Puppet and Vagrant community! Exclamation points! Many others!
- 51. Credits Layout by Nanoc Presentation by Reveal.js Consciousness by caffeine