SCALE 10x Build a Cloud Day
8 likes3,430 views
Chef automates cloud infrastructure management by enabling users to provision and configure resources programmatically. It provides a declarative interface to resources, allowing users to define policies and manage configurations efficiently through code. With community support and a wide range of cookbooks, Chef simplifies complex infrastructure management and integration tasks.
![That looks like this
extra_packages = case node['platform']
when "ubuntu","debian"
%w{
ruby1.8
ruby1.8-dev
rdoc1.8
ri1.8
libopenssl-ruby
}
end
extra_packages.each do |pkg|
package pkg do
action :install
end
end](https://image.slidesharecdn.com/scale2012-build-a-cloud-day-120123135449-phpapp01/85/SCALE-10x-Build-a-Cloud-Day-35-320.jpg)
![Or this
search(:users, '*:*') do |u|
user u['id'] do
uid u['uid']
shell u['shell']
home "/home/#{u['id']}"
end
directory "#{home_dir}/.ssh" do
owner u['id']
group u['gid']
mode "0700"
end
template "#{home_dir}/.ssh/authorized_keys" do
source "authorized_keys.erb"
owner u['id']
group u['id']
mode "0600"
variables :ssh_keys => u['ssh_keys']
end
end](https://image.slidesharecdn.com/scale2012-build-a-cloud-day-120123135449-phpapp01/85/SCALE-10x-Build-a-Cloud-Day-36-320.jpg)
![Build it somewhere
#EC2
knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-a7a97dce
-f m1.small -d omnibus -r 'role[base-cloud],role[tc],role[sample]'
#Rackspace
knife rackspace server create --image 110 --flavor 2 -i ~/.ssh/mray.pem -d omnibus -r
'role[base-cloud],role[tc],role[sample]'
#CloudStack
knife cs server create -S "small instance" -T "CentOS 5.5(64-bit) no GUI (KVM)" -i
~/.ssh/mray.pem -d omnibus -r 'role[base-cloud],role[tc],role[sample]'
#Ubuntu Linux
knife bootstrap test.lab -r 'role[webserver]' -i ~/.ssh/mray.pem -x ubuntu --sudo -d
omnibus -r 'role[base-cloud],role[tc],role[sample]'](https://image.slidesharecdn.com/scale2012-build-a-cloud-day-120123135449-phpapp01/85/SCALE-10x-Build-a-Cloud-Day-40-320.jpg)
![Pass results into Templates
pool_members = search("node","role:webserver”)
template "/etc/haproxy/haproxy.cfg" do
source "haproxy-app_lb.cfg.erb"
owner "root"
group "root"
mode 0644
variables :pool_members => pool_members.uniq
notifies :restart, "service[haproxy]"
end](https://image.slidesharecdn.com/scale2012-build-a-cloud-day-120123135449-phpapp01/85/SCALE-10x-Build-a-Cloud-Day-42-320.jpg)
![Pass results into Templates
# Set up application listeners here.
listen application 0.0.0.0:80
balance roundrobin
<% @pool_members.each do |member| -%>
server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check
<% end -%>
<% if node["haproxy"]["enable_admin"] -%>
listen admin 0.0.0.0:22002
mode http
stats uri /
<% end -%>](https://image.slidesharecdn.com/scale2012-build-a-cloud-day-120123135449-phpapp01/85/SCALE-10x-Build-a-Cloud-Day-43-320.jpg)
SCALE 10x Build a Cloud Day
- 1. Introduction to Chef SCALE Build a Cloud Day matt@opscode.com www.opscode.com @mattray
- 2. Congratulations!!! • U has a cloud • Now what? http://www.flickr.com/photos/ian_munroe/4758240536/
- 4. APIs are awesome! • You can provision compute resources in seconds • You can provision storage resources in seconds • That’s cool. http://www.flickr.com/photos/jdhancock/3634246981/
- 5. Chef can help with that • knife ec2 server • knife cloudstack create server create • knife rackspace • knife kvm server server create create • knife terremark • knife vsphere server create server create • knife voxel • knife eucalyptus server create server create • knife gandi • knife openstack server create server create http://www.flickr.com/photos/kyz/3122499444/
- 7. You need to configure them
- 9. See Nodes Application Server Application Database
- 10. See Nodes Grow Application Server Application Databases
- 11. See Nodes Grow Application Servers Application Databases
- 12. See Nodes Grow Load Balancer Application Servers Application Databases
- 13. See Nodes Grow Load Balancers Application Servers Application Databases
- 14. See Nodes Grow Load Balancers Application Servers Application Database Cache Application Databases
- 15. Tied together with Config Load Balancers Application Servers Application Database Cache Application Databases
- 16. Infrastructure is a Snowflake Load Balancers Application Servers Application Database Cache Floating IP? Application Databases
- 17. Evolving Complexity Load Balancers ApplicationCache Application Servers NoSQL Database Cache Database Slaves Database
- 18. Complexity Grows Quickly DC2 DC1 DC3
- 19. And it Continues to Evolve http://www.flickr.com/photos/16339684@N00/2681435235/
- 20. Golden Images are not the answer • Gold is heavy • Hard to transport • Hard to mold • Easy to lose configuration detail http://www.flickr.com/photos/garysoup/2977173063/
- 21. Typical Boring Infrastructure Graphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
- 22. New Compliance Mandate Graphite Nagios Jboss App • Move SSH off port 22 • Lets put it on 2022 Memcache Postgres Slaves Postgres Master
- 23. 6 Golden Image Updates Graphite 1 2 Nagios 3 Jboss App • edit /etc/ssh/sshd_config 4 Memcache 5 Postgres Slaves 6 Postgres Master
- 24. 12 Instance Replacements Graphite 1 2 Nagios 3 4 5 6 7 Jboss App • Delete, launch 8 • Repeat 9 Memcache • Typically manually 10 11 Postgres Slaves 12 Postgres Master
- 25. In a Maintenance Window Graphite 1 2 Nagios 3 4 5 6 7 Jboss App • Don't break anything! 8 9 Memcache 5 10 11 Postgres Slaves 12 Postgres Master
- 26. With Different IP Addresses? Graphite Nagios Jboss App • Invalid Configs • Bob just got fired :( Memcache Postgres Slaves Postgres Master
- 27. Configuration Desperation http://www.flickr.com/photos/francoforeshock/5716969942/
- 28. Configuration Management and Automated Systems Integration is the Answer http://www.flickr.com/photos/philliecasablanca/3354734116/
- 29. Chef Solves This Problem • But you already guessed that, didn’t you?
- 30. Collections of Resources • Networking • Routes • Users • Files • Groups • Directories • Tasks • Symlinks • Packages • Mounts • Software • Services • Configurations • Other Stuff http://www.flickr.com/photos/stevekeys/3123167585/
- 31. Acting in Concert http://www.flickr.com/photos/glowjangles/4081048126/
- 32. To Provide a Service http://www.flickr.com/photos/28309157@N08/3743455858/
- 33. Chef is Infrastructure as Code • Programmatically provision and configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources. http://www.flickr.com/photos/louisb/4555295187/
- 34. Declarative Interface to Resources • Define policy • Say what, not how • Pull not Push http://www.flickr.com/photos/bixentro/2591838509/
- 35. That looks like this extra_packages = case node['platform'] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 libopenssl-ruby } end extra_packages.each do |pkg| package pkg do action :install end end
- 36. Or this search(:users, '*:*') do |u| user u['id'] do uid u['uid'] shell u['shell'] home "/home/#{u['id']}" end directory "#{home_dir}/.ssh" do owner u['id'] group u['gid'] mode "0700" end template "#{home_dir}/.ssh/authorized_keys" do source "authorized_keys.erb" owner u['id'] group u['id'] mode "0600" variables :ssh_keys => u['ssh_keys'] end end
- 37. Recipes and Cookbooks • Recipes are collections of Resources • Cookbooks contain recipes, templates, files, custom resources, etc • Code re-use and modularity • Hundreds already on Community.opscode.com http://www.flickr.com/photos/shutterhacks/4474421855/
- 38. Nodes • Chef-Client generates configurations directly on nodes • Reduce management complexity through abstraction • Store the configuration of your programs in version control http://www.flickr.com/photos/ssoosay/5126146763/
- 39. Upload your infrastructure knife cookbook upload apt knife cookbook upload chef-client knife cookbook upload java knife cookbook upload jpackage knife cookbook upload ntp knife cookbook upload sudo knife cookbook upload tomcat knife cookbook upload users knife cookbook upload sample knife role from file base-cloud.rb knife role from file tc.rb knife role from file sample.rb knife data bag create users knife data bag from file users mray.json
- 40. Build it somewhere #EC2 knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-a7a97dce -f m1.small -d omnibus -r 'role[base-cloud],role[tc],role[sample]' #Rackspace knife rackspace server create --image 110 --flavor 2 -i ~/.ssh/mray.pem -d omnibus -r 'role[base-cloud],role[tc],role[sample]' #CloudStack knife cs server create -S "small instance" -T "CentOS 5.5(64-bit) no GUI (KVM)" -i ~/.ssh/mray.pem -d omnibus -r 'role[base-cloud],role[tc],role[sample]' #Ubuntu Linux knife bootstrap test.lab -r 'role[webserver]' -i ~/.ssh/mray.pem -x ubuntu --sudo -d omnibus -r 'role[base-cloud],role[tc],role[sample]'
- 41. Search • Search for nodes with Roles • Find configuration data • IP addresses • Hostnames • FQDNs http://www.flickr.com/photos/kathycsus/2686772625
- 42. Pass results into Templates pool_members = search("node","role:webserver”) template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end
- 43. Pass results into Templates # Set up application listeners here. listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%>
- 44. So when this Graphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
- 45. Becomes this Graphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
- 46. Updates can be automatic Graphite Nagios Jboss App Memcache Postgres Slaves Postgres Master
- 47. Count the resources • Load balancer config Graphite Nagios • Nagios host ping • Nagios host ssh Jboss App • Nagios host HTTP • Nagios host app health Memcache • Graphite CPU • Graphite Memory Postgres Slaves • Graphite Disk • Graphite SNMP • Memcache firewall • 12+ resource changes for 1 node addition • Postgres firewall Postgres authZ config
- 48. CLONING CANNOT COPE WITH THIS • Chef can. http://www.flickr.com/photos/evelynishere/2798236471/
- 49. Build anything • Simple internal applications • Complex external applications • Workstations • Hadoop clusters • IaaS infrastructure • PaaS infrastructure • SaaS applications • Storage systems • You name it http://www.flickr.com/photos/hyku/245010680/
- 50. And manage it simply • Automatically reconfigure everything • Linux, Windows, Unixes, BSDs • Load balancers • Metrics collection systems • Monitoring systems • Cloud migrations become trivial http://www.flickr.com/photos/helico/404640681/
- 51. The Chef Community • Apache License, Version 2.0 • 550+ Individual contributors • 100+ Corporate contributors • Dell, Rackspace,VMware, RightScale, Heroku, and many more • Nearly 400 cookbooks • http://community.opscode.com
- 53. Thanks! matt@opscode.com www.opscode.com