Infrastructure as code with Puppet and Apache CloudStack

Infrastructure as code
with Puppet and
Apache CloudStack
David Nalley <ke4qqq@apache.org> @ke4qqq

#whoami

• Recovering sysadmin
• Apache CloudStack Committer
• Designer of ugly slides

To set the stage...

• Apache CloudStack is...
• an open source IaaS platform
• proven in production at massive scale
• awesome

Decent API

• EC2 or native
• http://cloudstack.apache.org/docs/api

So IaaS removes one
constraint....

No longer waiting days/weeks to provision a
machine.

...but introduces another
constraint..

Now have to get a machine configured in a
timely manner.

People provision stuff

Often not ops folks
Often not intimately familiar with intricacies

Classification

Problem: We spin up, dynamically, 1-500 VMs at
any given time - how do we decide what
configurations apply.

Classification

The wrong way - dedicated images for each
purpose

Classification
editing nodes.pp

node 'foo-356.cloud.com' {
include httpd
}

Classification
globbing

node 'foo*' {
include httpd
}

Classification
Everything is default

node 'default' {
include httpd
}

Classification

External node classifier

Classification
Facts
class base {
case $::fact {
'httpd': {
include httpd
}
'otherrole': {
include nginx
}
}
}

One solution

During instance provisioning define metadata.
Custom fact for that metadata
Case statement based on that fact

Example metadata

role=webserver
location=datacenter1
environment=production

Corresponding
class base {
manifest
case $::fact {
'webserver': {
include httpd
}
'database': {
include postgresql
}
}
}

Links, et al

Fact:
http://s.apache.org/acs_userdata
Blog with details:
http://s.apache.org/acs_userdata2

Video is here, go watch it

• I only have 45 minutes - so can't delve into
everything, you should watch the video- it’s
great.
• http://youtu.be/c8YWctfOpwo

And then there was a
knife.....plugin

• So the folks at Edmunds.com wrote a knife
plugin for CloudStack.
• The knife plugin had the ability to define an
application stack, potentially hundreds of
nodes, that are interrelated, and provision
them with a single knife command.
• https://github.com/cloudstack-extras/knife-cloudstack

Deploying a machine with knife

knife cs server create

"name": "hadoop_cluster_a",
"description": "A small hadoop cluster with hbase",
"version": "1.0",
"environment": "production",
"servers": [
{
"name": "zookeeper-a, zookeeper-b, zookeeper-c",
"description": "Zookeeper nodes",
"template": "rhel-5.6-base",
"service": "small",
"port_rules": "2181",
"run_list": "role[cluster_a], role[zookeeper_server]",
"actions": [
{ "knife_ssh": ["role:zookeeper_server", "sudo chef-client"] }
]
},
{
"name": "hadoop-master",
"description": "Hadoop master node",
"service": "large",
"networks": "app-net, storage-net",
"port_rules": "50070, 50030, 60010",
"run_list": "role[cluster_a], role[hadoop_master], role[hbase_master]"
},
{
"name": "hadoop-worker-a hadoop-worker-b hadoop-worker-c",
"description": "Hadoop worker nodes",
"service": "medium",
"port_rules": "50075, 50060, 60030",
"run_list": "role[cluster_a], role[hadoop_worker], role[hbase_regionserver]",
"actions": [
{ "knife_ssh": ["role:hadoop_master", "sudo chef-client"] },
{ "http_request": "http://${hadoop-master}:50070/index.jsp" }
]
}

Deploy that with...

knife cs stack create hadoop_cluster_a

Then at FOSDEM 2012

• CloudStack user shows me Puppet types
and resources for OpenNebula.
• https://puppetlabs.com/blog/puppetizing-opennebula/

• They indicated they wanted this
awesomeness for CloudStack....

Why???
• They wanted to define each of their application
stacks in puppet, so that not only the
configuration of software on the machine, but
the machines themselves would be configured
by Puppet.
• Automated deployment of test environments
that are exactly the same
• Really gets outside of machine configuration to
entire infrastructure configuration

What we have grown used to

Puppet _defines_ the configuration within the
machine

What we want...

Puppet _defines_ the machine

What we want...

Puppet _defines_ a collection of machines

What we want...

Puppet _defines_ ALL the machines

...and then at PuppetConf

• There was Google Compute Engine types
and resources for Puppet.
• Dan Bode gave a presentation showing off
the work he had done... that presentation is
worth seeing...
• http://www.slideshare.net/bodepd/google-compute-presentation-puppet-conf

Infrastructure as code with Puppet and Apache CloudStack

So then for Christmas...

• puppet types and providers arrived -
courtesy of Dan Bode
• https://github.com/bodepd/cloudstack_resources

So how does this work

cloudstack_instance { 'foo1':
ensure => present,
flavor => 'Small Instance',
zone => 'FMT-ACS-001',
image => 'CentOS 5.6(64-bit) no GUI (XenServer)',
network => 'puppetlabs-network',
# domain
# account
# hostname
}

Setting defaults
Cloudstack_instance {
image => 'CentOS 6.3',
flavor => 'M1.medium',
zone => 'San Jose',
network => 'davids_net',
keypair => 'david_keys',
}
cloudstack_instance { 'baz':
ensure => $::ensure,
group => 'role=db',
}

A simple stack
class my_web_stack {
ensure => present,
group => 'role=apache',
}
ensure => present,
group => 'role=db',
}
}

Define all your
infrastructure

Resources/contact
Me: David Nalley <ke4qqq@apache.org> @ke4qqq
CloudStack: http://cloudstack.apache.org
users@cloudstack.apache.org
cloudstack_resources:
https://github.com/bodepd/cloudstack_resources

Jason Hancock Vids
http://youtu.be/c8YWctfOpwo
http://youtu.be/8W0BqCmNZQQ

Infrastructure as code with Puppet and Apache CloudStack

More Related Content

What's hot (20)

Similar to Infrastructure as code with Puppet and Apache CloudStack (20)

More from ke4qqq (14)

Recently uploaded (20)

Infrastructure as code with Puppet and Apache CloudStack