TechUG Glasgow talk 22/Feb/17 Configuration Management Best Practices

The Cloud Specialists
Configuration Management Best
Practices
Dag Sonstebo
Cloud Architect
dag.sonstebo@shapeblue.com
Twitter: @dagsonstebo

C l i c k t o e d i t
ShapeBlue.com @ShapeBlue
A b o u t M e
• Cloud Architect @ ShapeBlue
• Background:
• Cloud and virtualization architect with 19
years experience from the service provider,
financial and manufacturing industries.
• Specialize in:
• Cloud infrastructure architecture and
engineering.
• Virtualization - VMware vSphere, Citrix
XenServer, KVM.
• Automation and configuration
management.

A b o u t S h a p e B l u e
“ShapeBlue are expert builders of public &
private clouds. They are the leading global
Apache CloudStack
integrator & consultancy”

S h a p e B l u e c u s t o m e r s

Configuration management

W h a t i s c o n f i g u r a t i o n m a n a g e m e n t ?
• Wikipedia:
“Configuration management (CM) is a
systems engineering process for
establishing and maintaining consistency
of a product's performance, functional,
and physical attributes with its
requirements, design, and operational
information throughout its life.”
• Originated in the US military in the 1950’s,
has been adopted by a number of fields
since then – among these ITIL.
• Is becoming an important building block in
the devops and cloud era.

Ve r y g o o d . . s o h o w d o e s i t a c t u a l l y w o r k i n o u r w o r l d ?
• From our point of view Configuration Management needs to ensure
idempotency across your infrastructure….
• An idempotent element of a set does not change in value when
multiplied by itself…...
• Mathematical operation which can be applied multiple times
without changing the result beyond the initial application…..
• Or - for the non-mathematicians:
• If it ain’t broke, don’t fix it....
• Run it a 1000 times, outcome must be the same....

Ve r y g o o d . . s o h o w d o e s i t a c t u a l l y w o r k i n o u r w o r l d ?
Config management toolset needs to:
• maintain the state of your infrastructure
• be highly automated
• be consistent
• centralise configuration data and
procedures
• replace your manual processes

W h y d o I n e e d i t ?
We’re moving…
• into the devops / cloud era - bimodal IT adds extra
overhead
• from kitten-centric to chicken-centric infrastructure
• from old-school enterprise IT to maintaining our
commodity data and considering workloads as
disposable
• towards everything-as-a-service
• towards consistent Infrastructure-As-Code
Benefits:
• Reduction in cost – less effort, less manpower
• Increase in speed – faster execution of your
procedures
• Reduction of risk (remove errors and security
violations)

S o – h o w d o e s C o n f i g M a n a g e m e n t f i t i n t o t h i s ?
• Provides:
• High degree of automation
• Consistent procedures
• Requires:
• Change in company culture – from techies to change
management team and up to CTO office
• Change in technology staff skillsets
• A new approach to testing, auditing and
authorization

Best practices… or how do I get started....

A p p r o a c h
• Start small, build gradually.
• Rome wasn’t built in a day…..don’t
try to take on the world in the first week.
• Start off with the small tasks:
• Configure your NTP settings
• Add the latest DNS servers settings
• Standardize your SSH login policies.
• Then move on to the big tasks:
• Patch those 25k servers.
• Build a Continuous Integration workflow which does nightly software builds
and rolls these out to your environments automagically.

Ve r s i o n c o n t r o l
• Version control is no longer just for
developers…..
• It is a key component for
• Collaboration
• Traceability
• Quality control
• A starting point for your Continuous
Integration workflows….
• If you don’t know it – time to learn it!

M a k e s u r e e v e r y o n e i s o n b o a r d a n d p l a n y o u r a p p r o a c h … .
• Configuration
management requires a
change in culture - make
sure your team buys into
it.
• Plan your procedures
and make your code
reusable.

C h o o s e t h e r i g h t t o o l s f o r t h e j o b

C h o o s e t h e r i g h t t o o l s f o r t h e j o b
Things to consider:
• Command line or GUI?
• Agentless?
• Platform support?
• Runbook language skills required?
• Opensource or proprietary?

A s e l e c t i o n f r o m t h e o p e n s o u r c e t o o l b o x …
Agentless Platform support Runbooks
Ansible Yes Win / Lin / Unix / OSX YAML English
Chef No Win / Lin / Unix / OSX Ruby
Puppet No Win / Lin / Unix / OSX Ruby
Salt Both
Win / Lin / most Unix /
OSXs
YAML English
Hashicorp . . .

Tr e n d s

Ansible

W h y d i d I c h o o s e A n s i b l e ?
• I tried Puppet and did not like:
• Ruby runbooks
• Rolling out clients and assigning certificates
• I needed something simpler and more robust…
• Ansible ticked all the boxes:
• Simple and agentless.
• All configuration done over SSH or PowerShell using username + password or
SSH keys.
• Human readable configuration, near plain English.
• Support for:
• all Operating Systems
• Cloud and systems providers like AWS, Azure, CloudStack, Digitial Ocean,
Docker, Google, OpenStack, Rackspace, Vmware…
• Push or pull configuration
• Ad-hoc tasks or bigger playbooks

G e t t i n g s t a r t e d w i t h A n s i b l e
• Build Ansible control machine:
• Linux or OSX
• No major spec requirements
• Install Ansible
• Configure version control:
• E.g. configure your Github repo
• Get started:
• Create some playbooks, with tasks and roles, as well as a static inventory.
• On your control machine pull down your repo and run your
playbooks:
• # git pull
• # ansible-playbook –i inventoryfile playbookname.yml

A n s i b l e p l a y b o o k – h e l l o w o r l d - i n s t a l l A p a c h e
deployapache.yml:
---
- hosts: webservers
vars: http_port: 80
max_clients: 200
remote_user: root
tasks:
- name: Update all packages
yum: name: '*' state: latest
- name: ensure apache is at the latest version
yum: name: httpd state: latest
- name: write the apache config file
template: src: /srv/httpd.j2 dest: /etc/httpd.conf
notify:
- restart apache
- name: ensure apache is running
service: name: httpd state: started
handlers:
- name: restart apache
service: name: httpd state: restarted
myinventory:
---
[webfarm:children]
webservers
mysqlservers
[mysqlservers]
mysqllhost1
Mysqllhost1
[webservers]
apachehost1
apachehost2
#ansible-playbook –i myinventory deployapache.yml

A n s i b l e b u i l d i n g b l o c k s
Playbooks
Host Inventories
Hosts
Groups
Variables (hosts
or groups)
Tasks Modules
Roles Tasks Modules
Templates jinja2

H o s t i n v e n t o r i e s
• Static • Dynamic scripts, e.g.
• DYI….
• Ansible Tower
• LDAP
• Cobbler
• AWS EC2
• OpenStack
• BSD Jails
• DigitalOcean
• Google Compute Engine
• Linode
• OpenShift
• Ovirt
• SpaceWalk
• Vagrant
• Zabbix
myinventory:
---
[webfarm:children]
webservers
mysqlservers
[mysqlservers]
mysqllhost1
Mysqllhost1
[webservers]
apachehost1
apachehost2

A n s i b l e v a r i a b l e s
• Can be defined in:
• In the inventory.
• In the Ansible playbook.
• In include files for hosts or groups.
Inventory:
[webservers]
apachehost1 http_port=80
apachehost2 http_port=8080
Playbook:
- hosts: webservers
vars:
http_port: 80
Include file:
---
http_port: “80”

A n s i b l e v a r i a b l e s
• Facts:
• Information automatically gathered from the system (not user
defined):
• You can also write your own – just ensure the output is valid JSON!
# ansible kvmlab5-kvm1 -m setup -i hosts_kvmlab5ds-49kvmlab5-kvm1 | SUCCESS => {
"ansible_facts": {
"ansible_all_ipv4_addresses": [
"10.2.2.38",
"192.168.122.1",
"169.254.0.1”
],
"ansible_architecture": "x86_64",
"ansible_bios_date": "09/17/2015",
…..

A n s i b l e t a s k s a n d r o l e s
• Tasks:
- name: ensure yum cache is cleared
shell: command="yum clean all”
- name: Install MySQL
yum: name=mysql-server state=present
when: ansible_distribution_major_version == "7"
- name: Copy DNS settings template
template:
src=templates/resolveconf.j2
dest=/etc/resolv.conf
• Roles: re-usable collections of tasks – e.g. ”apache”, “mysql”, etc.

Te m p l a t e s
• Templates:
• used to populate configuration files, etc.
• Written in the Jinja2 python templating
language.
• Resolveconf.j2:
Variables:
dns_servers:
- 8.8.8.8
- 8.8.4.4
dns_search: “mylab.local”
resolveconf.j2:
{% for ns in dns_servers %}
nameserver {{ ns }}
{% endfor %}
search {{ dns_search }}
/etc/resolve.conf:
nameserver 8.8.8.8
nameserver 8.8.4.4
search mylab.local

Our use case

B a c k g r o u n d
• ShapeBlue:
• Apache CloudStack consultancy, designing and
developing IaaS solutions for our customers.
• Also develop features and provide bugfixes – both
for the open source community as well as our own
commercial clients.
• Problem:
• Development used to be done in everyone's home labs – using
VMware workstation / Fusion / XenServer /KVM / VirtualBox / etc.
• Long build time - 2-5 hours per lab – built per FR or PR.
• ”It worked in my lab…...”
• We needed an automated, consistent and efficient mechanism to
build development and test environments.

S o l u t i o n
• Solution:
• Trillian project – a fast, flexible, and consistent environment build
framework for building nested CloudStack clouds in our own CloudStack
lab – with any configuration of application servers, operating systems,
hypervisor choice, storage options and networking we want.
• All built using Ansible – 8000 lines of code.
• Each environment build require ~2 mins of user input, environments are
built in ~20 minutes.
• Full Continuous Integration workflow:
PRs or commits to
Github
Github build
bot – Blue
Orangutan
Jenkins –
schedules
build and
testing jobs
Trillian
environment
build
Jenkins
automated
test runs
Results fed
back to
Github
Trillian
automatic
decommision

R e s u l t
• Productivity increased 100x!
• 1500 environments and 6000 VMs built
over the last 5 months
• Time saving > 4000 man hours.
• Overall great success…
• But – lesson learnt – you can quickly
become a victim of your own success.
Ensure you have the resources and
stability for sustainable growth.

Questions?

TechUG Glasgow talk 22/Feb/17 Configuration Management Best Practices

More Related Content

What's hot (20)

Similar to TechUG Glasgow talk 22/Feb/17 Configuration Management Best Practices (20)

Recently uploaded (20)

TechUG Glasgow talk 22/Feb/17 Configuration Management Best Practices