Puppet and AWS is Easy...?
5 likes1,297 views
Puppet and AWS is easy ...... ? This document discusses how Puppet and AWS were used to automate infrastructure at Healthdirect Australia. It describes how manual deployments, inconsistent environments, and a lack of automation were initially problems, and how iterative improvements using Puppet for configuration management and the AWS Puppet module for provisioning infrastructure helped solve these issues and establish continuous delivery of automated testing environments. Key aspects discussed include development best practices, testing frameworks, and integration with deployment tools like Jenkins.
![29
require => Class['docker']
docker-compose.yml.erb](https://image.slidesharecdn.com/healthdirectpowerpointpuppetcamp-151109215714-lva1-app6892/85/Puppet-and-AWS-is-Easy-29-320.jpg)
Puppet and AWS is Easy...?
- 1. Puppet and AWS is easy …… ? Cara Crawford and Scott Coulton
- 2. Who are we? DevOps/Infrastructure Manager 2 Cara Crawford • 8 + years development experience • 6 + years operational sysadmin experience • Extensive experience across website solutions in both datacentre and cloud solutions across Linux and Windows environments @cara_p
- 3. Who are we? DevOps Solution Architect 3 Scott Coulton • 10 + years hosting solutions experience • Extensive experience in architecture across AWS, Puppet, Linux, Docker and all things open source Sc h#ps://github.com/sco#y-‐c h#ps:// registry.hub.docker.com /repos/sco#yc/ h#ps://forge.puppetlabs.com/ sco#yc @sco#coulton
- 4. Who are we? 4 Healthdirect Australia manages the following healthcare service: • Healthdirect nurse helpline and health information • After hours GP helpline • Pregnancy, Birth and Baby • Mindhealthconnect • National Health Service Directory • My Aged Care • Video Call Consulting
- 5. What we were trying to solve August 2013 5 • Majority of servers located in AWS • Manual deployments • Cloning for provisioning • Inconsistent environments • No CD • No CI • No Infrastructure as code • No Automated testing • Very limited automation Deployments would take weeks or months with a high level of developer input onto cloned servers
- 6. How did we solve it 6 First Iteration – Nov 2013 • Basic Combination approach of Boto and Python for AWS scripting • Basic Opensource Puppet for configuration, server management Third Iteration – Jan 2015 • AWS Puppet module with cloud formation • Advanced puppet coding standards with testing • Container services and schedulers Second Iteration – Feb 2014 • Improvements to scripts • Additional Puppet coding standards including upgrading to Enterprise • Heading towards 100% Puppet coverage
- 7. Our Environment 7 5 development environments 3 Puppet masters Dynamically controlling the VPC environments they are responsible for Each environment the access level reducing towards Stage and Prod being automated environments
- 8. So it all looks easy, Right !!!!! 8
- 9. What have we learnt ? 9 A solid foundation - Development … Where to start ??? - Unit and acceptance testing frameworks - CD with Puppet Deploying to AWS with Puppet - Tips and tricks What we are working on now to take Puppet to the next level
- 10. Development, Where to start ? 10
- 11. If you have never written a module start here !!! 11 h#ps://forge.puppetlabs.com/
- 12. Start with a solid development foundation !!! 12 h#ps://github.com/mitchellh/vagrant A must have for any Puppet development • Quick deployments • A massive community • Supports any OS you do • You can deploy a single machine or 100 • It is as complex or simple as you need it to be • Its all Ruby
- 13. If you have not developed Vagrant framework , Here are is a good one 13 h#ps://github.com/adrienthebo/oscar
- 14. What we have done with Vagrant 14 • Re useable VagrantFile with variables • Yaml box configuraMon • MulM-‐node test environments with both open source and Puppet Enterprise • Full Hiera integraMon • We have a full replicaMon of producMon on our laptops
- 15. VagrantFile 15
- 16. Yaml backend 16
- 17. Unit testing is a must !!! 17 • Picks up human error • Tests module dependancies • EssenMal for the CD/CI process Tools for the job h#p://rspec-‐puppet.com/ h#ps://github.com/rodjek/rspec-‐puppet
- 18. What do tests look like ? 18 A simple example A complex example execute the test • rake spec_prep && rspec
- 19. But wait what don't test cover ? 19 • Package versions, If a file exists etc So we need acceptance testing • This where Vagrant comes back in Run the module !!!! Puppetlabs have a very good framework for this h#ps://github.com/puppetlabs/beaker
- 20. So now we have our working modules … How do we integrate them with CD ? 20 • r10k h#ps://github.com/puppetlabs/r10k is a good place to start. This will give you a great transport mechanism from source control to the Puppet master. But what if we are using a CD tool (ie Jenkins etc) ?
- 21. Puppet code deployment flow 21
- 22. So some CD tricks and tips we use 22 Every module has its own repo
- 23. So some CD tricks and tips we use 23 Each pipeline runs different test before it can move to the next
- 24. AWS Deployment Tools 24 … And because we are not big fans of JSON h#ps://github.com/bazaarvoice/cloudformaMon-‐ruby-‐dsl
- 25. Example of the AWS Puppet module 25
- 26. Benefits of the AWS Puppet module 26 • Anyone on the team can develop a new AWS instance, elb etc • Out of the box we have great unit tesMng coverage • On every Puppet run reverts the environment to the desired state Number 1 2p for AWS h#ps://docs.puppetlabs.com/guides/rest_api.html The HTTP API AWS SQS + Puppetlabs HTTP API = Happy engineers
- 27. 27 What we are working on to take Puppet and AWS to the next level
- 28. 28
- 29. 29 require => Class['docker'] docker-compose.yml.erb
- 30. What we are working on now to take Puppet to the next level 30 Docker scheduling with Puppet Dynamic Hiera backend and service discovery
- 31. So that was easy right ???? 31