Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides

Rise of the Machines
Continuous Delivery at SEEK
CO PRESENTED BY:

Scorched Earth Deployments
● Create a pristine environment for every release.
● No need to migrate state of environment.
○ ∴ Configuration Management tools not
required.
● State should not be changed at runtime.
○ ∴ Immutable Server
● Blue/Green deployment strategy:
○ Monitor health during deployment.
○ Auto-rollback if new environment is unhealthy.

Automate all the things!
● Delivery team is responsible for the automation of:
○ Server installation
○ Infrastructure creation
○ Deployment
● Automation is impossible without team managed
infrastructure.

Infrastructure as Code
● SEEK use AWS CloudFormation to automate their infrastructure.
● Operations are responsible for the network-level infrastructure,
including:
○ VPC
○ Routing
○ Users and Roles
○ Shared services
● Delivery team is responsible for infrastructure within the VPC, such
as:
○ Elastic Load Balancer
○ Security Group
○ Auto Scaling Group
○ Launch Configuration
○ Instance Profile

Server automation
● Provisioning Linux:
○ SSH ⇒ Easy!
● Provisioning Windows:
○ WinRM ⇒ Hard!
○ AWS makes bootstrapping easier:
■ EC2 User Data
■ cfn-init
○ Packer Windows Plugins can be used to create Windows Amazon
Machine Images
● Combination of Packer and CloudFormation can be used regardless
of target platform

Packaging
● Applications should be self-installable.
● Application installation should be separate from server automation.
● Installation should be independent of Continuous Deployment tool.
● Native package manager is ideal solution:
○ Linux: apt-get or yum
○ Windows:
■ One-Get (available in next version of Windows)
■ Chocolatey (available now!)
● Package manager is platform independent pattern
○ <package-manager> install <package-name>
● Windows Coding Horror:
○ Interacting with Windows resources is not trivial, custom
PowerShell modules or PowerShell DSC Resources make this
easier.

Pre-Baked Images
● Install a package on a base image.
○ But… don’t include configuration.
● Then, create an image (AMI).
● AMI is our environment independent deployable
artifact.
● Future is containerised...
○ Docker is the next evolution of our approach.

Late-binding of context
● Keep your artifact environment agnostic:
○ Frameworks want you to apply configuration at build or
package-time.
○ Deploy tools want to apply configuration at install-time.
○ When deploying to the cloud, you should apply configuration at
launch-time.
● When AMI is launched it doesn’t know where it is:
○ Need to inject environment-specific context at launch-time.
○ How do you do this?

Environment Variables
A platform independent way of specifying environment-specific config that’s
been around since 1979
● Use environment variables for:
○ DB connection parameters.
○ Location of service endpoints.
● Don’t use them for:
○ Altering runtime behaviour.
● Overriding .NET configuration with environment variables:
○ .NET 4.5.x: write your own Configuration Provider.
○ .NET 4.6: works out of the box.
○ Azure + .NET 4.5.x: works out of the box.
● Service discovery is on the horizon:
○ SEEK is investigating.
○ Externalised config is first step.

Confidence to Release
● Put enough steps and checkpoints in pipeline to get enough
confidence to release.
● Too many checks will result in a slow pipeline and kill the benefits of
Continuous Delivery.
● Take things away until you cry.
○ Then... compensate some other way
○ For example, consumer-driven contract testing makes sure we
honour API contracts without the need to integrate.

Ship It!
● Distinction between Deployment and release:
○ Deploy: create Blue environment and deploy application.
○ Release: cut-over from Green to Blue environment.
● Smoke test in Blue environment before release using synthetic
transaction.
● Same smoke test can be used to monitor health after release.
● Choose to enable manual release if change set is risky and additional
manual testing is required.

Towards Polyglot Architecture
● Continuous Delivery + Microservices gives teams the ability to choose
the right stack for the job.
● For SEEK, CD is opening up the possibility migrating legacy apps from
Windows to Linux.
○ Who knows, in the future, running Windows containers will be a
viable alternative.
● SEEKs approach to Continuous Delivery provides better logical
separation automation steps.
● Time to first commit still a problem and this will only get worse with
the proliferation of different stacks and tools.

Manual Development Environments
■ Are a bad experience for your developers.
■ Are time consuming and therefore costly.
■ Make it difficult for devs to move between projects.
■ Are rarely accurately reconstructed.
■ Encourages manual configuration rather than the more
reliable and repeatable practice of Infrastructure-as-code.
Can lead to environments that have all of the parts, but not quite
in the right order.

Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides

“It works on my box”
■ Sign of environmental divergence.
■ The Butterfly effect: Minor differences in environment
configuration can produce obscure bugs in Prod.
■ This issue is compounded when deployments carried out by
another person (e.g. Ops).

Dev vs Production Parity
■ Environment Parity = Confidence.
■ Overcompensate lack of parity with more, possibly manual tests.
■ Environment Parity…
○ enables continuous delivery .
○ requires diligence and automation for accurate replication.
○ encourages infrastructure-as-code.

Wouldn’t it be great if we...
■ Could use declarative formats for setup automation, to
minimize time and cost for new developers joining the
project.
■ Had a clean contract with the underlying operating system,
offering maximum portability between execution
environments.

Wouldn’t it be great if we...
■ Built apps suitable for deployment on modern cloud
platforms, obviating the need for servers and systems
admin.
■ Could minimize divergence between development and
production, enabling continuous deployment.
■ Could scale up without significant changes to tooling,
architecture, or development practices.

The “12 Factor App”
■ These 5 principles are what loosely describe the 12 factor
App, a guideline for modern SaaS products.
■ 12 Factor Apps, Factor #10: Dev/prod parity
■ “Keep development, staging, and production as similar as
possible”.
■ Vagrant can help us solve the environment parity issue.

Vagrant, Docker Compose &
Friends
Onward to immutable development environments

Scorched Earth Dev Environments
■ Dev automation typically come in 2 flavours:
○ Use of CM Tools like Puppet, Chef (Boxen, Babushka)
○ Virtual Machine Runners (Vagrant, Docker Compose)
■ CM Tools fraught with challenges:
○ Not isolated
○ Hard to automate across OS/Platforms
○ Doesn’t force developers to automate

What is Vagrant?
“A tool to help create and configure lightweight, reproducible, and
portable development environments.”
■ A replacement for manually configured boxes.
○ Enforces infrastructure-as-code.
■ A machine runner (typically a VM or a Container)
■ Separates authoring environment from development runtime
■ Isolated & repeatable

What is Vagrant?
The following is a logical view of how Vagrant interacts with your
authoring environment:
■ Vagrant has spun up Virtualbox VM with the name ‘Web’,
and is running it on a Mac OSX desktop
■ Code is synchronised from /c/proj/x -> /vagrant of
the running environment

What is Vagrant?
■ The Web application spins up a server using the code in
/vagrant, which binds to a private network on the VM on
port 80
■ Port 8080 is exposed to the outside ‘world’ (in this case, a
Mac OSX desktop) which forwards requests to port 80 in the
running VM
■ The Developer is free to author in whichever IDE of her
choosing

The TAO of Vagrant
■ vagrant up to get a fully working dev runtime.
■ Development occurs in preferred IDE: Visual Studio,
Sublime, ViM, with your terminal, your shortcuts - etc.
■ vagrant destroy && vagrant up will destroy the
machine and create a new, clean, isolated environment.
■ Workflow repeatable and transferrable to other projects.

Developing on Ecosystems
From monoliths to ecosystems

From Monolith to Microservices
■ Applications are usually more interesting than a single UI,
they are usually connected to things.
○ They form part of a functioning ecosystem.
■ Vagrant good for Monoliths and ecosystems.
■ Explosion of microservices highlights this need.

Commentary on Architecture:
■ Multiple components, in disparate technologies
■ Distributed system
■ Layered architecture
■ = complexity
Increased complexity increases the number of things that can
go wrong. We can combat this with parity and local simulation.

■ Vagrant can replicate ecosystems locally, while still having
the benefits of isolation, repeatability, immutability etc.
■ Running ecosystems locally gives us flexibility.
■ Vagrant can simulate or test a number of scenarios, for
example to:
○ Model distributed systems behaviour
○ Perform disaster testing; outages, network partitions ...
○ Replace components independently
○ Test OS/DB upgrades
○ Run entire production replica ecosystems, locally

Local Ecosystem Setup
The following is a Logical View of this fictional ecosystem with
Web and API applications exposed to the ‘World’ and a database
hidden ‘behind the firewall’:

Local Ecosystem Setup
■ Note that we are running multiple virtualisation
technologies here as if they were all running locally
■ Vagrant will spin up 3 local VMs:
○ Web and API applications exposed to the ‘World’ and a
database box hidden ‘behind the firewall’:
■ The 4 VMs are able to communicate through a private
network on their native ports (80, 80 and 27017
respectively).

Native to Docker and is fast, but some caveats:
■ Unless on linux, networking is not automatically exposed to
the host machine.
■ You need to ensure that your apps are compiled/built within
the Docker containers - otherwise you’ll still need to
bootstrap your dev box anyway!
■ Hybrid VM/Docker setups not supported.
Docker Compose

Machine Image Pipelines with
Packer
Where environments are born

Cattle in Production, Pets elsewhere
■ Dev == Prod - but what about our build/other servers?
■ We don't want cattle in production and pets elsewhere.
■ Quality starts before coding begins
○ Must think about environments we develop, build and
deploy to.
■ How do we ensure other environments are configured
identically?
-> Immutable servers everywhere!

The Machine Factory
■ How do we go about creating and managing all of these
machine images?
■ At SEEK, we needed a way to generate all of these machine
images in a sensible, standardised and organised way.
■ Enter: The Machine Factory:
○ “A process for generating machine images for use in
different contexts such as development, test, build, prod
and so on.”
■ So what does this look like?

Machine Factory
■ Assembly-line style manufacturing:
○ Input images (AMI, ISO, Docker Image etc.)
○ Apply Build Steps with shared infrastructure code, using
CM tools, shell scripts etc.
○ Output Images (AMI, ISO, Docker etc.)
■ Images put in stasis for their destined environment/context
where they will be reanimated.
■ CM tools as a code smell:
○ Your application might be too big. For this step,
lightweight options usually suffice.

Tuning the assembling line: Pipelines
■ Booting machines and provisioning from scratch is slow.
■ 2 main strategies: Intermediate images & enrichment.
■ Create intermediate images to shortcut future builds:
○ Install common infra such as log forwarders, statsd, AV,
patches/updates etc.
○ Engineering wide (Platform), team wide (Base) or specific
Application images.

Tuning the assembling line: Enrichment
■ Prod should have only the runtime dependencies
○ Better Security.
○ Better Performance.
■ Dev will have the most. This can be expressed as:
Prod ⊊ Build Server ⊊ Dev
■ Enrich machines from left to right in your pipeline, feeding
previous image into next step.

Machine Factory @ SEEK
Walking the walk

Applying CD to Talent Search
■ Talentsearch previously part of .NET monolith.
■ Talent Search is a web-based tool that Recruiters use to
connect with candidates, and was previously part of the
monolith Warner discussed earlier.

Carving out the relevant bits
■ Isolate scope, cut out vertical slices from the monolith.
■ Enables us to Focus our automation efforts.
■ Broadly this is achieved using 2 approaches:
○ Mocking out downstream APIs for local dev.
○ Consumer Driven Contracts, using Pact.
■ These give us isolation and speed.

Automating Windows
■ Windows tooling support in past year is much improved.
■ AWS & Vagrant all have first class support.
■ Seek have contributed a number of plugins for Vagrant to
make Windows development easier,
■ ...a number of useful chocolatey packages,
■ ...and a large suite of DSC resources to assist configuring
Windows machines.
■ And then there is Packer...

Automating Windows: Packer
“A tool for creating identical machine images for multiple
platforms from a single source configuration”
■ Packer is the core of our Machine Factory.
○ It will boot a machine from an image such as an ISO
○ Configure it with scripts or CM tools.
○ Export it into multiple formats - such as a Vagrant box,
Docker image or an AMI.
■ Uses a simple JSON DSL to easily comprehend and visualise
the steps applied in the manufacturing process.

Automating Windows: Packer
■ Packer uses SSH, which is very fragile on Windows.
■ SEEK have helped create a port of Packer to communicate
natively to Windows over WinRM.
■ The result is Packer Community, a set of plugins for Packer
that make Windows machine automation a breeze.
■ Using Packer we have automated:
○ our local dev boxes in Vagrant,
○ our build/test/prod servers in AWS,
○ ...across Linux, Windows Server and Mac OSX.

#winning
■ Dev env < 10 mins for .NET API + UI
■ Dev -> Build -> Prod from Factory
○ Vagrant: Vbox, Parallels, Mac +
Windows
○ AWS
■ Commit to Release in 7 mins.
■ Proven ‘vertical slices’ strategy.

Bonus: Challenges &
Lessons on Windows
Tips on effective use of Vagrant, Packer and DSC.

Vagrant
■ Other authoring environment dependencies = anti-pattern.
○ Installing VM provider (e.g. Virtualbox) + Vagrant should
be the goal
○ Consider scripting the install (DSC, BoxStarter,
Powershell scripts etc.) if you can’t avoid.
■ Shared folders and IIS Applications don’t play nicely
○ Use rsync synced folder type instead
■ When provisioners get too big/slow, move into a Packer
script - vagrant up should be fast and reliable

Vagrant
■ Always set $VAGRANT_HOME on multi-partitioned machines
due to disk space restrictions (n+1 space requirements)
■ Distributing Images
○ Use Versions + Atlas, if possible
○ S3 also good option
■ GUI Testing (e.g. Protractor tests)
○ Consider having a GUI fallback (at the cost of slightly
bigger images)
○ X-Forwarding on *nix hosts not ideal

Vagrant
■ Cross-OS concerns do occasionally come into play:
○ NPM for example has issues writing to a linux
networked file system where the underlying host is
Windows.
■ Domain access can be problematic and increases
complexity - avoid if possible.
○ Use Vagrant Windows Domain plugin if you can’t.
■ Use multiple NICs with private_network if application
requires multiple SSL certificates.

Vagrant
Box cleanup:
■ Clean up after yourself and remove any
binaries/installers/cruft not required for day-to-day use
■ Defrag and compress virtual disk images if possible
■ Matt Wrock has a number of good tips on how to cut it right
down.

Vagrant
Build .NET apps in Vagrant without Visual Studio:
■ Install MS Build Tools and VS Agents
■ Package reference assemblies from VS installation into
guest machine.
Install the remote debugger and port forward on 4018 to debug
from Visual Studio

DSC
■ Expect pain/effort, it’s in its infancy and is immature.
■ Error messages can be very unhelpful/confusing, and you
should be prepared to get comfortable with Windows
debugging in the event viewer and the CLI.
■ IMHO, DSC is still a better choice than the *nix open source
alternatives such as Chef/Ansible (although support is
improving).

DSC
■ Use the Vagrant and the DSC Plugin to iterate on a box;
Vagrant allows you to test your DSC configurations in a
clean environment and test any potential issues with
conditions/ordering, without polluting your host
environment
■ Prefer the feature rich SEEK-Jobs/DSC community resources
over the Powershell team (experimental) xModules
■ Avoid the Script resource - It is rarely a good idea to use it.
Write your own custom resource instead.

Packer
■ Don't try to do it all in one go. Build complexity
■ When building a new image, create scripts, run them
manually and then automate once you’re comfortable they’
re working as expected.
■ Iteration times can be slow, particularly when booting and
installing an OS
○ Create intermediate boxes to build off to avoid that
overhead.
■ Using Vagrant to test those scripts is often a good starting
point.

Packer
■ Sysprep Windows machines so licenses don’t expire.
■ If you’re creating a Windows image, Use Packer Community
(only if you can’t use Linux! :)).
■ Don’t forget to disable Windows Updates (at least for dev
machines).
■ Windows boxes often need restarting, use the restart-
windows Packer Community provisioner to do so safely.

Packer
■ Build Packer images in pipelines - feed output of Packer
runs into the start of another Packer run.
■ For example, use the Virtualbox ISO builder to boot and
install a Windows machine with updates applied.
■ That box can now be fed into the Virtualbox OVF builder
which runs the application specific provisioners (typically
this is the more error prone step).

Thank you
@SEEK_Geek
CO PRESENTED BY:
@terrencebenade
@dius_au @warnergodfrey
@matthewfellows

Resources
■ Vagrant (http://vagrantup.com)
■ The TAO of Vagrant (http://mitchellh.com/the-tao-of-vagrant)
■ Vagrant DSC Plugin (https://github.com/mefellows/vagrant-dsc/)
■ Vagrant Windows Domain Plugin (https://github.com/SEEK-
Jobs/vagrant-windows-domain)
■ DSC (http://blogs.technet.
com/b/privatecloud/archive/2013/08/30/introducing-powershell-
desired-state-configuration-dsc.aspx)
■ Read the free DSC eBook (https://onedrive.live.com/?
cid=7f868aa697b937fe&id=7F868AA697B937FE!107) for a small but
comprehensive guide to DSC

Resources
■ SEEK-Jobs/DSC resource (https://github.com/SEEK-Jobs/DSC/)
■ Setting up a server for use with DSC (see packer templates https:
//github.com/mefellows/packer-community-templates)
■ Packer (http://packer.io)
■ Packer Windows Plugins (aka Packer for Windows https://github.
com/packer-community/packer-windows-plugins)
■ Sample Windows Packer Templates (https://github.
com/mefellows/packer-community-templates)
■ Legacy Windows Packer Templates and provisioning scripts (Uses
OpenSSH, however there is plenty of inspiration to be found -
https://github.com/joefitzgerald/packer-windows)

Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides (20)

More from DiUS (20)

Recently uploaded (20)

Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides