Domain's Robot Army

Editor's Notes

• #3: We are Domain.com.au One of Australia’s leading property classified websites About four million Australians check us out every month We’ve been around since ‘99 We’re historically a windows shop with a big, monolithic .NET application, but we’re currently in the middle of a huge project to overhaul almost everything we do, tech-wise And who am I? I’m Jason Brown, the DevOps lead at Domain. I’m the field marshall heading up the Robot Army. I came on board about six months ago, and it’s my job to drive the ‘ops’ side of DevOps, and to essentially automate myself of one job and into another, every few weeks. I was recently joined by a couple more skilled engineers, who make things significantly smoother, which means I can come out and talk to groups like this
• #4: We’ll cover the CF and Powershell needed to drive the Robot Army Supporting Code: We’re still mainly a Windows shop, so parts of this will be windows-centric, but the core concepts apply to any platform We’ll mention the CI and CD pipelines, but not in depth do jump in if you have questions, it’ll slow me down and stop me skimming over important points
• #5: Point one: we’re completely rearchitecting our software on a Microservice architecture, as well as migrating a number of other apps into the Cloud. This means our capacity planning is somewhat at the ‘guesswork’ end of educated guesswork We could respond to this in a couple of ways We could just guess We could over-engineer so that we never hit our limits We could leverage autoscaling capacities in AWS to make sure we’re at optimum scale all the time - which could also deliver us significant cost savings We may also incur a significant management overhead, with many different services spread across many different servers, so we need to automate as much as we can. Point two: We’re doing this with a continuous delivery model backing it up So we can’t predict what code will be on a given node on a given day. Which means our provisioning pipeline MUST integrate tightly with our delivery pipeline Point Three: we’re super-agile, and we run light. we need to do this stuff quickly. If we need a new environment, we don’t want to wait a week, but our ops team is very lean Point Four: We’re on Windows. There aren’t that many shops doing microservices in the cloud on Windows Platforms, and those that do aren’t leveraging every feature they possibly can There are lots of companies out there doing cloud. There are lots who use autoscaling. There are lots who do automation. And there are lots trying to implement CD. We don’t know many that are doing them all. The point is though, it’s combining continuous delivery with microservices that is the tricky part. We could be throwing code out literally hundreds of times per week, so making sure that our auto-scaling clusters are always on prime code is a difficult process, and we’ll touch on that towards the end because we think we nailed it
• #6: The Robot Army refers to our entire AWS server fleet it’s made up of smaller Robot Platoons - essentially microclusters consisting of an auto-scaling group, an ELB and two or more EC2 servers, spanning two availability zones. This is a fairly standard configuration - it’s what you see in, for example, Elastic Beanstalk containers. But we’ve added a whole lot of smarts under the covers, driven from the CloudFormation template
• #7: Worth mentioning on the constantly expanding bit that ELB tagging appeared YESTERDAY (30th Sept) along with the ability to update SNS topics - both of which have been “want” items for a while now.
• #8: Will drop this slide if the projector is terrible. CLoudformation is very texty so it’s not great for a slide presentation, but let’s see if it works Now this will be terribly hard to read but these slides will be available later, I just wanted to show a basic example of a CloudFormation resource With cross-zone load balancing, your load balancer nodes route traffic to the back-end instances across all Availability Zones. By default the CrossZone property is false.
• #9: https://s3-us-west-2.amazonaws.com/cloudformation-templates-us-west-2/AutoScalingMultiAZWithNotifications.template Why does every cluster have its own template and config script? Well, the template is actually pretty-much generic. But when we started this project, we weren’t sure how much variance would be needed between individual platoons. As it turns out, the robots are far more generic than we expected them to be Still, by maintaining separate templates for now, we also maintain a separate revision history for them in git, and we can see where we’ve made changes The intention is, eventually, to drive ALL our stacks - of which there will be many - from ONE CloudFormation template
• #10: This is a vital link in the chain because it’s where CloudFormation meets the Operating System. IN essence what we do here is write a lot of powershell commands and supporting files onto the disk from S3, then execute them Three key sections within AWS::CloudFormation::Init “files: : {} writes a few files to the disk, a couple of which came with the template "commands" : { } contains five powershell commands in total “services”: {} contains the standard cfn-hup It’s a bit tricky to go into depth on this LaunchConfig, but at some point in the future there’ll be a detailed blog post on this and we may even end up open-sourcing it, though we’ll go into the powershell on the next slide
• #11: cluster-config.json persists some of the parameters we’ve passed to the CloudFormation template onto disk, for later use - it’s written from the CF launchconfiguration directly, using parameters pushed into the CF template the common config script installs New Relic, The Sysinternals Suite, some DSC and Powershell modules we want common to every node in the entire army The cluster config DSC script is currently unique per-cluster, and allows us to make significant variations between individual setups. So if one cluster out of ten requires, say, MSMQ, or CGI support, or we can add it here with DSC. It allows the army as a whole to have minimal configurations across the entire army, with some platoons having extras here and there as needed. We could make quite radical changes here if we wanted (but we don’t) All of this is itself driven by a powershell script back at home base, making it an easy one-liner to stand up one of these clusters There’s then a final script, generic across the entire army, called cluster-deploy.ps1 - this picks up cluster-config.json and uses that to find the latest app code from…. <next slide>
• #12: This is the final piece in the puzzle. It’s no good having all these beautiful auto-scaling doohickeys if you end up deploying stale code to them - and “stale code” for us can mean literally minutes out of date This seems to be what makes continuous delivery and Auto Scaling daunting for some organisations - how do you get your latest code onto the servers as they scale (or indeed self-heal)? In linuxworld there’s things like Puppet, chef, mcollective etc. You could roll your own solution, perhaps using powershell and S3 (as we occasionally do to bring a platoon online for the first time) You could bake AMIs somewhere in your Continuous Integration pipeline, so your app code is already on-board. You could use EBS snapshots to attach an entire drive of app code. We chose to be more lean and flexible, so we went with OctopusDeploy To be honest, if it weren’t for the continuous delivery aspect, if we were still deploying only twice a week, then AMI baking or pulling zipfiles from S3 would be fine for us. But we want the ability to deploy many times a day, either automated or manually, and we want pretty much anyone in the organisation, from a first-day developer to the Technical Director, to be able to deploy code. Not that we want that happening a lot, but we want it to be possible. So Octopus it is. (and it’s good). We found we can drive many of the variances between platoons from Octopus, and therefore simplify the build pipeline
• #13: CFN-Init not loading profiles doesn’t appear to be a problem at first, until you need to use, say, the Data Protection API in windows. Octopus needs this to register its server agents (called tentacles). We eventually got round it using a DSC module for Octopus which was not even two weeks old when we found it. DSC always load a user profile. Good tip to know. There’s a balance to be struck between either having lots of parameters in your master template, or having lots of templates, and there are risks on either side. Finding the sweet spot is tricky. We’re starting with one and moving towards the other Note: multi-tenancy is tricky, but we’re also deliberately isolating our services - other organisations may want to multi-tenant, in which case they’d need to build some extra smarts. OctopusDSC, while it saved our bacon quite well, had to be modified. By default, it uses Amazon public IP addresses. Because we’ve designed the robot army to be highly secure, none of the individual soldiers has a public IP. So we changed it to use private IPs and sent the code to Octopus for integration into the product - yes, windows shops do open source too. And finally, autoscaling means a box could be terminated at any time, but Octopus does not automatically de-register dead servers. So we had to roll our own code to handle the cleanup. Luckily: API-first, so easy to do with powershell.
• #14: You can have your cake and eat it, CD-wise Without Cloudformation, this would be significantly harder We will be open-sourcing components of this pipeline, if not the whole thing
• #15: Follow domain’s tech blog, including career opportunities.