Packer, Terraform & Jenkins - DevOpsBelfast March 2019

Editor's Notes

• #3: Who are Axon – we’re a leading supplier of software & services to the blue light industry, also known as public safety, our first product was the Taser, something so good it became a verb, Taser continues to be an important product for us, but we also provide a Digital Evidence Management System that’s a key element to our body camera division, and that’s where I work, I’m an SRE at evidence.com, helping build & operate the platform that hosts our services all over the world, managing video footage and other evidential material for police forces and other blue light industries all over the world.
• #7: If you google enough or have been going to infrastructure management conferences for a few years, it would be easy to think that VM image baking ios a solved problem, we’ve been talking about it for ages, there are tools that do some of the essentials, but in my opinion, there was still a lot left as an exercise for the student. I started a new job recently, and we have a infrastructure that’s state of the art for the year 2010, terraform for cloud based infrastructure, puppet & salt for long lived VMs configuration management, but this leads to pain & bad practices, like manually resizing VMs & editing the terraform to keep sync, adding VMs to cope with a demand spike is painful, removing them is even more painful, you've got puppet certs, manual DNS, discovery via hieradata, all solved problems in some way.
• #8: Nope. Many of our services are moving to k8s managed containers, but not everything will, for good & bad reasons. There are also some environments & companies where they’re just not interested in the k8s & PaaS overhead, they just want somewhere better to run their LAMP stack apps that fits with their appetite for change.
• #9: As my new work environment was a little cumbersome to innovate & iterate in, I decided to do a POC outside our internal constraints & came up with a model of a simple linux web app stack & concentrate on what tools I would need, what the Jenkins pipeline might look like if I wanted to build & test on a PR branch and deploy to production from master.
• #11: Back to what triggered this in work, I was delighted when I found a GitHub repo in our org called ops/packer – yes I thought, somebody has already done the hard work, we had layered packer templates to build out various things, TeamCity build agents, base images for CentOS in our 2 main cloud providers. Then I looked at the AMIs & VHDs in our accounts – I had no idea where they came from, which template built them, from which branch, when? I started googling again, Netflix’s aminator has some hooks for some of that, but nothing concrete, and aminator is strictly AWS, we’re 90% Azure.
• #12: The next challenge I had was proving that an image was actually good – how do I know that I haven’t just broken something or broken some security policy by tweaking something, or worse, an assumption on an installed package & that installed package changed a default, Chef’s InSpec tools allows you to do much of that, it’s also great at ensuring that you’re in compliance with security baselines, either internal or external (CIS), I’m not an InSpec expert, but I think we have a couple of Chef employees in the room who are actually paid to work on InSpec, so you should find them & poke them for more details if you’re interested.