Evolving Infrastructure
- 1. Evolving Infrastructure Louis Dunne, Platform Engineering louis.dunne@workday.com
- 2. Workday Introduction • Background / Our Own Cloud 2.0 Architecture • Config Management / Chef Build Pipeline / Chef Development Tools 3.0 Architecture • Release & Deployment Changes / Image Build Pipeline / Release Management / Planned vs. Unplanned Changes
- 3. Enterprise HCM & Financials software in the cloud First release November 2006
- 6. Workday Introduction • Background / Our Own Cloud 2.0 Architecture • Config Management / Chef Build Pipeline / Chef Development Tools 3.0 Architecture • Release & Deployment Changes / Image Build Pipeline / Release Management / Planned vs. Unplanned Changes
- 8. One of the DevOps mantras is Infrastructure is Code
- 9. If it’s code… → you need to test it → you need a build & test pipeline
- 10. Local Development Code Review C.I. Unit Tests All triggered by developers pushing code Bronze Cookbook Artefacts Local Development Code Review C.I. Unit Tests Local Development Code Review C.I. Unit Tests
- 11. C.I. System Tests C.I. System Tests C.I. System Tests Failures (back to dev) System Tests Triggered on the Hour Bronze Cookbook Artefacts Silver Cookbook Artefacts
- 12. C.I. Integration Tests Gold Cookbook Artefacts Silver Cookbook Artefacts Failures (back to dev) Integration Tests Triggered Several Times a Day
- 15. Where Do We Run Our ServerSpec Tests? • Lab Hardware? • Vagrant? • AWS?
- 16. The Lab? • Hardware Based • Can’t set machine state before the test run • Can’t reset machine state after the test run
- 17. Run In Vagrant? • Good for simple cases • Harder for integration testing a few dozens Chef roles • Prefer a hosted platform with longer running nodes for some services like artefact repos
- 20. https://www.chef.io/delivery/ https://downloads.chef.io/chef-dk/
- 21. Workday Introduction • Background / Our Own Cloud 2.0 Architecture • Config Management / Chef Build Pipeline / Chef Development Tools 3.0 Architecture • Release & Deployment Changes / Image Build Pipeline / Release Management / Planned vs. Unplanned Changes
- 22. Platform Services Appliances Workday Linux Servers Image Based Deployment Chef / Cobbler Based Deployment
- 23. ■ Cobbler for the OS ■ Chef based deployments of system / infrastructure changes ■ Custom tooling for applications deployments ■ Cobbler / Chef for bare metal ■ Most services moving to image based deployments ■ Custom deployment tools to manage VM lifecycle 2.0 Deployments 3.0 Deployments
- 24. 1. Where in the build & test pipeline do the Machine Images get created? 1. What technology & processes are used to create them?
- 25. • Early in the pipeline • Application teams → image artefact • Image artefact → build & test pipeline
- 26. • Lots of tools to choose from: • Diskimage-builder • VMBuilder • Box Grinder • Packer • Imagefactory • We use Oz (https://github.com/clalancette/oz)
- 27. KickStart File OZ Template Base Image OZCentOS Core Image Manifest Unit + System Tests System RPMs
- 29. Gold Cookbook Artefacts Gold Application Artefacts Promote to Staging UNIT SYSTEM INTEGRATION UNIT INTEGRATION I N F R A S T R U C T U R E A P P L I C A T I O N S SYSTEM Promote to Production
- 30. Image Build Service SYSTEM TESTS UNIT INTEGRATION I N F R A S T R U C T U R E SYSTEM Promote to Staging A P P L I C A T I O N S Base Image Base Image Application RPMs Application Image Application Image
- 31. The ability to push code to environments easily and quickly - push button deploys A stable framework for development, testing, deployment and auditing Deploy applications and operational environments in the same way each time Source code, tools and all components that make up releaseIdentifiability Reproducibility Consistency Agility
- 32. ■ Images ■ Startup Properties ■ Cookbooks ■ Roles ■ Data Bags ■ Encrypted Data Bags ■ RPMs ■ Other artefacts (e.g. ruby & python packages) Config Management Image Deployment
- 33. ■ Destroy and recreate rather than change in place ■ All facets of the OS are captured by the image artefact ■ Server state mutated over time as updates are applied ■ Impractical to manage every last detail of the OS with config management Config Management Image Deployment
- 34. • The ideal is… • Push all changes through the image pipeline for both planned and unplanned changes • If your pipeline is reliable with a quick turnaround you can use this for all changes • We also like to have a Break The Glass option Planned Vs Unplanned Changes
- 35. • For low impact changes only • Bash script → RPM • Agent on each VM can deploy RPMs • Trigger remotely via secure channel • Testable, easy to roll out, good for auditability Break The Glass
- 38. “If it hurts, do more of it”