Operations in the Continuous Delivery Ecosystem
- 3. 11/8/17 1 Continuous Delivery in Operations Sumedha Ganjoo Project Manager, National Instruments What are we going to talk about? Adoption of continuous delivery principles in Operations
- 4. 11/8/17 2 What are we not going to talk about? Team Structure DevOps Vs Operations Context • Global team • Tools and infrastructure used by 200-400 devs • More than hundred thousand tests and ~50-100 submissions per day
- 5. 11/8/17 3 One year back… • Projects were long and by the time they finished, requirements or some tools had changed • Developers were running into issues and were complaining about tools • Fixes to tools couldn’t be made available to developers fast as roll outs were risky and took time
- 6. 11/8/17 4 It was time for change… If it’s painful, do it more often
- 7. 11/8/17 5 Switched to a weekly release cadence 2016 2017 Apr Jun Aug Oct Dec Feb Apr 0002 4/25/2016 0004* 5/19/2016 0006* 7/6/2016 0008 9/29/2016 0010 11/11/2016 0012 1/5/2017 0014 2/9/2017 0016 2/23/2017 0018 3/9/2017 0020 3/22/2017 0022 4/5/2017 0024 4/20/2017 0001* 4/21/2016 0003* 5/6/2016 0005* 5/27/2016 0007 8/29/2016 0009 10/24/2016 0011 12/16/2016 0013 1/19/2017 0015 2/16/2017 0017 3/2/2017 0019 3/16/2017 0021 3/29/2017 0023 4/12/2017 0025 5/4/2017 Dev Practices Tools Continuous Delivery Processes
- 8. 11/8/17 6 Dev Practices Tools Continuous Delivery Processes Key Principles • Keep the mainline healthy • Automated testing • Make core more testable • Low tolerance for failure
- 9. 11/8/17 7 Keep the mainline healthy • Release branch as close to mainline as possible Cherry-picking the fix from master Main line Release branch X is fixed quickly Feature X is bad Scenario 1 Bug Y is fixed in main line much later Feature X is bad R1 branch R2 branch Feature X is reverted from Release branch to be able to release and bug Y is filed Time of stabilizing the release branch increases. Main line z z Getting to a healthy mainline takes much longer now: • Longer the time from introduction, longer the bug fix takes • Other features build on top of buggy code Scenario 2 Automated testing • Standardized acceptance tests • Unit testing
- 10. 11/8/17 8 Make code more testable • Refactor & Re-architect • Componentize Low tolerance for failure • Quality first • Low infrastructure failure
- 11. 11/8/17 9 Dev Practices Tools Continuous Delivery Processes Key Principles • Balance long term vs short term • Minimal viable products
- 12. 11/8/17 10 Balance long term vs short term • Support vs feature requests • Short term enhancements vs strategic investments • Focused & regular prioritization Balance long term vs short term • Support vs feature requests • Short term enhancements vs strategic investments • Focused & regular prioritization
- 13. 11/8/17 11 Balance long term vs short term • Support vs feature requests • Short term enhancements vs strategic investments • Focused & regular prioritization Minimal viable products • Groom features into MVPs • Deliverable every iteration
- 14. 11/8/17 12 Dev Practices Tools Continuous Delivery Processes Key Principles • Automated pipeline • Reproducible, Reliable & Scalable Infrastructure – Infrastructure as code – Active resource monitoring
- 15. 11/8/17 13 Delivery Pipeline Practices • Only Build Your Binaries Once • Smoke-Test Your Deployments • Deploy into a Copy of Production • Each Change Should Propagate through the Pipeline Instantly • If Any Part of the Pipeline Fails, Stop the Pipeline Pipeline Manual Extensive Testing Smoke Testing Build Installers CI 2 CI 1 CI 3 SCC Pre GC
- 16. 11/8/17 14 Build Binaries Build Installers Build VMs Pipeline Pre SCC Commit & Test Staged Automated Testing Asynchronous Stages Manual Deploy Reproducible, Reliable & Scalable Infrastructure Build Binaries Build Installers Build VMs Pre SCC Commit & Test Staged Automated Testing Asynchronous Stages Manual Deploy >600 Systems >100 >400 >100
- 17. 11/8/17 15 Infrastructure as Code Services Virtual Machines Resource Monitoring
- 18. 11/8/17 16 Why should an Operations organization adopt CD practices? • Easier to adjust to changing priorities and requests • Faster turn around time on requests = Happy clients • Higher confidence in release and more autonomy • Less stress • More efficient team Benefits