Operations Support Workflow - Rundeck
- 3. NEIL MCCAUGHLEY BUILD AND RELEASE ENGINERING MANAGER
- 4. NEIL MCCAUGHLEY BUILD AND RELEASE ENGINERING MANAGER
- 5. Source code Change Unit Testing Quality Control Packaging Deploy/Test Dedicated Test Environment Deploy Integration Environment Integration Testing Deploy/Test Pre-Production Deploy/Test Performance Publish Release Candidates Feature complete Gate Release Candidate Gate Production Deployment Production Release Gate Deploy/Test UAT Source Control Management CI and Build Tools Deployment Environments Configuration management and Deployment Testing FrameworksBinaries and package storage Code Quality Analysis Packaging Tools CI/CD WORKFLOW
- 6. Source code Change Unit Testing Quality Control Packaging Deploy/Test Dedicated Test Environment Deploy Integration Environment Integration Testing Deploy/Test Pre-Production Deploy/Test Performance Publish Release Candidates Feature complete Gate Release Candidate Gate Production Deployment Production Release Gate Deploy/Test UAT Source Control Management CI and Build Tools Deployment Environments Configuration management and Deployment Testing FrameworksBinaries and package storage Code Quality Analysis Packaging Tools
- 7. Release uses cases Source Control Update dependencies: Ivy - NuGet - Maven - Npm Update project source Request a repo Create branch (feature, release, hotfix) Tag Merge branches pre-release or release Create a repo Migrate project Git user support Git access request Crucible Project create Crucible Project create Crucible user creation Build Resolve dependencies Inspect dependencies Run unit tests Run code coverage Push updated version files Compile project Package project Run code inspection analysis Test deployment Create Feature/CI/Release/Hotfix build configuration Build support TeamCity access request Build template config maintenance Create deployment build configuration Create functional test build configuration Integration Test Trigger Deployment Trigger Local deployment Remote deployment Generate Chef cookbook Chef environment maintenance Chef role maintenance Maintain runlist Target deployment nodes Specify deployment package version Stage artifacts for Production Stage cookbooks for Production 3rd party cookbook creation Bootstrap client for deployment Chef job tool maintenance Migrate role Migrate Cookbook Chef access request Deployment support Encrypt Configuration passwords Decrypt Configuration passwords Provision server Artifact Library Retrieve dependency artifacts Publish project installer package Publish project library artifacts Artifact promotion to release repo Publish 3rd party packages Proxy Repository management Key Automated task Self-Service option Release Engineering Ticket
- 8. Release uses cases Source Control Update dependencies: Ivy - NuGet - Maven - Npm Update project source Request a repo Create branch (feature, release, hotfix) Tag Merge branches pre-release or release Create a repo Migrate project Git user support Git access request Crucible Project create Crucible Project create Crucible user creation Build Resolve dependencies Inspect dependencies Run unit tests Run code coverage Push updated version files Compile project Package project Run code inspection analysis Test deployment Create Feature/CI/Release/Hotfix build configuration Build support TeamCity access request Build template config maintenance Create deployment build configuration Create functional test build configuration Integration Test Trigger Deployment Trigger Local deployment Remote deployment Generate Chef cookbook Chef environment maintenance Chef role maintenance Maintain runlist Target deployment nodes Specify deployment package version Stage artifacts for Production Stage cookbooks for Production 3rd party cookbook creation Bootstrap client for deployment Chef job tool maintenance Migrate role Migrate Cookbook Chef access request Deployment support Encrypt Configuration passwords Decrypt Configuration passwords Provision server Artifact Library Retrieve dependency artifacts Publish project installer package Publish project library artifacts Artifact promotion to release repo Publish 3rd party packages Proxy Repository management Key Automated task Self-Service option Release Engineering Ticket MANUAL STEPS = SLOW RELEASE CYCLE, MORE SUPPORT
- 9. From a collaboration stand point, If you spend most of your day raising tickets or waiting on tickets to be completed YOU ARE DOING IT WRONG! We wanted to promote Development, QA and Operations tasks that where Automated Or when automation is not possible/desired Self Service
- 10. From a collaboration stand point, If you spend most of your day raising tickets or waiting on tickets to be completed YOU ARE DOING IT WRONG! We wanted to promote Development, QA and Operations tasks that where Automated or failing that Self Service WAIT. DOESN’T SELF SERVICE INTRODUCE RISK?
- 11. 1. Understand what's being self-serviced Making self-service safe 4. Just because you've implemented self-service doesn't mean you've reduced cost 3. Put in checks that balance processes and procedures 2. Make education part of your plan
- 12. 1. Understand what's being self-serviced Making self-service safe 4. Just because you've implemented self-service doesn't mean you've reduced cost 3. Put in restrictions that balance processes and procedures 2. Make education part of your plan WHERE WOULD WE GET THE BEST VALUE?
- 14. COMMON TICKETS
- 15. TREND ANALYSIS
- 16. HOW?
- 17. RUNDECK
- 18. OPERATIONS PROCEDURES JOB SCHEDULER SELF-SERVICE TEST ENVIRONMENTS TRIGGER AUTOMATED DEPLOYMENT
- 23. CREATING FROM TEMPLATE PROMOTES STANDARDISATION
- 24. Release uses cases Before Source Control Update dependencies: Ivy - NuGet - Maven - Npm Update project source Request a repo Create branch (feature, release, hotfix) Tag Merge branches pre-release or release Create a repo Migrate project Git user support Git access request Crucible Project create Crucible Project create Crucible user creation Build Resolve dependencies Inspect dependencies Run unit tests Run code coverage Push updated version files Compile project Package project Run code inspection analysis Test deployment Create Feature/CI/Release/Hotfix build configuration Build support TeamCity access request Build template config maintenance Create deployment build configuration Create functional test build configuration Integration Test Trigger Deployment Trigger Local deployment Remote deployment Generate Chef cookbook Chef environment maintenance Chef role maintenance Maintain runlist Target deployment nodes Specify deployment package version Stage artifacts for Production Stage cookbooks for Production 3rd party cookbook creation Bootstrap client for deployment Chef job tool maintenance Migrate role Migrate Cookbook Chef access request Deployment support Encrypt Configuration passwords Decrypt Configuration passwords Provision server Artifact Library Retrieve dependency artifacts Publish project installer package Publish project library artifacts Artifact promotion to release repo Publish 3rd party packages Proxy Repository management Key Automated task Self-Service option Release Engineering Ticket
- 25. Release uses cases After Source Control Update dependencies: Ivy - NuGet - Maven - Npm Update project source Request a repo Create branch (feature, release, hotfix) Tag Merge branches pre-release or release Create a repo Migrate project Git user support Git access request Crucible Project create Crucible Project update Crucible user creation Build Resolve dependencies Inspect dependencies Run unit tests Run code coverage Push updated version files Compile project Package project Run code inspection analysis Test deployment Create Feature/CI/Release/Hotfix build configuration Build support TeamCity access request Build template config maintenance Create deployment build configuration Create functional test build configuration Integration Test Trigger Deployment Trigger Local deployment Remote deployment Generate Chef cookbook Chef environment maintenance Chef role maintenance Maintain runlist Target deployment nodes Specify deployment package version Stage artifacts for Production Stage cookbooks for Production 3rd party cookbook creation Bootstrap client for deployment Chef job tool maintenance Migrate role Migrate Cookbook Chef access request Deployment support Encrypt Configuration passwords Decrypt Configuration passwords Provision server Artifact Library Retrieve dependency artifacts Publish project installer package Publish project library artifacts Artifact promotion to release repo Publish 3rd party packages Proxy Repository management Key Automated task Self-Service option Release Engineering Ticket
- 26. Release uses cases After Source Control Update dependencies: Ivy - NuGet - Maven - Npm Update project source Request a repo Create branch (feature, release, hotfix) Tag Merge branches pre-release or release Create a repo Migrate project Git user support Git access request Crucible Project create Crucible Project update Crucible user creation Build Resolve dependencies Inspect dependencies Run unit tests Run code coverage Push updated version files Compile project Package project Run code inspection analysis Test deployment Create Feature/CI/Release/Hotfix build configuration Build support TeamCity access request Build template config maintenance Create deployment build configuration Create functional test build configuration Integration Test Trigger Deployment Trigger Local deployment Remote deployment Generate Chef cookbook Chef environment maintenance Chef role maintenance Maintain runlist Target deployment nodes Specify deployment package version Stage artifacts for Production Stage cookbooks for Production 3rd party cookbook creation Bootstrap client for deployment Chef job tool maintenance Migrate role Migrate Cookbook Chef access request Deployment support Encrypt Configuration passwords Decrypt Configuration passwords Artifact Library Retrieve dependency artifacts Publish project installer package Publish project library artifacts Artifact promotion to release repo Publish 3rd party packages Proxy Repository management Key Automated task Self-Service option Release Engineering Ticket REDUCTION OF QUEUE SUPPORT TIME FROM 60% FTE TO 20%
- 27. Source code Change Unit Testing Quality Control Packaging Deploy/Test Dedicated Test Environment Deploy Integration Environment Integration Testing Deploy/Test Pre-Production Deploy/Test Performance Publish Release Candidates Feature complete Gate Release Candidate Gate Production Deployment Production Release Gate Deploy/Test UAT Source Control Management CI and Build Tools Deployment Environments Configuration management and Deployment Testing FrameworksBinaries and package storage Code Quality Analysis Packaging Tools
- 28. EXTENSIONS: OPERATIONS OUTSIDE RELEASE MANAGEMENT
- 33. • Drives process and automation • Improves standardisation adoption • Enables shorter release cycle • Reduced complexity • Audit and monitoring improvement • Assigns ownership • Facilitates collaboration/handover between teams Engineering Wins
- 34. • Drives process and automation • Improves standardisation adoption • Enables shorter release cycle • Reduced complexity • Audit and monitoring improvement • Assigns ownership • Facilitates collaboration/handover between teams Engineering Wins CAVEAT: DISTRIBUTED OWNERSHIP REQUIRES ACCOUNTABILITY, CONTINUOUS IMPROVEMENT AND MAINTENANCE
Editor's Notes
- #11: How do we control/mitigate risk?
- #12: We wanted to make self-service safe - Making everybody admin is a common pitfall of Devops/Self service - There's a lot of risk in implementing self-service services. - So what can be done to mitigate the risks to acceptable limits? Here are some recommendations: 1. Understand what's being self-serviced: - Just because something can be self-serviced doesn't mean it should be. - We need to consider what happens to their processes and procedures when the end user becomes the owner of the process. 2. Make education part of your plan: - People make mistakes, and just because you understand the value of complete, accurate and timely information doesn't mean your users do. 3. Put in checks that balance processes and procedures: - You can't always ensure that an access request or self-service data stewardship is being handled correctly. Self-service doesn't equal self-management. - While the operational aspects may be delegated to end users, management of these systems must still be maintained. Eg Notifications 4. Just because you've implemented self-service doesn't mean you've reduced cost: - In many cases self-service doesn't mean costs will go down in the short term. - With the overhead of managing and monitoring these services, costs may actually go up in the short-run. - If you choose and design wisely. It should reduce future spending.
- #13: How de we determine this?
- #14: Deviating from standard means that any automation of self service tooling is at risk of being out of date. Covers: - Build configurations - Deployment scripts - Source code branching policy
- #15: If Build/IT/DBA Engineers are doing the same repeatable tasks. - Opportunity for automation or self service - The best engineers are often the ones who embrace the most tiresome, least glamorous, and most "never-completed" of tasks.
- #16: Various monitoring and event notifiers like Splunk or Zabbix can tell us common failure points or common manual tasks
- #17: So we know our - Pipeline - Manual tasks - We have a guideline of how to make self-service a realistic function - How or where are we going to run these tasks
- #18: Rundeck is an - open source automation service with a web console - It provides command line tools and a WebAPI - It lets you easily run automation tasks across a set of nodes - Give remote admin connectivity through ssh and winrm
- #35: distributed ownership and accountability (requires trust, communication, and continuous improvement)