Private Cloud Deployment Automation with XebiaLabs and OpenShift
Download as PPTX, PDF2 likes563 views
The document outlines the automation of private cloud deployment at KeyBank using XebiaLabs and OpenShift in response to operational challenges. It details the transition to Docker containers, Jenkins for continuous integration, and the implementation of a robust deployment pipeline which improved efficiency significantly. Key metrics include increasing deployment frequency to over 20 times a day and executing thousands of tests per build, all aimed at enhancing performance and reliability in banking operations.
Private Cloud Deployment Automation with XebiaLabs and OpenShift
- 1. Private Cloud Deployment Automation with XebiaLabs and OpenShift Tom Larrow DevOps Automation Engineer at Key Bank
- 2. 2 Corporate Bank Business Units • Real Estate Capital • Treasury Management • Derivatives/Foreign Exchange • Investment Banking • Equipment Finance • Institutional Asset Services • Broker Dealer • Pacific Crest Research • One of the Top 25 largest banks in US • Headquartered in Cleveland OH • $136 Billion in Assets • $5.8 Billion in Revenue • 20,000+ Employees • 2 Datacenters in US • 8 consecutive Outstanding ratings for lending under the Community Reinvestment Act • 4 times named a Best Place to Work for LGBT Equality by the HRC About KeyBank
- 3. Impetus for Change • Major outage revealed complexity of systems • One online banking login required 190 Network Hops across 2 datacenters • Lack of Standard Configurations and Automation allowed only quarterly releases • All hands on deck 4 times a year • Acquisition of First Niagara • Needed to update online banking sooner than anticipated 3
- 4. Updating our Platform • Moved to Docker containers for immutable infrastructure • Dramatically decreased time to provision a server • Run inside the same container that was tested • Kubernetes for container orchestration • Insure containers always up, in optimum amount to handle load • Technology that powers Google • Red Hat OpenShift • Red Hat supported version of Kubernetes • Self Serve PAAS abilities 4
- 6. Jenkins Continuous Integration for Dev 6 • Went from deploying to lower environments every few weeks to 20+ deployments a day. • Over 6,000 tests executed on every commit insure quality • Deploy to OpenShift via rolling deployments • Separate jobs to push images to higher environments
- 7. OpenShift Rolling Deployments • OpenShift has image triggers, when a new Docker image is pushed into the registry it will perform a rolling deployment 7
- 8. Early DevOps Success 8 1200 12 Test Execution Time (Minutes) 84 195 Scenarios Legacy Test Automation New Test Automation *Still Required Manual Testing *Executed Every 5 Minutes • During highest commit phase Key was identifying 10 defects / day • Averages roughly 7000 tests executing per build • 60% defects resolved within a day Bank Acquisition Customer Day One 10 # of Production Releases to Respond to Customer Input in the first 4 days 0 # of Defects Resulting in Production Releases Consistent Release Process that Guarantees Versions are Deployed in a Dependable Automated Manner
- 9. Maturing Required Changes • Jenkins Pipeline allowed for great CI and delivery velocity • Also exposed weaknesses in the system • Things accidentally made it to an environment, and from there were pushed to production • As a bank, needed more audit data • Needed better rollback abilities • Needed a full Enterprise Tool 9
- 10. Updating and going beyond CI Pipeline • Increased parallelization and tests run • Built XL Deploy packages, and used them for Deployments 10
- 11. Better Control of Deployments with XL Release and XL Deploy 11
- 13. OpenShift XL Deploy plugin • Existing community plugin used “oc new-app” and “oc delete” to perform deployments • During POC wrote our own plugin from scratch to do deployments • Because access to source code available, could add new type which performed rolling deployment • Changes merged into XL Deploy OpenShift plugin version 7.1.0 13
- 14. Using the Plugin • Jenkins builds Docker images and pushes into a registry tagged with a unique identifier, build number, commit hash etc. • XL Deploy package created with this image as the DockerPushApp type 14
- 15. How the DockerPushApp type works • Issues “docker pull” command to pull tagged image out of the registry • Retags the image with name of image running in OpenShift • Pushes image into OpenShift registry • This triggers OpenShift rolling deployment • Uses OpenShift API to monitor deployment and report status on completion 15
- 16. Next Steps • Update the plugin to allow the deployment of: • Deployment configurations • Change the number of pods running in each environment • Change the Readiness/Liveliness probes • Configuration Maps • Store data that gets injected into the pod for each environment • Secrets • Protected Config Maps for passwords and other sensitive material 16
- 17. 17 • Don’t spend too much time planning the perfect system, it will change as you learn. • Don’t ever think you’ve mastered something. Always a new and better way to do it. • Don’t forget to have fun. • Do question ”why” each step of your process exists. How can you streamline it? • Do, and fail. It is how you learn what works and what doesn’t. • Do get the feedback to the developers as soon as possible. Do’s and Don’t’s