VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Private Cloud- Operations Transformation
1 like367 views
The document discusses the transformation of VMware's internal operations by moving enterprise application development and testing to a private cloud, focusing on automation and efficiency gains. It highlights a significant reduction in process time from 4 weeks to 36 hours, a 20% increase in developer productivity, and an annual cost reduction of $6 million. Key lessons include investing in agility and the importance of proper blueprint management to streamline processes and improve service delivery.
VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Private Cloud- Operations Transformation
- 1. Moving Enterprise Application Dev/Test to VMware’s Internal Private Cloud- Operations Transformation Venkat Gopalakrishnan, VMware Kurt Milne, VMware OPT5194 #OPT5194
- 2. 2 Executive Summary Key Lesson Learned Invest in Agility, and Service Quality and Cost will Improve AppOps Team Deploy integrated, complex SDLC instances to support 600 developers. Challenge Process is manual, siloed, slow, unreliable. Reduces developer efficiency. Increases risk. Two Fundamentally Different Options 1. Fix the “human middleware” on traditional infrastructure 2. Replace and automate on private cloud SDDC Results From Choice to Replace and Automate on SDDC Process time – dropped from 4 weeks to 36 hours Developer productivity – increased 20% or more Project schedule risk - eliminated Annual infrastructure and operating costs - reduced by $6M annually
- 3. 3 Agenda The Challenge The Options The Solution Transforming Operations The Results Key Takeaways
- 4. 4 Corporate IT Application Group Manage portfolio of enterprise applications used by global business functions AppOps team 27 engineers Customer 600 developers Role Provision 16 different dev/test instances that include 80+ app components. Infrastructure footprint ~4,000 non-production VMs ~500 production VMs Enterprise Application Portfolio SaaS 65 IT tools 50 Business 100 Total 215
- 6. 6 Project “OneCloud” – Explosive Tenant Growth Corp IT AppOps = Tenant #4 Very low cost per VM “Cloud first” policy in IT AppOps SDLC provisioning Hands On Labs Hol.vmware.com Services & Support Customer environment reproduction Sales Engineering Demo Pods VMworld 2013 Management BU Field Testing TechSummit 2013 Tech Ops Mini R&D Cloud Training LiveFire Private Cloud IaaS Software Defined Data Center June 2012 Jan. 2013 Today End 2013 2014 Launched Built on vCloud Suite 4 tenants 10,000 VMs 9 tenants 38,000 VMs 12 tenants 50,000 VMs More services Timeline
- 7. 7 Private Cloud IaaS Software Defined Data Center Bring Your Own – Application Ops Three-tier Ops Model Different Tenants Different Application Ops Application Ops (Provided by Tenant) Now an infrastructure service consumer. Provisioning. Monitoring. Configuring. Upgrades. Maintenance. Many typical ops tasks still required. Infrastructure Ops (Provided by OneCloud infrastructure team) Network, storage, compute availability. Deliver to SLA. Tenant/Service Ops (Provided by OneCloud service team) Common service definitions, SLA, tenant onboarding, tenant management Private Cloud IaaS Software Defined Data Center
- 8. 8 Application Ops – Requirements Vary By Workload Type Ops requirements for this App… …are different than Ops requirements for this App Corp IT– App Dev/Test Provision: Complex App Stack Manage: Yes Duration: 6+ months Provision: VM Manage: No Duration: 3 weeks Sales – Demo Pod
- 9. 9 Bring Your Own – Cloud Automation and Management Tenant – needs automation and management capabilities IaaS – needs automation and management capabilities Service Manager Decides what goes in service catalog. Service Catalog Mechanism to request service. Policy Logic used to guide automation. Cloud Automation and Management Manage workloads and underlying services. Tenant 1 Tenant 2 Tenant 3 Private Cloud IaaS Software Defined Data Center
- 10. 10 Phased Project Approach Deploy automation and management capabilities Create 5+1 vDCs Blueprints for 80+ applications Service catalog with 16 instances Transition 2,800 VMs – Dev, Test, UAT Key Milestone – 4 months • 1st automated instance @ 172 hours Expand service profiles – using expanded virtual network and storage in IaaS Financial transformation – chargeback Advanced analytics, performance management Transition 1,200 VMs Stage, LoadTest Phase 2 - H1 2014 ProductionDev Test UAT Stage Load Test Phase 1 - Completed
- 11. © 2013 VMware Inc. All rights reserved Automation and Ops Transformation Venkat Gopalakrishnan – Director IT Operations
- 12. 12 Traditional Operations Functions – Provided by AppOps PeopleProcess Governance Extension via API and SDK 3rd Party Components Cloud Automation and Management vCloud Suite Private Cloud IaaS Software Defined Data Center
- 13. 13 Human Middleware Problem – AppOps Team View Global Team Management Project manage around PTO, holidays, variable skills Capacity Constrained Only 4-6 projects in parallel Slow and Error Prone Many manual steps. Ticketing systems. Human error Handoffs Silos. Globally distributed teams. Multiple application experts
- 14. 14 100% Task Automation – Not Going to Meet Needs Request Infrastructure Verification Hardware Setup Build VMs – New or Clone DNS Entries Install, Setup, Configure Workload Database Refresh Latest Code Deployment Load Balancer Entries Web Server Configuration Firewall Changes External Interface & Integration PPM Tasks Workload Monitoring Setup Security – VM access control Testing 1- 2 days 3- 5 days 2 – 4 weeks 3 – 5 days 1 – 2 days 4 – 7 days 2 – 3 days 2 – 5 days 2 – 5 days 1 -2 days 2 – 4 days 1 – 2 days 3 – 7 days 2 – 3 days 1 day 5 – 6 days Task timeWait time
- 15. 15 SDDC Eliminates Steps Request Infrastructure Verification Hardware Setup Build VMs – New or Clone DNS Entries Install, Setup, Configure Workload Database Refresh Latest Code Deployment Load Balancer Entries Web Server Configuration Firewall Changes External Interface & Integration PPM Tasks Workload Monitoring Setup Security – VM access control Testing 1- 2 days 1 – 2 days 4 – 7 days 2 – 3 days 2 – 5 days 2 – 5 days 1 -2 days 2 – 4 days 1 – 2 days 3 – 7 days 2 – 3 days 1 day 5 – 6 days Task timeWait time
- 16. 16 Automation Eliminates Wait Time… Request Infrastructure Verification Hardware Setup Build VMs – New or Clone DNS Entries Install, Setup, Configure Workload Database Refresh Latest Code Deployment Load Balancer Entries Web Server Configuration Firewall Changes External Interface & Integration PPM Tasks Workload Monitoring Setup Security – VM access control Testing Task timeWait time
- 17. 17 … and Manual Work Request Infrastructure Verification Hardware Setup Build VMs – New or Clone DNS Entries Install, Setup, Configure Workload Database Refresh Latest Code Deployment Load Balancer Entries Web Server Configuration Firewall Changes External Interface & Integration PPM Tasks Workload Monitoring Setup Security – VM access control Testing Task timeWait time
- 18. 18 Why Standardize and Automate Service Provisioning? Service Definition Blueprint Policy POC1 POC2 To Catalog Provision QA Staging Release 40 work weeks effort – Per Release… 20 work weeks effort – Once! Run Book 36 hours Service Request 4 weeks Virtual Data Center Virtual Server It takes less effort/time to convert the runbook into blueprints than it takes to “run” the runbook...
- 19. 19 Transformation – Process Challenges First version of automation solution did not meet all needs Actively deploying instances while building machines Difficulty in managing integration with SaaS apps High inflow of demand Action Automation capability – parallel provisioning Testing suite additional functions getting automated – environmental and functional Continuous process improvement in place, root cause action after every cycle Instance provisioning being treated as a ‘release’ Documentation is key to achieve predictability
- 20. 20 Total Cycle Time – Improvements 1. Re-provision instead of repair, and cross-training teams 2. Improve blueprints to drive down defects, automate functional and environmental testing 3. More automation and management capabilities Plan to get to 24 hour goal • Even more automation and management changes • Improve QA testing process Improvements Provision – 16 hours QA – 8 hours 2013 Goal 0 20 40 60 80 100 120 140 160 180 200 Test13 Dev14 Test14 Dev15 Test15 Dev16 Test16 1. 2. 3. Provisioningtime(hours) SDLC Instance - Oracle ERP with Portal (date) 05/07 05/22 05/27 06/19 06/25 07/22 08/05
- 21. 21 Process – Details Results 4 weeks to 36 hours. 24 hours (Provisioning 16 hours, Testing 8 hours) by Q4’13 Streamline demand intake process Created bandwidth to provision an instance per week Key Takeaways Automate end to end process, not focus on individual tasks Empower global team Don’t skimp on Blueprints
- 22. 22 Transformation – People Challenges New People Roles and Change in Skill Sets New role for Blueprint creation and management Automation requires global coverage to manage process Scarcity of skilled resources to perform new role Most top skills were in one location Action IT resources obtained vCloud certification Technical skills assessment, create a well balanced global team Created subject matter expertise in installation and configuration of tech stack/application Team got solid shell scripting, tasks automation and trouble shooting skills
- 23. 23 Ticketing Systems and Late Night Phone Calls – Gone!
- 24. 24 People – Details Results 27 – now 22 – goal 5 (old instances still in use) Provisioning can be initiated and executed from any of global location Employees performing high value work like blueprint management Key Takeaways Promote and help people internalize vision to get in lock step Mental shift – fix blue print and re-provision vs fix problem
- 25. 25 Transformation – Governance Challenges Functional test failures Blueprint changes resulting in manual work Lack of service definition and process to track cost per service Action Avoided changes during provisioning cycle Re-provision instead of repair Initiated programs to transform IT-as-a-Service (ITaaS)
- 26. 26 Blueprint Work – Governance Before Service Request
- 27. 27 Governance: Details Results Predictable delivery of 36 hours, targeting 24 hours by Q4’13 Improvement in functional testing, lower defect count 15 instances provisioned in 4 months Key Takeaways Spend time in Blue printing all apps, no shortcut “Disposable Infrastructure” reduce IT Capex
- 28. 28 Results Phase 1 Phase 2 Cycle Time Hours per SDLC instance 172 36 Today Phase 1 Phase 2 Virtual Machines Transitioned To Private Cloud Phase 1 Phase 2 AppOps Team # of Engineers Goal – 4000 2,800 2,200 Goal - 5 27 22 Reduced provision time 95% (4 weeks to 36 hours) Improved productivity of 600 developers 20% Reduced IT operations costs $1.5M /year Able to say “yes” to developer requests Reduced the cost of a VM/month 80% ($133 to $20) Reduced infrastructure costs $4.5M/year 672 hours (4 weeks) Goal – 24 hours Today Today
- 29. 29 Phase 1 Phase 2 Cycle Time Hours per SDLC instance 172 36 Today Phase 1 Phase 2 Virtual Machines Transitioned To Private Cloud Phase 1 Phase 2 AppOps team # of Engineers Goal – 4000 2,800 2,200 Goal - 5 27 22 672 hours (4 weeks) Goal – 24 hours Today Today Reduced provision time 95% (4 weeks to 36 hours) Improved productivity of 600 developers 20% Reduced IT operations costs $1.5M /year Able to say “yes” to developer requests Reduced the cost of a VM/month 80% ($133 to $20) Reduced infrastructure costs $4.5M/year Bottom Line Agility is Self-Sustaining
- 30. 30 Key Takeaways (Advice) Share results of early automation with developers (customers) Show how the effort will help them Training is key. Blueprint management role become key SME Help them become experts Don’t try to automate individual tasks Take holistic approach – system’s footprint view SDDC provides greater flexibility not possible with server virtualization Software controlled infrastructure
- 31. 31 Additional Resources Expert Lounge – Tues 12-1pm. Sign up in person at Expert Lounge. Group Discussion – Wed 11am – 12pm. OPT 1002-GD Related Architecture Session- Wed 3:30 – 4:30 (Full - look for repeat) VSVC4948 - Moving Enterprise Application Dev/Test to VMware’s internal Private Cloud – Architecture, Implementation and Integration Blogs.vmware.com/cloudops White paper and video/demo of this presentation IT Transformation webpage on vmware.com http://www.vmware.com/solutions/vmware-it-journey/ @vmwarecloudops #cloudops
- 32. THANK YOU
- 34. Moving Enterprise Application Dev/Test to VMware’s Internal Private Cloud- Operations Transformation Venkat Gopalakrishnan, VMware Kurt Milne, VMware OPT5194 #OPT5194