VMworld 2013: Part 2: How to Build a Self-Healing Data Center with vCenter Orchestrator
0 likes1,871 views
The document outlines how to build a self-healing data center using VMware's vCenter Orchestrator, detailing its features, integrations, and real-world use cases. It emphasizes the automation of infrastructure remediation, provides examples of customer implementations, and highlights the role of vCenter operations in monitoring and addressing operational issues. Key takeaways include the benefits of automation in IT processes and the importance of customer experiences in driving these initiatives.
VMworld 2013: Part 2: How to Build a Self-Healing Data Center with vCenter Orchestrator
- 1. Part 2: How to Build a Self-Healing Data Center with vCenter Orchestrator Nicholas Colyer, Catamaran RX Dan Mitchell, VMware VCM5695 #VCM5695
- 2. 22 Session Agenda vCenter Orchestrator Overview: A quick look at vCenter Orchestrator platform VMware Example: vCenter Operations Manager Remediation package - using vCenter Orchestrator and vCenter Operations Manager Customer Example: Real-world use cases addressed by one customer using vCenter Orchestrator Partner Example: vCenter Orchestrator plugins by partners like EMC address common use cases for remediation
- 3. 33 Key Takeaways Advice, considerations and implementation tips for real-world use cases Understand the concept of the self-healing data center and how vCenter Orchestrator supports it1 2 3 Hear from a customer regarding their experiences today and how they will continue to take advantage of vCO remediation capabilities
- 4. 44 VMware vCenter Orchestrator Product Overview
- 5. 55 vCenter Orchestrator Overview Features Drag-&-drop design • Create powerful workflows easily by drop-&-dragging pre- built actions Cloud scalability • Execute hundreds to thousands of workflows in parallel to meet cloud scale Flexible triggers • Launch workflows from the vSphere Web Client, vCloud Automation Center, browser, schedule, event, and API Automate VMware • 100% coverage of vSphere and vCloud APIs • Unmatched VMware content Key Benefits • Integrate VMware solutions into your IT environment and processes • Reduce IT OpEx and total cost of ownership of VMware solutions • Automate your cloud and accelerate transition to “IT as a Service” model Platform Plug-insEcosystem vSphere • Included with vSphere at no extra cost • Installed with vCenter OOTB Included with vCenter Server Fully Integrated with vCAC • Trigger vCO workflows from vCAC • Use vCO to configure and extend vCAC
- 6. 66 vCO Workflow Designer • Drag and drop actions • Conditional logic • Pause, wait until, counters, etc. • Exception handling • Version control • Role-based access control • And more ... ~500 workflows and actions for vCenter Server and vCloud Director
- 7. 77 • Windows • Mac & Linux Designer • SOAP • REST Web ServicesOperator • vSphere Web Client High-level vCO Product Architecture • Oracle • MS SQL Server • PostgreSQL Workflow Library Webview Library Workflow Engine … vCO Platform (Access points) Management Systems IT Infrastructure vCO Platform (Engine, 64-bit) vCO Plug-Ins …… • vCloud Automation Center • Service Catalogs • AMQP • SNMP External Notifications
- 8. 88 • vCenter Server 4.0, 4.1, 5.0 & 5.1 • vCloud Director 1.0, 1.5 & 5.1 • vCloud Automation Center 5.1 & 5.2 • vCenter Update Manager 4.1, 5.0 & 5.1 • vCenter Chargeback 2.0 • vCenter Configuration Manager 5.5 • vCenter Orchestrator Multi-node 5.0 & 5.1 • vSphere Auto Deploy • VMware Service Manager 9.1 • VMware Service Elasticity • Microsoft AD & PowerShell • AMQP / RabbitMQ • Email (POP3) • Email (SMTP) • HTTP-REST • JDBC • SOAP • SNMP v1, v2c, v3 • SQL • SSH • Telnet • XML • BMC Atrium CMDB & Remedy – NEW • EMC Unified Infrastructure Manager – NEW • Infoblox NIOS – UPDATED • Egenera PAN Manager - NEW • Radware vDirect • ServiceNow • Up.time Software Standard Protocols Partner Applications • F5 Networks BigIP – NEW • EMC ViPR – NEW • Cisco UCS Manager 2.x – NEW • NetApp storage • Bluecat Networks • VMware vCenter Network and Security • VMware Site Recovery Manager • HP ServiceManager Upcoming releases VMware Applications Thousands of Out of the Box Workflows & Actions
- 9. 99 • Improve scalability & availability • Built-in HA & clustering • Support external load balancers • Extend the vCO REST API to: • vCO server installation • vCO server configuration • Provide higher availability • Scale orchestration capacity along with the growth of your cloud • Enable dynamic scale-up and scale- down of orchestration capacity Overview Benefits Optimized for Growing Clouds Orchestration HA and dynamic elasticity!
- 10. 1010 VMware Cloud Automation vCloud Automation Center (IaaS, & DaaS Automation ) Infrastructure Integration • CMDB • DNS • IPAM • Load Balancers • Service Desk • Monitoring Systems • Databases • Web Services • Etc. Fabric Management Automation vCenterOrchestrator ITProcessAutomation Some Use Cases: o Automation of vSphere administrative tasks o Remediation of infrastructure failures o Automation of general IT admin tasks Primary Role & Use Cases for vCenter Orchestrator
- 11. 1111 VMware Example – vCenter Operations Manager with vCenter Orchestrator Automated Remediation
- 12. 1212 vCenter Operations Remediation Workflow Package What is its purpose? • The purpose of the vCenter Operations Manager Remediation Workflow Package is to be able to launch remediation workflows in vCenter Orchestrator, as response to alerts received from vCenter Operations Requirements on which the solution is based • Create a solution for the problem - to be launching workflows, when vCenter Operations alerts are received • This solution should be simple and should not need any programming or scripting from the user • The user should be able to launch any workflow, from the library, or his/her own creation, as a response to an alert • It should be easily configurable • The user should be able to filter the incoming events, based on different alert properties
- 13. 1313 vCenter Operations Remediation Workflow Package What do I need to use it? • vCenter Orchestrator virtual appliance. (v5.1 or later) • vCenter Orchestrator SNMP plugin • vCenter Operations integration package • vCenter Operations Manager How does it work? • vCenter Operations Manager sends SNMP traps to vCenter Orchestrator • vCenter Orchestrator acts on the appropriate traps by executing workflows
- 14. 1414 vCenter Orchestrator + vCOps Remediation 1. vCenter health and operational data is continually passed to vCOps for analysis 2. When vCOps identifies an operational issue, it throws an SNMP trap to vCO, triggering a vCO Policy to process the trap 3. vCO verifies the incoming trap is mapped to an alert definition 4. vCO verifies there are filter conditions defined for the trap 5. vCO launches the appropriate remediation workflow 6. The vCO remediation workflow corrects the operational issue
- 15. 1515 vCenter Orchestrator + vCOps Remediation 1. vCenter health and operational data is continually passed to vCOps for analysis 2. When vCOps identifies an operational issue, it throws an SNMP trap to vCO, triggering a vCO Policy to process the trap 3. vCO verifies the incoming trap is mapped to an alert definition 4. vCO verifies there are filter conditions defined for the trap 5. vCO launches the appropriate remediation workflow 6. The vCO remediation workflow corrects the operational issue
- 16. 1616 Example Use Case: Identify a Datastore Capacity Issue Datastore running out of capacity
- 17. 1717 Example Use Case: Identify Powered Off VMs Powered off VMs on the datastore
- 18. 1919 vCenter Operations Alerts Trigger Outbound Notification Alerts trigger outbound notification via Email and SNMP traps
- 19. 2020 vCenter Orchestrator SNMP Trap Policy Workflow SNMP Trap policy workflow Sample code that starts remediation workflow if capacity remaining alert is received
- 20. 2121 Automate Remediation Using vCenter Orchestrator Workflows Workflow to list powered off VMs and VM snapshots to resolve capacity issue Prepare report and send email notification
- 21. 2222 Email Notification from the Datastore Remediation Workflow Email listing powered off VMs and associated snapshots
- 22. 2323 Customer Example – CatamaranRX Nick Colyer Team Lead – Server Engineering CatamaranRX
- 23. 2424 Customer Examples of Automation – Nick Colyer, CatamaranRX Who is Nick Colyer? • Brief History • Blog: v-nick.com • Twitter: @vNickC How I got into automation My Examples: • Example #1 • Self Healing: Automating Configurations • Example #2 • Self Healing: Automating Incident responses
- 24. 2525 Example 1: Automating Configuration HA and DRS Settings Start with a Goal in mind: “I want to make sure that my ESXi Clusters are checked every day to ensure HA is on, DRS is fully automated.” Customer Example 1 – Automating Configuration for HA / DRS
- 25. 2626 Admission Control settings 1 Enable Host Monitoring 2 Break it down - HA Settings Customer Example 1 – Automating Configuration for HA / DRS
- 26. 2727 Break it down - DRS Settings DRS to Fully Automated 3 4 Ensure other settings remain! Affinity Rules etc. Customer Example 1 – Automating Configuration for HA / DRS
- 27. 2828 Customer Example 1 - Building the Workflow Feed in clusters1 Run corrective action2 Repeat for every cluster in your environment 3 Schedule workflow to run every night 4
- 28. 2929 Customer Example 1 - Create a Reusable Action Item Create scriptable tasks workflow or an action.
- 29. 3030 Action Item: Enable HA/DRS Javascript 1. Calculate HA % based on number of hosts //Get all the hosts in the cluster var Hosts = System.getModule("com.vmware.library.vc.cluster").getAllHostSystemsOfCluster(cluster); System.log("Number of Hosts in Cluster: " + Hosts.length); //Calculate HA Percentage to tolerate 1 host worth of resources being offline var HApercent = ((1/Hosts.length)*100); HApercent = HApercent.toFixed(0); //Log it System.log("HA Percent which will be used for cluster is: " + Hapercent)
- 30. 3131 2. Turn on HA and DRS (partial code) Action Item: Enable HA/DRS Javascript var clusterConfigSpec = new VcClusterConfigSpecEx(); clusterConfigSpec.drsConfig = new VcClusterDrsConfigInfo(); clusterConfigSpec.dasConfig = new VcClusterDasConfigInfo(); //Enable DRS/HA System.log("Setting HA and DRS to Enabled (even if they were already)"); clusterConfigSpec.dasConfig.enabled = true; clusterConfigSpec.drsConfig.enabled = true; //Reconfigure the cluster, by adding the True parameter this ensures any previous settings remain System.log("Executing Cluster Reconfiguration for " + cluster.name); task = cluster.reconfigureComputeResource_Task(clusterConfigSpec, true); IMPORTANT! If you don’t add the true option, it will remove all your other existing HA/DRS settings. i.e. affinity rules
- 31. 3232 Customer Example 1: Workflow in action Video Example: http://bit.ly/vco-hadrs-automate
- 32. 3333 Example 2: Automation in response to an event Start with a Goal in mind: “Enable repeatable scripted actions to be initiated in response to an SNMP Trap”
- 33. 3434 Customer Example 2: Breaking it Down… vCenter critical alarm for a datastore over 95% full1 Send trap to vCO2 Run Storage DRS on storage pool 3
- 34. 3535 Customer Example 2: How Do We Achieve This… 1. Configure SNMP Trap receiver on vCenter Orchestrator • http://blogs.vmware.com/orchestrator/2011/09/snmp-plug-in-integration-with-vcenter.html • http://www.vcoportal.de/2012/05/integrate-vcops-and-vco/ 2. Create Workflow which interprets traps 3. Create Workflows for repeatable automated corrective actions a. Locates Datastore Cluster which Datastore is a member of b. Executes SDRS a. Expand on it further: Auto provision a LUN from the SAN
- 35. 3636 Customer Example 2: Master Workflow That Feeds Corrective Action Workflows 1 Scriptable task to interpret trap data Does the trap contain something we know how to handle? 2 3 Run corrective action
- 36. 3737 Customer Example 2: Run SDRS Workflow in Detail 1. Search vCenter for a datastore with the same name as the one in the trap. 2. Check SDRS Pods to see if they contain the datastore object 3. Refresh Storage recommendations task = m.refreshStorageDrsRecommendation(podToRunSDRSOn) Full script on my web site: v-nick.com
- 37. 3838 Customer Example 2: Workflow in Action Video Example: http://bit.ly/vco-sdrs-automate
- 38. 3939 Taking It to the Next Level… 1. Instead of just running SDRS, create a workflow to auto-provision storage from the array when the space left in an SDRS pool gets below a threshold 2. Have a workflow that automatically creates the change order, but waits for someone to actually release the workflow 3. Corrective actions from other monitoring systems • i.e. Solarwinds/SCOM when a Windows 2008 Server drive is below critical amount. • vCO can automatically expand the disk in vSphere, and then expand it inside the OS.
- 39. 4040 Being Successful in a Corporate Environment How do you start? • Need upper leadership to be bought into the idea of automation. • Standardize > Write Procedures > Automate • Adopt an automate first approach Develop a team that will become “Stewards” of vCO • Empower others to automate Keep it simple • Re-use existing code • Look at the built in workflows Know what other tools in your environment can integrate with vCO (e.g. ServiceNow)
- 40. 4141 Partner Example – EMC Unified Infrastructure Manager
- 41. 4242 vCenter Orchestrator + EMC Unified Infrastructure Manager plug-in Use Case 1: vSphere Cluster capacity at maximum, need to add host • Virtual machines are running slow and you find out hosts are overloaded and running low on CPU and memory • VMware administrator can initiate adding a new server to the UIM VDI service, making it available as a new host to vCenter, either through the vCO interface, or through the vSphere web client
- 42. 4343 vCenter Orchestrator + EMC Unified Infrastructure Manager plug-in Use Case 2: Low remaining capacity on Oracle database server • An Oracle database is running out of storage, which could impact availability of the production applications • VMware administrator can initiate adding additional storage array LUNs to the UIM Oracle service, making them available as datastores within the vCenter cluster, either through the vCO interface, or through the vSphere web client
- 43. 4444 Advice, Considerations and Tips Map out your process • Before trying to automate anything, map out how YOU would fix the problem, step by step Factor in alert storms • Design your workflows to be aware of its active instances to prevent overlap Know when to give up • Remediation workflows only know as much as you teach them. If fixing an issue goes beyond the capabilities of your workflows, add notifications to let you know when manual intervention is required Establish credibility with the low-hanging fruit • Start out by fixing the easy stuff – stray snapshots, remounting of data stores Don’t reinvent the wheel! • Leverage the established community of vCenter Orchestrator experts – many have example workflows and packages to offer! .
- 44. 4545 Questions?
- 45. 4646 VMworld on Social Media @startswithv – Dan M #CloudMgmt #CloudAutomation #VMworld
- 46. 4747 Summary: SDDC Delivers Transformational Benefits * Claims being validated by the Taneja Group (final numbers expected August, 2013). Support for over 500 ISV solutions and 80 operating systems Choice Any App Anywhere Reduce IT capex by 75% and opex by 56%* Cloud Service Provider Economics Control Reduce downtime for tier 1 applications by 36%* Cloud on Your Terms Agility Increase IT productivity by 67%* Apps at Business Speed Start Your Journey with the VMware SDDC Today
- 47. 4848 Other VMware Activities Related to This Session HOL: HOL-SDC-1307 vCloud Automation Solutions
- 48. THANK YOU
- 50. Part 2: How to Build a Self-Healing Data Center with vCenter Orchestrator Nicholas Colyer, Catamaran RX Dan Mitchell, VMware VCM5695 #VCM5695