‘fsck’ for Openstack
Download as PPTX, PDF0 likes986 views
This document discusses tools developed by PayPal to maintain consistency in their large OpenStack private cloud. It introduces CloudBuilder and CloudSweeper. CloudBuilder uses a declarative "Blueprint" to automate the initial setup and configuration of the cloud, avoiding human errors. CloudSweeper includes periodic cleaning tools to resolve resource leaks and inconsistent states. These tools help address issues like misconfigurations, VM sprawl, orphaned ports, and inconsistent volume and instance states between OpenStack services.
‘fsck’ for Openstack
- 1. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. ‘fsck’ for Openstack Wei Tian -- Cloud Performance Lead at Paypal Zhenhua Feng -- Staff Software Engineer 10/ 27 / 2015 Detect Resource Leaking and Keep the Cloud Consistent
- 2. © 2014-15 PayPal Inc. All rights reserved. Confidential and proprietary. Agenda 2 • Some numbers about Paypal Cloud • What makes our cloud inconsistent • Our solutions to keep our cloud consistent
- 3. © 2014-15 PayPal Inc. All rights reserved. Confidential and proprietary. About PayPal Cloud 3 • Background – Started in July 2012 with 1 engineer and 16 decommissioned servers – Today, one of the world’s Largest OpenStack Private Cloud – Number of VMs : 82,000 – Number of Physical Servers: 8064 – Number of Racks: 84 – Total Cores: 386,000 – Block Storage: 2 peta bytes – Largest AZ with 2500+ hypervisors • Business Goals – Hosting ~100% of PayPal’s production traffic (except Databases and Messaging) – Powers 100% of PaaS, Dev/QA and M&As – First production workload on SDN in 2013
- 4. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. What Makes the Cloud Inconsistent? 4 • VPC • Flavor • Image properties • Host Aggregate metadata • Default security group • Networks • Volumes • VM Sprawl • inconsistent cinder volume states • Orphaned ports • Inconsistent DNS entries • Inconsistent states between neutron and NSX • Inconsistent states caused by RPC timeout • Inconsistent DB states between API and Compute cells Misconfiguration Resource Leaking
- 5. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. Misconfiguration • In Paypal cloud, administrator does the initial resource allocation and configuration. • The resources set up includes VPC, flavor, image, network, host aggregate, etc. • Administrator uses Openstack cli to create all those resources and make sure they match with each others. • As long as we are human, we are bound to make mistakes. 5
- 6. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. Scenario may have misconfiguration (1) 6
- 7. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. Scenario may have misconfiguration (2) 7
- 8. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. Resource Leaking 8 • When running Openstack, sometimes the state of a resource (volume, instance, port, etc.) can be inconsistent on the cluster. • Sometime, it is not able to correct the state through REST API alone. • You may need to manually edit the database or to run a shell script on hypervisor to correct the state. • Please note that it is important to find and fix the underlying issue, and to edit database or run shell script is a just a quick hack. • However, as a service operator, you also need to fix the issue right away to meet the SLA before the engineering fixes the code.
- 9. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. Resource Leaking (1) 9 VM Sprawl can cause major performance and capacity problems. The resource leaking includes zombie VMs and orphaned disk files. The state of a volume or an instance can be inconsistent. The volume shows attached in nova but not in cinder, or otherwise. Sometimes a volume deletion hangs, or a detach does not work. Neutron orphaned ports. Ports not deleted when VM deleted. Or ports without device_id. Ports leaking causes IP leaking and DNS leaking. Inconsistent state between neutron and NSX controller. A port is deleted from neutron but still exists in NSX.
- 10. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. Resource Leaking (2) 10 Inconsistent DNS entries. One IP with multiple DNS entries, or multiple IPs with same DNS entry, or fails to create/delete DNS entry Inconsistent states caused by RPC timeout. The caller says A RPC timeout, but the handler does the job but fails to reply. Inconsistent states between API and Compute cells DBs
- 11. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. Introduce CloudKeeper 11 CloudBuilder CloudSweeper • Resolve the misconfiguration. • Eliminating manual steps to setup Openstack cloud. • The CloudBuilder automates the entire setup process to avoid human errors. • Declarative instead of Imperative. All settings are described in a set of config files called Blueprint. • Like Puppet, the CloudBuilder continuously pushes the changes from BluePrint to Openstack cloud and keeps them in sync. • Resolve the resource leaking. • CloudSweeper has a task manager which triggers all plugin tools periodically. • CloudSweeper logs the results of each cleaning tool and report to dashboard for statistics and troubleshooting.
- 12. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. CloudBuilder -- Blueprint 12 Everything Data-Driven We define how the initial setup for the cloud in a set of JSON files. The CloudBuilder will create all the resources based on the JSON files: • VPC metadata • Flavor class • VPC networks • VPC host-aggregate • VPC images
- 13. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. CloudBuilder – Blueprint – VPC metadata
- 14. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. CloudBuilder – Blueprint – VPC Resources
- 15. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. CloudBuilder – Add Hypervisor to Host Aggregate New Hypervisor can be automatically add to the right host aggregates based on its characteristics The hypervisor asset information can be retrieved from CMS (Configuration Management System)
- 16. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. CloudSweeper
- 17. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. CloudSweeper – Neutron Port Cleaner
- 18. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. CloudSweeper – Volume Cleaner • Mismatched volume state in nova and cinder • Volume stuck in deleting state • Missing connection_info in block_device_mappimg table Symptom • Find the REAL state of the volume from hypervisor • Modify the nova and cinder DBs to reset the state. • Re-run “nova volume-delete” after cleaning state in DB for volume stuck in deleting state. Fix
- 19. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 19 Questions ?