2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration
Download as PPTX, PDF15 likes7,942 views
The document discusses the migration of production workloads from Open vSwitch (OVS) to LinuxBridge within Rackspace's private cloud environment, emphasizing the need for reliability and stability. It outlines various migration plans, challenges associated with the schema differences between OVS and LinuxBridge, and key preparation steps required for a successful migration. Additionally, it highlights benchmarks comparing performance metrics between OVS and LinuxBridge, concluding that while OVS has more features, stability is prioritized for customers.
2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration
- 1. Migrating production workloads from OVS to LinuxBridge Kevin Stevens (kevin.stevens@rackspace.com) James Denton (james.denton@rackspace.com)
- 2. Weโre operators! Kevin Stevens โข RPC Engineer since 2012 (Essex) โข IRC: k_stev on irc.freenode.net James Denton โข RPC Network Engineer since 2012 (Essex) โข IRC: busterswt on irc.freenode.net
- 3. What are we doing here? 1 A history of networking in Rackspace Private Cloud 2 3 4 Our experiences with Open vSwitch Swapping out OVS with LinuxBridge What to expect with each
- 4. 2011 Started building OpenStack-powered private clouds 2012 Began architecting, building and supporting private clouds in customer DCs 2013 Over 100 customers running Rackspace Private Clouds 2014 Released RPC v9 based on Icehouse. 99.99% API uptime SLA.
- 5. RPC v2.0/3.0 RPC v4.0/4.1 RPC v4.2 RPC v9.0 OpenStack Release Folsom Grizzly Havana Icehouse Network Stack nova-network Quantum Neutron Neutron L2 Connectivity flatDHCP Open vSwitch Open vSwitch LinuxBridge (ML2) L3 Agent Support N/A No Yes Yes Host OS Ubuntu 12.04 LTS Ubuntu 12.04 LTS CentOS 6.5 RHEL 6.5 Ubuntu 12.04 LTS Ubuntu 14.04 LTS The Evolution of Networking in RPC
- 6. Why Neutron?
- 7. Why Neutron w/ Open vSwitch? โขOpen vSwitch pushed by community โขOpen vSwitch pushed by packagers โขWanted overlay networks
- 9. The problems โขKernel panics (1.10) โขovs-vswitchd segfaults (1.11) โขBroadcast storms โขData corruption (2.01)
- 11. Why move to LinuxBridge? โขLooking for reliability and stability โขLess moving parts โขEasier to troubleshoot โขSupported by the community
- 12. โข Flexibility provided by overlay networking (if not using vxlan) โข Neutron Distributed Virtual Routers (Juno) โข Any customizability provided by OVS not implemented by Neutron itself www.rackspace.com 12 What do we lose by moving?
- 13. Planning
- 14. Plan A: Scorch the earth! โขSnapshot and delete all instances โขDelete all networks โขChange from OVS -> LB โขRecreate all networks โขBoot instances โขโฆ โข It works butโฆ
- 15. But waitโฆ these are production environments!
- 16. Plan B: Migration Environment โขDeploy LinuxBridge environment โขSnapshot all instances โขImport images into new environment โขBuild new instances โขCutover โขโฆ โข It works, butโฆ $$$
- 17. Plan C: Switch it out! โขStop services โขUpdate the database โขChange the configuration from OVS -> LB โขRestart services โขโฆ โข Profit!
- 18. Issues with migrating โข Neutron OVS DB schema != Neutron LinuxBridge DB schema โ Migration to OVS ML2 DB schema is required first โข Overlay networks may not supported โ LinuxBridge uses VXLAN rather than GRE โ Requires kernel >= 3.9 โข Means GRE networks must be converted to VLAN networks โ Didnโt want to introduce additional complexity โ VLANs easier to troubleshoot if something went wrong
- 19. The Process
- 20. Preparation โข Determine whatโs needed: โDependencies โSome method of converting database to ML2 schema โSome method of converting data to LB from OVS โWhich configuration files need mangling โWhich services need disabling โWhich services need restarting โRoll-back plan
- 21. Define a successful outcome โข Can instances gain a DHCP lease? โข Do instances have internal/external connectivity? โข Are security groups/other functions still operational? โขWere instances placed into the correct bridge? โข Will the changes survive a reboot?
- 22. Normal OVS Operation (Network Node)
- 23. Normal OVS Operation (Compute Node)
- 24. First steps: Database manipulation โข Backup! Backup! Backup! โข Use migrate_to_ml2.py (modified) to change the DB schema โข Update segments, ports and vlan tables โChange GRE to VLAN โChange segmentation id to real VLAN ID โSet a provider bridge
- 25. Next steps: Install and Configure โข Install the LinuxBridge plugin โข Update SQL connection strings โข Configure ml2_conf.ini / linuxbridge_conf.ini โข Change driver from OVS to ML2 in Neutron and Nova conf files
- 26. Next steps: Pull ports from bridges โข Stop Neutron services on all nodes โข Remove host data-plane port from the OVS bridge(s) โข Pull instance taps out of the OVS-related linux bridges โข Remove router and dhcp interfaces from OVS integration bridge โข Stop Openvswitch
- 27. Interfaces removed from bridges
- 29. Finally: Restart services โข Start Neutron services โข Restart compute services
- 30. Post Service Restart (Network Node)
- 31. Post Service Restart (Compute Node)
- 33. Failure Scenarios โขInstances unresponsive? โCheck traffic from tap->bridge->physical interface โVerify VLANs properly trunked through (and VLANs created on the switch) ๏
- 34. Failure Scenarios (Contโd) โขIPs disappear or taps placed in QBR bridges โCheck Nova instance_info_caches table. โCache can be regenerated with a hard reboot of instance, or by adding an interface to the instance
- 35. Failure Scenarios (Contโd Contโd) โข Unable to boot new instances? โ Usual troubleshooting techniques should be used โข DHCP Binding_failed error messages? โ Check /etc/default/neutron-server is referencing ML2 configuration file โข BRQ bridges not built? โ Verify New agents checking in? โ Verify the LinuxBridge agent is installed and running
- 36. Benchmarks
- 37. Compare all the things 8 4 2 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 * Host-to-host testing; no virtualization. Longer is better. 1 Aggregate Throughput (Gbps) # of Threads iPerf3 Benchmarks (TCP / 1500 MTU / 10G Data) โ Intel X520* (ixgbe driver) Open vSwitch (VXLAN) LinuxBridge (VXLAN) Open vSwitch (GRE) Open vSwitch (VLAN) LinuxBridge (VLAN)
- 38. Compare all the things SCP File Transfers (10G file)* 115.00 Seconds 104.00 Seconds * Host-to-host testing; no virtualization. Longer is better. 61.50 Seconds 59.75 Seconds 110.50 Seconds 0.00 20.00 40.00 60.00 80.00 100.00 120.00 140.00 160.00 180.00 200.00 OVS VXLAN LB VXLAN OVS GRE OVS VLAN LB VLAN Transfer Speed (MBps)
- 39. Compare all the things SCP File Transfers (10G file)* 115.00 Seconds 104.00 Seconds * Host-to-host testing; no virtualization. Longer is better. 61.50 Seconds 59.75 Seconds 110.50 Seconds 0.00 20.00 40.00 60.00 80.00 100.00 120.00 140.00 160.00 180.00 200.00 OVS VXLAN LB VXLAN OVS GRE OVS VLAN LB VLAN Transfer Speed (MBps)
- 40. Compare all the things SCP File Transfers (10G file)* 115.00 Seconds 104.00 Seconds * Host-to-host testing; no virtualization. Longer is better. 61.50 Seconds 59.75 Seconds 110.50 Seconds 0.00 20.00 40.00 60.00 80.00 100.00 120.00 140.00 160.00 180.00 200.00 OVS VXLAN LB VXLAN OVS GRE OVS VLAN LB VLAN Transfer Speed (MBps)
- 41. โข OVS provides a great deal of functionality โข Network stability more important for our customers than being on the cutting edge โข Linux bridge provides almost all of the features we might want to use โข How to migrate existing environments to LinuxBridge โข Improved stability and comparable performance with OVS achieved www.rackspace.com 41 In Summary
- 43. Questions?
Editor's Notes
- #3: Who are we Kevin since 2006 James since 2008
- #7: -Why did we originally go with Neutron
- #8: -Why did we go with OVS
- #10: -Kernel panics - Physical hosts would crash with a familiar stack trace -Segfaults โ ovs-vswitchd segfaults and restarts itself resulting in empty flow tables because the Neutron OVS plugin would not rebuild the flows. Instances on an affected node would lose connectivity. Still a problem in 1.11 if pushing enough traffic. -Broadcast storms โ If servers come up around the same time and the Neutron OVS plugin does not start for some reason, the servers begin forwarding everything to each other. Easily fixed by patching the startup script for Openvswitch but again extra complexity. -Data corruption. Eg. MYSQL transaction โBEGINโ seen as โBEGieโ on the receiving end -Relying on packaged version of OVS cripples your ability to stay current on something so fast-moving as OVS -Compiling OVS and kernel modules adds unnecessary complexity and requires frequent downtime
- #11: -What made us choose to move to Linux Bridge?
- #12: -tried and true -Been around for over a decade (2000). -Less complexity. -Allows us to take advantage of all of the features we wanted to use anyway.
- #13: -lose the flexibility provided by using overlay networking like GRE which for the most part has been pretty solid in itself. -wouldnโt be able to take advantage of the DVR functionality in Juno. -QOS customizations?
- #14: -how do we move to Linux Bridge
- #20: -*how* we went about the migration
- #21: Extensive testing, no cowboy shit
- #23: A glance at how our networking looks with OVS and GRE tenant overlay networks
- #24: A glance at how our networking looks with OVS and GRE tenant overlay networks
- #27: -Give the linuxbridge agent a clean slate to work with
- #28: look at the removal of the tap interfaces from the โqbrโ linux bridges, and removal of the physical interface from OVS provider bridge
- #29: When we stop the openvswitch service, the OVS bridges are no longer available
- #31: Restart LinuxBridge agent, L3 agent and DHCP agent
- #32: Restart linuxbridge and compute
- #35: Cache - This cache is used by nova when connecting instance to bridges. To rebuild, hard reboot or add an interface.
- #36: Binding failed - Check /etc/default/neutron-server (Ubuntu)
- #37: -does it actually pay off? -In the stability and supportability department definitely but how about performance-wise?
- #38: -Benchmarks should be taken with a grain of salt, given the disparity of hardware and software being evaluated. YMMV.
- #39: -Here we have a simple file transfer speed comparison (more is better in this graph)
- #40: -GRE on OVS
- #41: โฆand VLAN on linux bridge