Pushing Packets - How do the ML2 Mechanism Drivers Stack Up

Editor's Notes

• #2: https://etherpad.openstack.org/p/jjdenver One of the big use cased or needs for accelerated data plane is NFV. This talk introduces the viewer/reader to various ML2 drivers and virtual switching technologies supports by OpenStack, and compares features/functionality/performance. Our goal is to shed some light on a few of the mechanism drivers and virtual switching technologies available for OpenStack, and possibly help you determine which could be a good fit for your cloud.
• #3: JAMES Brief overview of what ML2 is Describe some of the most common mechanism drivers available to deployers Provide some comparisons between mech drivers and related vswitch technologies. When comparing these drivers and respective virtual switching technologies, we focused on an out of the box deployment with little no tuning using off-the-shelf tools like iPerf and limited Trex tests. The results seen here may not tell the whole story, but provide some interesting data nonetheless.
• #4: As new monolithic plugins came onboard to support their respective network technology, they had to implement the entire Neutron API or risk forgoing features. WHAT WERE THE FEATURES???
• #5: In theory, operators could deploy ML2 drivers that would be responsible for: Programming physical hardware switches SR-IOV vSwitch like LXB and/or OVS
• #6: The ML2 plugin relies on two types of drivers: type drives and mechanism drivers. TYPE drivers describe a type of network and its attributes -- VXLAN network type, with its VNI. Or VLAN network type with its 802.1q VLAN ID MECHANISM drivers are responsible for implementing a network type. -- linuxbridge driver implements networks using linux bridges. Not all type drivers are supported by all mechanism drivers! Multiple mechanisms can be used simultaneously within a cloud to access different ports of the same virtual network. The mechanisms, such as LinuxBridge, Open vSwitch, SRIOV, and others, can utilize agents that reside on the network/compute hosts to manipulate the virtual network implementation, or even interact with physical hardware such as a switch.
• #7: JONNY
• #8: JONNY
• #9: JAMES The test environment was composed of a 2-node openstack cloud (infra/compute) and a 3rd standalone node running an iperf client and T-Rex for traffic generation. The compute and standalone nodes were directly cross-connected. Each node contained a single dual-port Mellanox CX4-Lx En and CX5. We enabled jumbo frames and disabled port security/security groups for all ports involved in testing. Just to preface this - The tests demonstrated through this presentation do not focus on the ”potential” of any given driver and related technology. While some of the results surprised us, with enough tuning, we would likely have seen an increase in performance in some cases. Whether baremetal performance in attainable for some, is questionable.
• #10: JONNY In today’s talk we will be discussing 4 major mechanism drivers are their derivatives, including: linuxbridge open vswitch open vswitch + dpdk open vswitch + asap^2 sriov macvtap + sriov vpp
• #11: Like the name implies, the linuxbridge mechanism driver uses linux bridges to provide L2 connectivity to instances. For every Neutron network – flat, vlan, vxlan, local - a linux bridge is created. Instances are attached to the respective bridge, which in turn is connected to a tagged, untagged, or vxlan interface. As a new instance is created and scheduled to a compute node, a corresponding network bridge is either created (the first time) or an existing bridge is used.
• #12: The linuxbridge plugin is pretty meager in its requirements. It simply asks for a network interface (or bond) to be associated with a provider network. If using overlay tenant networks, an IP address is required for the VTEP address. The LXB agent handles the creation of linux bridges and virtual interfaces necessary to connect VM instances to the network.
• #13: For this test, we had a single virtual machine instance connected to four different flat provider networks simultaneously. The NICs were directly cabled to the standalone node, which acted as an iPerf client. The DUT was a VM instance with 8 cores and 16 GB of RAM. At 10G, the results were on par with what we saw against the baremetal compute node. At 25G, we start to see a drop off in performance and max out at around 18 Gbps. At 40G, the drop continues and the best we got was ~20 Gbps At 100G, there was no improvement over ~20 Gbps.
• #14: JAMES
• #15: JONNY
• #16: Open vSwitch is a software switch that when used in conjuction with OpenStack, uses openflow to influence traffic forwarding decisions. Openflow rules are applied to each bridge and perform things like vlan tag manipulation, qos, possible firewalling, and more.
• #17: We won’t go into the architecture of OVS, as that is much better described by others in the OVS and OpenStack community. We will say, however, that … ?
• #18: When installing OpenStack from something like OSA, Kolla, TripleO or others, necessary packages will likely be installed for you.
• #19: For the initial OVS tests, we used the same flavor of instance and repeated the iPerf tests. At 10G, the results were on par with what we saw against the baremetal compute node. At 25G, we start to see a drop off in performance and max out at around 18 Gbps. At 40G, the drop continues and the best we got was ~20 Gbps At 100G, there was little improvement over ~20 Gbps. So – it looked very much like LXB.
• #20: JAMES
• #22: With traditional network drivers, getting packets off the wire is interrupt-driven. When traffic is received by the NIC, an interrupt is generated and the CPU stops what its doing to grab the data and perform further processing. The more traffic, the more interruptions, resulting in less performance. DPDK uses the concept of ‘poll mode’ drivers vs interrupts, which means that one or more cores are constantly ‘polling’ the queue rather than relying on CPU interrupts for traffic processing
• #23: User can create applications using DPDK libraries. Will have different OVS binary (w/ DPDK support compiled in)
• #24: The bridge setup for OVS+DPDK looks just like vanilla OVS, except that additional attributes are configured to allow proper function of DPDK acceleration. Network interfaces are added to the bridge using the same ovs-vsctl add-port command, but DPDK-specific attributes are required. At 10G, the results were on par with what we saw against the baremetal compute node, LXB and vanilla OVS. At 25G, we start to see a drop off in performance, but not as drastic as the others. We were able to sustain ~22 Gbps. At 40G, we maxxed out around 27 Gbps At 100G, the max was again ~ 27 Gbps We show vanilla OVS against OVS+DPDK to demonstrate the improvement of DPDK acceleration for this particular test scenario.
• #25: Disadvantages: You must call out the number of cores to reserve for PMDs on each numa node, maybe even particular core numbers and sibling threads. You may need to isolate cores from the Linux scheduler You may need to reserve a certain number of cores for host operations Leaving you with a smaller subnet of cores available to virtual machines
• #27: JAMES With SR-IOV, a network card can be carved up and made to appear as multiple network interfaces. You use SR-IOV when you need I/O performance that approaches that of the physical bare metal interfaces. Different NICs support varying numbers of VFs, but usually enough to support a few dozen VMs on a host. Operators must configure the neutron-sriov-agent on compute nodes; network nodes must utilize LXB or OVS. The SRIOV agent can run in parallel to the other agents on a compute. https://www.redhat.com/en/blog/red-hat-enterprise-linux-openstack-platform-6-sr-iov-networking-part-i-understanding-basics
• #28: JONNY When testing SR-IOV, we attached a single virtual function per network to the instance. At 10G, the results were on par with what we saw against the baremetal compute node, LXB, OVS and DPDK At 25G, we saw performance comparible to the baremetal node with no dropoff in throughput At 40G, we saw a slight dropoff and we able to obtain approx 37 Gbps At 100G, the max ~ 70 Gbps, but right there with baremetal performance and near the max for PCIe 3.0
• #29: JAMES
• #30: In today’s talk we will be discussing 4 major mechanism drivers are their derivatives, including: linuxbridge open vswitch open vswitch + dpdk open vswitch + asap^2 sriov macvtap + sriov vpp
• #31: SR-IOV (indirect mode) uses an intermediary device, such as macvtap, to provide connectivity to instances. Rather than attaching the VF directly to an instance, the VF is attached to a macvtap interface which is then attached to the instance. This particular architecture addresses some of the shortcomings of direct mode, especially related to live migration. But as we’ll see in the next slide, the performance benefit over 10G is effectively lost.
• #32: Adding macvtap interfaces as an intermediary between the instance and the NIC resulted in a significant drop in performance at the 25/40/100 marks – comparable or WORSE to that of LXB or vanilla OVS. At 10G, the results were on par with what we saw against the baremetal compute node, LXB, OVS and DPDK and direct SRIOV At 25G, however, the results were even lower than that of LXV or OVS, about 16 Gbps sustained At 40G, we sat around 21 Gbps At 100G, nearly the same at around 20 Gbps. Any performance gained from using virtual function was lost in sriov indirect mode with macvtap.
• #33: JAMES
• #35: JAMES ASAP squared DIRECT works in conjunction with Open vSwitch, and offloads packet processing onto the embedded switch in the NIC using the switchdev API. ASAP FLEX works differently, in that some packets are still processed in software. Flex is not in scope or supported with openstack at this time. More ASAP info at: http://www.mellanox.com/related-docs/whitepapers/WP_SDNsolution.pdf
• #36: With ASAP squared, the concept of an eSwitch and VF representor port is introduced. An eSwitch is a switch that lives on the NIC. A representor port is a virtual representation of an eSwitch port. These ports are created and associated with a VF and attached to the OVS integration bridge. The instance itself is attached to the VF and requires drivers for the VF, much like SR-IOV direct mode. When you send traffic to the representor port, it arrives on the VF and to the VM. When the VF recieves a packet from the VM and the eSwitch doesn’t have a rule, the packet hits the SLOW PATH and the packet is processed in software. Subsequent packets are processed in hardware. ----------- At 10G, the results were on par with what we saw against the baremetal compute node, LXB, OVS and DPDK and direct SRIOV. Basically everything we’ve tested so far. At 25G, the results were as good as BM and similar to SR-IOV direct. At 40G and 100G, about the same and certainly within the margin of error. BENEFITS of ASAP??
• #37: If eSwitch resources are consumed, fallback mode is to the slow path (kernel)
• #38: In today’s talk we will be discussing 4 major mechanism drivers are their derivatives, including: linuxbridge open vswitch open vswitch + dpdk open vswitch + asap^2 sriov macvtap + sriov vpp
• #40: With VPP, there is a single vswitch connecting instance ports and network interfaces. At 10G, the results were on par with what we saw against the baremetal compute node and everything else tested. At 25G, the results were as good as BM and similar to SR-IOV direct. At 40G and 100G, significant drop off – worst performing of the bunch.
• #44: With the 25Gbps test, we start to see a drop in performance for LXB, OVS and Macvtap.
• #45: With 40Gbps, the drop continues. VPP took a significant hit.
• #46: Line rate on a 100G NIC in a PCIe 3.0 slot (16x) is slated to be roughly 80Gbps max. Here we see DPDK, SR-IOV and ASAP keeping up with baremetal performance.
• #47: With the T-Rex test, we setup a virtual machine as a router simply by enabling ip forwarding and configuring static routes described in the T-Rex configuration guide. We sought to run an SFR 1G profile that includes a mix of traffic of different protocols and payload sizes such as http/s, exchange, pop, smtp, sip, and dns.
• #48: For the 10G test, we executed the profile and multiplied it by 10 to have T-Rex sustain 10Gbps of traffic over a 120 second period. Against the baremetal node, there was zero packet loss. For SR-IOV/ASAP, we saw less than 1% packet loss over the 120 second period. VPP and DPDK followed with 24 and 40%, respectively. Indirect SR-IOV with macvtap, OVS and LXB led the rear with 80-96% packet loss. Common factor between the three worst performers? Virtio with no dataplane acceleration in place (i.e. DPDK).
• #49: For the 25G test, we executed the profile and multiplied it by 25 to have T-Rex sustain 25Gbps of traffic over a 120 second period. Against the baremetal node, there was roughly zero packet loss. For SR-IOV/ASAP, we saw between 18-22% packet loss over the 120 second period. VPP and DPDK followed with 67 and 76%, respectively. Indirect SR-IOV with macvtap, OVS and LXB flirted with nearly 100% packet loss
• #50: For the 40G test, it gets worse. Against the baremetal node, there was roughly zero packet loss. For SR-IOV/ASAP, we saw between 18-22% packet loss over the 120 second period. VPP and DPDK followed with 67 and 76%, respectively. Indirect SR-IOV with macvtap, OVS and LXB flirted with nearly 100% packet loss. ~~ It is possible we would have seen better performance splitting incoming and outgoing traffic across two different NICs ~~
• #51: With the 40 and 100 tests, T-Rex began to report ‘buffer full’ issues. Performance suffered across the board, but the SR-IOV and ASAP still came out ahead over the others. ~~ Like the 40G test, It is possible we would have seen better performance splitting incoming and outgoing traffic across two different NICs as well as being more attentive to tuning ~~
• #53: Upstream support: The built-in mechanism drivers see the highest adoption rates and are supported by a large community. IRC, Launchpad, and mailing lists exist to discuss problems and feature requests. Bug fix completion rates: Smaller projects may have less individuals available to triage and fix bugs. Community adoption: There is strength in numbers. A higher adoption rate of a given driver allows for more resources to be assigned to the driver. Feature completeness: A driver may excel at a few use cases, but may not provide features needed for a well-rounded cloud. This will vary from driver to driver and use-case to use-case. What’s important to one cloud may not be to another. Hardware compatibility: Some drives are limited to certain hardware – ASAP -> Mellanox, and DPDK -> Mellanox, Intel, and others (but only a subset). Ease of support: The most difficult to quantify – deployment tools may struggle with certain drivers, deploying in a one-size-fits-all style, or teams may resist new tooling. More interaction with vendors may be required, and heavy optimization is needed in some cases to eek out the best performance.
• #54: JAMES Each vSwitch requires its own tweaks: For DPDK-accelerated vSwitch, this may mean differences in hugepage configuration (2M vs 1G), hugepage allocation (static vs transparent) and (1G vs 4G based on MTU (DPDK docs)), NUMA considerations, etc. Different workloads, different settings: Great care should be taken to ensure an instance is leveraging same NUMA resources as NIC for best performance. Sometimes you must straddle NUMA nodes and performance could suffer. Can’t polish a turd: LXB and Vanilla OVS may do well for little over 10Gbps, but no amount of tuning will get them on-par with a DPDK or SR-IOV type performance.
• #55: JAMES Newer gen CPU PCIe matters for performance TAKE OUT SPECTRE
• #56: JAMES There is a tendancy to want to run the latest latest technology and live on the bleeding edge, but doing so comes at a cost. In the case of a DPDK-accelerated vSwitch, the lack of drivers for certain hardware in the upstream release may mean you’re rolling your own versions of DPDK, OVS, or VPP. Maintaining this configuration in the long run can be painful and not worth the trouble, especially if your use cases don’t really justify the cost.
• #57: If you’re looking for more of a set it and forget it approach, your best bet is to stay in the upstream lanes of LXB or OVS. Both are heavily contributed to and have been around since the beginning. While LXB may lack some features seen in an OVS-based solution, like DVR, Tap-aaS, FW-aaS v2,., it does its job well enough and remains a favorite of support engineers everywhere. Both LXB and vanilla OVS offer ”good enough” performance for < 10G networking.