Accelerated dataplanes integration and deployment
1 like416 views
This document discusses the integration of accelerated dataplanes, specifically Open vSwitch (OVS) with DPDK and FD.io (VPP), highlighting their configuration, common dependencies, and performance tuning techniques. Both solutions require specific settings such as hugepage configuration and PMD thread management, and they face integration challenges related to NIC binding and debugging. The document also addresses automated configuration options and outlines ongoing work to enhance support for these technologies within OpenStack deployments.
Accelerated dataplanes integration and deployment
- 2. Accelerated Dataplanes Integration Feng Pan (Red Hat) Tim Rozet (Red Hat)
- 3. To be covered •OVS DPDK and FD.IO (VPP) •Common Dependencies of OVS DPDK and FD.IO (VPP) •Configuration/Integration Differences between OVS DPDK and FD.IO •Integration Challenges •Performance Tuning •Automated Configuration options •OpenDaylight Integration - design and operation
- 4. High Level OVS DPDK •Runs as an application or system service in Linux userspace •Packets incoming to the host are not processed by the OVS kernel driver which handles packets in the non-DPDK scenario •Ports on the host are bound to DPDK, then assigned to bridges in OVS •VM ports on the host are created as vhostuser •Poll Mode Driver (PMD) threads continuously poll for new packets incoming to the DPDK dataplane
- 5. High Level FD.IO (VPP) •Runs as an application or system service in Linux userspace •Ports on the host are bound to DPDK, then assigned to interfaces in VPP •VM ports on the host are created as vhostuser •Support L2 and L3 features (NAT/ACL/NSH/IPSEC)
- 6. Common Dependencies •Both rely on PMD drivers, of which there are a few (uio_pci_generic, vfio-pci) •Both require hugepage configuration (required for DPDK) •IOMMU is required for vfio-pci •nova scheduler filters must be set to include NUMATopologyFilter (for KVM) •Host NIC must support DPDK •libvirt must be able to read vhostuser sockets •Though config is different, both require similar performance tuning
- 7. Integration Differences •OVS requires being put into DPDK mode, and then restarted: −ovs-vsctl –no-wait set Open_vSwitch . other_config:dpdk-init=true •DPDK ports are added live to a “netdev” type bridge in OVS DPDK •In FD.IO, DPDK ports are specified in startup config before FD.IO is run •With OVS DPDK, Host NIC is bound manually, while it is automatic in FD.IO
- 8. Integration Challenges •After binding to DPDK, NIC essentially does not exist in Linux kernel anymore •Packets flowing in userspace makes debugging more difficult •NUMA Topology and hardware properties about each Host should be considered before deploying accelerated dataplanes •Configuration changes may require host reboot and additional steps •Using existing Neutron agents (DHCP/L3) requires patches, uses kernel data path (allows VPP veth pair to Agent namespace)
- 9. Common Performance Tuning •PMD threads should be isolated and pinned to their own cores using isolcpus kernel argument •How many hugepages/which socket to assign to the dataplane? •Which NUMA node does the PCIe bus with DPDK NIC belong to? •Nova instances should be pinned to cores on the same socket as PMD •Use 1GB huge page if possible •BIOS settings for maximum performance: - Disable C state - Enable Turboboost/Speedstep
- 10. OVS DPDK Performance Tuning •PMD threads are pinned to cores via cpu mask: −ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x6 −If hyper-threading is enabled, care should be given to core sibling relationships −Cores should be allocated to the NUMA node that PCIe bus of the DPDK NIC belongs to −Allocating more than 1 core will result in multiple PMD threads spawned
- 11. OVS DPDK Performance Tuning (cont.) •Dataplane hugepages should be configured to same socket: −ovs-vsctl set Open_vSwitch . other_config:dpdk-socket- mem="1024,0" −format is “<MB of socket 0>, <MB of socket 1>, ...” •Pin IRQs away from isolated cores using tuna or tuned
- 12. VPP Performance Tuning • Run in multithreading mode cpu/workers <n>: Create n worker threads cpu/coremask-workers <mask> and cpu/corelist-workers <list>: Place worker threads according to mask or list cpu/main-core <n>: Assign main thread to specific core. Defaults to first available core. cpu/skip-cores <n>: Leave low n bits of process affinity mask clear dpdk/coremask <mask>: process level core mask dpdk/num-mbufs <n>: Number of I/O buffers. Defaults to 16384 dpdk/socket-mem <list>: Buffer memory allocation. Defaults to 256MB on each NUMA node • Tickless Kernel: nohz_full and rcu_nocbs kernel parameters, use with isolcpus
- 13. Automated Configuration Options •puppet-vswitch −manifests/dpdk.pp −Capable of configuring OVS with DPDK performance options −Part of OpenStack Puppet Modules •puppet-fdio −Capable of configuring fd.io with performance options −Can also configure honeycomb agent for OpenDaylight −Own project in FD.IO repository
- 14. OpenDaylight + OVS DPDK Computenode-0 Computenode-1 Tenantnetworki/f Tenantnetworki/f Tenant network i/f Bridge br-phy VM 2 vhost- user Controlnode-0 VXLAN OVS DPDK External network i/f Internet DHCP tap VXLAN VXLAN OVS OpenStack Services Network Control VM 1 Bridge br-int Bridge br-int Bridge br-phy Bridge br-int vhost- user OVS DPDK Bridge br-ex
- 15. OpenDaylight + FD.IO Computenode-0 Computenode-1 Tenantnetworki/f Tenantnetworki/f Tenant network i/f Bridge Domain VM 2 vhost- user Controlnode-0 VXLAN VPP External network i/f Internet DHCP tap VXLAN VXLAN VPP VPP OpenStack Services Network Control VM 1 Bridge Domain Bridge Domain vhost- user HoneyComb HoneyComb HoneyComb
- 16. OpenDaylight + FD.IO current work ● Add support for Nirvana stack (converged GBP+Netvirt) ● Add more complete performance tuning options to deployment ● Add interface bonding support ● Add DVR support ● Distributed DHCP to compute nodes (no VPP needed on control node anymore)
- 17. OPNFV Apex support •Apex is the only installer project that supports both ovs-dpdk and VPP data path •All deployment and performance tuning steps can be specified in config files and automated •Apex supports OpenDaylight with both dataplanes over multiple scenarios
- 18. Questions?