![Sabyasachi Sengupta, Nokia
[sabyasachi.sengupta@nuagenetworks.net]
DPDK Summit - San Jose – 2017
Linux FoundaIon
DPDK with KNI – Pushing the
Performance of an SDWAN gateway to
Highway Limits
#DPDKSummit](https://image.slidesharecdn.com/5-171118040218/85/LF_DPDK17_DPDK-with-KNI-Pushing-the-Performance-of-an-SDWAN-Gateway-to-Highway-Limits-1-320.jpg)
LF_DPDK17_DPDK with KNI – Pushing the Performance of an SDWAN Gateway to Highway Limits!
- 1. Sabyasachi Sengupta, Nokia [sabyasachi.sengupta@nuagenetworks.net] DPDK Summit - San Jose – 2017 Linux FoundaIon DPDK with KNI – Pushing the Performance of an SDWAN gateway to Highway Limits #DPDKSummit
- 2. 2#DPDKSummit Agenda u IntroducIon – 2 mins u Understanding a typical SDWAN Ecosystem, SDWAN gateway, Key Performance Requirements, Why DPDK? u SoluIon Overview – 3 mins u Dedicated DPDK apvs OVS-DPDK, KNI u SoluIon Design with DPDK – 5 mins u SoYware Architecture, High-level design, Component Design & Threading Model, ConfiguraIon Management u The Big Picture – 3 mins u Addressing SDWAN gateway requirements u Conclusion – 2 mins u Current status, Future Work, Credits, Further Reading & Q/A
- 3. 3#DPDKSummit u IntroducIon – 2 mins u Understanding a typical SDWAN Ecosystem, SDWAN gateway, Key Performance Requirements, Why DPDK? u SoluIon Overview u SoluIon Design with DPDK u The Big Picture u Conclusion
- 4. 4 Understanding the SDWAN Ecosystem u SoYware Defined WAN is centered around a gateway (usually a COTS hardware) through which a branch connects with other branches and its Enterprise Data center through IPSEC enabled broadband. u SDWAN gateway is centrally managed by Zero Touch Provisioning (ZTP) and oYen needs to provide high speed throughput. u The gateway hardware comprises of: u One or two ports face the WAN side (aka uplink), can support high speed Internet (each port can be 10Gbps Full Duplex) u Remaining ports face the LAN side (aka access link), usually can be Gigabit Ethernet – connects directly to hosts or other switches/routers depending on the size of the branch u The gateway soYware usually: u Runs Linux base OS (customized Redhat, Centos or Ubuntu) u VirtualizaIon soYware for supporIng VNFs u Virtual Switch (viz. Open vSwitch) – for switching packets to other overlay and underlay desInaIons (local and/or remote to the branch)
- 5. 5 The SDWAN Ecosystem A Simplified Illustration Hardware Switch Branch #N Machine Machine Machine Machine Machine Machine Access Links SDWAN Gateway Hardware Switch Branch #1 Machine Machine Machine Machine Machine Machine Access Links SDWAN Gateway Data Center (private cloud) Data Center (public cloud) CMS (openstack/ neutron plugin) Openflow Controller Border Gateway Border Gateway Access Links(Wired/Wireless) - LAN Uplinks (Broadband IPSEC enabled) – WAN – data traffic Legend WAN – control traffic (openflow, JSON) LAN side ports (usually 1GbE) WAN side ports (can be upto 10GbE) ovs-vswitchd openvswitch.ko & kernel network stack SDWAN gateway CoTS hardware (x86)
- 6. 6 Some SDWAN Datapath Considerations u For an Enterprise Branch, the SDWAN gateway must be capable of high speed data transfer. For cheaper capacity, Enterprises are adding Broadband links to MPLS. OYen a hybrid approach. This means: u Consistency of Performance – provide QoS (rate limiIng and policing) u According to IDC, “Today 40-60% of Enterprise data is migraIng between WAN to the Internet” u Example: Voice and Video Streaming capability across the WAN between hosts in different branches and main office u As data is transferred across broadband, security is the key u Typically done through Group Key Exchange – each gateway acts as an IPSEC end point.
- 7. 7 Why DPDK? Limita&ons of Kernel based forwarding u Using Linux Kernel for high speed data path typically has some inherent issues: u Linux Kernel default pagesize: 4K u This means for every three packets (1500 MTU) there could be a page fault. During large number of packets processed, this could introduce lot of delay. u No dedicated resources for packet processing u CPU and memory (pages) shared with rest of system u All ports are kernel managed u Packets arrive in kernel’s network stack and passes through several layers of kernel before reaching virtual switch (Open vSwitch). This can introduce boqlenecks. u First packet given to OVS user space for openflow rules table consultaIon leading to more boqlenecks. u Result: even though SDWAN gateway has 20G uplink, it cannot meet the performance requirements!
- 8. 8 Why DPDK? Limita&ons of Kernel based forwarding u Using Linux Kernel for high speed data path typically has some inherent issues: u Linux Kernel default pagesize: 4K u This means for every three packets (1500 MTU) there could be a page fault. During large number of packets processed, this could introduce lot of delay. u No dedicated resources for packet processing u CPU and memory (pages) shared with rest of system u All ports are kernel managed u Packets arrive in kernel’s network stack and passes through several layers of kernel before reaching virtual switch (Open vSwitch). This can introduce boqlenecks. u First packet given to OVS user space for openflow rules table consultaIon leading to more boqlenecks. u Result: even though SDWAN gateway has 20G uplink, it cannot meet the performance requirements! Advantages of DPDK u DPDK supports larger pagesize: u 2M or 1G hugepages u DPDK allows dedicated resources aqached to network ports (PMD). Also memory can set aside for packet processing. u DPDK allows packets to be received directly in the user space using PMD u What is specifically needed for SDWAN? u Add IPSEC capability u Add QoS capability
- 9. 9#DPDKSummit u IntroducIon u SoluIon Overview – 3 mins u Dedicated DPDK app vs OVS-DPDK, KNI u SoluIon Design with DPDK u The Big Picture u Future Work & Conclusion
- 10. 10 The Need for a Dedicated DPDK app Current State of Art Analysis u Open vSwitch has integrated DPDK (ovs-dpdk) as an Userspace Datapath u The main bridge is configured with datapath_type=netdev, which indicates packets are processed in user space instead of Linux kernel u Devices can be added to ovsdb with Interface type=dpdk and subsequently a PMD thread is spawned for polling packets u In SDWAN environment, this means a single virtual switch applicaIon (ovs-vswitchd) will have all capabiliIes of slow path (first packet processing for virtual switch features) as well as fast path (subsequent packets). What if? u Virtual Switch app that gets periodically refreshed by SDWAN vendors for new soYware features have a soYware glitch and crashes? Can we afford to be disconnected from the gateway? What are our opIons? u What about featureset like IP-tables (firewalling), connecIon tracking etc? Currently these are implemented by kernel. u Boqom-line: can we have best of both worlds?
- 11. 11 KNI as The Missing Link! u Kernel Network Interface is a programming technique provided by DPDK u KNI queues allow draining of packets between DPDK app (vrs-dpdk-datapath) to/from kernel u A set of KNI queues (slave network devices) that are aqached to a DPDK port u Associate them with same MAC addresses u Abstracted by dpdk_bondX device, where X = DPDK managed port ID u Upsides: u Leverage OVS for all slow-path and basic virtual switching funcIonaliIes as packets arriving from WAN can be fed into kernel resident openvswitch flow tables u Leverage kernel for all IPtables and conntrack funcIonaliIes u Downsides: u SIll introduces a copy between kernel and DPDK app – cannot be avoided u Agreed, but experimental data shows KNIs are rather fast!!
- 12. 12#DPDKSummit u IntroducIon u SoluIon Overview u SoluIon Design with DPDK – 5 mins u SoYware Architecture, High-level design, Component Design & Threading Model, ConfiguraIon Management u The Big Picture u Conclusion
- 13. 13 High Level Architecture alubr0 (ovs-vswitchd) openvswitch.ko Kernel TCP/IP rte_kni.ko DPDK Library 7 6 2 3 1 2 3 4 A KNI 1 2 3 4 5 6 8 7 7 vrs-dpdk-datapath (DPDK application) 4 User Space Kernel Space A Main Thread Receive Processing Transmit Processing NW to Access Traffic Path Access to NW Traffic Path Classifier/Flow Table QAT (ingress) NW/A Pipeline A/NW Pipeline Access vLAN VPORT Uplink VPORT (tunnel) Physical Port (Access/Uplink) 1 QAT (egress) 2 UDP/Mac Encap 3 QoS 4 Fragmentation LEGEND N NW/A Packet Hop N A/NW Packet HopControl Path (netlink) libopenvswitch (ovs-2.5+) libvrs-dpdk 5 81 Kernel Flow Table for switching
- 14. 14 Software Architecture libdpdk.a libopenvswitch-netdev libopenvswitch-netdev-dpdk vrs-dpdk-datapath-device libopenvswitch-dpif-netdev libopenvswitch-flow vrs-dpdk-datapath-flow libopenvswitch-netlink vrs-dpdk-datapath-cfg-mgr libopenvswitch-unixctl vrs-dpdk-datapath-unixctl vrs-dpdk-datapath-main rte_kni.ko User Space Kernel Space vrs-dpdk-datapath component – DPDK application code (Nokia developed) DPDK SDK (upgraded to support DPDK-17.02 – DPDK LTS) Open vSwitch library code LEGEND vrs-dpdk-datapath-qos vrs-dpdk-datapath-qat vrs-dpdk-datapath (DPDK app) igb_uio.ko
- 15. 15 Component Design & Threading Model vrs-dpdk-datapath Plugin Manager vrs-dpdk-datapath Device Manager (PMD & KNI) vrs-dpdk-datapath Configuration Manager Plugins (IPSec, Arp Snooper, QoS) vrs-dpdk-datapath Unixctl Manager vrs-dpdk-datapath Flow Manager Flow Table PMD Thread (RX/TX) KNI Thread (RX/TX) Main Thread (RX/TX) Physical Port Queue KNI Kernel Queues LEGEND Kernel RX/TX (rte_kni.ko) Netlink messaging (with ovs- vswitchd) Flow Table vrs-dpdk-datapath (DPDK app) Port1 (dpdk0) dpdk_bond0
- 16. 16 Security Configuration Management ovs-vswitchd IPSEC Key Server XFRM modules (xfrm4_tunnel, xfrm_ipcomp) vrs-dpdk-datapath Flow Tables (IPSEC_POLICY_TABLE, IPSEC_KEYS_TABLE) XFRM_MSG_NEWPOLICY XFRM_MSG_NEWSA XFRM_MSG_DELPOLICY XFRM_MSG_DELSA XFRM_MSG_GETPOLICY XFRM_MSG_GETSA XFRM_MSG_GETPOLICY XFRM_MSG_GETSA XFRM_MSG_NEWPOLICY XFRM_MSG_NEWSA XFRM_MSG_DELPOLICY XFRM_MSG_DELSA VSD UI (IPSec Policy Configuration) User Space Kernel Controller (VSC) nuage-rpc nuage-nsg-ipsec-cfg ALUFF_FLOW_ECMP_ROUTE OFPTYPE_IPSEC_POLICY_MOD JSON OFPTYPE_IPSEC_SEED_UPDATE Netlink with Kernel Datapath Netlink with DPDK Datapath Legend
- 17. 17#DPDKSummit u IntroducIon u SoluIon Overview u SoluIon Design with DPDK u The Big Picture – 3 mins u Addressing SDWAN gateway requirements u Conclusion
- 18. 18 Meeting SDWAN gateway specific requirements u Enabling/Disabling DPDK on WAN ports in the gateway u Could be dynamically done through SDN UI by Icking off Network Accelera@on u DPDK app starts, sets up hugeTLB pages, scans PCI bus and configures DPDK ports from list of whitelisted devices u Note: SDWAN gateways should NOT have downImes, meaning “no reboots” or traffic loss while enabling/disabling DPDK u SDWAN gateway underlay networking configuraIon u DPDK enabled WAN ports get their IP addresses automaIcally from DHCP server u DHCP server works seamlessly: modificaIon in KNI infra to ensure rte_kni_alloc accepts and assigns DPDK physical port’s mac address u Network Manager hooks added to setup underlay IP tables and routes u Tuning gateway for best results u Need to judiciously balance IRQs so as to ensure KNIs get enough cores and PMD threads get full CPU cycles u Other interesIng configuraIons in KNI devices: Packet Steering Parameters (rps_cpus/xps_cpus) in /proc, txqlen, ring parameters
- 19. 19 DPDK Profiles – Monetization opportunity! Profile # PMD / uplink # KNI / uplink Usage Normal (Small) 1 1 Regular (day-to-day) processing Accelerated (Medium) 1 3 Encrypt/decrypt happens in same PMD thread. Three KNIs drain packets from PMD. Useful for higher workload than regular profile. Performance (Large) 2 4 One dedicated CPU (core) each for encrypt and decrypt of packets, four KNIs associated with each uplink. Excellent throughput (upto 7Gbps HD) – useful for high performance WAN traffic (voice/video). • Customers can enable Profiles dynamically at Cloud Director UI and add / remove CPU allocation • Switching from one profile to another often works seamlessly without gateway reboots! • Just a restart of the DPDK application.
- 20. 20 Branch–Cloud–Branch: DPDK everywhere!! Branch-F Branch-A Branch-B Branch-C Branch-D Branch-E vrs-dpdk-datapath Cloud (Enterprise Datacenter) SDWAN border GW w/ 10G ports Branch Hosts Internet Router (Juniper MX, ALU SR7550 etc) 1G FD link (LAN) 10G FD IPSEC Broadband LEGEND 1G FD IPSEC Broadband SDWAN branch GW Stitching together disjoint underlays (terminates and reistarts IPSEC across WANs)
- 21. 21#DPDKSummit u IntroducIon u SoluIon Overview u SoluIon Design with DPDK u The Big Picture u Conclusion – 2 mins u Current status, Future Work, Credits, Further Reading & Q/A
- 22. 22 Current State & Future Work u Current status u Upto 7Gbps Half Duplex with IPSEC on 10Gbps WAN link u Highway: 55mph, Freeway: 65+ mph u That copy between user space and kernel space! u Kernel IRQ processing becomes the boqleneck aYer that rate u SIll way way beqer than original: ~2Gbps H/D with IPSEC on 10Gbps WAN link u SoluIon: u Move all LAN side ports to DPDK. u vrs-dpdk-datapath app acts as fastpath app and sends first packet to the slow path ovs-vswitchd app u Implement flow cache inside vrs-dpdk-datapath along with the pipeline.
- 23. 23 Credits u Engineering u Sabyasachi Sengupta – sabyasachi.sengupta@nokia.com - SDWAN ecosystem & DPDK/OVS Infrastructure u Paul Hong – paul.hong@nokia.com - DPDK Plugin Management Infrastructure u Limin Wang – limin.wang@nokia-bell-labs.com - IPSEC plugin, FragmentaIon u Ravilochan Shamanna – ravilochan.samanna@nokia.com - IPSEC plugin u John Shirron – john.shirron@nokia.com - QoS plugin u Rohit PaIl Bagli – rohit.paIl-bagli@nokia.com - FuncIonal test u Priyanka Kumar – priyanka.kumar@nokia.com - FuncIonal test u Ankush Singh – ankush.singh@nokia.com - Performance test u Product Management u Prasad Nellipudi – prasad.nellipudi@nokia.com u Program Management u Raymond Zhang – raymond.zhang@nokia.com