NFV Infrastructure Manager with High Performance Software Switch Lagopus
1 like410 views
This document discusses NFV infrastructure manager Lagopus, a high performance software switch developed by NTT. It begins with an agenda that covers NFV, OpenStack, performance requirements for NFV, and a comparison of Lagopus and Open vSwitch. It then provides details on Lagopus, including that it uses DPDK for high performance packet processing in userspace without kernel involvement. The document aims to demonstrate Lagopus's advantages over Open vSwitch for NFV workloads through its ability to achieve high packet processing speeds needed for telco use cases.
NFV Infrastructure Manager with High Performance Software Switch Lagopus
- 1. Copyright©2015 NTT corp. All Rights Reserved. NFV Infrastructure Manager with High Performance Software Switch Lagopus Hiorufmi Ichihara NTT Software Innovation Center Collaborator: Tomoya Hibi(NTT), Hiroki Kumazaki(NTT)
- 2. 2Copyright©2015 NTT corp. All Rights Reserved. • NFV • NFV Infrastructure Manager • OpenStack • Performance requirement for NFV • OpenVSwtich and Lagopus • Lagopus vs OVS • Lagopus advantage • Demo(video) Agenda
- 3. 3Copyright©2015 NTT corp. All Rights Reserved. Hirofumi Ichihara • Engineer at NTT Software Innovation Center • OpenStack developer • Neutron and Devstack contributor in OpenStack projects Who
- 4. 4Copyright©2015 NTT corp. All Rights Reserved. • NFV(Network Function Virtialization) What is NFV? ref: https://portal.etsi.org/nfv/nfv_̲white_̲paper.pdf • Rapid development • Reducing CAPEX • Reducing OPEX • Avoiding vendor lock in
- 5. 5Copyright©2015 NTT corp. All Rights Reserved. ETSI NFV Today main topic ref: http://www.etsi.org/deliver/etsi_̲gs/nfv/001_̲099/002/01.01.01_̲60/gs_̲nfv002v010101p.pdf
- 6. 6Copyright©2015 NTT corp. All Rights Reserved. Storage • Manage infrastructure for VNF • Compute • Network • Storage • IaaS Softwares • OpenStack • CloudStack • Eucalyptus • VMware NFV Infrastructure Manager Hypervisor Network Physical Switch Virtual Network VM VM LRFW VM VM LR LV FW LR Logical Router Logical Volume Firewall VM Virtual Machine LV LV LV
- 7. 7Copyright©2015 NTT corp. All Rights Reserved. OpenStack • Cloud Operating System • Provides API to users • Multi Hypervisor support • KVM, QEMU, Xen, VMware, Hyper-‐‑‒V, LXC and others • Integrated many network appliance and storage appliance Ref: http://www.openstack.org/software/
- 8. 8Copyright©2015 NTT corp. All Rights Reserved. OpenStack Conceptual Architecture Ref: http://docs.openstack.org/admin-‐‑‒guide-‐‑‒cloud/content/conceptual-‐‑‒architecture.html
- 9. 9Copyright©2015 NTT corp. All Rights Reserved. OpenStack Release Schedule 2010 2011 2012 2013 2014 Major release Provided functions ▲ Oct 21 Austin ▲ Feb 3 Bexar ▲ Apr 15 Cactus ▲ Sep 22 Diablo ▲ Apr 5 Essex ▲ Sep 27 Folsom ▲ Apr 4 Grizzly ▲ Oct 17 Havana ▲ Apr 1７ Icehouse Nova Swift Glance Keystone Horizon Cinder Quantum Nova Swift Nova Swift Glance Virtual machine image Object storage Virtual machine Nova Swift Glance Keystone Horizon Cinder Neutron Ceilometer HeatVirtual environment provisioning Metering/Monitoring Nova Swift Glance Keystone Horizon Cinder Neutron Ceilometer Heat Trove ▲ Oct 16 Juno Nova Swift Glance Keystone Horizon Cinder Quantum Block storage Database Nova Swift Glance Keystone HorizonWeb user interface Authentication service Virtual network Nova Swift Glance Keystone Horizon Cinder Neutron Ceilometer Heat Trove SaharaHadoop ▲ Apr 30 Kilo Nova Swift Glance Keystone Horizon Cinder Neutron ・ ・ ・ ・ ・ ・ • 1 release / half a year • Apr 30, 2015 Kilo release
- 10. 10Copyright©2015 NTT corp. All Rights Reserved. • Provides API enables to define Virtual Machine • Supporting a wide variety of virtualization technologies, including KVM, Xen, Docker, etc OpenStack Nova KVM Host Docker Host VM VM VM VM Container Container ContainerVM VM
- 11. 11Copyright©2015 NTT corp. All Rights Reserved. • Provides API enables to define virtual network • Neutron core resources • Network: Virtual netowork （L2 Switch） • Subnet: Manage IP address assign to network • Port: Port connected with network • Service plugin resources • Router(include Floating IP) • Load balancer • VPN • Firewall OpenStack Neutron L2 Switch Virtual Machine Router VPN Firewall Load balancer Tenant A Tenant B
- 12. 12Copyright©2015 NTT corp. All Rights Reserved. • OpenStack has been designed for Data Center use cases NFV History in OpenStack Hong Kong Summit 2014 Past Present Telco and vendor expect to use OpenStack to build NFV infrastructure They realized OpenStack NOT NFV orchestrator for CARRIER They didnʼ’t realized Cloud Provider Telco Carrier What is NFV requirement?
- 13. 13Copyright©2015 NTT corp. All Rights Reserved. • Canʼ’t help failure • User must HA • A few network down • User gets angry • Guaranteed by service contract • Sometimes Fail Gaps between Cloud and Telco • Must help failure • Provider must HA • Donʼ’t down network • Government gets angry too • Guaranteed by law • 24/365 on Ready! Cloud Provider Telco Carrier What is NFV requirement?
- 14. 14Copyright©2015 NTT corp. All Rights Reserved. • Performance • Packet processing speed with short packet • Low latency • High Availability • Interface for management • Stable • Monitoring • Fault detection • Security • Fault tolerance NFV Telco Requirements
- 15. 15Copyright©2015 NTT corp. All Rights Reserved. • OpenStack has been designed for Data Center use cases NFV History in OpenStack Hong Kong Summit 2014 Past Present Telco and switch vendor expects to use OpenStack to build NFV infrastructure They realized OpenStack NOT NFV orchestrator for CARRIER They didnʼ’t realized After Summit Telco Working Group and NFV subteam was organized Oct. 2014 OPNFV was organized Juno, Kilo release Some NFV requirements was merged in OpenStack
- 16. 16Copyright©2015 NTT corp. All Rights Reserved. • Proposed in OpenStack • VLAN aware VM • VM Scheduler • High Availability method • Service chaining API • Driver for network high performance • Liberty summit sessions related to NFV • 41 sessions (my grep) • OPNFV Days OpenStack Activity for NFV
- 17. 17Copyright©2015 NTT corp. All Rights Reserved. # of packet to be proceeded for 10Gbps with 1 CPU core 0 2,000,000 4,000,000 6,000,000 8,000,000 10,000,000 12,000,000 14,000,000 16,000,000 0 256 512 768 1024 1280 # of packets per seconds Packet size (Byte) Short packet 64Byte 14.88 MPPS, 67.2 ns • 2Ghz: 134 clocks • 3Ghz: 201 clocks Computer packet 1KByte 1.2MPPS, 835 ns • 2Ghz: 1670 clocks • 3Ghz: 2505 clocks
- 18. 18Copyright©2015 NTT corp. All Rights Reserved. • OpenvSwitch • Multilayer software switch • Supports OpenFlow 1.4 protocol • Including DPDK implement • Source code: https://github.com/openvswitch/ovs • Lagopus • High performance software switch with DPDK • Supports OpenFlow 1.3 protocol • Source code: https://github.com/lagopus/lagopus OpenvSwitch/Lagopus
- 19. 19Copyright©2015 NTT corp. All Rights Reserved. OpenFlow OpenFlow controller OpenFlow switch Data-‐‑‒plane Flow Table Control plane (Routing / swithching) Data-‐‑‒plane (ASIC, FPGA) Control plane (routing/ switching) Flow match action Flow match action counter OpenFlow Protocol Flexible flow match pattern (Port #, VLAN ID, MAC addr, IP addr, TCP port #) Action (frame processing) (output port, drop, modify/pop/push header) Flow statistics (# of packet, byte size, flow duration,… ) counter Conventional NW node OpenFlow Flow Table #2 Flow Table #3 OpenFlow switch agent
- 20. 20Copyright©2015 NTT corp. All Rights Reserved. high-‐‑‒performance user-‐‑‒space packet processing with Intel DPDK NIC sk_̲buf Ethernet Driver API Socket API vswitch packet buffer Dataplane 1. Interrupt & DMA 2. system call (read) User space Kernel space Driver 4. DMA 3. system call (write) NIC Ethernet Driver API Socket API vswitch packet buffer agent agent 1. DMA Write DPDK dataplane Userspace packet processing (Event-‐‑‒based) DPDK apps (polling-‐‑‒based) 2. DMA READ
- 21. 21Copyright©2015 NTT corp. All Rights Reserved. • Many rx interrupt causes overhead • Polling-‐‑‒based packet receiving • Lock with multiple thread causes bottleneck • Thread local storage and lockless-‐‑‒queue • High TLB miss rates causes slowing performance • Huge DTLB for miss of memory controller • Context switch and memory copy causes overhead • Direct data copy from NIC buffer to CPU • Kernel stack bypass of network Issues about high performance packet processing on x86
- 22. 22Copyright©2015 NTT corp. All Rights Reserved. • OpenFlow software switch • Run on x86 server • High performance packet processing • > 10Gbps • Multiple protocol • Conform OpenFlow1.3 • Protocol(MPLS, PBB) for WAN • OSS About Lagopus OpenFlow Controller Control Plane OpenFlow Protocol OpenFlow Switch Data Plane Flow Table Flow Pattern Action Flow Pattern Action Counter Counter Flow Table #2 Flow Table #3 Flow Table #4
- 23. 23Copyright©2015 NTT corp. All Rights Reserved. • OVS-‐‑‒DPDK master • Back for DPDK 1.8.0 • commit 66cabc46ecc09eeae536277a0fc7d5e44836f845 • Lagopus v0.1.2 • DPDK 1.8.0 • System • CPU Intel(R) Xeon(R) CPU E5-‐‑‒2697 v2 @ 2.70GHz • Memory 64GB(fugapages: 1GBx8) • NIC Intel X520 10GNIC • OpenFlow Controller • Ryu (https://github.com/osrg/ryu) • Traffic Generator • IXIA Switch Performance Conditions commit 543342a41cbceffaac30ace2c66b6e489eb359c8 Author: Mark Kavanagh <mark.b.kavanagh@intel.com> Date: Mon Apr 20 12:37:14 2015 -‐‑‒0700 DPDK: add support for v2.0.0 Update relevant artifacts to add support for DPDK v2.0.0 -‐‑‒ INSTALL.DPDK.md -‐‑‒ travis build script -‐‑‒ acinclude.m4: add 'mssse3' flag to OVS_̲CFLAGS -‐‑‒ netdev-‐‑‒dpdk: fix build with unified offload types in DPDK v2.0.0 Note that this breaks compatibility with DPDK v1.8.0
- 24. 24Copyright©2015 NTT corp. All Rights Reserved. diff -‐‑‒-‐‑‒git a/config /common_̲linuxapp b /config /common_̲linuxapp index 2f9643b..78738dc 100644 -‐‑‒-‐‑‒-‐‑‒ a/config /common_̲linuxapp +++ b /config /common_̲linuxapp @@ -‐‑‒81,7 +81,7 @@ CONFIG_̲RTE_̲BUILD_̲SHARED_̲LIB=n # # Combine to one single library # -‐‑‒CONFIG_̲RTE_̲BUILD_̲COMBINE_̲LIBS=n +CONFIG_̲RTE_̲BUILD_̲COMBINE_̲LIBS=y CONFIG_̲RTE_̲LIBNAME="intel_̲dpdk" # @@ -‐‑‒160,7 +160,7 @@ CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲DEBUG_̲TX_̲FREE=n CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲DEBUG_̲DRIVER=n CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲PF_̲DISABLE_̲STRIP_̲CRC=n CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲RX_̲ALLOW_̲BULK_̲ALLOC=y -‐‑‒CONFIG_̲RTE_̲IXGBE_̲INC_̲VECTOR=y +CONFIG_̲RTE_̲IXGBE_̲INC_̲VECTOR=n CONFIG_̲RTE_̲IXGBE_̲RX_̲OLFLAGS_̲ENABLE=y # @@ -‐‑‒372,7 +372,7 @@ CONFIG_̲RTE_̲KNI_̲VHOST_̲DEBUG_̲TX=n # fuse-‐‑‒devel is needed to run vhost. # fuse-‐‑‒devel enables user space char driver development # -‐‑‒CONFIG_̲RTE_̲LIBRTE_̲VHOST=n +CONFIG_̲RTE_̲LIBRTE_̲VHOST=y CONFIG_̲RTE_̲LIBRTE_̲VHOST_̲DEBUG=n # DPDK config
- 25. 25Copyright©2015 NTT corp. All Rights Reserved. • Full route • Set about 51,000 flow rule (IPv4) • packets: 1flow, random ipv4_̲dst 100000flow • Port VLAN • Set port VLAN to two port (2 flow rule for round trip) • packets: 1flow, random ipv4_̲dst 10000flow Test scenario
- 26. 26Copyright©2015 NTT corp. All Rights Reserved. Evaluation -‐‑‒ full route 64 512 1024 1500 0 1 2 3 4 5 6 7 8 9 10 0 500 1000 1500 Gbps byte/packet Wire-‐‑‒rate OVS-‐‑‒DPDK Lagopus • Lagopus wins with 1flow packets • OVS-‐‑‒DPDK wins with 100000flow packets 64 512 1024 1500 0 1 2 3 4 5 6 7 8 9 10 0 500 1000 1500 Gbps byte/packet Wire-‐‑‒rate OVS-‐‑‒DPDK Lagopus Result 1flow packets Result 100000flow packets
- 27. 27Copyright©2015 NTT corp. All Rights Reserved. Evaluation – port VLAN 64 512 1024 1500 0 1 2 3 4 5 6 7 8 9 10 0 500 1000 1500 Gbps byte/packet Wire-‐‑‒rate OVS-‐‑‒DPDK Lagopus 64 512 1024 1500 0 1 2 3 4 5 6 7 8 9 10 0 500 1000 1500 Gbps byte/packet Wire-‐‑‒rate OVS-‐‑‒DPDK Lagopus • Lagopus wins with 1flow packets • Lagopus wins with 10000flow packets Result 1flow packets Result 10000flow packets
- 28. 28Copyright©2015 NTT corp. All Rights Reserved. • Lagopus has some advantages • Telco friendly network protocol • QinQ, PBB, MPLS • Comfortable more OpenFlow protocols than OVS • Important protocol is Meter for NFV • OVS is used by general services but lagopus is designed for network services Lagopus vs OVS-‐‑‒DPDK
- 29. 29Copyright©2015 NTT corp. All Rights Reserved. OpenFlow 1.3 Conformance Status Type Action Set field Match Group Meter Total # of test scenario (mandatory, optional) 56 (3 , 53) 161 (0 , 161) 714 (108 , 606) 15 (3 , 12) 30 (0 , 30) 991 (114 , 877) Lagopus 2015.3.19 56 (3, 53) 161 (0, 161) 714 (108, 606) 15 (3, 12) 30 (0, 30) 976 (114, 862) OVS (kernel) 2014.08.08 34 (3, 31) 96 (0, 96) 534 (108, 426) 6 (3, 3) 0 (0, 0) 670 (114, 556) OVS (netdev) 2014.11.05 34 (3, 31) 102 (0, 102) 467 (93, 374) 8 (3, 5) 0 (0, 0) 611 (99, 556) IVS 2015.02.11 17 (3, 14) 46 (0, 46) 323 (108, 229) 3 (0, 2) 0 (0, 0) 402 (111, 291) ofswitch 2015.01.08 50 (3, 47) 100 (0, 100) 708 (108, 600) 15 (3, 12) 30 (0, 30) 962 (114, 848) LINC 2015.01.29 24 (3, 21) 68 (0, 68) 428 (108, 320) 3 (3, 0) 4 (0, 4) 523 (114, 409) Trema 2014.11.28 50 (3, 47) 159 (0 , 159) 708 (108, 600) 15 (3, 12) 34 (0, 34) 966 (114, 854) http://osrg.github.io/ryu/certification.html
- 30. 30Copyright©2015 NTT corp. All Rights Reserved. • System • Intel(R) Core(TM) i7-‐‑‒4790 CPU @ 3.60GHz • Memory 16GB(fugapages: 1GBx8) • NIC Intel X520 10GNIC • Traffic Generator • Pktgen DPDK(master 2015.4.28) Meter Demo Lagopus Data Plane Flow Table Flow Pattern Action in:port1 Meter 100000kbps out:port2 Pktgen DPDK port1 port2
- 31. 31Copyright©2015 NTT corp. All Rights Reserved. • How do we setup lagopus for NFV? • Lagopus is designed as network switch not switch for VM on hypervisour • Big issue • Canʼ’t add/delete port • OpenFlow controller essential • Canʼ’t try easily Using Lagopus for NFV
- 32. 32Copyright©2015 NTT corp. All Rights Reserved. • Made by me + other developers • Tomoya Hibi: Lagopus developer • Hiroki Kumazaki: Lock free master • What is Gondola? • Manage KVM and Docker with Hybrid tenants and support for various tunneling protocols(VLAN, MPLS) • Try to run Lagopus as Hypervisor switch easily • Exhibited in SDN/Cloud Program Contest 2014 Okinawa • Not OSS yet Gondola
- 33. 33Copyright©2015 NTT corp. All Rights Reserved. • Using etcd for DB and notification • Master node • REST/GUI • Scheduler • Alive monitoring • Manage agent nodes • Agent node • Manage VMs and containers Gondola Architecture HV（KVM+Docker） HV（KVM+Docker） gondola gondola HV（KVM+Docker） HV（KVM+Docker） gondola master HV（KVM+Docker） lagopus Container VM Ryu Libvirt API NW IF API REST API VM management db etcd Docker API gondola agent DC Network Virtual/Phisical port Physical machine GUI User App heart beat change notification write Network Gondola SW App.
- 34. 34Copyright©2015 NTT corp. All Rights Reserved. • Demo1 • Create VM/Container and ping/ssh between them • Isolation between multi-‐‑‒tenant networks • Demo2 • OpenStack Integration Demo video Host (ubuntu02) kvm-‐‑‒test 192.168.0.10 Host (ubuntu03) docker-‐‑‒test 192.168.0.5 lagopus lagopus docker-‐‑‒test2 192.168.0.5 Host (ubuntu02) docker-‐‑‒kvm Host (ubuntu03) docker-‐‑‒openstack lagopus lagopus kvm-‐‑‒openstack2OpenStack API/GUI mistake -> kvm-openstack
- 35. 35Copyright©2015 NTT corp. All Rights Reserved. • OpenStack as NFV Infrastructure Manager • NFV requirements discussion • DPDK for high performance packet processing • Advantage of Lagopus Summary