DPDK Summit - 08 Sept 2014 - NTT - High Performance vSwitch

0
Copyright©2014 NTT corp. All Rights Reserved.
High-performance vSwitch of the user, by the user, for the user
A bird in cloud
Yoshihiro Nakajima, Wataru Ishida, Tomonori Fujita, Takahashi Hirokazu, Tomoya Hibi, Hitoshi Matsutahi, Katsuhiro Shimano
NTT Labs

1
Agenda
vSwitch of the user
Background and motivation
vSwitch by the user
High-performance software-switch
•Concept
•Design
•Implementation & techniques
•Performance Evaluation
vSwitch for the user
PoC of carrier usecase
•Segment Routing or Service chaining (Tentative)
Open source community

2
Our mission
Open innovation with open source
for networking

3
vSwitch of the user
•Background
•Motivation

4
Trend shift in networking
Closed (Vender lock-in)
Yearly dev cycle
Waterfall dev
Standardization
Protocol
Special purpose HW / appliance
Distributed cntrl
Custom ASIC / FPGA
Open (lock-in free)
Monthly dev cycle
Agile dev
DE fact standard
API
Commodity HW/ Server
Logically centralized cntrl
Merchant Chip

5
What is Software-Defined Networking?
5
Innovate services and applications in software development speed.
Reference: http://opennetsummit.org/talks/ONS2012/pitt-mon-ons.pdf
Decouple control plane and data plane →Free control plane out of the box （OpenFlow is one of APIs）
Logically centralized view →Hide and abstract complexity of networks, provide entire view of the network
Programmability via abstraction layer
→Enables flexible and rapid service/application development
SDN conceptual model

6
OpenFlow vs conventional NW node
OpenFlow controller
OpenFlow switch
Data-plane
Flow Table
Control plane (Routing / swithching)
Data-plane
(ASIC, FPGA)
Control plane (routing/ switching)
Flow match
action
Flow match
action
counter
OpenFlow Protocol
Flexible flow match pattern (Port #, VLAN ID, MAC addr, IP addr, TCP port #)
Action (frame processing)
(output port, drop, modify/pop/push header)
Flow statistics (# of packet, byte size, flow duration,… )
counter
Conventional NW node
OpenFlow
Flow Table
#2
Flow Table
#3

7
7
Evaluate the benefits of SDN by implementing our control plane and switch

8
Motivation of vSwitch
Agile and flexible networking
Full automation in provisioning, operation and management
Seamless networking for customers
Server virtualization and NFV needs a high-performance software switch
Short latency
Wire-rate in case of short packet (64B)
No High-performance OpenFlow 1.3 software switch for wide-area networks
1M flow rules
10Gbps-wire-rate
Multiple table supports

9
vSwitch requirement from user side
1.Run on the commodity PC server and NIC
2.Provide a gateway function to allow connect different various network domains
Support of packet frame type in DC, IP-VPN, MPLS and access NW
3.Achieve 10Gbps-wire rate with >= 1M flow rules
low-latency packet processing
flexible flow lookup using multiple-tables
High performance flow rule setup/delete
4.Run in userland and decrease tight-dependency to OS kernel
easy software upgrade and deployment
5.Support various management and configuration protocols.

10
Why Intel x86 server for networking?
Strong processing power
Many CPU cores
•8 cores/CPU (2012), 12 cores/CPU (2013),…
Rapid development cycle
Available everywhere
Can be purchased in Akihabara!
3 month lead time in case of special purpose HW
Reasonable price and flexible configuration
8 core CPU (ATOM) with 1,000 USD
Can chose configuration according to your budget
Many programmers and engineers
Porting to a NPU needs special skill and knowledge

11
vSwitch by the user
•Concept
•Design
•Implementation & techniques
•Performance Evaluation

12
vSwitch by the user Concept

13
Target of Lagopus vSwitch
High performance soft-based OpenFlow switch
10Gbps wire-rate packet processing / port
1M flow rules
Expands the SDN apps to wide-area networks
Not only for data centers
WAN protocols, e.g. MPLS and PBB
Various management /configuration interfaces
Open Innovation
Community-based development

14
Approach for vSwitch development
Simple is better for everything
Packet processing
Protocol handling
Straight forward approach
No OpenFlow 1.0 support
Full scratch (No use of existing vSwitch code)
User land packet processing as much as possible, keep kernel code small
Kernel module update is hard for operation
Every component & algorithm can be replaced
Flow lookup, library, protocol handling

15
vSwitch design
Simple modular-based design
Switch control agent
Data-plane
Switch control agent
Unified configuration data store
Multiple management IF
Data-plane control with HAL
Data-plane
High performance I/O library (Intel DPDK)
Raw socket for test

16
Implementation strategy for vSwitch
Massive RX interrupts handling for NIC device
=> Polling-based packet receiving
Heavy overhead of task switch
=> Thread assignment (one thread/one physical CPU)
Lower performance of PCI-Express I/O and memory bandwidth compared with CPU
=> Reduction of # of access in I/O and memory
Shared data access is bottleneck between threads
=> Lockless-queue, RCU, batch processing

17
Processing bypass for speed
NIC
skb_buf
Ethernet Driver API
Socket API
vswitch
packet buffer
packet buffer memory
Standard linux application
1. Interrupt & DMA
2. system call (read)
User space
Kernel space
Driver
4. DMA
3. system call (write)
NIC
Ethernet Driver API
User-mode I/O & HAL
vswitch
packet buffer
Application with intel DPDK
1. DMA Write
2. DMA READ
DPDK Library
Polling-base
packet handling
Event-base packet handling

18
Packet processing using multi core CPUs
Exploit many core CPUs
Reduce data copy & move (reference access)
Simple packet classifier for parallel processing in I/O RX
Decouple I/O processing and flow processing
Improve D-cache efficiency
Explicit thread assign to CPU core
NIC 1
RX
NIC 2 RX
I/O RX
CPU0
I/O RX CPU1
NIC 1
TX
NIC 2 TX
I/O TX
CPU6
I/O TX
CPU7
Flow lookup packet processing
CPU2
CPU4
CPU3
CPU5
NIC 3
RX
NIC 4 RX
NIC 3
TX
NIC 4 TX
NIC RX buffer
Ring buffer
Ring buffer
NIC TX buffer

19
Packet batching
Reduce # of lock in flow lookup table
Frequent locks are required
•Switch OpenFlow agent and counter retrieve for SNMP
•Packet processing
Input packet
Lookup table
Input packet
Lookup table
Lock
Unlock
ロック
Unlock
●Naïve implementation
●Packet batching implementation
Lock is required for each packet
Lock can be reduced due to packet batching

20
Bypass pipeline with flow $
20
●Naïve implementation
table1
table2
table3
Input packet
table1
table2
table3
Input packet
Flow cache
1. New flow
2. Success flow
Output packet
Multiple flow $ generation & mngmt
Write flow $
●Lagopus implementation
Output packet
Packet
Reduce # of flow lookup in multiple table
Exploit flow $ for

21
vSwitch by the user Performance Evaluation

22
Performance Evaluation
Summary
Throughput: 10Gbps wire-rate
Flow rules: 1M flow rules
Evaluation models
WAN-DC gateway
•MPLS-VLAN mapping
L2 switch
•Mac address switching

23
Performance Evaluation
Evaluation setup
Server spec.
CPU: Dual Intel Xeon E5-2660
•8 core(16 thread), 20M Cache, 2.2 GHz, 8.00GT/s QPI, Sandy bridge
Memory: DDR3-1600 ECC 64GB
•Quad-channel 8x8GB
Chipset: Intel C602
NIC: Intel Ethernet Converged Network Adapter X520-DA2
•Intel 82599ES, PCIe v2.0
Server
Lagopus
Flow table
tester
Flows
Throughput (bps/pps/%)
Flow
rules
Packet size
Flow cache (on/off)

24 Copyright©2014 NTT corp. All Rights Reserved.
WAN-DC Gateway
0
1
2
3
4
5
6
7
8
9
10
0 200 400 600 800 1000 1200 1400 1600
Throughput (Gbps)

Packet size (byte)

10 flow rules
100 flow rules
1k flow rules
10k flow rules
100k flow rules
1M flow rules
Throughput vs packet size, 1 flow, flow-cache
0
1
2
3
4
5
6
7
8
9
10
1 10 100 1000 10000 100000 1000000
Throughput (Gbps)

flows

10k flow rules
100k flow rules
1M flow rules
Throughput vs flows, 1518 bytes packet

25
L2 switch performance (Mbps) 10GbE x 2 (RFC2889 test)
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
LINC
OVS
(netdev)
OVS
(kernel)
Lagopus
(software)
Mbps
72
128
256
512
1024
1280
1518
Intel xeon E5-2660 (8cores, 16threads), Intel X520-DA2, DDR3-1600 64GB
Packet size

26
vSwitch for the user
PoC of carrier usecase
Open source community

27
SPRING
alternative way of IP forwarding
use “labels” to forward packets
much simpler architecture than MPLS network
can utilize traditional MPLS routers
good chemistry with NFV
For what ?
policy-based routing
•explicit routing instead of shortest path routing
service-chaining (NFV)
•apply services to specific traffic pattern

Career Network with SPRING
Source
Destination
NFV server
 use lagopus as NFV server with career grade
routers
 all nodes include NFV server are labeled

Career Network with SPRING
Source
Destination
NFV server
 at the ingress router, label stack is pushed
 label stack represents the policy
IP

30
PoC Implementation

31
Demo

32
Open Source Activity for the user
Lagopus goes wild since this summer
Your contribution are very welcome!
http://lagopus.github.io/
Twitter: lagopusvswitch
Ryu: component-based SDN framework
OpenFlow 1.3, 1.4, BGP, BMP, etc..
http://osrg.github.io/ryu/

33
Summary
We contribute vSwitch for user
Improve performance and scalability
Enhance the functionality and capability
DPDK is great library for user
Easy to develop & implement
Easy to deploy like UNIX apps

34
Thank you!
This research is a part of the project for “Research and Development of Network Virtualization Technology” supported by the Ministry of Internal Affairs and Communications.

DPDK Summit - 08 Sept 2014 - NTT - High Performance vSwitch

More Related Content

What's hot (20)

Viewers also liked (17)

Similar to DPDK Summit - 08 Sept 2014 - NTT - High Performance vSwitch (20)

More from Jim St. Leger (8)

Recently uploaded (20)

DPDK Summit - 08 Sept 2014 - NTT - High Performance vSwitch