Mellanox for OpenStack - OpenStack最新情報セミナー 2014年10月

Mellanox for OpenStack
Senior System Engineer | Mellanox Technologies Japan KK
友永和総 (Kazusa Tomonaga)
2014年10月8日 – OpenStack最新情報セミナー
※本内容は予告なく変更となる場合があります。

Agenda
 1. Mellanox Overview
 2. Mellanox CloudX
 3. CloudX Deep Dive
• Network Virtualization Acceralation – VXLAN NICハードウェアオフロード
• High Speed Storage Interconnect – iSER (iSCSI over RDMA)
• SR-IOV / NFV
 7. Cloud Application Performance
 8. Demo
 9. まとめ
© 2014 Mellanox Technologies 2

Mellanox Overview

メラノックス会社概要
Ticker: MLNX
 高帯域、低レイテンシーなサーバ・ストレージ間インターコネクトのリーディングカンパニー
• FDR 56Gb/s InfiniBandと10/40/56ギガビットEthernet
• アプリケーションのデータ処理時間を大幅に削減
• データセンタ・サービス基盤のROIを劇的に向上
 会社概要
• 本社：イスラエルYokneam, 米国Sunnyvale, California
• 従業員数：全世界で1,428名（2014年6月末時点）
 堅実な財務内容
• 2013年度売上： $390.9M
• 2014年度第二四半期売上： $102.6M
• 2014年度第三四半期売上： $114M~$118M
• Cash + investments @ 6/30/14 = $343.7M

Exponential Data Growth – The Best Platforms Are Needed
We Live in a World of Data
More Devices More Applications More Data
Data Needs to be Accessible Always and in Real-Time

Leading Supplier of End-to-End Interconnect Solutions
Analyze Store
Enabling the Use of Data
Comprehensive End-to-End InfiniBand and Ethernet Portfolio
ICs Adapter Cards Switches/Gateways Software and Services Metro / WAN Cables/Modules
At the Speeds of 10, 40 and 100 Gigabit per Second

メラノックスのコアテクノロジー : 高性能・高集積ASIC
 VPI (Virtual Protocol Interconnect) テクノロジー
• IB and EN with single chip (ConnectX-3、SwitchX-2)
• IB and EN port by port (ConnectX-3、SwitchX-2)
• IB/EN Bridging (SwitchX-2)
 高スループット、低レイテンシー、超低消費電力 (Ultra Low Power)
 RDMA (Remote Direct Memory Access) 対応、高速データ転送
 VXLAN/NVGREオフロード (ConnectX-3 Pro)
Typ power: 7.9W
3.0 x8
17mm
2 x IB FDR (56Gbps)
3.0 x16
45mm
2 x 56Gbps
Ethernet mode:
1/10/40/56GbE
InfiniBand/Ethernet
InfiniBand/Ethernet
144組のネットワークSerDesを搭載
36 x 40/56GbE
64 x 10GbE
48 x 10GbE + 12 x 40/56GbE
Ethernet mode:
1/10/40/56GbE
• InfiniBand or Ethernet
• InfiniBand + Ethernet
• InfiniBand / Ethernet Bridging
36x 40GbE: 83W
64x 10GbE: 63W
2pt 40GbE (100% load power)

メラノックス製品の3つの特長
1) 高スループット
 アダプタ/スイッチ - 10/40/56Gbps
2) 低レイテンシー
 スイッチ : 220ns (L2 40GbE), 330ns (L3 40GbE)
 アダプタ : ~1us
3) 超低消費電力
 スイッチ : 83W (36 x 40GbE 100% load)
 アダプタ : 7.9W (2x 40GbE typ. Power)
エンドトゥエンドソリューション：
 アダプタ向けASIC、スイッチ向けASICを自社開発（フルカスタムASIC）
 アダプタ、スイッチ、ケーブルを自社開発
• 100Gbps実現に向けた光モジュール/ケーブル関連技術を自社で保有
(2013年、Kotura社とIPtronics社を買収)
 ドライバ、高速化ソフトウェア、管理ソフトウェアを自社開発

Mellanox CloudX

Making the Cloud Easy
The World is Moving to the Cloud
But Building a Cloud is a Challenge
Private Cloud Public Cloud
Mellanox Makes it Easy: Deploy, Use and Maintain

CloudX is the Most Efficient Cloud Platform
Lower Your IT Cost by 50%!
The Platform for Creating the Applications of Tomorrow

CloudX: Optimized Cloud Platform
 CloudX is a group of reference architectures which
allow building the most efficient, high performance
and scalable Infrastructure As A Service (IaaS)
clouds based on Mellanox superior interconnect and
off the shelf building blocks
 Supports the most popular cloud software
• OpenStack
• Windows Azure Pack (WAP)
• VMware

Mellanox OpenStack Reference Documents
http://www.mellanox.com/openstack/
Designing CloudX Solution using Mirantis
Fuel OpenStack Software
http://community.mellanox.com/docs/DOC-1464
HowTo Configure iSER Block Storage for
OpenStack Cloud with Mellanox ConnectX-3
Adapters
http://community.mellanox.com/docs/DOC-1462
今後、随時拡充されていく予定です。

Mellanox CloudX
• メラノックスが最先端の大規模クラウドシステムでグローバルに培ったテクノロジーを
• リファレンスアーキテクチャのドキュメントを無償提供、プロフェッショナルサービス
• 世界最先端のクラウドシステム構築を実現
• エンドトゥエンドの高性能ネットワークを実現する
ハードウェア
• Mellanox ネットワークアダプタ
• Mellanox ネットワークスイッチ
• Mellanox ネットワークケーブル
• ハードウェアサポートサービス
• CloudXシステム構築プロフェッショナルサービス
• CloudXシステム運用・保守テクニカルサポート
パッケージ化
メニューも用意

Mellanox CloudX ハードウェア : Ethernetスイッチ
SX1036
The Ideal 40GbE ToR/Aggregation
SX1024
Non-blocking 10GbE 40GbE ToR
SX1016
Highest density 10GbE ToR
SX1012
Ideal storage/Database 10/40GbE Switch
 Highest Capacity in 1RU
• 12 to 36 QSFP
• 64 x 10GbE
 Value
 Latency
 Power (SX1036)
• VPI 56Gb/s InfiniBand & Ethernet
• End to end solution
• 220ns L2 latency
• 330ns L3 latency
• Under 1W per 10GbE interface
• 0.6W per 10GbE of throughput
SX1036 – 83W
SX1016 – 62W
SX1024 – 75W
SX1012 – 50W
For 100% L oad Power

Shattering Ethernet Switching Performance Records
 Leading throughput: 2.5X better
• 2.88Tb/s throughput on a single chip, running Full Wire Speed at any packet size
 Leading L2 unicast/multicast latency for L2/L3 switches: 2X better
• 198-223ns for any packet size
 Leading L3 latency: 2X better
• 321-337ns for any packet size
 Industry record power efficiency: 6X better than competition
• Sub 0.6Watt per 10GbE throughput with 100% load at Full Wire Speed
‘Zero’ Jitter!
206 209 220 216 219 223 222 222 223
1,000
800
600
400
200
0
64 128 256 512 1,024 1,280 1,518 2,176 9,216
L2 Min
L2 Average
L2 Max
Packet Size (Bytes)
Latency (ns)

Best ROI – Switch Silicon Example
What matters in switch silicon?
• Highest switching capacity
• Lowest power
• Lowest latency
x1.7 Better x2.5 Better x5 Better
Switching Capacity (Bpps)
0.95
1.44
2.5
1.2
1
0.8
0.6
0.4
0.2
Latency (μsec)
3.5
3
2.5
2
1.5
1
0.5
Power (Watt/Gb)
3
2.5
2
1.5
1
0.5
0
Trident+ Trident2 SwitchX
1
0.5
0.2
0
3
2
0.4
0

Mellanox CloudX ハードウェア : Ethernetネットワークアダプタ
• 10/40/56ギガビットEthernetサポート
• HPCシステムでも広く活用される世界トップレベルの性能
• 低レイテンシー、高スループット、超低消費電力
• RDMA (Remote Direct Memory Access) サポート
• RoCE (L2 Ethernet RDMA)
• RoCEv2 (L2/L3 Ethernet RDMA)
• オーバーレイネットワークオフロード
• VXLAN (Linux, VMware*)
• NVGRE (Windows) *Available soon
The Foundation of Cloud 2.0
The World’s First NVGRE / VXLAN Offloaded NIC

Mellanox CloudX プロフェッショナルサービス
Mellanox CloudXテクノロジーを活用した200ノードまでのクラウドシステム構築を実現する技術
サービス
• Mellanox CloudX Technical implementation package up to 200 nodes
• CloudX Planning and design, Installation, Configuration, Performance Tuning, Testing
and Knowledge Transfer
GPS-0200-ONST-CLOUDX
オンサイトサポート
GPS-0200-REMT-CLOUDX リモートサポート
※価格等、詳細は個別にお問い合わせください。

Mellanox CloudX テクニカルサポート
Ordering P/N Description Comments
SUP-CLOUDX-1S CloudxX™ System Support
1 YEAR SILVER SUPPORT
Includes support for CloudX™ operational assistance, including
integrated components for Mellanox Opensack plugins: Neutron
Mellanox Plug-in, eswitchd, Cinder over iSER and/or other
CloudX supported architecture plug-in’s operational assistance,
Mellanox OFED CloudX™ operational assistance
下記のようなMellanox CloudX使用にあたって必要となる
コンポーネント群の運用についての技術サポートを提供
・Mellanox Neutron Plug-in
・Mellanox eSwitchd (Embedded Switch)
・Mellanox iSER for Cinder
・Other CloudX components
・MLNX OFED
3 YEAR SILVER SUPPORT
SUP-CLOUDX-1G CloudxX™ System Support
1 YEAR GOLD SUPPORT
3 YEAR GOLD SUPPORT
※価格等、詳細は個別にお問い合わせください

CloudX Deep Dive

Next Generation Software Defined Networks
EVN: Efficient Virtualized Network
Fully Integrated Solution Combines
RDMA, Convergence, & SDN/Virtualization
Efficient Virtualized Network (EVN)
Fully Integrated, World Class SDN Solution
VIRTUALIZATI ACCELERATION CONVERGENCE
ON
EVN: More than SDN - Efficient Clouds Need an Efficient Virtualized Network

Accelerate All Three Elements Required for SDN Networks
= + +
Software Defined Networks Virtual Network Management Overlay Network Tunnels OpenFlow
1.Centralized Software Based Control Plane
• Enables network virtualization
2.Overlay Networks – NVGRE/VXLAN
• Isolation, Scalability, Simplicity
• Mellanox accelerates overlay networks to offer bare metal speed
3.Industry Standard API – OpenFlow
• Enables an industry ecosystem and innovation

Comprehensive OpenStack Integration for Switch and Adapter
Integrated with Major
OpenStack
Distributions
In-Box With Havana &
Ice House
Neturon-ML2
support for
mixed VM
environment
(VXLAN, PV,
SRIOV)
Ethernet
Neutron :
Hardware
support for
security and
isolation
Accelerating
storage
access by up
to 5x
OpenStack Plugins Create Seamless Integration, Control, & Management

Network Virtualization Acceleration

ConnectX-3 Pro Accelerates Overlay Networks
VM1 VM2 VM3 VM4
Server
VXLAN Overlay Networks
Virtual Domain 1
Virtual Domain 2
Virtual Domain 3
OpenFlow
Virtual Network
Management API
Virtual
View
Physical
View
Virtual Overlay Networks Simplifies
Management and VM Migration
ConnectX-3 Pro
Overlay Accelerators Enable
Bare Metal Performance
VM5 VM6 VM7 VM8
Server
Mellanox SDN Switches & Routers
Overlay Network Virtualization: Isolation, Simplicity, Scalability

VXLAN Performance
VXLAN Offload Engine – 5X higher throughput, 75% lower CPU utilization
CPU Usage Per Gbit/sec with VxLAN
4.50
4.00
3.50
3.00
2.50
2.00
1.50
1.00
0.50
0.00
1 VM 2 VMs 3 VMs
CPU% / Bandwidth (Gbit/sce)
Lower is Better
VxLAN in software 3.50 3.33 4.29
VxLAN HW
0.90 0.89 1.19
Offload
Total VM Bandwidth when using VxLAN
25
20
15
10
5
0
1 VM 2 VMs 3 VMs
Bandwidth [Gbit/sec]
Higher is Better
VxLAN in software 2 3 3.5
VxLAN HW Offload 10 19 21

Turbocharge Your OVS with Mellanox ConnectX-3 Pro
Source: PLUMGrid white paper
“Mellanox ConnectX-3 Pro card is the only way to scale-out PLUMgrid’s
Virtual Network Infrastructure (VNI) overlay-based Architecture”

High Performance Storage Interconnect

iSER – iSCSI Extension for RDMA
 Zero copy using RDMA
 IB and Ethernet (RoCE)
 Transport protocol implemented in hardware (minimal CPU cycles per IO)
 Open stack integration
 Support for T10/DIF
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
iSCSI/TCP iSCSI/RDMA
IO Latency @ 4K IO [micsec]
iSCSI (TCP/IP)
1 x FC 8 Gb
port
4 x FC 8 Gb
port
iSER 1 x
40GbE/IB Port
iSER 2 x
40GbE/IB Port
(+Acceleration)
2500
2000
1500
1000
500
0
KIOPs 130 200 800 1100 2300
K IOPs @ 4K IO Size
5-10% the latency under 20x the workload

Mellanox Accelerates Storage: More than 4X Greater Throughput
OpenStack Storage Performance *
1.3
5.5
6
5
4
3
2
1
0
iSCSI over TCP iSER
GBytes/s
* iSER patches are available on OpenStack
branch: https://github.com/mellanox/openstack
Built-in OpenStack Components/Management &
Cinder/iSER to Accelerate Storage Access

SR-IOV / NFV

Mellanox Single Root I/O Virtualization (SR-IOV)
 PCIe device presents multiple instances to the OS/Hypervisor
 Enables Application Direct Access (ADA)
• Reduces CPU overhead and improves application performance
 Eliminates virtualization penalty with RDMA & ADA
• Low latency applications benefit from the Virtual infrastructure
VM
VF Device Driver
VM
VF Device Driver
VMn
VF Device Driver
VM
Virtual NIC
Physical Function Device Driver
PF VF VF VF

SR-IOV Boosts Ethernet Performance
 SR-IOV Accelerates RoCE
• Enables native RoCE performance in virtualized environments
40
35
30
25
20
15
10
RoCE – SR-IOV Throughput
1 VM 2 VM 4 VM 8 VM 16 VM
Throughput (Gb/s)
Throughput (Gb/S)
1 VM 2 VM 4 VM 8 VM
No Performance Compromise in Virtualized Environment
3
2.5
2
1.5
1
0.5
0
Latency (us)
RoCE - SR-IOV Latency
Message Size 2B Message Size 16B Message Size 32B

SR-IOV and eSwitch
VM
OS
tap tap SR-IOV
Para-virtual
VM
OS
VM
OS
VM
OS
to the
VM
Embedded
Switch
Mellanox
Neutron
Agent
Create/delete,
configure policy
per VM vNIC
OpenStack Manager
Neutron
Plug-Ins
Servers
Manager
Compare eSwitch vs OVS
Qperf (TCP) Latency
Provision VM & Fabric Policy in hardware, through standard APIs
Benefits: Isolation, functionality, performance & offload, simpler SDN

ConnectX-3 Family QoS
 Port based ETS
• Max bandwidth per TClass
• Bandwidth reservation per TClass
 Per Function rate limiter
• Average bandwidth
• Peak bandwidth
• Maximum burst at peak bandwidth
 Packet pacing
• Low jitter packet pacing for work
queues
Work
Queue
Work
Queue
Work
Queue
RR
arbiter
QoS
Queue
Work
Queue
Work
Queue
Work
Queue
RR
arbiter
QoS
Queue
Work
Queue
Work
Queue
Work
Queue
RR
arbiter
QoS
Queue
Work
Queue
Rate Shaper Rate Shaper
Priority 0
Arbiter
Work
Queue
Work
Queue
RR
arbiter
QoS
Queue
Priority 1
Arbiter
Priority ...
TC2
Flow
Ctrl
TC Group 1
DWRR
Strict Priority
Flow
Ctrl
TC Group 0
DWRR
TC Group 7
DWRR
TC0
TC1
Flow
Ctrl
TC3
Flow
Ctrl
TC7
Flow
Ctrl ...
L H
...
0
Priority
1
Priority
2
Priority
3
Priority
7
Rate Limiter
Rate Limiter
Function Rate Limiter Enhanced ETS Packet Pacing

Record Breaking 195Gbs for Guest VM over DPDK
 High performance 10/40/56Gbps
 SR-IOV enabled
 OpenFlow enabled eSwitch
 OpenStack Neutron Plug-in
 PMD for DPDK: VM OS bypass
 Multi cores and RSS support
 Delivering bare-metal performance
Hypervisor
Legacy Software
vSwitches
Neutron Plug-in
OpenStack integration
VM
OS
VM
OS
High-performance packet processing solutions for
• Gateways
• Security appliances
• UTMs
• Virtual appliances
• etc.
• Data Plane libraries
• Optimized NIC drivers
Client’s Application Software
librte_pmd_mlx4 librte_crypto_nitrox VMware … 6WIND addons
……
Multicore Processor
SR-IOV eSwitch
Hardware Offload
VM
6WIND or Intel® DPDK
OpenFlow enabled
10/40/56Gbps

Remote HWA as a Service in NFV Cloud Model
Fat-tree SDN Switch Network
40GbE
56Gbps
IB FDR
Ethernet Ethernet Ethernet
Platform 1
40Gbps
DPI
BRAS
SGSN
GGSN
PE Router
STB
Firewall
RDMA / RoCE
CG-NAT SBC
RDMA / RoCE
RDMA / RoCE
Nx40Gbps Nx40Gbps Nx40Gbps
SX1024 Ethernet Switch
Fabric
HWA /
Signal
Processing
40Gbps
Platform 2
Fabric
HWA /
Signal
Processing
40Gbps
Platform X
Fabric
HWA /
Signal
Processing

iSCSI SAN/NAS Storage Architecture in an NFV Cloud Model
Fat-tree SDN Switch Network
Aggregation
ToR
Compute Storage
10/40/100Gbps
Compute Storage
10/40/100Gbps
RDMA / RoCE
RDMA / RoCE
Rack 1 Rack 2 10/40/100Gbps
12x10/40/100Gbps 12x10/40/100Gbps 12x10/40/100Gbps
Ethernet Switch
SAN/NAS Storage
Compute Storage
10/40/100Gbps
10/40/100Gbps
Ethernet Switch
SAN/NAS Storage
10/40/100Gbps
Rack n
RDMA / RoCE
Ethernet Switch
SAN/NAS Storage
iSCSI SAN/NAS Storage over Standard Ethernet Network: Shared Resource

Cloud Application Performance

CloudX Delivers Unbounded Cloud Performance
4X Faster Runtime!
Benchmark: TestDFSIO (1TeraByte, 100 files)
2X Higher Performance!
Benchmark: 1M Records Workload (4M Operations)
2X faster runtime and 2X higher throughput
2X Faster Runtime!
Benchmark: MemCacheD Operations
3X Faster Runtime!
Benchmark: Redis Operations

Accelerating Cloud Performance
SCSI Write Example, Linux KVM
6200
1200 800
8000
6000
4000
2000
0
I/O Size - 64 [KB]
Bandwidth
[MB/s]
iSER 16 VMs Write
40GbE VMs Write
10GbE
Fiber Channel - 8Gb
Migration of Active VM
38
10
50
40
30
20
10
0
Time [s]
Fibre 10 GbE
40 GbE
10GE-A 40GE-A
Storage
Migration
Virtualization
6X
3.5X
20X
VM-to-VM Latency Performance
40
50
40
30
20
10
2 0
Message Size - 256 [bytes]
Latency [us]
TCP ParaVirtualization
RDMA Direct Access

Demo

OpenStack VXLAN offload Demo – Mellanox ConnectX-3 Pro

OpenStack Cinder iSER (iSCSI over RDMA) Demo

まとめ – Mellanox CloudXで効率の良い先進クラウドが構築可能
• メラノックスが最先端の大規模クラウドシステムでグローバルに培ったテクノロジーを
• リファレンスアーキテクチャのドキュメントを無償提供、プロフェッショナルサービス
• 世界最先端のクラウドシステム構築を実現
• エンドトゥエンドの高性能ネットワークを実現する
ハードウェア
• Mellanox ネットワークアダプタ
• Mellanox ネットワークスイッチ
• Mellanox ネットワークケーブル
• ハードウェアサポートサービス
• CloudXシステム構築プロフェッショナルサービス
• CloudXシステム運用・保守テクニカルサポート
パッケージ化
メニューも用意

Mellanox for OpenStack - OpenStack最新情報セミナー 2014年10月

More Related Content

What's hot (20)

Viewers also liked (17)

Similar to Mellanox for OpenStack - OpenStack最新情報セミナー 2014年10月 (20)

More from VirtualTech Japan Inc. (20)

Recently uploaded (20)

Mellanox for OpenStack - OpenStack最新情報セミナー 2014年10月