Sharing High-Performance Interconnects Across Multiple Virtual Machines
Download as PPTX, PDF3 likes939 views
The document discusses sharing high-performance devices across multiple virtual machines using technologies such as VMware ESXi's direct path I/O and SR-IOV for device virtualization. It highlights the importance of achieving near bare-metal performance for various applications in high-performance computing, big data analytics, and machine learning. The document provides technical details and results from benchmarks that demonstrate the effectiveness of this approach in optimizing performance for demanding workloads.
Sharing High-Performance Interconnects Across Multiple Virtual Machines
- 1. Sharing High-Performance Devices Across Multiple Virtual Machines
- 2. Preamble • What does “sharing devices across multiple virtual machines” in our title mean? • How is it different from virtual networking / NSX, which allow multiple virtual networks to share underlying networking hardware? • Virtual networking works well for many standard workloads, but in the realm of extreme performance we need to deliver much closer to bare-metal performance to meet application requirements • Application areas: Science & Research (HPC), Finance, Machine Learning & Big Data, etc. • This talk is about achieving both extremely high performance and device sharing 2
- 3. Sharing High-Performance PCI Devices 1 Technical Background 2 Big Data Analytics with SPARK 3 High Performance (Technical) Computing CONFIDENTIAL 3
- 4. Direct Device Access Technologies Accessing PCI devices with maximum performance
- 5. VMware ESXi VM Direct Path I/O • Allows PCI devices to be accessed directly by guest OS – Examples: GPUs for computation (GPGPU), ultra-low latency interconnects like InfiniBand and RDMA over Converged Ethernet (RoCE) • Downsides: No vMotion, No Snapshots, etc. • Full device is made available to a single VM – no sharing • No ESXi driver required – just the standard vendor device driver CONFIDENTIAL 5 Virtual Machine Guest OS Kernel Application DirectPath I/O
- 6. • The PCI standard includes a specification for SR-IOV, Single Root I/O Virtualization • A single PCI device can present as multiple logical devices (Virtual Functions or VFs) to ESX and to VMs • Downsides: No vMotion, No Snapshots (but note: pvRDMA feature in ESX 6.5) • An ESXi driver and a guest driver are required for SR-IOV • Mellanox Technologies supports ESXi SR-IOV for both InfiniBand and RoCE interconnects 6 SR-IOV Virtual Machine Guest OS Kernel Application PF VF vSwitch VMXNET3 Device Partitioning (SR-IOV)
- 7. Enabling Technology: Full-featured Virtual Functions • Device capabilities – Multi-queue NIC – RDMA (user / kernel) – User-space networking • All capabilities may be used concurrently – For example, kernel NIC together with RDMA applications • Scales to any number of applications or vCPUs – Millions of hardware queues – Multiple interrupt vectors – Thousands of application address spaces PF VF VF Hypervisor Guest HW VF TCP/IP RDMA core RDMA storage User-space VF driver NIC RDMA RDMA Application VF driver
- 8. Remote Direct Memory Access (RDMA) • A hardware transport protocol – Optimized for moving data to/from memory • Extreme performance – 600ns application-to-application latencies – 100Gbps throughput – Negligible CPU overheads • RDMA applications – Storage (iSER, NFS-RDMA, NVMoF, Lustre) – HPC (MPI, SHMEM) – Big data and analytics (Hadoop, Spark) 8
- 9. How does RDMA achieve high performance? • Traditional network stack challenges – Per message / packet / byte overheads – User-kernel crossings – Memory copies • RDMA provides in hardware: – Isolation between applications – Transport • Packetizing messages • Reliable delivery – Address translation • User-level networking – Direct hardware access for data path 9 Kernel User RDMA-capable hardware NVMeF iSER Buf Buf Buf AppA AppB Buf Buf
- 10. • Mellanox SN2700 Spectrum Switch • Each Host: – OS: ESXi6.5, Memory: 256GB RAM, CPU: dual socket, Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GH (14 cores per socket) – Network: Mellanox ConnectX-4 100Gb (SR-IOV RoCE capable), Hypervisor Driver: MLNX- NATIVE-ESX-ConnectX-4-5_4.16.10.2-10EM-650.0.0.459867, Firmware: 12.20.10.10 – 1 VM, 20 vCPU and 220GB memory (NOTE: Running multiple NUMA-aligned VMs will generally yield higher performance – sometimes better than bare-metal) – Guest OS: Linux CentOS7.3x64 – VM Network: SR-IOV RoCE VF, VM Driver: MLNX OFED 4.1-1.0.2.0 – Running Spark benchmark (HiBench/Terasort) – Spark version https://github.com/Mellanox/SparkRDMA/, Java version 1.8.0-131 • Name Node Server: – OS: Linux CentOS7.3x64, Memory: 256GB RAM, CPU: dual socket, Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GH (14 cores per socket) – Network: Mellanox ConnectX-4 100Gb, Driver: Mellanox OFED 4.1-1.0.2.0, Firmware: 12.20.10.10 • For detailed configuration of Spark on vSphere, see: – Fast Virtualized Hadoop and Spark on All-Flash Disks – Best Practices for Optimizing Virtualized Big Data Applications on VMware vSphere 6.5 • https://blogs.vmware.com/performance/2017/08/big-data-vsphere65-perf.html Test Setup – SR-IOV 16 ESXi6.5 hosts, one Spark VM per host 1 Server used as Named Node
- 11. SPARK Test Results – vSphere with SR-IOV Runtime samples SR-IOV TCP (sec) SR-IOV RDMA (sec) Improvement Average 222 (1.05x) 171 (1.01x) 23% Min 213 (1.07x) 165 (1.05x) 23% Max 233 (1.05x) 174 (1.0x) 25% 0 50 100 150 200 250 Average Min Max Runtime(secs) TCP vs. RDMA (Lower Is Better) SR-IOV TCP SR-IOV RDMA
- 12. High Performance Computing Research, Science, and Engineering applications on vSphere
- 13. MPI Workloads 14 MPI (Message Passing Interface) Examples: • Weather forecasting • Molecular modelling • Jet engine design • Spaceship, airplane & automobile design
- 14. ESXi ESXiESXi 15 InfiniBand SR-IOV MPI Example Cluster 2 Host VM VM Host Host IB IB VM VM VM IB IB VM IB Cluster 1 SR-IOV SR-IOV IB IB SR-IOV SR-IOV IB IB SR-IOV SR-IOV • SR-IOV InfiniBand • All VMs: #vCPU = #cores • 100% CPU overcommit • No memory overcommit
- 15. ESXi ESXiESXi LinuxLinuxLinux 16 InfiniBand SR-IOV MPI Performance Test Cluster 2 Host VM VM Host Host IB IB VM VM VM IB IB VM IB Cluster 1 SR-IOV SR-IOV IB IB SR-IOV SR-IOV IB IB SR-IOV SR-IOV 93.4 93.4 98.5 169.3 169.3 Run time (seconds) Application: NAMD Benchmark: STMV 20-vCPU VMs for all tests 60 MPI processes per job Bare metal One vCluster Two vClusters 10%
- 16. Summary • Virtualization can support high-performance device sharing for cases in which extreme performance is a critical requirement • Virtualization supports SR-IOV device sharing and delivers near bare-metal performance – High Performance Computing – Big Data SPARK Analytics • The VMware platform and partner ecosystem address the extreme performance needs of the most demanding emerging workloads 17