2012 workshop wed_ethernet_servicesoveri_poib
- 1. Ethernet Services over IPoIB Ali Ayoub, Mellanox Technologies March, 2012 www.openfabrics.org 1
- 2. Background • What is IPoIB? – IP encapsulation over InfiniBand – RFC 4391/4392 – Provides IP services over InfiniBand fabric • Benefits: – Acts like an data-link within the TCP/IP Stack • Socket-based apps run transparently – Allow users to run IP-based unmodified applications on InfiniBand – Supports NIC offloads: CSUM, TSO, etc.. And other performance features such as: NAPI, TSS, RSS, etc.. www.openfabrics.org 2
- 3. IPoIB Packet Format • IPoIB integrated as Data Link layer • Sockets API is not affected TCP/IP Stack L5 Sockets API L4 L3 L2 (IPoIB) L1 (IB) • Packet frame: IB Header www.openfabrics.org IPoIB Header IP Header Payload 3
- 4. Limitations • IPoIB is limited for IP applications only – No Ethernet Header encapsulation • IPoIB network interface doesn’t act like a standard Ethernet Network Device • • • • Non-standard Ethernet interface utilities & characteristics 20 Bytes link-layer address Ethernet link-layer (MAC) address is 6 Bytes DHCP requires using dhcp-client identifier – Host administrator must be aware of PKEYs • While Ethernet interfaces support VLANs • vconfig command cannot be used www.openfabrics.org 4
- 5. Limitations cont. • Link Layer setting, modification, migration – IPoIB Link Layer address is based on QPN/GID • Cannot be controlled by the user – Host administrator cannot set or re-configure the link layer address (MAC) IPoIB Link Layer (RFC 4391) www.openfabrics.org 5
- 6. Limitations cont. • IPoIB cannot be used in fully-virtualized or para-virtualized environments – Hypervisor networking model normally use “bridged mode” – Virtual Switch (vSwitch) is Ethernet L2 switch – IPoIB NIC cannot be enslaved to a vSwitch • Promiscuous mode is unsupported – Simplifies vSwitch functionality www.openfabrics.org 6
- 7. IPoIB & Network Virtualization VM “sees” an Ethernet Network Device (e1000, virtio, vmxnet, etc..) Hypervisor Packet sent out by the VM includes an Ethernet header Virtual Interface(s) (vifX) Virtual Switch(s) (vSwitchX) Uplink(s) (pifX) vSwitch0 vSwitch1 pif0 Network Virtualization Model vSwitch forwards the Ethernet packet to the host Network Interface pif1 Native IPoIB driver serves only IP packets (doesn’t expect Ethernet Header) Reference http://www.vmware.com/resources/techresources/997 www.openfabrics.org 7
- 8. Solution • Create a new Ethernet Network Device over IPoIB interface • Managed by “eIPoIB” kernel module • Registers a standard Ethernet Interface into the Operating System • Same “Look & Feel” as an Ethernet NIC TCP/IP IP Frame Ethernet Ethernet Frame eIPoIB Shim Layer Strip Ethernet Header – ifconfig, vconfig, ethtool, promiscuous, etc.. • eIPoIB is a shim layer driver between the TCP/IP stack and the native IPoIB www.openfabrics.org Push Ethernet Header IP Frame Strip IPoIB Header IPoIB Push IPoIB Header IPoIB Frame IB layer 8
- 9. eIPoIB vs. IPoIB Interface • ifconfig ib0 www.openfabrics.org 9
- 10. eIPoIB vs. IPoIB Interface • ifconfig eth3 www.openfabrics.org 10
- 11. eIPoIB Operations in a Nutshell • Initialization: TCP/IP – Register Ethernet network interface (eth0) – Map it to native IPoIB interface (ib0) IP Frame • Transmit: – Packet to be sent has Ethernet Header – eIPoIB strips the Ethernet Header – Pass the IP packet to the underlying IPoIB interface. Ethernet Ethernet Frame • Non IP/ARP/RARP packets are dropped – Native IPoIB interface sends the packet out eIPoIB Shim Layer • Receive: – – – – Packet received is an IP packet Native IPoIB handovers the IP packet to eIPoIB layer eIPoIB pushes the Ethernet header Packet forwarded to upper layers as regular Ethernet frame Push Ethernet Header Strip Ethernet Header IP Frame Strip IPoIB Header IPoIB Push IPoIB Header IPoIB Frame IB layer www.openfabrics.org 11
- 12. eIPoIB Frame Flow IP Header IP Data TCP/UDP TCP/IP IP Frame Ethernet Header Ethernet Ethernet Data Ethernet Frame IP Header IP Data TCP/UDP eIPoIB Shim Layer Strip Ethernet Header IPoIB Header IPoIB IP Data Push IPoIB Header www.openfabrics.org IPoIB Header IP Data CRC InfiniBand Header IP Frame IPoIB Frame IB layer 12
- 13. eIPoIB/IPoIB Interoperability TCP/IP • Same wire protocol • eIPoIB “talks” IPoIB IP Frame Ethernet Ethernet Frame TCP/IP eIPoIB Shim Layer Strip Ethernet Header IPoIB www.openfabrics.org IP Frame Strip IPoIB Header IPoIB InfiniBand Header IPoIB Header Ethernet Data CRC IB layer Push Ethernet Header Push IPoIB Header IPoIB Frame IB layer 13
- 14. eIPoIB Networking Model (KVM) eth0 ib0 Reference http://www.linux-kvm.org/wiki www.openfabrics.org 14
- 15. eIPoIB Design (KVM) vm1 vif0.1 vm2 vif0.2 vm.x vif0.3 vif0.x QEMU TCP/IP tap0.1 tap0.2 tap0.3 tap0.x br0 eth0 ib0.0 ib0.1 ib0.2 CMA ib0.3 ib0.x ib0 IB port 0 www.openfabrics.org 15
- 16. eIPoIB – Closer Look • Promiscuous mode • MAC Translation www.openfabrics.org 16
- 17. Promiscuous Mode • Normally vSwitch uplink is put in promiscuous mode – InfiniBand doesn’t support promiscuous mode • Promiscuous mode is simulated by: a. Snooping the src.mac and vlan of outgoing packets b. OS notifies the driver when a new MAC/VLAN need to be “served”. For example: the driver can get a notification from libvirt library (available for KVM/XEN) when a new VM virtual NIC is created • Multicast promiscuous support requires IGMP/MLD Snooping in eIPoIB level www.openfabrics.org 17
- 18. MAC Translation • Requirements: – eIPoIB exposes an Ethernet MAC • Local Ethernet MAC (LEMAC) 6 bytes length – eIPoIB neighbors are seen as Ethernet neighbors • Remote Ethernet MAC (REMAC) 6 bytes length – IPoIB Local MAC used on the wire • Local IPoIB MAC (LIMAC) 20 bytes – IPoIB neighbors’ MAC used on the wire • Remote IPoIB MAC (RIMAC) 20 bytes www.openfabrics.org 18
- 19. MAC Translation cont. Receive Flow • Receiver QP => dst.mac • SQPN/SLID => src.mac – Remember QPN/LID to IPoIB-MAC (QPN/GID) mapping • IPoIB header => Ethernet ethertype • Replace IPoIB header by Ethernet header Transmit Flow • src.mac => QP (child interface) – Source MAC normally controlled by the host/hypervisor admin • dst.mac => IPoIB-MAC • Ethernet header ethertype => IPoIB Header • Strip Ethernet header, and handover packet to IPoIB ARP/NDP packets are modified in TX/RX flow, so the MAC addresses in the packet payload are updated accordingly www.openfabrics.org 19