How To Install OFED Linux/VMware/Windows
9 likes2,305 views
The document provides installation instructions for Mellanox OFED software on Linux, VMware ESXi, and Windows systems, focusing on Infiniband technology for high-speed networking. It includes commands for diagnosing Infiniband fabrics and managing connections, alongside benchmark results for data transfers. The source of the information is from the Sakura Internet Research Center's research documentation.
![Install OFED for VMware ESXi 5.1
1) Enable ESXi Shell & SSH
Troubleshooting Options > Enable ESXi Shell, Enable SSH and SSH login.
# vmware -v
VMware ESXi 5.1.0 build-799733
2) Download/Install MLNX_OFED
# cd /opt
# wget http://mellanox.com/downloads/Drivers/MLNX-OFED-ESX-1.8.0.0.zip
# esxcli software vib install -d /opt/MLNX-OFED-ESX-1.8.0.0.zip
# sync; sync; sync; reboot –f
# esxcfg-nics –l
vmnic_ib0 0000:01:00.00 ib_ipoib Up 56252Mbps Full
00:02:c9:34:1c:f1 1500 Mellanox Technologies MT27500 Family [ConnectX-3]](https://image.slidesharecdn.com/how-to-install-ofed2012-matsumoto-b-ibday06-130117024019-phpapp02/85/How-To-Install-OFED-Linux-VMware-Windows-5-320.jpg)
![IB Diagnostic Commands
# ibhosts *show InfiniBand host nodes in topology
Ca : 0x0002c9030010c36c ports 2 "hostname HCA-2"
Ca : 0x0002c9030010c360 ports 2 "hostname HCA-2"
Ca : 0x0002c9030010c328 ports 2 "sl6 HCA-1
# iblinkinfo *report link info for all links in the fabric
Switch 0x000b8cffff006f38 MT43132 Mellanox Technologies:
:
2 6[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 8 1[ ] "sl6 HCA-1" ( )
2 7[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 9 1[ ] "hostname HCA-2" ( )
2 8[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 10 1[ ] "hostname HCA-2" ( )
# ibtracert 9 10 *trace InfiniBand path
From ca {0x0002c9030010c360} portnum 1 lid 9-9 "hostname HCA-2"
[1] -> switch port {0x000b8cffff006f38}[7] lid 2-2 "MT43132 Mellanox Technologies"
[8] -> ca port {0x0002c9030010c36d}[1] lid 10-10 "hostname HCA-2"
To ca {0x0002c9030010c36c} portnum 1 lid 10-10 "hostname HCA-2"](https://image.slidesharecdn.com/how-to-install-ofed2012-matsumoto-b-ibday06-130117024019-phpapp02/85/How-To-Install-OFED-Linux-VMware-Windows-9-320.jpg)
How To Install OFED Linux/VMware/Windows
- 1. 01 Feb, 2013 SAKURA Internet Research Center Senior Researcher / Naoto MATSUMOTO
- 2. Infiniband Device list 40/56Gbit/s Infiniband mode 10/40Gbit/s Ethernet mode *Mellanox VPI (Virtual Protocol Interconnect) Only Infiniband HCA (Host Channel Adapter) Infiniband 40/56Gbit/s QDR Switch QSFP Cable (Copper/Fibre) + Infiniband OFED (OpenFabrics Enterprise Distribution) Software
- 3. IB Networking Example SERVER OpenSM *OpenSM InfiniBand subnet manager and administration Loop-Free IB-SWITCH OpenSM OpenSM IB-SWITCH IB-SWITCH IB-SWITCH IB-SWITCH IB-SWITCH
- 4. Install OFED for Linux 1) Download MLNX_OFED # uname -a Linux 2.6.32-220.el6.x86_64 #1 SMP Sat Dec 10 17:04:11 CST 2011 # wget http://mellanox.com/downloads/ofed/MLNX_OFED_LINUX-1.5.3-3.1.0 -rhel6.2-x86_64.tgz 2) Install MLNX_OFED Package # yum install tcl tk glibc-develglibc-devel.i686 # tar xzvf ./MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.2-x86_64.tgz # ./MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.2-x86_64/mlnxofedinstall # chkcnfig opensmd on; opensm –c /etc/opensm/opensm.conf # sync; sync; sync; reboot
- 5. Install OFED for VMware ESXi 5.1 1) Enable ESXi Shell & SSH Troubleshooting Options > Enable ESXi Shell, Enable SSH and SSH login. # vmware -v VMware ESXi 5.1.0 build-799733 2) Download/Install MLNX_OFED # cd /opt # wget http://mellanox.com/downloads/Drivers/MLNX-OFED-ESX-1.8.0.0.zip # esxcli software vib install -d /opt/MLNX-OFED-ESX-1.8.0.0.zip # sync; sync; sync; reboot –f # esxcfg-nics –l vmnic_ib0 0000:01:00.00 ib_ipoib Up 56252Mbps Full 00:02:c9:34:1c:f1 1500 Mellanox Technologies MT27500 Family [ConnectX-3]
- 6. Install OFED for Windows 1) DownloadMLNX_OFED from Mellanox Web page. 2) Install MLNX_OFED Package.
- 7. IB Fabric Example VM VM VM VM VMware VMware Windows Linux Windows Linux Windows Linux 40/56Gbit/s Windows Linux IP over Infiniband Network High Speed Server Interconnect Fabric for Mixed PHYSICAL and VIRTUAL.
- 8. 10/40/56Gbit/s Benchmarks RAMDISK SERVER SERVER RAMDISK 5.58 Gbit/sec using 8.00 Gbit/sec THTTPD 13.68 Gbit/sec RCOPY 18.23 Gbit/sec SOURCE: SAKURA Internet Research Center. 12/2012 rev2 Project THORN.
- 9. IB Diagnostic Commands # ibhosts *show InfiniBand host nodes in topology Ca : 0x0002c9030010c36c ports 2 "hostname HCA-2" Ca : 0x0002c9030010c360 ports 2 "hostname HCA-2" Ca : 0x0002c9030010c328 ports 2 "sl6 HCA-1 # iblinkinfo *report link info for all links in the fabric Switch 0x000b8cffff006f38 MT43132 Mellanox Technologies: : 2 6[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 8 1[ ] "sl6 HCA-1" ( ) 2 7[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 9 1[ ] "hostname HCA-2" ( ) 2 8[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 10 1[ ] "hostname HCA-2" ( ) # ibtracert 9 10 *trace InfiniBand path From ca {0x0002c9030010c360} portnum 1 lid 9-9 "hostname HCA-2" [1] -> switch port {0x000b8cffff006f38}[7] lid 2-2 "MT43132 Mellanox Technologies" [8] -> ca port {0x0002c9030010c36d}[1] lid 10-10 "hostname HCA-2" To ca {0x0002c9030010c36c} portnum 1 lid 10-10 "hostname HCA-2"
- 10. Thanks for your interest. SAKURA Internet Research Center.