x86_64 Hardware Deep dive
0 likes315 views
The document provides a detailed guide on x86_64 hardware, including setup commands, installation tips for various Linux components, and performance testing strategies. It covers kernel updates, PCIe configurations, and NVMe SSD management with various commands and tools. Additionally, it includes benchmarks for performance evaluation of different storage solutions and configurations.
![PCIe 4.0 NVMe SSD(M.2) RAID0 TIPS
5
# lscpu
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 5 3600 6-Core Processor
# apt install nvme-cli fio -y
# update-pciids
# lspci -tv
# lspci -vv
# nvme list
Node Model
------------ ---------------------------
/dev/nvme0n1 GIGABYTE GP-ASM2NE6100TTTD
/dev/nvme1n1 GIGABYTE GP-ASM2NE6100TTTD
:
/dev/nvme3n1 Force MP600
# mdadm --create /dev/md0 --raid-devices=3 --level=0 /dev/nvme[013]n1
# mdadm --detail /dev/md0
Number Major Minor RaidDevice State
0 259 0 0 active sync /dev/nvme0n1
1 259 1 1 active sync /dev/nvme1n1
2 259 5 2 active sync /dev/nvme3n1
# mkfs.ext4 /dev/md0
# mkdir /mnt/gen4
# mount /dev/md0 /mnt/gen4/
# fio --directory=/mnt/gen4/ --bs=4k --rw=read --size=20G --numjobs=16 --group_reporting --name=test
read: IOPS=2553k, BW=9973MiB/s (10.5GB/s) ...
# lsmem
Total online memory: 16G
SOURCE: SAKURA Internet Research Center (2020/05)](https://image.slidesharecdn.com/src20200506-6v1-200506123716/85/x86_64-Hardware-Deep-dive-5-320.jpg)
x86_64 Hardware Deep dive
- 1. x86_64 Hardware Deep dive 6th May, 2020 SAKURA Internet, Inc. Research Center SR / Naoto MATSUMOTO (C) Copyright 1996-2020 SAKURA Internet Inc
- 2. x86_64 Hardware Deep dive cheatsheet 2 # uname -sr; tail -1 /etc/lsb-release Linux 4.15.0-72-generic DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS" Setup commands # apt update # apt upgrade -y # apt install -y hwloc hwloc-nox # tar xzvf ./mlc_v3.7.tgz (Download from Intel.com) # update-pciids Usage commands # lscpu # lshw # lstopo # lspci –tv # Linux/mlc SOURCE: SAKURA Internet Research Center (2020/05)
- 3. GCC 9.2 Install TIPS 3 # wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc4/linux-headers-5.3.0-050300rc4_...201908111734_all.deb # wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc4/linux-headers-5.3.0-050300rc4-...201908111734_amd64.deb # wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc4/linux-modules-5.3.0-050300rc4-...201908111734_amd64.deb # wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc4/linux-image-unsigned-5.3.0-050300rc4-...11734_amd64.deb # dpkg -i linux-headers-5.3.0-050300rc4_5.3.0-050300rc4.201908111734_all.deb # dpkg -i linux-headers-5.3.0-050300rc4-generic_5.3.0-050300rc4.201908111734_amd64.deb # dpkg -i linux-modules-5.3.0-050300rc4-generic_5.3.0-050300rc4.201908111734_amd64.deb # dpkg -i linux-image-unsigned-5.3.0-050300rc4-generic_5.3.0-050300rc4.201908111734_amd64.deb # sync; sync; sync; reboot # uname -sr; tail -1 /etc/lsb-release Linux 5.3.0-050300rc4-generic DISTRIB_DESCRIPTION="Ubuntu 19.04" # apt install build-essential libgmp-dev libmpfr-dev libmpc-dev zlib1g-dev numactl -y # wget http://ftp.tsukuba.wide.ad.jp/software/gcc/releases/gcc-9.2.0/gcc-9.2.0.tar.gz # tar xzvf ./gcc-9.2.0.tar.gz # ../configure --prefix=/usr --disable-multilib --with-system-zlib --enable-languages=c,c++,d,fortran,go,objc,obj-c++ # make -j12 # make install # gcc --version gcc (GCC) 9.2.0 SOURCE: SAKURA Internet Research Center (2020/05)
- 4. PCIe P2P DMA Install TIPS 4 # wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.2-rc6/linux-headers-5.2.0-050200rc6...201906222033_all.deb # wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.2-rc6/linux-headers-5.2.0-050200rc6...201906222033_amd64.deb # wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.2-rc6/linux-modules-5.2.0-050200rc6...201906222033_amd64.deb # wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.2-rc6/linux-image-unsigned-5.2.0-050200rc6...22033_amd64.deb # dpkg -i linux-headers-5.2.0-050200rc6_5.2.0-050200rc6.201906222033_all.deb # dpkg -i linux-headers-5.2.0-050200rc6-generic_5.2.0-050200rc6.201906222033_amd64.deb # dpkg -i linux-modules-5.2.0-050200rc6-generic_5.2.0-050200rc6.201906222033_amd64.deb # dpkg -i linux-image-unsigned-5.2.0-050200rc6-generic_5.2.0-050200rc6.201906222033_amd64.deb # sync; sync; sync; reboot # apt install -y nvme-cli libbobcat-dev # git clone http://www.github.com/sbates130272/p2pmem-test # cd p2pmem-test/ # ./p2pmem-test /dev/nvme0n1 /dev/nvme1n1 /dev/p2pmem0 -s 1 -4k --check Copy one 4KB chunk from /dev/nvme0n1 to /dev/nvme0n1 via the memory exposed by /dev/p2pmem0. Perform a check on the data (i.e. write know data to /dev/nvme0n1 and validate that by reading /dev/nvme1n1 after the p2pmem based transfer). https://github.com/sbates130272/donard (DRAFT) # cd /nvme_donard # cd /libdonard # ./waf # cd /libdonard/build/speed # dd if=/dev/zero of=/<nvme_drive>/test1.dat bs=1K count=100K # ./nvme2gpu_read -b 128M -D /<nvme_drive>/test1.dat # ./nvme2gpu_read -b 128M =/<nvme_drive>/test1.dat SOURCE: SAKURA Internet Research Center (2020/05)
- 5. PCIe 4.0 NVMe SSD(M.2) RAID0 TIPS 5 # lscpu Vendor ID: AuthenticAMD Model name: AMD Ryzen 5 3600 6-Core Processor # apt install nvme-cli fio -y # update-pciids # lspci -tv # lspci -vv # nvme list Node Model ------------ --------------------------- /dev/nvme0n1 GIGABYTE GP-ASM2NE6100TTTD /dev/nvme1n1 GIGABYTE GP-ASM2NE6100TTTD : /dev/nvme3n1 Force MP600 # mdadm --create /dev/md0 --raid-devices=3 --level=0 /dev/nvme[013]n1 # mdadm --detail /dev/md0 Number Major Minor RaidDevice State 0 259 0 0 active sync /dev/nvme0n1 1 259 1 1 active sync /dev/nvme1n1 2 259 5 2 active sync /dev/nvme3n1 # mkfs.ext4 /dev/md0 # mkdir /mnt/gen4 # mount /dev/md0 /mnt/gen4/ # fio --directory=/mnt/gen4/ --bs=4k --rw=read --size=20G --numjobs=16 --group_reporting --name=test read: IOPS=2553k, BW=9973MiB/s (10.5GB/s) ... # lsmem Total online memory: 16G SOURCE: SAKURA Internet Research Center (2020/05)
- 6. PCIe 4.0 NVMe SSD(M.2) SMART-LOG TIPS 6 # uname -sr; tail -1 /etc/lsb-release Linux 4.15.0-58-generic DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS" # nvme list Node SN Model Format FW Rev ---------------- -------------------- ---------------------------------------- ---------------- -------- /dev/nvme0n1 SN192608967864 GIGABYTE GP-ASM2NE6100TTTD 512 B + 0 B EGFM11.0 /dev/nvme1n1 SN192608967963 GIGABYTE GP-ASM2NE6100TTTD 512 B + 0 B EGFM11.0 /dev/nvme2n1 1924822900012855002D Force MP600 512 B + 0 B EGFM11.0 /dev/nvme3n1 PHBT7160022A032E INTEL MEMPEK1W032GA 512 B + 0 B K3110300 # nvme smart-log /dev/nvme2n1 Smart Log for NVME device:nvme2n1 namespace-id:ffffffff critical_warning : 0 temperature : 36 C available_spare : 100% available_spare_threshold : 5% percentage_used : 0% data_units_read : 4,493,212 data_units_written : 1,336,869 host_read_commands : 41,019,727 host_write_commands : 4,432,398 controller_busy_time : 22 power_cycles : 78 power_on_hours : 44 unsafe_shutdowns : 45 media_errors : 0 num_err_log_entries : 98 Warning Temperature Time : 0 SOURCE: SAKURA Internet Research Center (2020/05)
- 7. Null block device High Performance I/O TIPS 7 # apt update # apt upgrade; sync; sync; sync; reboot # uname -sr; tail -1 /etc/lsb-release Linux 4.15.0-58-generic DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS" # lscpu Vendor ID: AuthenticAMD Model name: AMD Ryzen 5 3600 6-Core Processor # rmmod null_blk # modprobe null_blk nr_devices=8 # fio --bs=4k --size=128G --rw=read --numjobs=8 --runtime=40 --group_reporting=1 --name=nullb0 --filename=/dev/nullb0 --name=nullb1 --filename=/dev/nullb1 --name=nullb2 --filename=/dev/nullb2 --name=nullb3 --filename=/dev/nullb3 --name=nullb4 --filename=/dev/nullb4 --name=nullb5 --filename=/dev/nullb5 --name=nullb6 --filename=/dev/nullb6 --name=nullb7 --filename=/dev/nullb7 : read: IOPS=7201k, BW=27.5GiB/s (29.5GB/s)(384GiB/13980msec) # mount -t tmpfs -o size=8g tmpfs /mnt/ram # fio --bs=4k --size=500m --rw=read --numjobs=8 --runtime=40 --group_reporting=1 --name=tmpfs --directory=/mnt/ram/ : read: IOPS=6169k, BW=23.5GiB/s (25.3GB/s)(6000MiB/249msec) SOURCE: SAKURA Internet Research Center (2020/05)