Mastering in troubleshooting nfv
3 likes660 views
This document discusses troubleshooting Network Functions Virtualization (NFV) architecture and OpenStack with DPDK. It describes potential failures in configuring OVS-DPDK, including instances becoming unreachable over the DPDK network. Methodologies are provided for investigating and solving VNF issues, OVS-DPDK failures and performance problems. Solutions involve adjusting MTUs, enabling port security, and changing virtual network device drivers.
Mastering in troubleshooting nfv
- 2. AGENDA ● ● ●
- 8. VNF PANICS VNF CRASHES WHEN IT STARTS SENDING NETWORK PACKETS ● ● ● COMPUTE-1 COMPUTE-2 VNF-1 VNF-1VNF-1
- 9. VNF PANICS: WHO SHOULD INVESTIGATE INVESTIGATION THROUGH COLLABORATION ● ○ ● ○ ● COMPUTE-1 COMPUTE-2 VNF-1 VNF-1 ● ○
- 10. INVESTIGATION: HINT INVESTIGATION THROUGH COLLABORATION ● ● ● ● ● ● COMPUTE-1 COMPUTE-2 VNF-1 VNF-1 Packets defragmented here 1500 1500eth0 eth0
- 11. PACKET FLOW AND MTU tapx brx qvb qvo br-int int-br ph-br enox enox ph-br int-br br-int qvo qvb brx tapx VNF eth0 VNF eth0 9000 1500 ● ● ● ○ enox ph-br int-br br-int qvo qvb brx tapx tapx
- 12. POSSIBLE SOLUTIONS tapx brx qvb qvo br-int int-br ph-br enox enox ph-br int-br br-int qvo qvb brx tapx VNF eth0 VNF eth0 9000 1500 enox ph-br int-br br-int qvo qvb brx eth0 tapx tapx eth0eth0 eth0eth0 eth0
- 13. SOLUTION Change the mtu of vnf to 9000 Change the mtu of tapx to 1500 Enable port security Set bridge-nf-call-iptables to 0 Inspect what can be changed on vnf. Change e1000 to virtio Add support for per port mtu in Neutron
- 16. OVS-DPDK FAILURE TROUBLESHOOTING METHODOLOGY ● ○ ● ○ ○ ○ ● ● ○ driverctl -v list-devices | grep vfio ○ for i in `ps -eLo tid,comm | grep pmd | cut -f2 -d ' '` ; do grep huge /proc/$i/numa_maps ;done ○ ●
- 17. OVS-DPDK PERFORMANCE TROUBLESHOOTING METHODOLOGY ● ● ● ○ ○ ○ ○ ● # tuned-adm profile cpu-partitioning # grep tuned_params /boot/grub2/grub.cfg set tuned_params="nohz=on nohz_full=2,4,8-11 rcu_nocbs=2,4,8-11 intel_pstate=disable nosoftlockup isolcpus=2,4,8-11" linux16 /boot/vmlinuz-3.10.0-514.2.2.el7.x86_64 root=UUID=a69bf0c7-8d41-42c5-b1f0-e64719aa7ffb ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet default_hugepagesz=1GB hugepagesz=1G hugepages=12 iommu=pt intel_iommu=on selinux=0 $tuned_params
- 20. OVS-DPDK FAILURES ● After configuring ovs-dpdk , the instances cannot be reached via dpdk based network. DPDK NIC core core core core core core core core core core core core Numa Node 0 Numa Node 1 core core core core
- 22. OVS-DPDK FAILURES ● ● After configuring ovs-dpdk , the instances cannot be reached via dpdk based network. Symptom - I
- 24. ● ● OVS-DPDK Failures DPDK NIC core core core core core core core core core core core core Numa Node 0 Numa Node 1 PMD THREAD PMD THREAD core core core core
- 26. OVS-DPDK Failures ● After configuring ovs-dpdk , the instances cannot be reached via dpdk based network. Symptom - II
- 29. OVS-DPDK Failures ● ● DPDK NIC core core core core core core core core core core core core Numa Node 0 Numa Node 1 core core core core echo 4 > /sys/devices/system/node/node1/hugepage s/hugepages-1048576kB/nr_hugepages