VMware Advance Troubleshooting Workshop - Day 2

Introduction to vSphere Networking
Day 2
VMware vSphere:
Install, Configure, Manage

Content
• Virtual Networking
• Introduction to vSphere Standard Switches
• Troubleshooting of vSS
• Scenarios
• Introduction to vSphere Distributed Switches
• Troubleshooting of vDS
• NSX

Introduction to vSphere
Standard Switches

Learner Objectives
By the end of this lesson, you should be able to meet the following
objectives:
• Describe the virtual switch connection types
• Configure and view standard switch configurations, such as virtual machine
port group, VMkernel port, VLAN, and so on

About Virtual Networks
Server Hypervisor
Requirement: x86
Virtual
Machine
Virtual
Machine
Virtual
Machine
Application Application Application
x86 Environment
Decoupled
Hardware
Software
Network Hypervisor
Requirement: IP Transport
Virtual
Network
Virtual
Network
Virtual
Network
Workload Workload Workload
L2, L3, L4-7 Network Services
General Purpose Server Hardware
(Dell, HP, IBM, OpenCompute, Quanta)
General Purpose IP Hardware
(Arista, Cisco, HP, Juniper, Accton)
A virtual network is a network of virtual machines running on a physical machine
that are connected logically to one another so that they can send data to and
receive data from one another. Virtual machines can be connected to the virtual
networks that you create when you add a network.

Types of Virtual Switch Connections
A virtual switch has specific connection types:
• Virtual machine port groups
• VMkernel port:
– For IP storage, VMware vSphere® vMotion® migration, VMware vSphere® Fault
Tolerance, VMware Virtual SAN™, and VMware vSphere® Replication™
– For the ESXi management network
Virtual Switch
Production TestDev DMZ vSphere
vMotion
Management
Uplink Ports
Virtual Machine Port Groups VMkernel Ports

Virtual Switch Connection Examples
More than one network can coexist on the same virtual switch. Or
networks can exist on separate virtual switches.
Virtual Switch
Production TestDev iSCSIvSphere vMotionManagement
Virtual Switch Virtual Switch Virtual Switch Virtual Switch Virtual Switch
Production TestDev iSCSIvSphere vMotionManagement

Types of Virtual Switches
A virtual network supports these types of virtual switches:
• Standard switches:
– Virtual switch configuration for a single host
• Distributed switches:
– Virtual switches that provide a consistent network configuration for virtual machines
as they migrate across multiple hosts

Standard Switch Components
A standard switch provides connections for virtual machines to
communicate with one another, whether they are on the same host or on
different hosts.
VM
1
VM
2
VM
3
VMkernel
Test VLAN 101
Production VLAN 102
IP Storage VLAN 103
Management VLAN 104
Management
Network
IP
storage
VNIC VNIC VNIC VNIC

Viewing the Standard Switch Configuration
You can view a host’s standard switch configuration by clicking
Networking on the Manage tab.
Delete the
port group.
Display Cisco Discovery
Protocol information.
Display port group
properties.

About VLANs
Virtual Local Area Networks
(VLANs) allow you to have many
virtual networks running over a
single physical network
Uses a standard format to “tag”
Ethernet frames IEEE 802.1Q.This
information in the header tells the
network device which VLAN the
frame belongs in.
While VLANs do logically separate
traffic if someone has access to a
network segment they can see all
traffic on all VLANs in that
segment
Virtual Switch
VM
VLAN
105
VLAN
106
VM
VMkernel
Physical Switch
Physical
NIC
Trunk Port

VLAN RECOMMENDATIONS
VLAN Recommendations
The number of VLANs you can use is often
dictated by your networking equipment
VLAN 0 is reserved
VLAN 1 is usually the default but recommended not to use
Most equipment can go over 4,000 VLANs, you may not be able to use
them all at once
Make sure you know which (if any) VLAN on the physical uplink
switch is set as the native VLAN
Native VLANs are sent out untagged which can cause problems
Suggested to only trunk the VLANs you actually need
Does require more administration to add a pass an additional VLAN
As mentioned, it is suggested not to use VLAN 1 unless necessary

VLAN TAGGING
There are three places where a frame can be tagged with a VLAN.
•All VLAN tagging of packets is performed on the physical switch.
•ESXi/ESX host network adapters are connected to access ports on the physical
switch.
•The portgroups connected to the virtual switch must have their VLAN ID set to 0.
External Switch
Tagging
•All VLAN tagging of packets is performed by the virtual switch before leaving the
ESXi/ESX host.
•The ESXi/ESX host network adapters must be connected to trunk ports on the
physical switch.
•The portgroups connected to the virtual switch must have an appropriate VLAN ID
specified.
Virtual Switch
Tagging
•All VLAN tagging is performed by the virtual machine.
•You must install an 802.1Q VLAN trunking driver inside the virtual machine.
•VLAN tags are preserved between the virtual machine networking stack and
external switch when frames are passed to/from virtual switches.
•Physical switch ports are set to trunk port.
Virtual Guest
Tagging

Network Adapter Properties
A physical adapter can become a bottleneck for network traffic if the
adapter speed does not match application requirements.

Review of Learner Objectives
You should be able to meet the following objectives:
• Describe the virtual switch connection types
• Configure and view standard switch configurations, such as virtual machine
port group, VMkernel port, VLAN, and so on

Configuring Standard Switch
Policies

Learner Objectives
By the end of this lesson, you should be able to meet the following
objectives:
• Explain how to set the security policies for a standard switch port group
• Explain how to set the traffic shaping policies for a standard switch port group
• Explain how to set the NIC teaming and failover policies for a standard switch
port group

Network Switch and Port Policies
Policies that are set at the standard switch level apply to all port groups
on the standard switch by default.
Available network policies:
• Security
• Traffic shaping
• NIC teaming and failover
Policies are defined at the following levels:
• Standard switch level:
– Default policies for all the ports on the standard switch.
• Port group level:
– Effective policies: Policies defined at this level override the default policies that are
set at the standard switch level.

Configuring Security Policies
Administrators can define security policies at both the standard switch
level and the port group level:
• Promiscuous mode: Allows a virtual switch or port group to forward all traffic
regardless of the destination.
• MAC address changes: Accept or reject inbound traffic when the MAC
address has been altered by the guest.
• Forged transmits: Accept or reject outbound traffic when the MAC address
has been altered by the guest.

Traffic-Shaping Policy
Network traffic shaping is a mechanism for limiting a virtual machine’s
consumption of available network bandwidth.
Average rate, peak rate, and burst size are configurable.
Peak Bandwidth
Average
OutboundBandwidth
Time
Burst Size = Bandwidth x Time

Configuring Traffic Shaping
A traffic-shaping policy is defined by average bandwidth, peak
bandwidth, and burst size. You can establish a traffic-shaping policy for
each port group and each distributed port or distributed port group:
• Traffic shaping is disabled by default.
• Parameters apply to each virtual NIC in the standard switch.
• On a standard switch, traffic shaping controls only outbound traffic.

NIC Teaming and Failover Policies
Administrators can edit the NIC teaming and failover policy by
configuring specific options.

Load-Balancing Method: Originating Virtual Port ID
The diagram shows routing based on the originating port ID, called virtual
port ID load balancing.
Virtual NICs Physical NICs
Virtual
Switch
Physical
Switch

Load-Balancing Method: Source MAC Hash
The diagram shows routing based on source MAC hash.
Virtual
NICs
Physical
NICs
Virtual
Switch
Internet
Physical
Switch

Load-Balancing Method: Source and Destination IP
Hash
The diagram shows routing based on IP hash.
Virtual NICs Physical NICs
Virtual
Switch
Internet
Physical
Switch

Detecting and Handling Network Failure
The VMkernel can use link status or beaconing or both to detect a
network failure.
Network failure is detected by the VMkernel, which monitors the link state
and performs beacon probing.
VMkernel notifies physical switches of changes in the physical location of
a MAC address.
Failover is implemented by the VMkernel based on configurable
parameters:
• Failback: How the physical adapter is returned to active duty after recovering
from failure.
• Load-balancing option: Use explicit failover order. Always use the vmnic uplink
at the top of the active adapter list.

SR-IOV (Single Root IO Virtualization)
SR-IOV is a specification that allows a single Peripheral
Component Interconnect Express (PCIe) physical device under a single
root port to appear as multiple separate physical devices to the
hypervisor or the guest operating system.
SR-IOV uses physical functions (PFs) and virtual functions (VFs)
to manage global functions for the SR-IOV devices. PFs are full PCIe
functions that are capable of configuring and managing the SR-IOV
functionality. It is possible to configure or control PCIe devices using PFs,
and the PF has full ability to move data in and out of the device. VFs are
lightweight PCIe functions that support data flowing but have a restricted
set of configuration resources.
The number of virtual functions provided to the hypervisor or the
guest operating system depends on the device. SR-IOV enabled PCIe
devices require appropriate BIOS and hardware support, as well as SR-
IOV support in the guest operating system driver or hypervisor instance.

Switch Discovery Protocol
Switch discovery protocols help vSphere administrators to
determine which port of the physical switch is connected to a vSphere
standard switch or vSphere distributed switch.
vSphere 5.0 and later supports Cisco Discovery Protocol (CDP)
and Link Layer Discovery Protocol (LLDP). CDP is available for vSphere
standard switches and vSphere distributed switches connected to Cisco
physical switches. LLDP is available for vSphere distributed switches
version 5.0.0 and later.
When CDP or LLDP is enabled for a particular vSphere
distributed switch or vSphere standard switch, you can view properties of
the peer physical switch such as device ID, software version, and timeout
from the vSphere Web Client.

CDP
Cisco Discovery Protocol (CDP) enables vSphere administrators
to determine which port of a physical Cisco switch connects to a vSphere
Standard Switch or vSphere Distributed Switch. When CDP is enabled
for a vSphere Distributed Switch, you can view the properties of the
Cisco switch such as device ID, software version, and timeout.

LLDP
The Link Layer Discovery Protocol (LLDP) is a vendor-neutral link layer
protocol in the Internet Protocol Suite used by network devices for
advertising their identity, capabilities, and neighbors on an IEEE 802
local area network, principally wired Ethernet.

Jumbo Frames
Jumbo frames let ESXi hosts send larger frames out onto the physical network. The
network must support jumbo frames end-to-end that includes physical network adapters,
physical switches, and storage devices.
Before enabling jumbo frames, check with your
hardware vendor to ensure that your physical
network adapter supports jumbo frames.
You can enable jumbo frames on a vSphere
distributed switch or vSphere standard switch by
changing the maximum transmission unit (MTU) to
a value greater than 1500 bytes. 9000 bytes is the
maximum frame size that you can configure.

Review of Standard Switch
If a virtual machine loses network connectivity, the cause of the problem
might be anywhere from the virtual machine’s NIC to the ESXi host’s
physical network.
Physical NICs
Virtual
NIC
Virtual
NIC
Virtual
NIC
vmnic0 vmnic1
ESXi
Host Management
Network

• Explain how to set the security policies for a standard switch port group
• Explain how to set the traffic shaping policies for a standard switch port group
• Explain how to set the NIC teaming and failover policies for a standard switch
port group

ESXCLI Command
To troubleshoot networking configurations from the ESXi command line,
ESXCLI is the tool to use or PUTTY can be used.
There are a number of options available when running ‘esxcli’ in terms of
network settings:
~ esxcli network

Netcat Command
Netcat can be used to test connectivity to and from your ESXi host.
~ nc -h

VMPING Command
You can test connectivity to remote ESXi host using the ping and
vmkping utilities. Using vmkping to test connectivity via vMotion
interfaces is a common practice. For example:
~ vmkping 192.168.1.20

OpenSSL Command
You can use the open ssl client present on an ESXi host to test
connectivity to an ssl port – for example to vCenter or to another host. To
do so:
~ openssl s_client -connect 192.168.1.100:443

TCPDUMP Command
This command is used to identify the packet flow in an NIC. To display
packets on interface vmk0 you can run:
~ tcpdump-uw -i vmk0 | more

ESXCFG Command
The esxcfg-nics command provides information about the physical NICs
in use by the VMkernel.
This prints the VMkernel name for the NIC, its PCI ID, driver, link state,
speed, duplex, and a short PCI description of the card. It also allows
users to set speed and duplex settings for a specific NIC.
~ esxcfg-nics <options> [nic]

PKTCAP-UW Tool
The pktcap-uw tool is an enhanced packet capture and analysis tool that
can be used in place of the legacy tcpdump-uw tool. The pktcap-uw tool
is included by default in ESXi 5.5.

vMA
The vSphere Management Assistant (vMA) allows administrators
and developers to run scripts and agents to manage ESXi hosts and
vCenter Server systems. vMA is a virtual machine that includes
prepackaged software, a logging component, and an authentication
component that supports non-interactive login.
As an alternative to esxcli, you can also use the vicfg-dns command from
the vMA or vSphere CLI. Running the command without any parameters
will display a host’s DNS configuration:
vi-admin@vma:~> vifptarget --set
192.168.88.134
vi-admin@vma:~[192.168.88.134]>
vicfg-dns
DNS Configuration
Host Name esxi1
Domain Name vmlab.local
DHCP false
DNS Servers 10.0.0.1

Network Problem 1
As an initial check from VMware vSphere® ESXi™ Shell, ping a system
that is known to be up and accessible by the ESXi host.
The ESXi host has intermittent or no network connectivity to other
systems.
DCUI
Command
Prompt

Identifying Possible Causes
If you know that your hardware is functioning correctly, take the top-down
approach to troubleshooting, starting with the ESXi host configuration.
ESXi
Host
Hardware
(Network, Server)
The ESXi host network configuration is incorrect.
The VLAN ID of the port group is incorrect.
The speed and duplex of the network links are not
consistent.
The network link is down.
NIC teaming is not configured properly.
Possible Causes
The network adapter or server hardware is not supported.
The physical hardware is faulty or misconfigured.
Network performance is slow.

Possible Cause: ESXi Network Misconfiguration (1)
Verify that your ESXi host network is configured properly:
• Check vSphere standard switches, vmnics, port groups, and VMkernel ports:
– In VMware vSphere® Management Assistant, use vicfg-vswitch –l
– In vSphere ESXi Shell, use esxcfg-vswitch –l and esxcfg-vmknic –l
• Check VLAN IDs of port groups:
– esxcli network vswitch standard portgroup list

Possible Cause: ESXi Network Misconfiguration (2)
Verify that your ESXi host network configuration is configured properly:
• Speed and duplex:
– vicfg-nics –l
• Network uplink and NIC status (up or down):
– esxcfg-nics –l
– vicfg-nics –l
– esxcli network nic list

Resolving ESXi Network Misconfiguration
Adjust settings in your ESXi network configuration that are not configured
properly:
• Standard switches, vmnics, port groups:
– Add standard switch: vicfg-vswitch –a vswitch#
– Add port group: vicfg-vswitch –A pg_name vswitch#
– Add uplink: vicfg-vswitch –L vmnic# vswitch#
• VLAN IDs of port groups:
– esxcli network vswitch standard portgroup set
–p pg_name -v vlan_ID
• Speed and duplex:
– vicfg-nics –d duplex -s speed vmnic#
• Network link status (up or down):
– Connect network adapters to the intended physical switch ports.

Possible Cause: NIC Teaming Misconfiguration
Verify that NIC teaming is configured properly.

Possible Cause: Unsupported or Faulty Hardware
Verify that you are not encountering the following ESXi network hardware
issues:
• The network adapter or server hardware is not supported:
– vicfg-nics –l
– Verify that the network hardware is listed in VMware Compatibility Guide.
• The physical hardware is faulty or misconfigured:
– lspci –p

Possible Cause: Slow Network Performance
Use esxtop (or resxtop) to view key network metrics that can help
identify network performance problems.
The esxtop command is available in both vSphere ESXi Shell and
VMware vSphere® Command-Line Interface.
Sample resxtop Output

Review of Virtual Machine Connectivity
If your virtual machine loses network connectivity, the cause of the
problem might be in the physical layer, the virtual layer, or the guest
operating system itself.
OS
APP
FIREWALL
Virtual
Machine
Virtual NIC
Port Groups
Uplink Ports
Virtual
Switch
Physical NICs

Network Problem 2
As an initial check, ping the virtual machine from another system.
If the ping command fails, ping other virtual machines on the same
network to determine the scope of the problem.
The virtual machine has no network connectivity.

Take a top-down approach to troubleshooting, from the guest operating
system to the virtual machine and the ESXi host.
Application or
Guest OS
Virtual
Machine
ESXi
Host
Possible Causes
The port group name does not exist.
The virtual network adapter is not connected.
Underlying issues with ESXi network connectivity exist.
Storage or resource contention on the ESXi host exists.
IP settings are misconfigured.
The firewall in the guest OS is blocking traffic.

Possible Cause: IP Settings and Firewall Problems
IP settings and problems with firewalls might cause the problem.
Check IP settings to ensure that the TCP/IP settings in the guest
operating system are correct.
The firewall in the guest operating system might be blocking traffic.
Ensure that the firewall does not block required ports.

Possible Cause: Port Group Misconfiguration
The port group name that the virtual machine uses is incorrect:
• View the standard switch port group names on the ESXi host:
– vicfg-vswitch –l
• Verify that the virtual machine is using the correct port group.
The virtual network adapter is not connected to the port group:
• Verify that the network adapter is connected to the correct port group.

Possible Cause: ESXi Network Connectivity Problems
Storage or resource contention on the ESXi host can cause network
connectivity issues:
• Ensure that the virtual machine has no underlying issues with storage and that
it is not in resource contention.
Problems might exist with the ESXi host network, the port group ID, the
speed or duplex settings, the physical network link, or the NIC teaming
configuration.
To eliminate a NIC failure or physical configuration issue, connect the
virtual machine to a virtual switch that uses NIC teaming.

Possible Cause: No Available Ports on Virtual Switch
If your vSphere version is earlier to version 5.5, you might encounter a
problem. The virtual switch might not have an available port for the virtual
machine to connect to:
• This situation can occur during a VMware vSphere® vMotion® migration.
Verify the number of configured ports:
The vicfg-vswitch command also shows the number of used ports.

Resolving the Issue of Unavailable Ports on a Virtual Switch
If the virtual switch does not have available ports for virtual machines to
connect to, resolve the issue in one of the following ways:
• Increase the number of ports for the virtual switch and reboot the host to make
the changes effective.
• Create a new virtual switch and spread the virtual machines and port groups
across the two switches.
This problem is not relevant to vSphere version 5.5 and above.

Network Problem 3
Another symptom is that the ESXi host is successfully added to the
vCenter Server inventory but disconnects 30 to 90 seconds after the task
completes.
The problem is that dropped, blocked, or lost heartbeat packets are
occurring between vCenter Server and the ESXi host.
An ESXi host frequently disconnects from VMware vCenter Server™.

Heartbeat Communication Between vCenter Server and ESXi
The ESXi host sends a heartbeat to vCenter Server to signal that the
host is accessible by the management network.
Windows
Firewall
Management
Network
(vmk0)
ESXi
vCenter Server
Heartbeat Sent over UDP Port 902

Take a top-down approach to troubleshooting, from the vCenter Server
system to the ESXi host and the hardware.
vCenter Server
ESXi
Host
Windows Firewall is enabled on the vCenter Server system,
and UDP port 902 is blocked.
Possible Causes
vCenter Server is not using port 902 for receiving heartbeats,
or the ESXi firewall is blocking that port.
Hardware
(CPU, Memory,
Network, Storage)
The network between ESXi and vCenter Server is congested.

Possible Cause: Port Blocked by Windows Firewall
If Windows Firewall is enabled and UDP port 902 is blocked, view the
ports blocked by Windows Firewall.
To resolve this problem, adjust Windows Firewall settings:
• If ports are not configured, disable Windows Firewall.
• If the firewall is configured to affect ports, ensure that Windows Firewall is not
blocking UDP port 902.

Possible Cause: vCenter Server Not Using Port 902
By default, the vpxa agent on the ESXi host sends heartbeats to vCenter
Server (vpxd) through UDP port 902.
A problem might exist if the host is configured to send heartbeats over a
port other than 902.
Use the less /etc/vmware/vpxa/vpxa.cfg command on the host
to determine the port that is used to send heartbeats:
…

Resolving the Use of a Port Other Than 902 (1)
If you prefer to use a nondefault port for heartbeats, ensure that the ESXi
firewall is not blocking that port.
Contents of heartbeat.xml

Resolving the Use of a Port Other Than 902 (2)
Check the vCenter Server configuration to verify the port number used
for heartbeats.

Possible Cause: Network Congestion
If the network between ESXi and vCenter Server is congested, dropped
heartbeats might occur.
To verify whether your management network is congested, use a network
packet analyzer:
• You can use the resxtop utility or graphical views to analyze traffic.
• The pktcap-uw command is an enhanced packet capture and analysis tool.
• The tcpdump-uw command is a legacy network traffic capture tool:
– Available in vSphere ESXi Shell, based on the standard tcpdump utility
– For example, to display packets on the VMkernel interface vmk0, run the command:
• tcpdump-uw –i vmk0
• Wireshark is a publicly available network analyzer:
– Captures live network traffic
– Displays packets with detailed protocol information

Resolving Network Congestion
Resolving network congestion has both short-term and long-term
solutions.
Short-term solution to this problem:
• Increase the timeout limit in vCenter Server to keep the ESXi host continuously
connected.
Long-term solution to this problem:
• Resolve the underlying network congestion problems.
• If using distributed switches, use VMware vSphere® Network I/O Control to
reprioritize traffic and increase the number of shares for management traffic.

Network Problem 4
This problem can occur if the ESXi host’s management network was
misconfigured or manipulated from the command line.
For example, you can bring a physical network card up or down with the
esxcli command:
• esxcli network nic up –n vmnic0
• esxcli network nic down –n vmnic0
• esxcli network nic list
The ESXi host cannot be managed by vCenter Server.

Preventing Loss of Management Network Connectivity
vSphere network rollback prevents accidental misconfiguration of
management networking and loss of connectivity:
• For example, if you try to change the IP address of your management
VMkernel interface, VMware vSphere® Web Client returns the error message
in the screenshot.

Host Networking Rollback
Rollback enables you to roll back to a previous valid configuration.
The host networking rollback is triggered when a network configuration
change is made that disconnects the host
Several events can trigger a host networking rollback:
• Updating DNS and routing settings
• Updating the speed or duplex of a physical NIC
• Changing the IP settings of a management VMkernel network adapter
• Updating teaming and failover policies to a port group that contains the
management VMkernel network adapter
If a network disconnects for any of these reasons, the task fails and the
host reverts to the last valid configuration.

Recovering a Lost Management Network: Standard Switch
If your management network is on a standard switch and you lose
management network connectivity, the solution uses the Configure
Management Network option in the DCUI.

Network Restore Options in the DCUI
To restore the network through the DCUI:
1. Select Network Restore Options.
2. Perform a full network restore.
3. Repair the Management network on a misconfigured standard or
distributed switch.
The Restore Network Settings option deletes all the current network
settings except for the Management network.

• Provide a network troubleshooting overview
• Analyze and troubleshoot standard switch problems
• Analyze and troubleshoot virtual machine connectivity problems
• Analyze and troubleshoot management network problems

Key Points
• Virtual network connectivity problems might occur with standard switches,
distributed switches, virtual machines, or management networks.
• A virtual machine connectivity problem might exist in the physical layer, the
virtual layer, or the guest operating system.
• The ping command is useful when troubleshooting ESXi host and virtual
machine connectivity issues.
• When an ESXi host frequently disconnects from vCenter Server, heartbeat
packets are being lost between vCenter Server and the ESXi host.
• vSphere network rollback prevents accidental misconfiguration of management
networking and loss of connectivity.
• A good practice is to back up your distributed switch configuration with the
vSphere Web Client whenever you make a change to the configuration.
• You can use the restore or the import function to reset the distributed switch
configuration.
Questions?

VMware Advance Troubleshooting Workshop - Day 2

More Related Content

What's hot (20)

Viewers also liked (18)

Similar to VMware Advance Troubleshooting Workshop - Day 2 (20)

Recently uploaded (20)

VMware Advance Troubleshooting Workshop - Day 2