SlideShare a Scribd company logo
Operational Best Practices for NSX in VMware
Environments
Ray Budavari, VMware
Thomas Kraus, VMware
NET5790
#NET5790
22
Agenda
 Introduction - Network Virtualization
 Operational Impacts
 NSX for vSphere Components
 Operational Tools
 Demonstrations
 Conclusion
33
Introduction - Network Virtualization
1. Decouple
Physical
Virtual
2. Reproduce 3. Automate
Network
Operations
Cloud
Operations
Hardware
independence
Operational benefits
of virtualization
No change to network
from end host perspective
Virtual
Physical
44
Agenda
 Introduction - Network Virtualization
 Operational Impacts
 NSX for vSphere Components
 Operational Tools
 Demonstrations
 Conclusion
55
Operational Impacts - Questions
 If a Virtual Machine has a network outage where do I start?
 How does network virtualization map to our operating model?
 What tools exist to correlate logical and physical networks to
assist in troubleshooting?
 What opportunities does network virtualization provide to improve
how we operate our environment?
 Are we adding complexity
by adopting network
virtualization?
 Does network virtualization
create a ‘black box’?
66
Operational Impacts - Answers
Capability Physical Virtual Capability Physical Virtual
Packet Capture
✔ ✔
VM level
visibility
✗ ✔
NetFlow
✔ ✔
Network
Snapshot
✗ ✔
RSPAN/ERSPA
N ✔ ✔ CLI
✔ ✔
Performance
Statistics ? ✔ UI
? ✔
Syslog
✔ ✔ API
? ✔
77
Operational Impacts - Opportunities
 All NSX components such as the NSX Controller, NSX vSwitch and
NSX Edge provide detailed network visibility and data
 Simplify the underlying physical network
• One consistent physical transport network to manage for virtual machine traffic
• Greatly reduces the number of MAC/ARP table entries to manage
• Enables you to build the network you want, while still meeting application and
workload connectivity requirements
DC Networks
 Centralized reporting and
monitoring, distributed
performance and scale
 Designed for automation
• NSX is built on a REST API
provided by NSX Manager
• All operations can be performed
programmatically via scripting or
higher-level languages
88
Operational Impacts - Examples
 If a Virtual Machine has a network outage where do I start
to troubleshoot?
 Before Network Virtualization:
• Validate VLAN trunk configuration across multiple devices and ports
• Verify VM visibility on each path of the network
• Troubleshooting requires accessing different devices and interfaces
• vSphere Web Client
• Hypervisor CLI
• Access Switch CLI
• Distribution Switch CLI
• Firewall
• Load Balancer
• These devices are typically managed by different teams
• Virtualization Administrators
• Network Administrators
• Security Administrators
• Service Providers
99
Operational Impacts - Examples
 After Network Virtualization:
• VXLAN network tests determine if the issue is related to the transport network
• If VTEPs are reporting issues, engage the network team to troubleshoot physical
transport network
• Provide VTEP IP/MAC information
• Otherwise virtualization team validates VM logical networking
• Verify NSX Components and Controller state information
• Verify Source and Destination Hypervisors
• Enhanced toolset is available for troubleshooting
• Reduced number of components and resources required
• NSX components can be queried or configured via REST API
1010
Agenda
 Introduction - Network Virtualization
 Operational Impacts
 NSX for vSphere Components
 Operational Tools
 Demonstrations
 Conclusion
1111
NSX for vSphere Components
Consumption
• Self Service Portal
• Cloud Management
• vCloud Automation Center
Data
Plane
NSX Edge
Services
Gateway
ESXi
VDS
Hypervisor Kernel Modules
FirewallDistributed
Logical Router
VXLAN
NSX vSwitch
• NSX Edge
• VM form factor
• Data Plane for North South
traffic
• Routing and Advanced
services
• NSX vSwitch
• Distributed network edge
• Line Rate performance
Management
Plane
NSX Manager
• Single point of configuration
• REST API and UI interface
vCenter Server
Control
Plane
NSX Controller • Manages Logical networks
• Run-time state
• Does not sit in the Data Path
• Control-Plane Protocol
NSX Edge
Logical Router User World Agent
1212
Components – NSX Manager
• NSX for vSphere centralized management plane
• 1:1 mapping between an NSX Manager and vCenter Server
• Provides the management UI and API for NSX
• vSphere Web Client Plugin
• Deploys NSX Controller and NSX Edge Virtual Appliances (OVF)
• Installs VXLAN, Distributed Routing and Firewall kernel modules and UW
Agent on ESXi hosts
• Configures Controller
Cluster via a REST API
and hosts via a
message bus
• Generates certificates to
secure control plane
communications
1313
Components – NSX Controller
 A reliable and secure control plane to distribute VXLAN and Logical
Routing network information to ESXi hosts
 NSX Controllers are clustered for scale out and high availability
 Network information is sliced across nodes in a Controller Cluster
 Enables dependency on multicast routing/PIM in the physical network to
be removed
 Provides suppression of ARP broadcast traffic in VXLAN networks
VXLAN
Logical Router
VXLAN
Logical Router
VXLAN
Logical Router
Controller
VXLAN Directory
Service
MAC table
ARP table
VTEP table
1414
Components – UW agent
 UW agent is a TCP (SSL) client that communicates with the Controller using
the control plane protocol
 May connect to multiple controllers
 Mediator between the ESXi Hypervisor Kernel Modules and NSX Controllers
 Also communicates with message bus agent to retrieve information from|
NSX Manager
 Runs as a service daemon on ESXi: netcpa
• Logs to: /var/log/netcpa.log
Controller
Cluster
Controller Controller Controller
ESXi Host
Kernel
Modules
Client ClientUser World
Agent
LR
NSX
MGR
Client
VXLAN
1515
Components – NSX vSwitch and NSX Edge
 NSX vSwitch (VDS)
 VMkernel Modules
 VXLAN
 Distributed Routing
 Distributed Firewall
 Switch Security
 Message Bus
 L3-L7 Services:
 NAT, DHCP, LB, VPN, Interface based
FW
 Dynamic Routing
 VM form factor
 High Availability
vSphere NSX Edge Services GW
 Control Functions only
 Dynamic Routing &
updates to Controller
 Determines active ESXi
host for L2 Bridging
NSX Edge Logical Router
ESXi
VDS
Hypervisor Kernel Modules
(vSphere VIBs)
FirewallLogical RouterVXLAN
NSX vSwitch
NSX Edge
Services
Gateway
NSX Edge
Logical
Router
1616
Agenda
 Introduction - Network Virtualization
 Operational Impacts
 NSX for vSphere Components
 Operational Tools
 Demonstrations
 Conclusion
1717
Operational Tools – ESXi
 pktcap-uw
 New with vSphere 5.5
 Enhanced tool that provides a framework for packet capture and tracing at the
Uplink, vSwitch, vmknic, vnic and port level at any stage in a packet’s lifecycle
1818
Operational Tools – ESXi
 pktcap-uw
 Extensive range of filters
such as source/destination
mac, IP, Protocol, VLAN,
VXLAN, ports etc.
 Supports pcap format
output for use with protocol
analyzers such as
Wireshark
1919
Operational Tools – NSX vSwitch Backup & Restore
 vSwitch Backup & Restore
2020
Operational Tools – NSX vSwitch Netflow
 NetFlow / IPFIX
NetFlow collector
address and port
2121
Operational Tools – NSX vSwitch RSPAN/ERSPAN
 RSPAN/ERSPAN, Port Mirroring
2222
Operational Tools – NSX vSwitch Alarms
 vSwitch Alarms allow for alerting on VDS related events. Some of the
available preconfigured Triggers include Health Status, Reconfiguration,
Port blocked, Port Deleted, Link Down and Host removal
 SNMP Network MIBs provide standards based visibility of NSX vSwitch
objects
2323
Operational Tools – NSX vSwitch Health Check
 Network Health Check feature helps to detect common
configuration errors
 Mismatched VLAN trunks between virtual switch and physical switch
 Mismatched MTU setting between vNIC, virtual switch, physical adapter, and
physical switch ports.
 Mismatched Teaming Configurations
 vSphere admins can provide failure data to the Network admins to
facilitate problem resolution  Health Check uses
L2 Echo protocol to
send Ethernet
broadcast frames to
the physical switch
 If reply packets are
not received,
warnings are
highlighted in the
vSphere Web Client
2424
vSphere Cluster B
UWA VTEP
UWA VTEP
UWA VTEP
Operational Tools – NSX Controller
 Control Plane basics
 ESXi hosts and NSX Edge Logical
Router VMs collect network
information, which is then reported
to the Controller via User World
Agent (UWA)
 The NSX Controller CLI provides a
consistent, centralized interface to
verify VXLAN and Logical Routing
network state information
 NSX Manager also provides APIs to
programmatically retrieve data from
the controller nodes
NSX Manager
NSX
Controller
Cluster
vSphere Cluster A
UWA VTEP
UWA VTEP
UWA VTEP
2525
Operational Tools – NSX Controller VTEP Report
vSphere Host
VM
vSphere Distributed Switch
MAC1
Management
Network
10.20.10.10
vSphere Host
VM
MAC2
vSphere Host
Send VNI,VTEP
Mapping to
Controller
1
2
5
10
VXLAN 5001
3
VNI VTEP IP
5001 10.20.10.10
9
IP1 IP
2
Controller
VNI VTEP IP
5001 10.20.10.10
VNI VTEP IP
5001 10.20.10.11
10.20.10.11 10.20.10.12
11
6
7
VNI VTEP IP
5001
10.20.10.10
10.20.10.11
VNI VTEP IP
5001
10.20.10.10
10.20.10.11
48
10
VNI VTEP IP
5001
10.20.10.10
10.20.10.11
11
Report the new
VNI,VTEP
Mapping to the
Hosts
2626
Operational Tools – NSX Controller
 General NSX Controller troubleshooting
steps:
• Verify Controller cluster status and roles
• Verify Controller node network connectivity
• Check Controller API service
• Validate VXLAN and Logical Router mapping
table entries to ensure they are consistent
• Review source and destination netcpa logs and
CLI to determine control plane connectivity issues
between ESXi hosts & NSX Controller
 The first set of commands relates to NSX
Controller CLI cluster status and health:
• nsx-controller # show control-cluster status
• nsx-controller # show control-cluster startup-
nodes
• nsx-controller # show control-cluster roles
• nsx-controller # show control-cluster connections
• nsx-controller # show control-cluster vnet core
stats
• nsx-controller # show network <arg>
• nsx-controller # show log cloudnet/cloudnet_java-
vnet-controller.<start-time-stamp>.log
2727
Operational Tools – NSX Controller
 General NSX Controller VXLAN operations:
• List VNIs
• VXLAN connection table
• Verify VXLAN VTEP, MAC and ARP mapping tables
• View VXLAN statistics
 NSX Controller CLI VXLAN commands:
• # show control-cluster vnet vxlan vni <vni>
• # show control-cluster vnet vxlan connection-table <vni>
• # show control-cluster vnet vxlan vtep-table <vni>
• # show control-cluster vnet vxlan mac-table <vni>
• # show control-cluster vnet vxlan arp-table <vni>
• # show control-cluster vnet vxlan vni-stats <vni>
 Note: VXLAN Logical Switches and Logical
Router instances are distributed across
Controller Nodes (slicing), so you will need to
run the CLI commands on the node which is
active for a given object
2828
Operational Tools – NSX Controller
 General NSX Controller Logical Routing operations:
• List Logical Router instances
• Verify Logical Router interface and route mapping tables
• Verify active controller connections
• View Logical Router statistics
 NSX Controller CLI Logical Routing commands:
• # show control-cluster vnet logical-router instance-table <lr-id>
• # show control-cluster vnet logical-router lif-table <lr-id>
• # show control-cluster vnet logical-router route-table <lr-id>
• # show control-cluster vnet logical-router lr-stats <lr-id>
 Note that the NSX Controller CLI is still not final
2929
Operational Tools – VXLAN
 Common VXLAN issues:
• Connectivity between VXLAN VTEPs on Transport Network
• MTU on Transport Network not set to 1600 bytes or greater
• Teaming mismatch between dvUplinks and upstream switch
• Preparation - either with the installation of VXLAN kernel modules or creation of VTEP VMkernel
interfaces
• ESXi host communication with Controller
 NSX for vSphere provides a new workflow for preparation and configuration
 Supports multiple VTEPs per host
 NSX leverages the vSphere 5.5 multi-instance TCP/IP stack
3030
Operational Tools – VXLAN
 At Logical Switch level on the Monitoring Tab you can use the
unicast or broadcast test to verify the connectivity between VTEPs
VXLAN standard tests
with a 1600 byte MTU
3131
Operational Tools – VXLAN
 The VXLAN Replication Mode will also determine transport network
connectivity requirements
 Unicast Mode
• All replication occurs using
unicast
 Hybrid Mode
• Local replication offloaded to
physical network, while remote
replication occurs via unicast
 Multicast Mode
• Requires IGMP for a Layer 2
topology and Multicast Routing
for L3 topology
 All modes require an MTU of
1600 bytes
3232
Operational Tools – VXLAN
 VXLAN namespace for esxcli provides detailed network
information and statistics.
• # esxcli network vswitch dvs vmware vxlan list
• # esxcli network vswitch dvs vmware vxlan network list --vds-name=Compute_VDS
• # esxcli network vswitch dvs vmware vxlan network mac list –vds-name=Compute_VDS --
vxlan-id=5001
• # esxcli network vswitch dvs vmware vxlan network arp list --vds-name Compute_VDS --vxlan-id=5001
• # esxcli network vswitch dvs vmware vxlan network port list --vds-name Compute_VDS --vxlan-id=5001
• # esxcli network vswitch dvs vmware vxlan network stats list --vds-name Compute_VDS --vxlan-id=5001
3333
Operational Tools – Logical Routing
 Use the net-vdr command on
ESXi hosts to view Logical
Routing configuration and
statistics
• Display Logical Router instances
~ # net-vdr -I –l
• List Logical Interface and Routing
Tables
~ # net-vdr -l –lif <instance-name>
~ # net-vdr -l --route <instance-name>
LIFs and routes are pushed by the NSX
controller to the ESXi hosts and should
be consistent across the environment
• View L2 Bridging information
~ # net-vdr -b –mac <instance-name>
~ # net-vdr -l –stats <instance-name>
3434
Operational Tools – NSX Edge Services
 NSX Edge VM CLIs
• NSX Edge provides a familiar CLI command
set for troubleshooting network services
• Documented in a dedicated CLI guide
 Sample Configuration Commands
• show configuration {ospf|bgp|isis|static-routing}
• show configuration {firewall|nat|dhcp|dns}
• show configuration {loadbalancer|ipec|sslvpn-plus}
 Sample Status Commands
• show interface [IFNAME]
• show firewall
• show ip {route|ospf|bgp|forwarding}
• show arp
• show system {cpu|memory|network-stats|storage|uptime}
• show service {dhcp|dns|highavailability|ipsec|loadbalancer|sslvpn-plus}
3535
Operational Tools – NSX Edge Services
 API provides based statistics (for interfaces and services)
 UI also provides interface statistics and graphs
3636
Operational Tools – NSX Edge Services
 Logging commands
• show log {follow|reverse}
• show flowtable
 Debug/troubleshooting commands
• traceroute <ip_address or dns_name>
• ping <ip address> or ping interface addr <alternate_src_ip> <ip_address>
• debug packet display interface <vNic_0-9> <EXPRESSION>
• debug packet display interface vNic_0 host_192.168.1.2
• debug packet display interface vNic_2 host_192.168.1.3_and_port_80
• debug packet display interface vNic_1 src_192.168.1.2_and_dst_192.168.1.3
• debug packet capture interface <vNic_0-9> <EXPRESSION>
• debug show files
• debug copy {scp|ftp} <URL>
 debug messagebus {forwarder|messages}
3737
Operational Tools – Flow Monitoring
 Flow monitoring provides vNIC level visibility of VM traffic flows
 Reporting on Top Flows, Destinations and Sources
 Detailed Flow Data for both Allowed and Blocked Flows
3838
Operational Tools – Flow Monitoring
 Flow data easily available through UI or via API for orchestration
 Per flow granularity for Allowed and Blocked with ability to add or
edit firewall rules related to the flow.
3939
Operational Tools – NSX Manager
 Perform Backup & Restore
operations (both scheduled
and on-demand)
 NSX Manager Appliance
Upgrades
 Verify status of NSX
Manager Services
 Generate Tech support logs
 View appliance CPU,
Memory and Disk usage
4040
Operational Practices – vCenter Operations Manager
 Dashboard based view of environment
 Monitor overall health of NSX vSphere Management and Control
Layer components and diagnose issues quickly
 Networking and Security metrics
4141
Operational Tools – Syslog
 Syslog is supported across all NSX components
• 1) NSX Manager
• 2) NSX Controllers
• 3) NSX Edges
• 4) ESXi Hosts
1
2
3
4
4242
Operational Practices – Log Insight
 Consolidation, visualization, and correlation of syslog data from
multiple related components in a Software Defined Datacenter
 Build Custom Dashboards for real time monitoring and trending
 Customize Log interpretation Logic to parse using regex, int, str
4343
Operational Tools – REST API
 NSX Manager exposes web service API over HTTPS (TCP 443)
 API request and response data is formatted in XML
 Simple “single-user” authentication using password
 REST principles:
• Leverages HTTP to send data between Clients and Servers (Requests and Responses)
• Resources, Global Permanent Identifiers, Constraints
4444
Agenda
 NSX Platform / Network Virtualization
 Operational Impacts
 NSX for vSphere Components
 Operational Tools
 Demonstrations
 Conclusion
4545
NSX for vSphere Demonstrations
1. Packet capture of encapsulated VXLAN frames
2. Flow Monitoring
4646
Key Takeaways & Best Practices
 VMware NSX provides a unified platform for administering,
monitoring, and supporting your virtual networks and services
 NSX enables a similar operational model for virtual networks as
vSphere does for virtual machines
 Moving network features to logical space simplifies physical
networks and troubleshooting
 Start with the basics when troubleshooting (transport network and
control plane)
 Understanding the component interactions and toolset is key to
NSX operations
 Enable logging on all components ‘before’ you have issues and
familiarize yourself with how to collect support logs
 Automate repeatable steps via the REST API to reduce error
 Take the NSX for vSphere Hands on Lab: HOL-SDC-1303 to
reinforce concepts from this session
4747
Questions
4848
Backup Slides
4949
Network Virtualization - Operations
• Overall Logical network
health/stats
• VM to VM connectivity
• Per VM flow visibility
• Traffic Analysis – Packet
Capture
• Transport/Tunnel health
• Inventory/Fault Mgmt
• Multi-level Logging, Event
tracking and Auditing
• Physical network
troubleshooting/visibility
• Upgrade Management
Cloud Ops or Network Ops
vSwitch
NSX EdgeESXi ESXi ESXi ESXi
vSwitch vSwitch vSwitch vSwitch
L2
Logical Topology
L2
WAN/Inter
net
What are the key capabilities required for
operating a Logical world?
Controller Cluster
NSX Manager
5050
NSX Operations – Capabilities
NSX Optimized for vSphere
Logical Network Health
UI: NSX Manager
CLI: Central NSX Controller, NSX Edge
VM to VM connectivity (Logical) NSX Controller Central CLI, Host level CLI
Traffic Flow visibility
IPFIX (VDS)
NSX Edge – Flow Monitoring
Traffic Analysis per VM
RSPAN/ERSPAN (VM Traffic)
Host Packet Capture (Overlay)
Network Inventory, Fault Management NSX Manager, SNMP (MIBS for ports, Switch etc)
Multi-level logging, Event tracking &
Auditing
Syslog Export (NSX controller, NSX Manager, NSX
Edge etc.)
Transport (Overlay) Health
NSX Manager Connectivity Check
NSX Controller Central CLI, Per host CLI
Upgrade Management
NSX Manager (Automated VIB and Controller
upgrades)
API visibility NSX Manager API
External Tools Custom, VCOPs, Log Insight
5151
NSX System Architecture
Cloud Management System
vCenter Server NSX Manager
Management Plane
Control Plane
NSX Edge
Logical
RouterNSX Controller
Data Plane
NSX Edge
Services
Gateway
VXLAN DR DFWSecurity VXLAN DR DFWSecurityVXLAN DR DFWSecurity
vSphere API
MessageBus
vSphereAPI
REST API
NSX vSwitch
Control Plane Protocol
Control Plane
Protocol
REST APIvSphere API
UWA
CP Protocol
5252
Control Plane Protocol
 Control plane protocol
• All messages are TLVs
• Categorized into primitives:
• Connection management, negotiation, etc:
• Hello, bye, keepalive
• App specific:
• Open, close, notification, update, query
• Extensible.
• App ID in message common header.
 VXLAN sub protocol
• Update and query messages contains one
or more TLVs for different data types:
 VM IP, VM MAC, VTEP
5353
VXLAN Control Plane Security
UW
Agent
VTEP
UW
Agent
VTEP
UW
Agent
VTEP
vSphere Cluster B
UW
Agent
VTEP
UW
Agent
VTEP
UW
Agent
VTEP
vSphere Cluster A
Controller Cluster
NSX Manager DB
1
Certificate
Generation
2
OVF
Deployment3 Message Bus
4 REST API
NSX Manager
5 SSL 5 SSL
5 SSL
5454
Operational Practices – NSX Controller MAC Report
vSphere Host
VM
vSphere Distributed Switch
MAC1
Management
Network
10.20.10.10
vSphere Host
VM
MAC2
vSphere Host
Send VNI,VM
MAC Mapping
and VTEP IP to
Controller
1
2
5 VXLAN 5001
3
IP1 IP
2
Controller
VNI VM MAC
5001 MAC1
VNI VM MAC
5001 MAC2
10.20.10.11 10.20.10.12
6
7
48
VNI VM MAC VTEP
5001 MAC1 10.20.10.10
VNI VM MAC VTEP
5001 MAC1
10.20.10.1
0
5001 MAC2
10.20.10.1
1
5555
Operational Tools – NSX Controller IP Report
vSphere Host
VM
vSphere Distributed Switch
MAC1
Management
Network
10.20.10.10
vSphere Host
VM
MAC2
vSphere Host
Send VM MAC,
IP Mapping and
VNI to Controller
1
2
5 VXLAN 5001
3
IP1 IP
2
Controller
VNI
VM
IP
VM
MAC
5001 IP1 MAC1
VNI
VM
IP
VM
MAC
5001 IP2 MAC2
10.20.10.11 10.20.10.12
6
7
48
VNI VM IP VM MAC
5001 IP1 MAC1
VNI VM IP VM MAC
5001 IP1 MAC1
5001 IP2 MAC2
5656
Controller Based VXLAN – ARP Request
vSphere Host
VM
vSphere Distributed Switch
MAC1
Management
Network
10.20.10.10
vSphere Host
VNI
VM
IP
VM
MAC
VTEP
5001 IP1
MAC
1
10.20.10.1
0
VM
MAC2
vSphere Host
1
2
VXLAN 5001
3
IP1 IP2
Controller
10.20.10.11 10.20.10.12
6 5
VNI
VM
IP
VM
MAC
VTEP
5001 IP1 MAC1 10.20.10.10
5001 IP2 MAC2 10.20.10.11
4
PayloadL2
DA: Broadcast
SA: MAC1
ARP Request for
VM IP2 sent to
Controller
ARP Report for
VM IP2, MAC2
sent to VTEP
10.20.10.10
VNI
VM
IP
VM
MAC
VTEP
5001 IP1 MAC1 10.20.10.10
5001 IP2 MAC2 10.20.10.11
5757
Controller Based VXLAN – Communication after ARP Resolution
vSphere Host
VM
vSphere Distributed Switch
MAC1
VXLAN Transport
Network
10.20.10.10
vSphere Host
VM
MAC2
vSphere Host
7
8
VXLAN 5001
IP1 IP2
Controller
10.20.10.11 10.20.10.12
9
VNI
V
M
IP
VM
MAC
VTEP
5001 IP1 MAC1 10.20.10.10
5001 IP2 MAC2 10.20.10.11
PayloadL2
DA: MAC2
SA: MAC1
L2 IP UDP VXLAN PayloadL2
DA: 10.20.10.11
SA: 10.20.10.10
5001
10
DA: MAC1
SA: MAC2
VNI
VM
IP
VM
MAC
VTEP
500
1
IP1 MAC1 10.20.10.10
500
1
IP2 MAC2 10.20.10.11
VNI
VM
IP
VM
MAC
VTEP
500
1
IP2 MAC2 10.20.10.11
VNI
VM
IP
VM
MAC
VTEP
500
1
IP2 MAC1 10.20.10.11
500
1
IP1 MAC2 10.20.10.10
5858
Operational Tools – NSX Edge Services
 Download Edge Gateway Tech Support Logs using the Web Client
 Or from NSX Edge CLI using the following command
 NSX-Edge1-0# export tech-support scp user@scpserver:file
5959
Operational Tools – REST API
• VERB = GET
• URI = https://<NSX Manager Hostname>/api/2.0/vdn/scopes
• HEADERS = Authorization
• HTTP Body = N/A
• RESPONSE: Search for the id of scope:
<id>vdnscope-X</id>
6060
Operational Tools – REST API
• VERB = POST
• URI = https://<NSX Manager
Hostname>/api/2.0/vdn/scopes/vdnscope-1/virtualwires
• HEADERS = Authorization, Content-Type
• HTTP Body =
<virtualWireCreateSpec>
<name>Test-Logical-Switch-01</name>
<description>Created via REST API</description>
<tenantId>virtual wire tenant</tenantId>
<multicastProxy>true</multicastProxy>
<disableMulticast>true</disableMulticast>
</virtualWireCreateSpec>
6161
Operational Tools – REST API
• Response: 201 Created
• The Response Body provides the virtualwire-id, which can be used for additional
operations (eg, attaching to a Logical Router LIF or for Distributed Firewall rules)
6262
Demo 1 (3 mins) - Script
 Component Installation
• NSX Manager
• NSX Controller Cluster
 Preparation
• Login to ESXi host (destination)
• Add Logical Switch
• Connect VMs to Logical Switch
 Data Collection
• Start data collection on destination host, output to a share (that is also accessible on
analyzer)
• Connect VMs to Logical Switch
• Generate some traffic
• Stop data collection
• Start Wireshark and open pcap file
• Enable VXLAN decoder
• Walk through packet data format (VXLAN headers, unicast mode etc)
• Show anything else ? Controller CLI/esxcli ?
6363
References
 Other VMworld breakouts – VXLAN troubleshooting,
Security operations
 VMware Networking and Security Booth
 Hands on Lab: HOL-SDC-1303 VMware NSX to gain hands
on experience
 Expert Bar/Group Discussions
6464
Other VMware Activities Related to This Session
 HOL:
HOL-SDC-1303
VMware NSX Network Virtualization Platform
THANK YOU
VMworld 2013: Operational Best Practices for NSX in VMware Environments
Operational Best Practices for NSX in VMware
Environments
Ray Budavari, VMware
Thomas Kraus, VMware
NET5790
#NET5790

More Related Content

PDF
VMworld 2013: Advanced VMware NSX Architecture
PPTX
VMUGbe 21 Filip Verloy
PDF
VMware NSX primer 2014
PDF
VMware NSX + Cumulus Networks: Software Defined Networking
PDF
VMworld 2013: Virtualized Network Services Model with VMware NSX
PDF
VMware NSX - Lessons Learned from real project
PDF
VMworld 2014: VMware NSX and vCloud Automation Center Integration Technical D...
PDF
Network Virtualization with VMware NSX
VMworld 2013: Advanced VMware NSX Architecture
VMUGbe 21 Filip Verloy
VMware NSX primer 2014
VMware NSX + Cumulus Networks: Software Defined Networking
VMworld 2013: Virtualized Network Services Model with VMware NSX
VMware NSX - Lessons Learned from real project
VMworld 2014: VMware NSX and vCloud Automation Center Integration Technical D...
Network Virtualization with VMware NSX

What's hot (20)

PDF
VMworld 2013: Bringing Network Virtualization to VMware Environments with NSX
PPTX
VMworld 2016: How to Deploy VMware NSX with Cisco Infrastructure
PDF
VMworld Europe 2014: Advanced Network Services with NSX
PPTX
nsx overview with use cases 1.0
PDF
NSX Reference Design version 3.0
PPTX
VMworld 2015: VMware NSX Deep Dive
PDF
The Future of Cloud Networking is VMware NSX
PDF
VMworld 2013: Deploying VMware NSX Network Virtualization
PDF
VMworld 2015: The Future of Network Virtualization with VMware NSX
PDF
VMworld 2013: NSX PCI Reference Architecture Workshop Session 1 - Segmentation
PDF
VMworld 2014: Introduction to NSX
PPTX
#NET5488 - Troubleshooting Methodology for VMware NSX - VMworld 2015
PPTX
VMworld 2016: Advanced Network Services with NSX
PDF
VMworld 2013: Technical Deep Dive: Build a Collapsed DMZ Architecture for Opt...
PDF
VMworld 2013: VMware NSX Integration with OpenStack
PDF
VMworld 2014: Virtualize your Network with VMware NSX
PPTX
Nsx security deep dive
PDF
An Introduction to VMware NSX
PDF
VMware NSX 101: What, Why & How
VMworld 2013: Bringing Network Virtualization to VMware Environments with NSX
VMworld 2016: How to Deploy VMware NSX with Cisco Infrastructure
VMworld Europe 2014: Advanced Network Services with NSX
nsx overview with use cases 1.0
NSX Reference Design version 3.0
VMworld 2015: VMware NSX Deep Dive
The Future of Cloud Networking is VMware NSX
VMworld 2013: Deploying VMware NSX Network Virtualization
VMworld 2015: The Future of Network Virtualization with VMware NSX
VMworld 2013: NSX PCI Reference Architecture Workshop Session 1 - Segmentation
VMworld 2014: Introduction to NSX
#NET5488 - Troubleshooting Methodology for VMware NSX - VMworld 2015
VMworld 2016: Advanced Network Services with NSX
VMworld 2013: Technical Deep Dive: Build a Collapsed DMZ Architecture for Opt...
VMworld 2013: VMware NSX Integration with OpenStack
VMworld 2014: Virtualize your Network with VMware NSX
Nsx security deep dive
An Introduction to VMware NSX
VMware NSX 101: What, Why & How
Ad

Viewers also liked (19)

PDF
VMworld 2013: vSphere Distributed Switch – Design and Best Practices
PDF
Vtsp nv-certificate
PDF
VMworld 2013: VMware Compliance Reference Architecture Framework Overview
PDF
VMworld 2013: NSX PCI Reference Architecture Workshop Session 2 - Privileged ...
PDF
VMworld 2013: VMware Virtual SAN Technical Best Practices
PPTX
VMworld 2016 Recap
PDF
もう一つのHCI VxRackとVBlock
PPTX
Emc vmax3 technical deep workshop
PPTX
Self service it with v realizeautomation and nsx
PPTX
Blue Medora - VMware vROps Management Pack for VCE Vblock Overview
PPTX
NSX for vSphere Logical Routing Deep Dive
PPTX
Emc recoverpoint technical
PPTX
SEC8022_Securing_SDDC_NSX_Hammad_Shahzad
PPTX
VMworld 2016: Enforcing a vSphere Cluster Design with PowerCLI Automation
PDF
VMware NSX for vSphere - Intro and use cases
PPTX
VMworld 2016: Migrating from a hardware based firewall to NSX to improve perf...
PDF
VMUG - NSX Architettura e Design
PPTX
VMworld 2016: vSphere 6.x Host Resource Deep Dive
PPTX
VMware Site Recovery Manager - Architecting a DR Solution - Best Practices
VMworld 2013: vSphere Distributed Switch – Design and Best Practices
Vtsp nv-certificate
VMworld 2013: VMware Compliance Reference Architecture Framework Overview
VMworld 2013: NSX PCI Reference Architecture Workshop Session 2 - Privileged ...
VMworld 2013: VMware Virtual SAN Technical Best Practices
VMworld 2016 Recap
もう一つのHCI VxRackとVBlock
Emc vmax3 technical deep workshop
Self service it with v realizeautomation and nsx
Blue Medora - VMware vROps Management Pack for VCE Vblock Overview
NSX for vSphere Logical Routing Deep Dive
Emc recoverpoint technical
SEC8022_Securing_SDDC_NSX_Hammad_Shahzad
VMworld 2016: Enforcing a vSphere Cluster Design with PowerCLI Automation
VMware NSX for vSphere - Intro and use cases
VMworld 2016: Migrating from a hardware based firewall to NSX to improve perf...
VMUG - NSX Architettura e Design
VMworld 2016: vSphere 6.x Host Resource Deep Dive
VMware Site Recovery Manager - Architecting a DR Solution - Best Practices
Ad

Similar to VMworld 2013: Operational Best Practices for NSX in VMware Environments (20)

PPTX
VMworld 2015: VMware NSX Deep Dive
PPTX
VMware nsx network virtualization tool
PDF
VMworld 2013: Troubleshooting VXLAN and Network Services in a Virtualized Env...
PDF
NSX_Poster.pdf
PDF
NSX: La Virtualizzazione di Rete e il Futuro della Sicurezza
PDF
VMworld 2013: Datacenter Transformation with Network Virtualization: Today an...
PPTX
Reference design for v mware nsx
PDF
VMworld 2013: VMware NSX Extensibility: Network and Security Services from 3r...
PDF
VMworld 2013: vSphere Networking and vCloud Networking Suite Best Practices a...
PDF
vVMworld 2013: Deploying, Troubleshooting, and Monitoring VMware NSX Distribu...
PDF
Nsx t reference design guide 3-0
PDF
Net1674 final emea
PDF
OVHcloud Hosted Private Cloud Platform Network use cases with VMware NSX
PDF
The Vision for the Future of Network Virtualization with VMware NSX
PDF
VMworld 2013: An Introduction to Network Virtualization
PDF
Deploying Elastic Self-Service Load Balancing
PDF
GAMO VMware vCloud Air
PPTX
VXLAN Practice Guide
PDF
VMworld 2014: Advanced Topics & Future Directions in Network Virtualization w...
VMworld 2015: VMware NSX Deep Dive
VMware nsx network virtualization tool
VMworld 2013: Troubleshooting VXLAN and Network Services in a Virtualized Env...
NSX_Poster.pdf
NSX: La Virtualizzazione di Rete e il Futuro della Sicurezza
VMworld 2013: Datacenter Transformation with Network Virtualization: Today an...
Reference design for v mware nsx
VMworld 2013: VMware NSX Extensibility: Network and Security Services from 3r...
VMworld 2013: vSphere Networking and vCloud Networking Suite Best Practices a...
vVMworld 2013: Deploying, Troubleshooting, and Monitoring VMware NSX Distribu...
Nsx t reference design guide 3-0
Net1674 final emea
OVHcloud Hosted Private Cloud Platform Network use cases with VMware NSX
The Vision for the Future of Network Virtualization with VMware NSX
VMworld 2013: An Introduction to Network Virtualization
Deploying Elastic Self-Service Load Balancing
GAMO VMware vCloud Air
VXLAN Practice Guide
VMworld 2014: Advanced Topics & Future Directions in Network Virtualization w...

More from VMworld (20)

PPTX
VMworld 2016: Troubleshooting 101 for Horizon
PPTX
VMworld 2016: What's New with Horizon 7
PPTX
VMworld 2016: Virtual Volumes Technical Deep Dive
PPTX
VMworld 2016: Advances in Remote Display Protocol Technology with VMware Blas...
PPTX
VMworld 2016: The KISS of vRealize Operations!
PPTX
VMworld 2016: Getting Started with PowerShell and PowerCLI for Your VMware En...
PPTX
VMworld 2016: Ask the vCenter Server Exerts Panel
PPTX
VMworld 2016: Virtualize Active Directory, the Right Way!
PPTX
VMworld 2015: Troubleshooting for vSphere 6
PPTX
VMworld 2015: Monitoring and Managing Applications with vRealize Operations 6...
PPTX
VMworld 2015: Advanced SQL Server on vSphere
PPTX
VMworld 2015: Virtualize Active Directory, the Right Way!
PPTX
VMworld 2015: Site Recovery Manager and Policy Based DR Deep Dive with Engine...
PPTX
VMworld 2015: Building a Business Case for Virtual SAN
PPTX
VMworld 2015: Explaining Advanced Virtual Volumes Configurations
PPTX
VMworld 2015: Virtual Volumes Technical Deep Dive
PPTX
VMworld 2015: Networking Virtual SAN's Backbone
PPTX
VMworld 2015: The Best SDDC!
PPTX
VMworld 2015: Conversation with the VMware CIO Suggestions on being an IT Leader
PPTX
VMware 2015: Next Horizon for Cloud Networking and Security
VMworld 2016: Troubleshooting 101 for Horizon
VMworld 2016: What's New with Horizon 7
VMworld 2016: Virtual Volumes Technical Deep Dive
VMworld 2016: Advances in Remote Display Protocol Technology with VMware Blas...
VMworld 2016: The KISS of vRealize Operations!
VMworld 2016: Getting Started with PowerShell and PowerCLI for Your VMware En...
VMworld 2016: Ask the vCenter Server Exerts Panel
VMworld 2016: Virtualize Active Directory, the Right Way!
VMworld 2015: Troubleshooting for vSphere 6
VMworld 2015: Monitoring and Managing Applications with vRealize Operations 6...
VMworld 2015: Advanced SQL Server on vSphere
VMworld 2015: Virtualize Active Directory, the Right Way!
VMworld 2015: Site Recovery Manager and Policy Based DR Deep Dive with Engine...
VMworld 2015: Building a Business Case for Virtual SAN
VMworld 2015: Explaining Advanced Virtual Volumes Configurations
VMworld 2015: Virtual Volumes Technical Deep Dive
VMworld 2015: Networking Virtual SAN's Backbone
VMworld 2015: The Best SDDC!
VMworld 2015: Conversation with the VMware CIO Suggestions on being an IT Leader
VMware 2015: Next Horizon for Cloud Networking and Security

Recently uploaded (20)

PDF
Review of recent advances in non-invasive hemoglobin estimation
PDF
NewMind AI Weekly Chronicles - August'25 Week I
PDF
Electronic commerce courselecture one. Pdf
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
PDF
Spectral efficient network and resource selection model in 5G networks
PDF
Encapsulation_ Review paper, used for researhc scholars
PPT
Teaching material agriculture food technology
PDF
Network Security Unit 5.pdf for BCA BBA.
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
PDF
Machine learning based COVID-19 study performance prediction
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
PDF
Chapter 3 Spatial Domain Image Processing.pdf
PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
PPTX
MYSQL Presentation for SQL database connectivity
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
Review of recent advances in non-invasive hemoglobin estimation
NewMind AI Weekly Chronicles - August'25 Week I
Electronic commerce courselecture one. Pdf
Digital-Transformation-Roadmap-for-Companies.pptx
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
Spectral efficient network and resource selection model in 5G networks
Encapsulation_ Review paper, used for researhc scholars
Teaching material agriculture food technology
Network Security Unit 5.pdf for BCA BBA.
Per capita expenditure prediction using model stacking based on satellite ima...
Machine learning based COVID-19 study performance prediction
Mobile App Security Testing_ A Comprehensive Guide.pdf
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
Chapter 3 Spatial Domain Image Processing.pdf
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
MYSQL Presentation for SQL database connectivity
Dropbox Q2 2025 Financial Results & Investor Presentation
The Rise and Fall of 3GPP – Time for a Sabbatical?
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
20250228 LYD VKU AI Blended-Learning.pptx

VMworld 2013: Operational Best Practices for NSX in VMware Environments

  • 1. Operational Best Practices for NSX in VMware Environments Ray Budavari, VMware Thomas Kraus, VMware NET5790 #NET5790
  • 2. 22 Agenda  Introduction - Network Virtualization  Operational Impacts  NSX for vSphere Components  Operational Tools  Demonstrations  Conclusion
  • 3. 33 Introduction - Network Virtualization 1. Decouple Physical Virtual 2. Reproduce 3. Automate Network Operations Cloud Operations Hardware independence Operational benefits of virtualization No change to network from end host perspective Virtual Physical
  • 4. 44 Agenda  Introduction - Network Virtualization  Operational Impacts  NSX for vSphere Components  Operational Tools  Demonstrations  Conclusion
  • 5. 55 Operational Impacts - Questions  If a Virtual Machine has a network outage where do I start?  How does network virtualization map to our operating model?  What tools exist to correlate logical and physical networks to assist in troubleshooting?  What opportunities does network virtualization provide to improve how we operate our environment?  Are we adding complexity by adopting network virtualization?  Does network virtualization create a ‘black box’?
  • 6. 66 Operational Impacts - Answers Capability Physical Virtual Capability Physical Virtual Packet Capture ✔ ✔ VM level visibility ✗ ✔ NetFlow ✔ ✔ Network Snapshot ✗ ✔ RSPAN/ERSPA N ✔ ✔ CLI ✔ ✔ Performance Statistics ? ✔ UI ? ✔ Syslog ✔ ✔ API ? ✔
  • 7. 77 Operational Impacts - Opportunities  All NSX components such as the NSX Controller, NSX vSwitch and NSX Edge provide detailed network visibility and data  Simplify the underlying physical network • One consistent physical transport network to manage for virtual machine traffic • Greatly reduces the number of MAC/ARP table entries to manage • Enables you to build the network you want, while still meeting application and workload connectivity requirements DC Networks  Centralized reporting and monitoring, distributed performance and scale  Designed for automation • NSX is built on a REST API provided by NSX Manager • All operations can be performed programmatically via scripting or higher-level languages
  • 8. 88 Operational Impacts - Examples  If a Virtual Machine has a network outage where do I start to troubleshoot?  Before Network Virtualization: • Validate VLAN trunk configuration across multiple devices and ports • Verify VM visibility on each path of the network • Troubleshooting requires accessing different devices and interfaces • vSphere Web Client • Hypervisor CLI • Access Switch CLI • Distribution Switch CLI • Firewall • Load Balancer • These devices are typically managed by different teams • Virtualization Administrators • Network Administrators • Security Administrators • Service Providers
  • 9. 99 Operational Impacts - Examples  After Network Virtualization: • VXLAN network tests determine if the issue is related to the transport network • If VTEPs are reporting issues, engage the network team to troubleshoot physical transport network • Provide VTEP IP/MAC information • Otherwise virtualization team validates VM logical networking • Verify NSX Components and Controller state information • Verify Source and Destination Hypervisors • Enhanced toolset is available for troubleshooting • Reduced number of components and resources required • NSX components can be queried or configured via REST API
  • 10. 1010 Agenda  Introduction - Network Virtualization  Operational Impacts  NSX for vSphere Components  Operational Tools  Demonstrations  Conclusion
  • 11. 1111 NSX for vSphere Components Consumption • Self Service Portal • Cloud Management • vCloud Automation Center Data Plane NSX Edge Services Gateway ESXi VDS Hypervisor Kernel Modules FirewallDistributed Logical Router VXLAN NSX vSwitch • NSX Edge • VM form factor • Data Plane for North South traffic • Routing and Advanced services • NSX vSwitch • Distributed network edge • Line Rate performance Management Plane NSX Manager • Single point of configuration • REST API and UI interface vCenter Server Control Plane NSX Controller • Manages Logical networks • Run-time state • Does not sit in the Data Path • Control-Plane Protocol NSX Edge Logical Router User World Agent
  • 12. 1212 Components – NSX Manager • NSX for vSphere centralized management plane • 1:1 mapping between an NSX Manager and vCenter Server • Provides the management UI and API for NSX • vSphere Web Client Plugin • Deploys NSX Controller and NSX Edge Virtual Appliances (OVF) • Installs VXLAN, Distributed Routing and Firewall kernel modules and UW Agent on ESXi hosts • Configures Controller Cluster via a REST API and hosts via a message bus • Generates certificates to secure control plane communications
  • 13. 1313 Components – NSX Controller  A reliable and secure control plane to distribute VXLAN and Logical Routing network information to ESXi hosts  NSX Controllers are clustered for scale out and high availability  Network information is sliced across nodes in a Controller Cluster  Enables dependency on multicast routing/PIM in the physical network to be removed  Provides suppression of ARP broadcast traffic in VXLAN networks VXLAN Logical Router VXLAN Logical Router VXLAN Logical Router Controller VXLAN Directory Service MAC table ARP table VTEP table
  • 14. 1414 Components – UW agent  UW agent is a TCP (SSL) client that communicates with the Controller using the control plane protocol  May connect to multiple controllers  Mediator between the ESXi Hypervisor Kernel Modules and NSX Controllers  Also communicates with message bus agent to retrieve information from| NSX Manager  Runs as a service daemon on ESXi: netcpa • Logs to: /var/log/netcpa.log Controller Cluster Controller Controller Controller ESXi Host Kernel Modules Client ClientUser World Agent LR NSX MGR Client VXLAN
  • 15. 1515 Components – NSX vSwitch and NSX Edge  NSX vSwitch (VDS)  VMkernel Modules  VXLAN  Distributed Routing  Distributed Firewall  Switch Security  Message Bus  L3-L7 Services:  NAT, DHCP, LB, VPN, Interface based FW  Dynamic Routing  VM form factor  High Availability vSphere NSX Edge Services GW  Control Functions only  Dynamic Routing & updates to Controller  Determines active ESXi host for L2 Bridging NSX Edge Logical Router ESXi VDS Hypervisor Kernel Modules (vSphere VIBs) FirewallLogical RouterVXLAN NSX vSwitch NSX Edge Services Gateway NSX Edge Logical Router
  • 16. 1616 Agenda  Introduction - Network Virtualization  Operational Impacts  NSX for vSphere Components  Operational Tools  Demonstrations  Conclusion
  • 17. 1717 Operational Tools – ESXi  pktcap-uw  New with vSphere 5.5  Enhanced tool that provides a framework for packet capture and tracing at the Uplink, vSwitch, vmknic, vnic and port level at any stage in a packet’s lifecycle
  • 18. 1818 Operational Tools – ESXi  pktcap-uw  Extensive range of filters such as source/destination mac, IP, Protocol, VLAN, VXLAN, ports etc.  Supports pcap format output for use with protocol analyzers such as Wireshark
  • 19. 1919 Operational Tools – NSX vSwitch Backup & Restore  vSwitch Backup & Restore
  • 20. 2020 Operational Tools – NSX vSwitch Netflow  NetFlow / IPFIX NetFlow collector address and port
  • 21. 2121 Operational Tools – NSX vSwitch RSPAN/ERSPAN  RSPAN/ERSPAN, Port Mirroring
  • 22. 2222 Operational Tools – NSX vSwitch Alarms  vSwitch Alarms allow for alerting on VDS related events. Some of the available preconfigured Triggers include Health Status, Reconfiguration, Port blocked, Port Deleted, Link Down and Host removal  SNMP Network MIBs provide standards based visibility of NSX vSwitch objects
  • 23. 2323 Operational Tools – NSX vSwitch Health Check  Network Health Check feature helps to detect common configuration errors  Mismatched VLAN trunks between virtual switch and physical switch  Mismatched MTU setting between vNIC, virtual switch, physical adapter, and physical switch ports.  Mismatched Teaming Configurations  vSphere admins can provide failure data to the Network admins to facilitate problem resolution  Health Check uses L2 Echo protocol to send Ethernet broadcast frames to the physical switch  If reply packets are not received, warnings are highlighted in the vSphere Web Client
  • 24. 2424 vSphere Cluster B UWA VTEP UWA VTEP UWA VTEP Operational Tools – NSX Controller  Control Plane basics  ESXi hosts and NSX Edge Logical Router VMs collect network information, which is then reported to the Controller via User World Agent (UWA)  The NSX Controller CLI provides a consistent, centralized interface to verify VXLAN and Logical Routing network state information  NSX Manager also provides APIs to programmatically retrieve data from the controller nodes NSX Manager NSX Controller Cluster vSphere Cluster A UWA VTEP UWA VTEP UWA VTEP
  • 25. 2525 Operational Tools – NSX Controller VTEP Report vSphere Host VM vSphere Distributed Switch MAC1 Management Network 10.20.10.10 vSphere Host VM MAC2 vSphere Host Send VNI,VTEP Mapping to Controller 1 2 5 10 VXLAN 5001 3 VNI VTEP IP 5001 10.20.10.10 9 IP1 IP 2 Controller VNI VTEP IP 5001 10.20.10.10 VNI VTEP IP 5001 10.20.10.11 10.20.10.11 10.20.10.12 11 6 7 VNI VTEP IP 5001 10.20.10.10 10.20.10.11 VNI VTEP IP 5001 10.20.10.10 10.20.10.11 48 10 VNI VTEP IP 5001 10.20.10.10 10.20.10.11 11 Report the new VNI,VTEP Mapping to the Hosts
  • 26. 2626 Operational Tools – NSX Controller  General NSX Controller troubleshooting steps: • Verify Controller cluster status and roles • Verify Controller node network connectivity • Check Controller API service • Validate VXLAN and Logical Router mapping table entries to ensure they are consistent • Review source and destination netcpa logs and CLI to determine control plane connectivity issues between ESXi hosts & NSX Controller  The first set of commands relates to NSX Controller CLI cluster status and health: • nsx-controller # show control-cluster status • nsx-controller # show control-cluster startup- nodes • nsx-controller # show control-cluster roles • nsx-controller # show control-cluster connections • nsx-controller # show control-cluster vnet core stats • nsx-controller # show network <arg> • nsx-controller # show log cloudnet/cloudnet_java- vnet-controller.<start-time-stamp>.log
  • 27. 2727 Operational Tools – NSX Controller  General NSX Controller VXLAN operations: • List VNIs • VXLAN connection table • Verify VXLAN VTEP, MAC and ARP mapping tables • View VXLAN statistics  NSX Controller CLI VXLAN commands: • # show control-cluster vnet vxlan vni <vni> • # show control-cluster vnet vxlan connection-table <vni> • # show control-cluster vnet vxlan vtep-table <vni> • # show control-cluster vnet vxlan mac-table <vni> • # show control-cluster vnet vxlan arp-table <vni> • # show control-cluster vnet vxlan vni-stats <vni>  Note: VXLAN Logical Switches and Logical Router instances are distributed across Controller Nodes (slicing), so you will need to run the CLI commands on the node which is active for a given object
  • 28. 2828 Operational Tools – NSX Controller  General NSX Controller Logical Routing operations: • List Logical Router instances • Verify Logical Router interface and route mapping tables • Verify active controller connections • View Logical Router statistics  NSX Controller CLI Logical Routing commands: • # show control-cluster vnet logical-router instance-table <lr-id> • # show control-cluster vnet logical-router lif-table <lr-id> • # show control-cluster vnet logical-router route-table <lr-id> • # show control-cluster vnet logical-router lr-stats <lr-id>  Note that the NSX Controller CLI is still not final
  • 29. 2929 Operational Tools – VXLAN  Common VXLAN issues: • Connectivity between VXLAN VTEPs on Transport Network • MTU on Transport Network not set to 1600 bytes or greater • Teaming mismatch between dvUplinks and upstream switch • Preparation - either with the installation of VXLAN kernel modules or creation of VTEP VMkernel interfaces • ESXi host communication with Controller  NSX for vSphere provides a new workflow for preparation and configuration  Supports multiple VTEPs per host  NSX leverages the vSphere 5.5 multi-instance TCP/IP stack
  • 30. 3030 Operational Tools – VXLAN  At Logical Switch level on the Monitoring Tab you can use the unicast or broadcast test to verify the connectivity between VTEPs VXLAN standard tests with a 1600 byte MTU
  • 31. 3131 Operational Tools – VXLAN  The VXLAN Replication Mode will also determine transport network connectivity requirements  Unicast Mode • All replication occurs using unicast  Hybrid Mode • Local replication offloaded to physical network, while remote replication occurs via unicast  Multicast Mode • Requires IGMP for a Layer 2 topology and Multicast Routing for L3 topology  All modes require an MTU of 1600 bytes
  • 32. 3232 Operational Tools – VXLAN  VXLAN namespace for esxcli provides detailed network information and statistics. • # esxcli network vswitch dvs vmware vxlan list • # esxcli network vswitch dvs vmware vxlan network list --vds-name=Compute_VDS • # esxcli network vswitch dvs vmware vxlan network mac list –vds-name=Compute_VDS -- vxlan-id=5001 • # esxcli network vswitch dvs vmware vxlan network arp list --vds-name Compute_VDS --vxlan-id=5001 • # esxcli network vswitch dvs vmware vxlan network port list --vds-name Compute_VDS --vxlan-id=5001 • # esxcli network vswitch dvs vmware vxlan network stats list --vds-name Compute_VDS --vxlan-id=5001
  • 33. 3333 Operational Tools – Logical Routing  Use the net-vdr command on ESXi hosts to view Logical Routing configuration and statistics • Display Logical Router instances ~ # net-vdr -I –l • List Logical Interface and Routing Tables ~ # net-vdr -l –lif <instance-name> ~ # net-vdr -l --route <instance-name> LIFs and routes are pushed by the NSX controller to the ESXi hosts and should be consistent across the environment • View L2 Bridging information ~ # net-vdr -b –mac <instance-name> ~ # net-vdr -l –stats <instance-name>
  • 34. 3434 Operational Tools – NSX Edge Services  NSX Edge VM CLIs • NSX Edge provides a familiar CLI command set for troubleshooting network services • Documented in a dedicated CLI guide  Sample Configuration Commands • show configuration {ospf|bgp|isis|static-routing} • show configuration {firewall|nat|dhcp|dns} • show configuration {loadbalancer|ipec|sslvpn-plus}  Sample Status Commands • show interface [IFNAME] • show firewall • show ip {route|ospf|bgp|forwarding} • show arp • show system {cpu|memory|network-stats|storage|uptime} • show service {dhcp|dns|highavailability|ipsec|loadbalancer|sslvpn-plus}
  • 35. 3535 Operational Tools – NSX Edge Services  API provides based statistics (for interfaces and services)  UI also provides interface statistics and graphs
  • 36. 3636 Operational Tools – NSX Edge Services  Logging commands • show log {follow|reverse} • show flowtable  Debug/troubleshooting commands • traceroute <ip_address or dns_name> • ping <ip address> or ping interface addr <alternate_src_ip> <ip_address> • debug packet display interface <vNic_0-9> <EXPRESSION> • debug packet display interface vNic_0 host_192.168.1.2 • debug packet display interface vNic_2 host_192.168.1.3_and_port_80 • debug packet display interface vNic_1 src_192.168.1.2_and_dst_192.168.1.3 • debug packet capture interface <vNic_0-9> <EXPRESSION> • debug show files • debug copy {scp|ftp} <URL>  debug messagebus {forwarder|messages}
  • 37. 3737 Operational Tools – Flow Monitoring  Flow monitoring provides vNIC level visibility of VM traffic flows  Reporting on Top Flows, Destinations and Sources  Detailed Flow Data for both Allowed and Blocked Flows
  • 38. 3838 Operational Tools – Flow Monitoring  Flow data easily available through UI or via API for orchestration  Per flow granularity for Allowed and Blocked with ability to add or edit firewall rules related to the flow.
  • 39. 3939 Operational Tools – NSX Manager  Perform Backup & Restore operations (both scheduled and on-demand)  NSX Manager Appliance Upgrades  Verify status of NSX Manager Services  Generate Tech support logs  View appliance CPU, Memory and Disk usage
  • 40. 4040 Operational Practices – vCenter Operations Manager  Dashboard based view of environment  Monitor overall health of NSX vSphere Management and Control Layer components and diagnose issues quickly  Networking and Security metrics
  • 41. 4141 Operational Tools – Syslog  Syslog is supported across all NSX components • 1) NSX Manager • 2) NSX Controllers • 3) NSX Edges • 4) ESXi Hosts 1 2 3 4
  • 42. 4242 Operational Practices – Log Insight  Consolidation, visualization, and correlation of syslog data from multiple related components in a Software Defined Datacenter  Build Custom Dashboards for real time monitoring and trending  Customize Log interpretation Logic to parse using regex, int, str
  • 43. 4343 Operational Tools – REST API  NSX Manager exposes web service API over HTTPS (TCP 443)  API request and response data is formatted in XML  Simple “single-user” authentication using password  REST principles: • Leverages HTTP to send data between Clients and Servers (Requests and Responses) • Resources, Global Permanent Identifiers, Constraints
  • 44. 4444 Agenda  NSX Platform / Network Virtualization  Operational Impacts  NSX for vSphere Components  Operational Tools  Demonstrations  Conclusion
  • 45. 4545 NSX for vSphere Demonstrations 1. Packet capture of encapsulated VXLAN frames 2. Flow Monitoring
  • 46. 4646 Key Takeaways & Best Practices  VMware NSX provides a unified platform for administering, monitoring, and supporting your virtual networks and services  NSX enables a similar operational model for virtual networks as vSphere does for virtual machines  Moving network features to logical space simplifies physical networks and troubleshooting  Start with the basics when troubleshooting (transport network and control plane)  Understanding the component interactions and toolset is key to NSX operations  Enable logging on all components ‘before’ you have issues and familiarize yourself with how to collect support logs  Automate repeatable steps via the REST API to reduce error  Take the NSX for vSphere Hands on Lab: HOL-SDC-1303 to reinforce concepts from this session
  • 49. 4949 Network Virtualization - Operations • Overall Logical network health/stats • VM to VM connectivity • Per VM flow visibility • Traffic Analysis – Packet Capture • Transport/Tunnel health • Inventory/Fault Mgmt • Multi-level Logging, Event tracking and Auditing • Physical network troubleshooting/visibility • Upgrade Management Cloud Ops or Network Ops vSwitch NSX EdgeESXi ESXi ESXi ESXi vSwitch vSwitch vSwitch vSwitch L2 Logical Topology L2 WAN/Inter net What are the key capabilities required for operating a Logical world? Controller Cluster NSX Manager
  • 50. 5050 NSX Operations – Capabilities NSX Optimized for vSphere Logical Network Health UI: NSX Manager CLI: Central NSX Controller, NSX Edge VM to VM connectivity (Logical) NSX Controller Central CLI, Host level CLI Traffic Flow visibility IPFIX (VDS) NSX Edge – Flow Monitoring Traffic Analysis per VM RSPAN/ERSPAN (VM Traffic) Host Packet Capture (Overlay) Network Inventory, Fault Management NSX Manager, SNMP (MIBS for ports, Switch etc) Multi-level logging, Event tracking & Auditing Syslog Export (NSX controller, NSX Manager, NSX Edge etc.) Transport (Overlay) Health NSX Manager Connectivity Check NSX Controller Central CLI, Per host CLI Upgrade Management NSX Manager (Automated VIB and Controller upgrades) API visibility NSX Manager API External Tools Custom, VCOPs, Log Insight
  • 51. 5151 NSX System Architecture Cloud Management System vCenter Server NSX Manager Management Plane Control Plane NSX Edge Logical RouterNSX Controller Data Plane NSX Edge Services Gateway VXLAN DR DFWSecurity VXLAN DR DFWSecurityVXLAN DR DFWSecurity vSphere API MessageBus vSphereAPI REST API NSX vSwitch Control Plane Protocol Control Plane Protocol REST APIvSphere API UWA CP Protocol
  • 52. 5252 Control Plane Protocol  Control plane protocol • All messages are TLVs • Categorized into primitives: • Connection management, negotiation, etc: • Hello, bye, keepalive • App specific: • Open, close, notification, update, query • Extensible. • App ID in message common header.  VXLAN sub protocol • Update and query messages contains one or more TLVs for different data types:  VM IP, VM MAC, VTEP
  • 53. 5353 VXLAN Control Plane Security UW Agent VTEP UW Agent VTEP UW Agent VTEP vSphere Cluster B UW Agent VTEP UW Agent VTEP UW Agent VTEP vSphere Cluster A Controller Cluster NSX Manager DB 1 Certificate Generation 2 OVF Deployment3 Message Bus 4 REST API NSX Manager 5 SSL 5 SSL 5 SSL
  • 54. 5454 Operational Practices – NSX Controller MAC Report vSphere Host VM vSphere Distributed Switch MAC1 Management Network 10.20.10.10 vSphere Host VM MAC2 vSphere Host Send VNI,VM MAC Mapping and VTEP IP to Controller 1 2 5 VXLAN 5001 3 IP1 IP 2 Controller VNI VM MAC 5001 MAC1 VNI VM MAC 5001 MAC2 10.20.10.11 10.20.10.12 6 7 48 VNI VM MAC VTEP 5001 MAC1 10.20.10.10 VNI VM MAC VTEP 5001 MAC1 10.20.10.1 0 5001 MAC2 10.20.10.1 1
  • 55. 5555 Operational Tools – NSX Controller IP Report vSphere Host VM vSphere Distributed Switch MAC1 Management Network 10.20.10.10 vSphere Host VM MAC2 vSphere Host Send VM MAC, IP Mapping and VNI to Controller 1 2 5 VXLAN 5001 3 IP1 IP 2 Controller VNI VM IP VM MAC 5001 IP1 MAC1 VNI VM IP VM MAC 5001 IP2 MAC2 10.20.10.11 10.20.10.12 6 7 48 VNI VM IP VM MAC 5001 IP1 MAC1 VNI VM IP VM MAC 5001 IP1 MAC1 5001 IP2 MAC2
  • 56. 5656 Controller Based VXLAN – ARP Request vSphere Host VM vSphere Distributed Switch MAC1 Management Network 10.20.10.10 vSphere Host VNI VM IP VM MAC VTEP 5001 IP1 MAC 1 10.20.10.1 0 VM MAC2 vSphere Host 1 2 VXLAN 5001 3 IP1 IP2 Controller 10.20.10.11 10.20.10.12 6 5 VNI VM IP VM MAC VTEP 5001 IP1 MAC1 10.20.10.10 5001 IP2 MAC2 10.20.10.11 4 PayloadL2 DA: Broadcast SA: MAC1 ARP Request for VM IP2 sent to Controller ARP Report for VM IP2, MAC2 sent to VTEP 10.20.10.10 VNI VM IP VM MAC VTEP 5001 IP1 MAC1 10.20.10.10 5001 IP2 MAC2 10.20.10.11
  • 57. 5757 Controller Based VXLAN – Communication after ARP Resolution vSphere Host VM vSphere Distributed Switch MAC1 VXLAN Transport Network 10.20.10.10 vSphere Host VM MAC2 vSphere Host 7 8 VXLAN 5001 IP1 IP2 Controller 10.20.10.11 10.20.10.12 9 VNI V M IP VM MAC VTEP 5001 IP1 MAC1 10.20.10.10 5001 IP2 MAC2 10.20.10.11 PayloadL2 DA: MAC2 SA: MAC1 L2 IP UDP VXLAN PayloadL2 DA: 10.20.10.11 SA: 10.20.10.10 5001 10 DA: MAC1 SA: MAC2 VNI VM IP VM MAC VTEP 500 1 IP1 MAC1 10.20.10.10 500 1 IP2 MAC2 10.20.10.11 VNI VM IP VM MAC VTEP 500 1 IP2 MAC2 10.20.10.11 VNI VM IP VM MAC VTEP 500 1 IP2 MAC1 10.20.10.11 500 1 IP1 MAC2 10.20.10.10
  • 58. 5858 Operational Tools – NSX Edge Services  Download Edge Gateway Tech Support Logs using the Web Client  Or from NSX Edge CLI using the following command  NSX-Edge1-0# export tech-support scp user@scpserver:file
  • 59. 5959 Operational Tools – REST API • VERB = GET • URI = https://<NSX Manager Hostname>/api/2.0/vdn/scopes • HEADERS = Authorization • HTTP Body = N/A • RESPONSE: Search for the id of scope: <id>vdnscope-X</id>
  • 60. 6060 Operational Tools – REST API • VERB = POST • URI = https://<NSX Manager Hostname>/api/2.0/vdn/scopes/vdnscope-1/virtualwires • HEADERS = Authorization, Content-Type • HTTP Body = <virtualWireCreateSpec> <name>Test-Logical-Switch-01</name> <description>Created via REST API</description> <tenantId>virtual wire tenant</tenantId> <multicastProxy>true</multicastProxy> <disableMulticast>true</disableMulticast> </virtualWireCreateSpec>
  • 61. 6161 Operational Tools – REST API • Response: 201 Created • The Response Body provides the virtualwire-id, which can be used for additional operations (eg, attaching to a Logical Router LIF or for Distributed Firewall rules)
  • 62. 6262 Demo 1 (3 mins) - Script  Component Installation • NSX Manager • NSX Controller Cluster  Preparation • Login to ESXi host (destination) • Add Logical Switch • Connect VMs to Logical Switch  Data Collection • Start data collection on destination host, output to a share (that is also accessible on analyzer) • Connect VMs to Logical Switch • Generate some traffic • Stop data collection • Start Wireshark and open pcap file • Enable VXLAN decoder • Walk through packet data format (VXLAN headers, unicast mode etc) • Show anything else ? Controller CLI/esxcli ?
  • 63. 6363 References  Other VMworld breakouts – VXLAN troubleshooting, Security operations  VMware Networking and Security Booth  Hands on Lab: HOL-SDC-1303 VMware NSX to gain hands on experience  Expert Bar/Group Discussions
  • 64. 6464 Other VMware Activities Related to This Session  HOL: HOL-SDC-1303 VMware NSX Network Virtualization Platform
  • 67. Operational Best Practices for NSX in VMware Environments Ray Budavari, VMware Thomas Kraus, VMware NET5790 #NET5790