VMware NSX - Lessons Learned from real project
6 likes4,073 views
This document provides an overview and agenda for a presentation on implementing end-to-end quality of service (QoS) for VMware vSphere with NSX on Cisco UCS. It discusses the project requirements of guaranteeing network traffic for FCoE storage, vSphere management, vMotion and VM backups. It then presents three design options for implementing QoS by marking and prioritizing different classes of service on the virtual network interface cards, VMware distributed virtual switch port groups, Cisco UCS fabric interconnects and Nexus switches. The optimal solution must meet requirements within the constraints of the Cisco and VMware infrastructure components.
![NSX-v Security Concept
vRA Business Group: HR
Logical Network
Micro Security Zone
Technical Service - SAP
[NSX Security Group of all VMs
having tag MSZ-SAP]
Micro Security Zone
Technical Service - A
[NSX Security Group of all VMs
having tag MSZ-A]
vRA Business Group: FINANCE
Logical Network
Micro Security Zone
Technical Service - B
[NSX Security Group of all VMs
having tag MSZ-B]
NSX Distributed
Logical Router
MSZ-SAP MSZ-SAP MSZ-SAP
MSZ-SAP MSZ-SAP
MSZ-A MSZ-A
MSZ-B MSZ-B MSZ-B
Default NSX Security Policy
NAME SOURCE DESTINATION SERVICE ACTION
Default Any Any Any Block
SECURITY TAGS
Security tags for technical services:
MSZ-<Technical-Service-from-CMDB>
For example: MSZ-SAP, MSZ-A, MSZ-B
Security tags for applications:
APP-<gkpke.APP-SEC-TAG[x]>
For example: APP-MSSQL, APP-IIS, APP-EXCHANGE
APP-MSSQL
APP-MSSQL NSX SECURITY GROUPS
We have NSX Security Group for each Technical Service.
This security group forms Micro Security Zone for particular
Technical Service.
For example: MSZ-SAP, MSZ-A, MSZ-B
All VMs tagged with the Security Group name will belong to
this security group.
NSX Security Policy for Micro Security Zones
NAME SOURCE DESTINATION SERVICE ACTION
Inside MSZ-A MSZ-A MSZ-A Any Allow
Inside MSZ-B MSZ-B MSZ-B Any Allow
Inside MSZ-SAP MSZ-SAP MSZ-SAP Any Allow
Other NSX Security Groups and Policies
Other NSX security groups and polices can be created based on
applications tags and other metadata available for NSX.
Physical or Hyper-V Server
belonging in to Micro
Security Zone](https://image.slidesharecdn.com/nsxrealprojectusecase-160129162248/85/VMware-NSX-Lessons-Learned-from-real-project-6-320.jpg)
VMware NSX - Lessons Learned from real project
- 1. © 2014 VMware Inc. All rights reserved. NSX Architecture Design Lessons Learned from real project David Pasek Infrastructure Architect VCDX #200 End to End QoS Solution for VMware vSphere with NSX on top of CISCO UCS
- 2. Agenda 1 Project Overview 2 NSX Conceptual & Logical Design 3 Deep Dive in to Network QoS – Design Decision Point 4 Q & A CONFIDENTIAL 2
- 3. • Private Cloud – EMC FEHC-CA with custom enhancements • vSphere VM as a Service • Hyper-V VM as a Service • Physical Server as a Service • Backup as a Service • Storage as a Service • Environment / Facilities • Two datacenters in metro distance (<5 ms) • Remote Offices (Technical Rooms) in MPLS distance • Products and Technologies • CMP: vRealizeAutomation, vRealize Orchestrator, vRealize Business • Infrastructure Virtualization: VMware vSphere, Hyper-V, NSX-v • Servers: Cisco UCS • Networking: Cisco Nexus • Storage: EMC VIPR, EMC VPLEX, EMC VNX, VMware VSAN • Backup: EMC Avamar, EMC Networker, EMC DataDomain • Security: NSX + PaloAlto Networks Project Overview
- 4. Overall Project High Level Concept Datacenter A Datacenter B vSphere Resource Pool - GOLD TIER VMware vSphere Metro Cluster Stretched across two datacenters Storage Stretched across two datacenters (VPLEX) Technical Room Resource Pool - TR TIER (vSphere + VSAN) Remote Location Existing Core Network Cloud Consumers Cloud Administrators vRealize Automation vRealize Business Std. + Adv. IT Finance vRealize Automation vCenter Orchestrator vRealize LogInsight vRealize Operations Manager vSphere Resource Pool - SILVER TIER Cluster in single datacenter Storage in single datacenter (different storage tiers) vSphere Resource Pool - SILVER TIER Cluster in single datacenter Storage in single datacenter (different storage tiers) Cloud Management Infrastructure Cluster VMware vSphere Metro Cluster Stretched across two datacenters Storage Stretched across two datacenters (VPLEX) Cloud Management Software Stack Cloud Management Platform, vSphere Management, NSX Management workloads Hyper-V Resource Pool Cluster in single datacenter Storage in single datacenter Hyper-V Resource Pool Cluster in single datacenter Storage in single datacenter Physical Servers Resource Pool Server in single datacenter Storage in single datacenter Physical Servers Resource Pool Server in single datacenter Storage in single datacenter
- 5. NSX-v Conceptual Architecture Datacenter A (CDP-A) Datacenter B (CDP-B) CORE NETWORK (dynamic routing protocol has to be implemented) PaloAlto FW Physical Appliance PaloAlto FW Physical Appliance NSX Edge GW NSX Edge GW NSX Edge GW NSX Edge GW VIRTUALNETWORKOVERLAY PHYSICAL NETWORK UNDERLAY ESXi Host ESXi HostESXi Host ESXi Host GOLD vSphere Cluster - STRETCHED ESXi HostESXi Host ESXi Host SILVER vSphere Cluster LOCAL ESXi Host ESXi HostESXi Host ESXi Host LOGICAL SWITCH (VXLAN SEGMENT) vNIC vPaloAlto L7 FW vPaloAlto L7 FW vPaloAlto L7 FW vPaloAlto L7 FW vPaloAlto L7 FW vPaloAlto L7 FW vPaloAlto L7 FW vPaloAlto L7 FW LOGICAL SWITCH (VXLAN SEGMENT) NSX FW NSX FW NSX FW NSX FW NSX FW NSX FW NSX FW NSX FW NSX DLR Distributed Logical Router East-West Routing in DCs Traffic Steering NSX DISTRIBUTED LOGICAL FIREWALL NSX DISTRIBUTED LOGICAL FIREWALL L3 Fabric ECMP + Dynamic Routing between PAN, NSX Edge GWs and NSX DLRs PaloAlto Panorama Centralized Security Management VMware NSX Manager Centralized Virtual Network Management NSX Edge L2 VPN NSX Edge L2 VPN NSX Edge L2 VPN Highly Available TR VPN Termination NSX Edge Services GWs Highly Available North South Routing NSX Edge GW L2 VPN LOGICAL SWITCH (VXLAN SEGMENT) Technical Room (TR) L2 VPN TUNNEL (TR<—>DC) FEHC Management vSphere Cluster - STRETCHED ESXi Host ESXi Host SILVER vSphere Cluster - LOCAL LOGICAL SWITCH (VLAN SEGMENT) ESXi HostESXi Host ESXi Host SILVER vSphere Cluster LOCAL LOGICAL SWITCH (VXLAN SEGMENT) NSX DISTRIBUTED LOGICAL FIREWALL
- 6. NSX-v Security Concept vRA Business Group: HR Logical Network Micro Security Zone Technical Service - SAP [NSX Security Group of all VMs having tag MSZ-SAP] Micro Security Zone Technical Service - A [NSX Security Group of all VMs having tag MSZ-A] vRA Business Group: FINANCE Logical Network Micro Security Zone Technical Service - B [NSX Security Group of all VMs having tag MSZ-B] NSX Distributed Logical Router MSZ-SAP MSZ-SAP MSZ-SAP MSZ-SAP MSZ-SAP MSZ-A MSZ-A MSZ-B MSZ-B MSZ-B Default NSX Security Policy NAME SOURCE DESTINATION SERVICE ACTION Default Any Any Any Block SECURITY TAGS Security tags for technical services: MSZ-<Technical-Service-from-CMDB> For example: MSZ-SAP, MSZ-A, MSZ-B Security tags for applications: APP-<gkpke.APP-SEC-TAG[x]> For example: APP-MSSQL, APP-IIS, APP-EXCHANGE APP-MSSQL APP-MSSQL NSX SECURITY GROUPS We have NSX Security Group for each Technical Service. This security group forms Micro Security Zone for particular Technical Service. For example: MSZ-SAP, MSZ-A, MSZ-B All VMs tagged with the Security Group name will belong to this security group. NSX Security Policy for Micro Security Zones NAME SOURCE DESTINATION SERVICE ACTION Inside MSZ-A MSZ-A MSZ-A Any Allow Inside MSZ-B MSZ-B MSZ-B Any Allow Inside MSZ-SAP MSZ-SAP MSZ-SAP Any Allow Other NSX Security Groups and Policies Other NSX security groups and polices can be created based on applications tags and other metadata available for NSX. Physical or Hyper-V Server belonging in to Micro Security Zone
- 7. End to End Network QoS - Design Decision Point • Requirements • End to end network QoS is required to achieve guarantees for particular network traffics. These traffics are • FCoE Storage • vSphere Management • vSphere vMotion • VM production • VM guest OS agent based backup <== this is the most complex requirement in context of QoS • Constraints • CISCO Nexus 7k • VMware NSX-v • CISCO UCS servers B200 M4 with virtual interface card VIC1340 (2x10Gb ports - each port connected to different fabric interconnect) • Cloud Automation (vRA, vRO)
- 8. End to End Network QoS – Option 1 of 3 UCS Blade Server B200 M4 NIC-A1 - 10Gb NIC port vHBA0 FCoE CoS 3 40% Mark as CoS 3 vNIC0 Mgmt VLAN 100 CoS 1 10% Mark CoS 1 vNIC2 vMotion VLAN 101 CoS 2 10% Mark CoS 2 vNIC4 VM Traffic VLAN 102 CoS 0 20% Mark CoS 0 NIC-B1 - 10Gb NIC port vHBA1 FCoE CoS 3 40% Mark as CoS 3 vNIC1 Mgmt VLAN 100 CoS 1 10% Mark CoS 1 vNIC3 vMotion VLAN 101 CoS 2 10% Mark CoS 2 vNIC5 VM Traffic VLAN 102 CoS 0 20% Mark CoS 0 UCS Fabric Interconnect A (EHM) UCS Fabric Interconnect B (EHM) vFC vEth vEth vEth vFC vEth vEth vEth CISCOUCS CISCO Nexus 7k CISCO Nexus 7k Eth Eth Eth EthFc Fc SAN A SAN B vPC Domain vPCvPC vNIC7 Backup VLAN 103 CoS 4 20% Mark CoS 4 vNIC6 Backup VLAN 103 CoS 4 20% Mark CoS 4 VMwarevSphere-ESXi vmkernel Mgmt (Native VLAN) vmkernel vMotion (Native VLAN) vmkernel VTEP VMware Distributed vSwitch DVS portgroup (Native VLAN) VTEP DVS portgroup (native VLAN) Backup vEth vEth VMwareNSX NSX Logical Switch (VXLAN) logical segment - Business Group VM vNIC Production VM vNIC Backup UCS uplink & N7K downlink QoS Settings CoS 0: 50% (VM Traffic) CoS 1: 10% (Mgmt) CoS 2: 10% (vMotion) CoS 4: 30% (Backup) vmnic0 vmnic2 vmnic4 vmnic6 vmnic1 vmnic3 vmnic5 vmnic7 VMware Standard vSwitch VMware Standard vSwitch VMware Distributed vSwitch Cisco VIC 1340 (4x10Gb port) DVS portgroup Virtual Wire - Business Group 1 CISCO UCS QoS Polices Bandwidth Management & QoS Marking UCS QoS Policy UP (Uplinks): CoS 0: 50% (VM Traffic) CoS 1: 10% (Mgmt) CoS 2: 10% (vMotion) CoS 4: 30% (Backup) UCS QoS Policy 1 (vNIC): CoS 0: 20% (VM Traffic) CoS 1: 10% (Mgmt) CoS 2: 10% (vMotion) CoS 3: 40% (FCoE) CoS 4: 20% (Backup) UCS all vNIC Templates: Host Control: None
- 9. End to End Network QoS – Option 2 of 3 UCS Blade Server B200 M4 10Gb NIC port (NIC-A1) vHBA0 FCoE CoS 3 40% Mark as CoS 3 10Gb NIC port (NIC-B1) vHBA1 FCoE CoS 3 40% Mark as CoS 3 UCS Fabric Interconnect A (EHM) UCS Fabric Interconnect B (EHM) vFC vEth vEth vEth vFC vEth vEth vEth CISCOUCS CISCO Nexus 7k CISCO Nexus 7k Eth Eth Eth EthFc Fc SAN A SAN B vPC Domain vPCvPC VMwarevSphere-ESXi vmkernel Mgmt vmkernel vMotion vmkernel VTEP DVS portgroup VLAN 102, Mark as CoS 0 VTEP DVS portgroup VLAN 103, Mark as COS 4 Backup vEth vEth VMwareNSX NSX Logical Switch (VXLAN) logical segment - Business Group VM vNIC Production VM vNIC Backup UCS uplink & N7K downlink QoS Settings CoS 0: 40% (VM Traffic) CoS 1: 10% (Mgmt) CoS 2: 10% (vMotion) CoS 4: 40% (Backup) vmnic0 vmnic1 VMware Distributed vSwitch (DVS) DVS portgroup VLAN 100, Mark as CoS 1 Mgmt DVS portgroup VLAN 101, Mark as CoS 2 vMotion Cisco VIC 1340 (4x10Gb port) DVS portgroup Virtual Wire - Business Group 1 DVS per PortGroup Marking CoS 0: System: VM Traffic CoS 1: System: Mgmt CoS 2: System: vMotion CoS 4: User-def: Backup vmnic2 vmnic3 CISCO UCS QoS Polices Bandwidth Management & QoS Marking UCS QoS Policy UP (Uplinks): CoS 0: 40% (VM Traffic) CoS 1: 10% (Mgmt) CoS 2: 10% (vMotion) CoS 4: 40% (Backup) UCS QoS Policy 1 (vNIC 0,1): CoS 0: 20% (VM Traffic) CoS 1: 10% (Mgmt) CoS 2: 10% (vMotion) CoS 3: 40% (FCoE) CoS 4: 20% (Backup) UCS all vNIC Templates: Host Control: None vNIC0 trunk CoS0 20% CoS1 10% CoS2 10% CoS4 20% vNIC1 trunk CoS0 20% CoS1 10% CoS2 10% CoS4 20%
- 10. End to End Network QoS – Option 3 of 3 UCS Blade Server B200 M4 10Gb NIC port (NIC-A1) vHBA0 CoS 3 40% FCoE Mark as CoS 3 10Gb NIC port (NIC-B1) vHBA1 CoS 3 40% FCoE Mark as CoS 3 UCS Fabric Interconnect A (EHM) UCS Fabric Interconnect B (EHM) vFC vEth vEth vEth vFC vEth vEth vEth CISCOUCS CISCO Nexus 7k CISCO Nexus 7k Eth Eth Eth EthFc Fc SAN A SAN B vPC Domain vPCvPC VMwarevSphere-ESXi vmkernel Mgmt vmkernel vMotion vmkernel VTEP DVS portgroup VLAN 102 VTEP vEth vEth VMwareNSX NSX Logical Switch (VXLAN) logical segment - Business Group VM vNIC Production & Backup UCS uplink & N7K downlink QoS Settings CoS 0: 40% (VM Traffic) CoS 1: 10% (Mgmt) CoS 2: 10% (vMotion) CoS 4: 40% (Backup) vmnic0 vmnic1 VMware Distributed vSwitch (DVS) DVS portgroup VLAN 100, Mark as CoS 1 Mgmt DVS portgroup VLAN 101, Mark as CoS 2 vMotion Cisco VIC 1340 (4x10Gb port) DVS portgroup Virtual Wire - Business Group 1 if DST IP = Backup Server mark as CoS 4 else CoS 0 DVS per PortGroup Marking CoS 0: System: VM Traffic CoS 1: System: Mgmt CoS 2: System: vMotion CoS 4: User-def: Backup vmnic2 vmnic3 CISCO UCS QoS Polices Bandwidth Management & QoS Marking UCS QoS Policy UP (Uplinks): CoS 0: 40% (VM Traffic) CoS 1: 10% (Mgmt) CoS 2: 10% (vMotion) CoS 4: 40% (Backup) UCS QoS Policy 1 (vNIC 0,1): CoS 0: 20% (VM Traffic) CoS 1: 10% (Mgmt) CoS 2: 10% (vMotion) CoS 3: 40% (FCoE) CoS 4: 20% (Backup) UCS all vNIC Templates: Host Control: None vNIC0 trunk CoS0 20% CoS1 10% CoS2 10% CoS4 20% vNIC1 trunk CoS0 20% CoS1 10% CoS2 10% CoS4 20%
- 11. End to End Network QoS – Final Decision • Decision • Option 3 – QoS (802.1p) marking in VDS and end-2-end bandwidth management in UCS • Justification • Decision is fully compliant with End to end network QoS requirement • VXLAN protocol is designed to keep L2 CoS tags by copying inner Ethernet header into outer Ethernet header => virtual overlay CoS tag is kept even in physical network underlay and it can be leveraged in Cisco UCS bandwidth management (aka DCB ETS - Enhanced Transmission Selection) to guarantee bandwidth for particular CoS traffics. • Single vNIC in VM has positive impact on • NSX Security Policies • Simple In-guest OS routing (default gateway only) without need for additional static routes • vRealize Automation Custom Integrations are simpler (single hostname, simpler integration with IPAM, etc.) • Impact • DVS QoS Policy (conditional 802.1p marking) has to be configured manually for each DVS portgroup used as NSX virtual wire (aka VXLAN) – can be automated by custom integration (SOLUTION IMPROVEMENT) • Detail Test Plan has to be prepared to validate correct QoS behavior (RISK MITIGATION)
- 12. Questions and Answers Blog post with additional details: http://blog.igics.com/2015/12/end-to-end-qos-solution-for-vmware.html Twitter: @david_pasek Blog: http://blog.igics.com