Dell VLT reference architecture v2 0

A Dell Reference Architecture
Virtual Link Trunking in Dell Networking
A Reference Architecture Document
Dell CTO Office and Systems Engineering
September 2013

2 Virtual Link Trunking in Dell Networking
Revisions
Date Description
September,
2013
Initial release

THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND
TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF
ANY KIND.
© 2013 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express
written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.
PRODUCT WARRANTIES APPLICABLE TO THE DELL PRODUCTS DESCRIBED IN THIS DOCUMENT MAY BE FOUND
AT: http://www.dell.com/learn/us/en/19/terms-of-sale-commercial-and-public-sector Performance of network
reference architectures discussed in this document may vary with differing deployment conditions, network loads, and
the like. Third party products may be included in reference architectures for the convenience of the reader. Inclusion
of such third party products does not necessarily constitute Dell’s recommendation of those products. Please consult
your Dell representative for additional information.
Trademarks used in this text:
Dell™, the Dell logo, Dell Boomi™, Dell Precision™ ,OptiPlex™, Latitude™, PowerEdge™, PowerVault™,
PowerConnect™, OpenManage™, EqualLogic™, Compellent™, KACE™, FlexAddress™, Force10™ and Vostro™ are
trademarks of Dell Inc. Other Dell trademarks may be used in this document. Cisco Nexus®, Cisco MDS
®
, Cisco NX-
0S
®
, and other Cisco Catalyst
®
are registered trademarks of Cisco System Inc. EMC VNX
®
, and EMC Unisphere
®
are
registered trademarks of EMC Corporation. Intel
®
, Pentium
®
, Xeon
®
, Core
®
and Celeron
®
are registered trademarks of
Intel Corporation in the U.S. and other countries. AMD
®
is a registered trademark and AMD Opteron™, AMD
Phenom™ and AMD Sempron™ are trademarks of Advanced Micro Devices, Inc. Microsoft
®
, Windows
®
, Windows
Server
®
, Internet Explorer
®
, MS-DOS
®
, Windows Vista
®
and Active Directory
®
are either trademarks or registered
trademarks of Microsoft Corporation in the United States and/or other countries. Red Hat
®
and Red Hat
®
Enterprise
Linux
®
are registered trademarks of Red Hat, Inc. in the United States and/or other countries. Novell
®
and SUSE
®
are
registered trademarks of Novell Inc. in the United States and other countries. Oracle
®
is a registered trademark of
Oracle Corporation and/or its affiliates. Citrix
®
, Xen
®
, XenServer
®
and XenMotion
®
are either registered trademarks or
trademarks of Citrix Systems, Inc. in the United States and/or other countries. VMware
®
, Virtual SMP
®
, vMotion
®
,
vCenter
®
and vSphere
®
are registered trademarks or trademarks of VMware, Inc. in the United States or other
countries. IBM
®
is a registered trademark of International Business Machines Corporation. Broadcom
®
and
NetXtreme
®
are registered trademarks of Broadcom Corporation. Qlogic is a registered trademark of QLogic
Corporation. Other trademarks and trade names may be used in this document to refer to either the entities claiming
the marks and/or names or their products and are the property of their respective owners. Dell disclaims proprietary
interest in the marks and names of others.

Table of contents
Revisions............................................................................................................................................................................................. 2
1 Overview......................................................................................................................................................................................6
2 Introduction to VLT ................................................................................................................................................................... 7
2.1 VLT Implementation....................................................................................................................................................... 7
2.2 Features in the FTOS 9.2(0.0) Release.........................................................................................................................8
2.3 VLT Terminology.............................................................................................................................................................8
2.4 Advantages of VLT ..........................................................................................................................................................8
2.5 Best Practices for Implementing VLT ..........................................................................................................................9
2.6 Typical VLT Topology...................................................................................................................................................10
3 Packet Flow in VLT ...................................................................................................................................................................11
4 Link Failover Scenarios............................................................................................................................................................ 13
4.1 Split-Brain in VLT........................................................................................................................................................... 15
4.2 VLTi Traffic...................................................................................................................................................................... 16
5 Peer-routing ............................................................................................................................................................................. 17
5.1.1 Secondary IP Address Support for VLANs ................................................................................................................ 18
5.1.2 Peer-routing-timeout .................................................................................................................................................. 18
5.1.3 VLAN Scalability............................................................................................................................................................. 18
5.1.4 Routing Protocols......................................................................................................................................................... 18
5.2 Graceful Restart for LACP............................................................................................................................................ 19
5.3 Layer-3 Routing in VLT ................................................................................................................................................20
5.4 OSPF Configured VLT Domain ................................................................................................................................... 21
5.5 IS-IS Configured VLT Domain.....................................................................................................................................22
5.6 BGP Configured VLT Domain.....................................................................................................................................23
6 Layer-2 Traffic in VLT Domain ..............................................................................................................................................24
6.1 Layer-3 Traffic in VLT Domain.....................................................................................................................................26
6.2 IPv6 Addressing in VLT.................................................................................................................................................29
6.3 mVLT in MXL Switches.................................................................................................................................................30
6.4 Auto VLT in IOM ............................................................................................................................................................ 31
6.5 LAG Features..................................................................................................................................................................35
7 VLT Peers Connected with 10G VLTi ...................................................................................................................................36
8 VLT Peers Connected with 40G VLTi...................................................................................................................................37

9 VLT Peers Connected to Standalone Switches ....................................................................................................................38
10 VLT Peers Connected to Stacked Switches ........................................................................................................................39
11 Single VLAN across Two ToRs.............................................................................................................................................. 40
12 Multi VLANs across Multiple ToRs ........................................................................................................................................ 41
13 Inter-VLAN Routing between ToRs......................................................................................................................................42
14 VLT in Distributed Core ..........................................................................................................................................................43
14.1 VLT in Leaf layer ............................................................................................................................................................44
14.2 VLT in Spine Layer.........................................................................................................................................................44
15 STP Flavors across ToR...........................................................................................................................................................45
16 RSTP in VLT Peers and MSTP in the ToR .............................................................................................................................47
17 RSTP in VLT peer and PVST+ within the ToR's...................................................................................................................48
18 BGP on the Upstream Links from VLT Peers ...................................................................................................................... 51
19 Scenarios with non-VLT Ports...............................................................................................................................................52
20 VRRP in VLT ..............................................................................................................................................................................53
20.1 VRRP Configuration......................................................................................................................................................53
21 Multicast in VLT........................................................................................................................................................................54
21.1 Multicast in Hierarchical VLT Topology ....................................................................................................................55
21.2 Multicast configuration and Outputs from Leaf and Spine VLT Nodes ..............................................................59
22 S5000 Switches in VLT ...........................................................................................................................................................67
23 Data Centre interconnect through VLTi with DWDM optics...........................................................................................68
23.1 DWDM Interconnect multiple VLT domains aggregated with hierarchical VLT................................................69
23.2 VLT Scalability................................................................................................................................................................70
23.3 Use cases........................................................................................................................................................................70
23.4 Example Configuration ................................................................................................................................................ 71
24 VLT Troubleshooting.............................................................................................................................................................. 80
24.1 Debug outputs...............................................................................................................................................................83
24.2 Syslog messages ...........................................................................................................................................................84
24.3 SNMP Traps....................................................................................................................................................................85
25 FTOS Upgrade in VLT topology ............................................................................................................................................86
26 Conclusion................................................................................................................................................................................87

1 Overview
The rapid growth of data centers, enterprise networks, campus networks, and so on,
necessitates a paradigm shift in the rebuilding of the underlying network to cater to
the ever-expanding business needs. As networks scale, extending and managing the
layer-2 fabric gets more complex. Extending multiple links for adding bandwidth may
not be an optimum solution with Spanning Tree Protocol (STP) running across the
network. Any link failure with such STP deployed networks is fraught with network
convergence resulting in sub optimal performance.
Virtual Link Trunking (VLT) overcomes the issue by providing access to the underlying
Top of Rack (ToR) switches in cascade. VLT offers high resiliency and availability,
while effectively utilizing the multipath feature. Diligent design in VLT also eliminates
the over-subscription of traffic from the access to the aggregation.
Migration from the existing STP-based network to a VLT-based one involves
meticulous planning. The Peer-routing’ feature introduced in the Dell™ Networking
operating system FTOS 9.2(0.0) release is highly recommended for active-active load
sharing across multiple VLAN’s within VLT peers, ensuring high-availability. FTOS
9.2(0.0) adds supporting the routed (Peer-routing) VLT with unique features
eliminating VRRP. With IPv6 and IPv4 dual stack, VRRP may be deployed for effective
resiliency.
If you are looking for a solution that is of service to your rapidly growing network
requirements, you can deploy VLT or multi-domain VLT (mVLT) in your networks.
This document explains the dual node VLT deployment strategies with its associated
network reference architecture. Various VLT deployment topologies are also
explained with emphasis on best practices and recommendations for some of the
network scenarios. This document also covers the configuration and troubleshooting
of VLT using relevant show commands and different outputs.

2 Introduction to VLT
VLT, in general terms, configures two physical switches as a single logical switch.
With physical links as a port-channel, connecting two individual switches configured
with VLT logically groups the switches as a single entity only for the access switches
that connect to the VLT domain. Both the VLT peers have their own configuration,
control, and data planes. VLT allows creating port-channels across the two switches,
eliminating the need for spanning tree blocking states across its connectivity and
effectively utilizing its entire VLT links connected to the access/ToR switches. The
access device is a switch, switch stack, dual NIC server or any other device supporting
Link Aggregation Protocol (LACP) port-channels. High-availability and redundancy in
the network is achieved by the duality in physical connectivity.
2.1 VLT Implementation
Periodic hello messages are sent through the VLT Interconnect (VLTi). The VLT
control messages are sent in TLV format through the VLTi links for synchronizing the
L2/L3 control planes across the two VLT peers. The MAC, ARP tables, IGMP states are
synchronized between the VLT peers ensuring traffic flow across the links and
seamless failover in case of VLT link or node failure. VLT ensures that the local traffic
on a VLT switch takes the shortest path to the destination using the VLT links and not
the VLTi links. However, VLTi carries the traffic during the link failure states. See
Virtual Link Trunking Topology.
A backup link is established between the VLT peers through the management
interface to exchange the periodic backup heartbeat messages through the out of
band network. The backup link is of any physical interface. However, as a best
practice, the management interface is used to exchange the heartbeat messages and
its significance (Section 3.1) is noted when all the members of the VLTi port-channel
fail. With the priority election in VLT domain, the primary VLT peer takes control of
handling LACP and RSTP control states and exchanging messages with the secondary
VLT peer.
For all the VLANs configured on the VLT port-channels, the VLANs are dynamically
mapped on the VLTi links. When Virtual Router Redundancy Protocol (VRRP) is
implemented within the VLT domain, the VLT peers offer active-active load sharing.
Traffic is locally switched to its destination without being redirected to the VRRP
master. Similarly, peer routing enables the active-active load sharing for the L3 VLAN
traffic without enabling VRRP. With peer-routing the VLANs in the VLT fabric can
scale beyond the limitation of VRRP groups.
Any broadcast traffic flooded across the VLTi links prevents duplicate copies at each
of the peer in the VLT domain. This is ensured by installing a port block. The packets
coming in through the VLTi are blocked from egressing on the VLT ports. This block is
installed only if both the peers have active ports on that VLT. If all the ports on one

peer for that VLT are down, then this state is indicated to the other peer. The second
peer removes the block so that the remote node continues to receive the data.
2.2 Features in the FTOS 9.2(0.0) Release
 ‘VLT Min-loss’ feature reduces the traffic loss and improve convergence time
during failovers.
 VLT supports IPv6 with VRRPv3. Dual stack of IPv4 and IPv6 are supported
benefitting the customers migrating to IPv6.
 Multicast features with PIM-SM are supported.
 MXL Switches could form VLT within the chassis improving the resiliency of
servers deployed within the M1000e chassis.
 IOA Switch forms auto VLT with a single CLI.
NOTE: Latest VLT features are supported on the Dell Force10 S4810, S4820T, MXL,
IOM/IOA and Z9000 switches in the FTOS 9.2(0.0) release.
2.3 VLT Terminology
Term Description
Virtual link trunk (VLT) The combined port channel between an attached device (ToR
switch) and the VLT peer switches.
VLT backup link Monitors the vitality of a VLT peer switch. The backup link sends
configurable, periodic keep alive messages between the VLT peer
switches.
VLT interconnect (VLTi) or
Inter Chassis Link (ICL)
Used to synchronize states between the VLT peer switches. Both
ends must be on 10G or 40G interface.
VLT domain Includes both VLT peer devices, the VLT interconnect, and all the
port-channels in the VLT connected to the attached devices. It is
also associated to the configuration mode that is used to assign VLT
global parameters.
Non-VLT (Orphan) Ports Any port not connected to the VLT port-channel in the VLT node
VLT nodes A pair of switches that is connected to the port channel known as
the VLT inter-connect (VLTi).
2.4 Advantages of VLT
 Loop free connectivity in layer-2 domain
 Faster network convergence
 High-availability and redundancy
 Effective utilization of all the links

 Link level resiliency
 Active-active load sharing with VRRP.
 Active-active load sharing with peer-routing for layer-3 VLAN
 Graceful failover of LACP during reload
 Agility in VM Migration under VLT domain
 Unified access for virtualization, web applications and cloud computing
 High performance for big data networks
 Easier design and manageability of fabric with AFM
2.5 Best Practices for Implementing VLT
 FTOS version must be the same for both the VLT peers.
 Backup link must pass through the out of band management network.
 Enable Rapid Spanning Tree Protocol (RSTP) in both the VLT peers and adjust the
timers for minimum values.
 Keep the primary VLT node as root-bridge and the secondary node as backup root-
bridge.
 Use identical system MAC address on both the VLT peers to avoid minimum traffic
loss.
 Configure unique unit-id on the VLT peers.
 Adjust the delay-restore timer from the default 90 seconds based on the servers.
 Configure LLDP to assist during the troubleshooting of link failures.
 Use VLTi links configured with static port-channel of more than one member.
 Configure VLT links with LACP port-channels.
 In a hierarchical mVLT domain with OSPF, configure the Spine/Core primary node as
DR.
 Enter description fields of the VLT and VLTi links configuration for easier
identification.
 Configure RSTP, “LACP ungroup” CLI on VLT nodes, and ToR for an easier BMP boot
process.
 Configure VLTi can as mrouter port, when PIM is deployed on VLT VLAN.
 In a PIM-SM environment, deploy a non-VLT core node as Rendezvous Point (RP).

2.6 Typical VLT Topology
Figure 1 Virtual Link Trunking Topology
The VLT domain has VLTi links connecting between the VLT peers and the VLT port-
channels connecting to single access switch, a switch stack, and a server supporting
LACP on its dual NIC. Alternatively, it can connect to another VLT domain as shown
in the figure above. The backup-link gets connected through the out of band
management network. Some hosts can directly connect through the non-VLT ports.
VLT Domain
Backup Link
VLT Domain
Server
Port Channel
Servers
Uplinks
Non-VLT Port
VLTi Links

3 Packet Flow in VLT
Due to the inherent layer-2 multi-path hashing in the VLT port-channel links, the
traffic might pass through any of the VLT peers before reaching its destination. The
various colors of traffic stream flow are shown in the figures below.
NOTE: During converged stable state, traffic does not flow across VLTi links.
Figure 2 North-South Traffic Flow in VLT Topology
Layer-3
Layer-2
VLT Domain
Unicast Traffic Flows (North-South)
Various modes of traffic flow in VLT
Topology

Figure 3 East-West Traffic Flow in VLT Topology
During the converged stable state, with MAC and ARP in sync for both the VLT peers,
the traffic flow (north-south) from the core layer (layer-3) to the VLT peer with ECMP
path reaches its destination through the shortest path. Similarly the traffic flow (east-
west) between the servers in the ToR switches, within the layer-2 domain always
takes the optimal path.
Layer-3
Layer-2
VLT Domain
Various modes of traffic flow in VLT Topology

4 Link Failover Scenarios
Any failure in the upstream layer-3 links forces the traffic towards the alternate ECMP
path to the VLT domain and local switch to reach its destination. The link failure in
the VLT port-channel might result in traffic passing through the VLTi as shown in the
figure below, since the MAC address learnt on the failed VLT is now mapped to the
VLTi port.
Figure 4 Traffic Flow During Link Failure
For the north bound traffic with default gateway configured for a specific layer-3
VLAN in a VLT peer, the hashed traffic reaches the other VLT node without diverting
to the appropriate gateway node.
Layer-3
Layer-2
VLT Domain

Figure 5 Figure 1: Traffic Flow During Link Failure
NOTE: The link failover feature with local switching mechanism is adopted in the latest
releases of FTOS 9.2(0.0).
Layer-3
Layer-2
VLT Domain

4.1 Split-Brain in VLT
Figure 6 : Traffic Flow During VLTi Failure
The backup heartbeat messages are exchanged between the VLT peers through the
backup links of the out of band management network. When the VLTi link (port-
channel) fails, the MAC/ARP entries cannot be synchronized between the VLT peers
through the failed VLTI link. Hence, the secondary VLT peer shuts the VLT port-
channel, forcing the traffic from the ToR switches to flow only through the primary
VLT peer to avoid a traffic black-hole. Similarly, the return traffic on layer-3 also
reaches the primary VLT node. This is called split-brain scenario. When the VLTi link
is restored, the secondary VLT peer waits the pre-configured time period (delay-
restore) for the MAC/ARP tables to synchronize before passing the traffic. In case of
both the VLTi and backup link failure, both the VLT nodes take primary role and
continue to pass the traffic. This is only possible if the system MAC address is
configured on both the VLT peers. However, there is no MAC/ARP synchronization.
Layer-3
Layer-2
VLT Domain
VLT Secondary
VLT Ports Shut
Due to Split brain
VLT Primary

4.2 VLTi Traffic
The VLTi links normally carry the following traffic:
 VLT hello (domain-id, system-mac, unit-id, priority, version), election and other
related messages in TLV format
 MAC, ARP table, IGMP state, iSCSI synchronization and such types of messages
 Traffic towards the non-VLT ports from one VLT peer to another VLT peer
 Multicast and broadcast traffic in the domain
 Diverted traffic due to VLT link failure
 Layer-3 protocol traffic forming adjacency over VLTi
 VRRP hello messages.

5 Peer-routing
With FTOS 9.2(0.0) release, VLT supports “peer-routing” feature. This feature enables the
VLT node to act as a proxy gateway for the other VLT peer.
As shown in the figure below, the packets are sent to either of the VLT port-channel
members. In the earlier release, due to the hashing algorithm in the port-channel, if the
packet is sent to Peer-2, which is not the destined gateway for the hosts under the ToR
Switch, the packet gets switched to the destined peer.
Figure 2: Traffic Flow without and with Peer-Routing
With Peer-routing feature in the FTOS 9.2(0.0) release, the VLT node acts as a proxy
gateway only for its connected VLT peer. The packet received by the non-destined VLT
node acts as a gateway and passes it upstream and without tunneling the packet to the
destined peer. Peer-routing has the following advantages:
1. Avoiding sub optimal routing.
2. Reduced latency by avoiding another hop in the traffic path.
3. Reduced number of VLTi port-channel members based on the specific design.
With peer-routing, VRRP need not be configured for the participating VLANs. Since both
VLT nodes act as a gateway for its peer, irrespective of the gateway IP address, the traffic
tends to flow upstream without any latency. There is no limitation in the scaling of
VLAN’s.
NOTE: With VRRP, there is a limitation of 255 VLANs.
VLAN 10
10.10.10.3/24
GW- 10.10.10.2
VLAN 10
10.10.10.1/24
WAN
VLAN 10
10.10.10.2/24
Packets would be
switched to destined
peer
Peer-1 Peer-2
Without Peer-Routing
VLAN 10
10.10.10.3/24
GW- 10.10.10.2
VLAN 10
10.10.10.1/24
WAN
VLAN 10
10.10.10.2/24
Although packets reach to
the non-destined vlt peer,
traffic is not switched to the
destined VLT peer.
Peer-1 passes the traffic
upstream on behalf of Peer-2
with Peer-routing enabled.
Peer-1 Peer-2
With Peer-Routing

5.1.1 Secondary IP Address Support for VLANs
The VLANs are configured with different subnets as secondary IP addresses, ensuring the
IP connectivity across different subnets for the underlying server nodes. However, these
VLANs must be symmetrically configured on both the VLT peers in the same mode.
Asymmetric configuration (layer-2 VLAN in node-1 and layer-3 VLAN in node-2) of same
VLANs is not permitted in the VLT peers.
Both the VLT nodes have to be configured for ‘peer-routing’ under VLT configuration.
5.1.2 Peer-routing-timeout
By default, the VLT nodes continuously act as proxy for its peer until the ARP times out.
With the ‘peer-routing-timeout ‘ enabled, if the VLT peer fails (or reboots during
maintenance), the VLT node acts as gateway only for the configured duration (maximum
of 65535 seconds). However, if the ‘peer-routing-timeout’’ is not configured, the active
node continues to pass the traffic on behalf of its failed peer, until the connected hosts
ARP times out.
As a best practice, do not disable ‘peer-routing’ when the VLT nodes are active and
passing traffic through multiple VLAN’s. Also ‘peer-routing’ must be enabled before
configuring VLANs.
5.1.3 VLAN Scalability
The ‘peer-routing’ feature also covers the VLAN wildcarding internal architecture
approach, which helps in scaling the number of L3 VLANs supported with the current
FTOS. In the current release, with ‘peer-routing’ enabled, each VLAN-based local entry in
the table is removed and the VLAN field is wild carded .A separate VLAN profile table is
used to set the status of routing and switching capability of each VLAN. Activating VLAN
wildcarding in FTOS scales more number of L3 VLANs with VLT.
5.1.4 Routing Protocols
Routing protocols (RIP, OSPF, IS-IS, BGP) are supported with routed VLT. As all the VLANs
in the VLT are part of broadcast network, point-to-point network configurations are not
supported in OSPF and IS-IS. These routing protocols by default enables only broadcast
network, hence, no specific configuration is required on the routed VLAN interfaces. Also
as a best practice, the Core/Spine VLT domain must be configured as DR/DIS for effective
convergence.

5.2 Graceful Restart for LACP
Figure 3: Traffic Flow with and without Graceful LACP
With FTOS 9.2(0.0) release, ‘graceful restart of LACP’ feature is introduced.
In earlier FTOS releases, if the VLT node gets reloaded, there is a definite time delay in
switching the traffic towards the other active peer. The interface status changes and the
detection of link status at hardware level with the subsequent trigger to switchover leads
to transient traffic loss. However, with FTOS 9.2(0.0) release, whenever the VLT nodes are
reloaded, the graceful restart LACP sends a special PDU on all the ports of the VLT port-
channel. On receiving the graceful LACP PDU, the ToR switch sends an acknowledgement
to the VLT node and detaches the ports from the VLT port-channel. Simultaneously, the
port-channel in the ToR seamlessly switches the traffic to the other member of the VLT
port-channel, forcing the deterministic traffic towards the other VLT peer. This process
ensures there is no traffic loss or blackholing due to the VLT node being reloaded.
When the node comes up after reload, the VLT port-channels take a definite ‘restore-
delay’ time to pass the traffic. This ensures all the MAC/ARP synchronization occurs
between the peers. After the delay-restore time, the VLT port-channel from the reloaded
node comes up and continues to pass the traffic. With the above features, seamless traffic
flow is ensured for the end-stations. Any firmware upgrade is done without major
downtime which is essential for high availability and high SLA networks.
VLT node
on Reload
WAN
When the port goes ‘down’
it takes minimal time to
switchover the traffic to
other available port
Peer-1 Peer-2
Traffic gets blackholed until
the source port is ‘down’.
Traffic is lost during this
state transition.
Without Graceful LACP
VLT node
on Reload
WAN
On receipt of the graceful
LACP message, re-hashing
happens to switchover the
traffic to other port without
loss
Peer-1 Peer-2
Graceful shutdown of
LACP Ports, switching the
traffic to other port
without any traffic loss
With Graceful LACP
Graceful LACP PDU

VLT-10-PEER-1#reload
Proceed with reload [confirm yes/no]: yes
All VLT LAG's gracefully shut down...!!!
LACP-5-PORT-UNGROUPED: PortChannel-033-Ungrouped: Interface Te 0/12
exited port-channel 33
5.3 Layer-3 Routing in VLT
Spanning the same VLANs across the racks, pods, and clusters is a major challenge in a
data center. Deploying a flat layer-2 network has its own limitations with sub optimal
utilization of the links, fraught with perils of loop, restricted scalability etc. With FTOS
9.2(0.0) release, the same VLAN extension across racks is possible in a unique way of
configuring layer-3 VLANs across the VLT nodes and the ToR switches. Spanning the
VLANs in mVLT architecture can interconnect and aggregate multiple racks with same
VLAN. With the interspersing of layer-2 and layer-3 VLANs in the mVLT, the ARP table can
considerably extend the scalability of the domain.
OSPF, IS-IS, BGP is configured as the routing protocol for the extended VLANs. This
ensures routes learnt by specific VLT node are advertised to all other participating nodes
in the same VLAN. Since the IGP is configured on the VLAN interfaces, only the default
broadcast mode is supported. Point-to-point mode of OSPF/IS-IS is not supported. BGP
form adjacencies with the VLAN IP address between the VLT nodes and advertise the
routes across different peers.
For a highly virtualized data center, the host gateway is interspersed across multiple VLT
domains, distributing the ARP tables for optimal scalability requirements.
In case of static routes, the same static routes with the next-hop are configured on both
the VLT peers for optimal routing. However, if the next-hop reachability for the uplink is
different, the appropriate NH has to be configured for each VLT peer.
Layer-3 VLT supports both IPv4 and IPv6 dual stack. However, ‘peer-routing’ is currently
supported only for IPv4. In case of IPv4 and IPv6 dual stack requirement, VRRP is
configured for resiliency in operation for the end-stations.
FTOS 9.2(0.0) release scales more number of VLANs (layer-3 and layer-2). With L3-VLT,
ARP/MAC addresses learnt by one of the VLT peer is synchronized in both the VLT nodes
for the corresponding VLANs.
The following sections highlight the IGP/BGP configuration in VLT domains:
 OSPF Configured VLT Domain
 IS-IS configured VLT domain
 BGP configured VLT domain

5.4 OSPF Configured VLT Domain
Figure 4: OSPF in VLT
OSPF is configured on VLAN interfaces as broadcast network (default OSPF network). No
other OSPF network (point-to-point) type is supported. All other VLT peers form
adjacency with the core VLT peers. As a best practice, configure the primary node of the
core VLT domain as DR and the secondary node as BDR for optimal routing adjacencies.
By configuring all the VLAN interfaces in the primary VLT peer of the aggregation layer,
the priority is incremented to act as OSPF DR. The number of OSPF adjacencies from each
peer depends upon the VLAN subnets.
With OSPF converged, all the nodes have the routes and next-hop details to reach other
nodes. Although, the VLAN interfaces are configured with OSPF, as a same broadcast
domain, and underlying layer-2 dependency is achieved. For instance, VM migration
within the rack and across VLT domains could be orchestrated.
The OSPF configurations in VLT domain are as follows:
router ospf 1
router-id 10.10.10.1
network 192.168.0.3/16 area 0
network 192.169.0.3/16 area 0
auto-cost reference-bandwidth
40000
interface Vlan 10
ip address 192.168.0.3/16
tagged Port-channel 33
ip ospf priority 100
no shutdown
!
interface Vlan 20
ip address 192.169.0.3/16
no shutdown
VLAN 10
10.10.10.4/24
VLAN 10
10.10.10.5/24
VLAN 10
10.10.10.6/24
VLAN 10
10.10.10.3/24
VLAN 10
10.10.10.2/24
VLAN 10
10.10.10.1/24
OSPF
OSPF Area 0
Broadcast NetworkOSPF is enabled on all
the VLAN interfaces.
Core VLT Peers act as
DR/BDR

5.5 IS-IS Configured VLT Domain
Figure 5: IS-IS in VLT
With multi-topology IS-IS, both IPv4 and IPv6 is configured with single IS-IS process. In
addition, the primary node of the core VLT domain is configured as DIS, with a priority of
100 across all VLAN interfaces. Similarly, a priority of 90 is configured for all the interfaces
in the secondary node. However, there is no concept of Backup DIS in IS-IS. This DIS
positioning reduces the LSP flooding in the LAN.
As a best practice set the ‘metric-type’ as ‘wide’ under IS-IS configuration for all the VLT
nodes. Also, configure the level as L1 or L2 or L1L2 (default).
NOTE: Configuring both level-1 and level-2 in the same VLT domain have reachability
issues, unless route-leaking is configured.
The IS-IS configuration in VLT domain is as follows:
interface Vlan 10
ip address 100.1.1.1/24
ip router isis 1
isis priority 100 level-1
isis priority 100 level-2
no shutdown
router isis 1
description ISIS_VLT_Peer1
log-adjacency-changes
metric-style wide level-1
metric-style wide level-2
net 49.0001.1111.2222.3333.4444.00
For the leaf VLT domains, the IS-IS priority need not be configured. All other leaf VLT
domains form the IS-IS Level1/2 adjacencies with the Spine/Core VLT domain.
VLAN 10
10.10.10.4/24
VLAN 10
10.10.10.5/24
VLAN 10
10.10.10.6/24
VLAN 10
10.10.10.3/24
VLAN 10
10.10.10.2/24
VLAN 10
10.10.10.1/24
IS-IS
IS-IS Level1Level2
Broadcast NetworkIS-IS is enabled on all
the VLAN interfaces.
Core VLT Primary node
act as DIS

5.6 BGP Configured VLT Domain
Figure 6: BGP in VLT
IBGP adjacencies are formed between the core VLT domain nodes and all the other VLT
nodes. As a best practice, configure both the core VLT peers as route-reflector (RR). With
core VLT domain acting as RR, BGP adjacencies are formed only between the RR and the
clients of other VLT domains. Adjust the timers for faster convergence with keep-alive
time and hold time at a minimum.
BGP Configuration for Spine/Core VLT domain acting as Route-reflector(RR)
router bgp 65000
network 100.100.100.0/24
neighbor 192.169.0.1 remote-as 65000
neighbor 192.169.0.1 route-reflector-client
neighbor 192.169.0.1 no shutdown
neighbor 192.169.0.2 remote-as 65000
neighbor 192.169.0.2 route-reflector-client
neighbor 192.169.0.2 no shutdown
neighbor 100:100:100:100::2 remote-as 65000
neighbor 100:100:100:100::2 route-reflector-client
neighbor 100:100:100:100::2 no shutdown
neighbor 100:100:100:100::3 remote-as 65500
neighbor 100:100:100:100::3 no shutdown
!
address-family ipv6 unicast
neighbor 100:100:100:100::2 activate
neighbor 100:100:100:100::2 route-reflector-client
neighbor 100:100:100:100::3 activate
exit-address-family

6 Layer-2 Traffic in VLT Domain
Figure 7: Layer-2 Traffic in VLT Domain
Most of the next generation data centers envisage a continuous growth to meet their
virtualization business needs while scaling the layer-2 domain in progressive phases. In
the figure above, VLAN 10, 30 and 50 are configured as layer-2 spanning in all the VLT
domains. Any host in either of the VLT node is synchronized with its VLT peer. MAC
addresses of all the VLANs are synchronized between the VLT peers.
VLT-10-PEER-1#show mac-address-table count
MAC Entries for all vlans :
Dynamic Address Count : 1007
Static Address (User-defined) Count : 1
Sticky Address Count : 0
Total Synced Mac from Peer(N): 503
Total MAC Addresses in Use: 1008
VLT-10-PEER-1#show vlt counter mac
Total MAC VLT counters
----------------------
L2 Total MAC-Address Count: 1007
VLT-10-PEER-1#show mac-address-table
Codes: *N - VLT Peer Synced MAC
VlanId Mac Address Type Interface State
10 00:00:4c:54:8b:f6 Dynamic Po 11 Active
10 00:01:e8:95:ec:97 Dynamic Po 33 Active
10 00:01:e8:b3:ba:47 Dynamic Po 33 Active
30 a0:00:a1:00:00:01 Dynamic Po 11 Active
30 a0:00:a1:00:00:04 Dynamic (N) Po 11 Active

30 a0:00:a1:00:00:0a Dynamic (N) Po 11 Active
30 a0:00:a1:00:00:0b Dynamic (N) Po 11 Active
30 a0:00:a1:00:00:0c Dynamic Po 11 Active
VLT-10-PEER-2#show vlt statistics mac
VLT MAC Statistics
--------------------
L2 Info Pkts sent:0, L2 Mac-sync Pkts Sent:7
L2 Info Pkts Rcvd:0, L2 Mac-sync Pkts Rcvd:9
L2 Reg Request sent:0
L2 Reg Request rcvd:0
L2 Reg Response sent:0
L2 Reg Response rcvd:0
Any MAC address learnt from the connected host on the VLT node is instantaneously
synchronized to its VLT peer. The VLT peer synchronized MAC addresses are indicated
with a (N) flag to identify the MAC addresses learnt from its VLT peer. This MAC address
synchronization handles the traffic flow even if it is hashed and forwarded through the
other member of the port-channel.
Network orchestration, which assists the seamless migration of VMs, is achieved with the
hierarchical multi VLT (mVLT) architecture that scales to the user needs, while effectively
utilizing all the multipath links.

6.1 Layer-3 Traffic in VLT Domain
The following figure illustrates the interspersing of the layer-3 VLANs and effectively
scaling the ARP requirements within the designed VLT network fabric.
Figure 8: Layer-3 Traffic in VLT Domain
In the figure above, three VLT domains VLT-1, VLT-2, and VLT-3 are configured with three
VLANs (VLAN10, 20, and 30). With routed-VLT, a VLAN is configured as layer-3 in a VLT
domain and as a layer-2 VLAN in all other VLT domains.
In this example, VLAN-10 is configured as layer-3 in VLT-1 and as layer-2 in other VLT
domains. Similarly, VLAN-30 configured as layer-3 in VLT-2 domain and as layer-2 in
other VLT domains. The benefit of interspersing layer-3 IP addressing for these VLAN’s
distributes the ARP tables for the respective VLANs in the relevant VLT domains. Assuming
48 x 10G ports from each VLT node in a VLT domain connecting to 48 physical servers of
dual NIC with approximate 50 VM’s for each of the physical server necessitates 2400 ARP
entries. By spreading the layer-3 VLANs across VLT domains, the ARP table requirement in
each of the VLT domain is optimally utilized and scale beyond the limitation of a single
switch. ARP entries for VLAN 10 are confined to VLT-1. Similarly, ARP entries for VLAN-30
are configured in VLT-2 domain only.
At the core/aggregation layer VLT domain, common Layer-3 VLANs are configured for
inter VLAN routing within the VLT domain. ARP scaling is well achieved by this design,
while extending the VLAN connectivity across all the nodes.
The spine/core VLT domain forms adjacencies with the border routers advertising the
end-station routes. The north bound traffic flows through the VLT node uplinks with the
route table information.

The following table illustrates the ARP synchronization between the VLT nodes.
VLT-10-PEER-1#show vlt detail
Local LAG Id Peer LAG Id Local Status Peer Status Active
VLANs
------------ ----------- ------------ ----------- ------
-------
11 11 UP UP 10,
30, 50
33 33 UP UP 10,
30, 50
VLT-10-PEER-1#show arp summary
Total Entries Static Entries Dynamic Entries CPU
------------------------------------------------------------
---------------------
2007 0 2007 CP
VLT-10-PEER-1#show vlt counter arp interface port-channel ?
<1-128> Port channel identifier
VLT-10-PEER-1#show vlt counter arp interface port-channel 11
VLT Port-ID: 11 ARP Counter
-----------------------
Total Arp Entries Learnt : 1000
Total Arp Entries Synced : 1000
Total Non-VLT Arp entries Learnt: 0
Total Non-VLT Arp Entries Synced 0
VLT-10-PEER-1#show vlt statistics arp
VLT ARP Statistics
--------------------
ARP Tunnel Pkts sent:6
ARP Tunnel Pkts Rcvd:0
ARP Tunnel Pkts sent Non Vlt:0
ARP Tunnel Pkts Rcvd Non Vlt:0
ARP-sync Pkts Sent:83478
ARP-sync Pkts Rcvd:12442
ARP Reg Request sent:4
ARP Reg Request rcvd:3
VLT-10-PEER-2#show vlt brief
VLT Domain Brief
------------------
Domain ID: 10
Role: Secondary
Role Priority: 1000
ICL Link Status: Up
HeartBeat Status: Up
VLT Peer Status: Up
Local Unit Id: 1
Version: 6(1)
Local System MAC address: 00:01:e8:8b:24:62
Remote System MAC address: 00:01:e8:8b:24:2c
Configured System MAC address: a0:10:10:aa:aa:aa
Remote system version: 6(1)
Delay-Restore timer: 10 seconds
Peer-Routing : Enabled

Peer-Routing-timeout timer: 0 seconds
Multicast peer-routing timeout: 150 seconds
VLT-10-PEER-2#show vlt statistics arp
VLT ARP Statistics
--------------------
ARP Tunnel Pkts sent Non Vlt:0
ARP Tunnel Pkts Rcvd Non Vlt:0
VLT-10-PEER-2#show vlt counter
Total VLT Counters
-------------------
L2 Total MAC-Address Count: 2008
IGMP MRouter Vlans count : 1
IGMP Mcast Groups count : 0
Total VLT Ndp Entries Learnt : 0
Total VLT Ndp Entries Synced : 0
Total Non-VLT Ndp Entries Learnt : 0
Total Non-VLT Ndp Entries Synced : 0
VLT-10-PEER-2#show vlt counter arp ?
interface Interface statistics
| Pipe through a command
<cr>
VLT-10-PEER-2#show vlt counter arp
Total ARP VLT counters
----------------------
VLT-10-PEER-2#show vlt counter arp interface port-channel 11
VLT Port-ID: 11 ARP Counter
-----------------------
VLT-10-PEER-2#

6.2 IPv6 Addressing in VLT
IPv6 addressing is supported in VLT domains in the FTOS 9.2(0.0) release. The VLT
domains are configured with IPv4 or IPv6 address or with a dual stack of IPv4 and IPv6.
However, “peer-routing” is currently supported for IPv4 only. In case of dual stack
requirement, it is recommended to configure VRRP for resiliency between the VLT peers
for both the IPv4 and IPv6 hosts. With IPv6 in VLT, NDP messages are synchronized
between the VLT peers. The “show vlt statistics” command indicates the NDP message
statistics.
VLT-100-PEER-1#show running-config interface vlan 50
!
interface Vlan 50
ip address 50.50.50.1/24
ipv6 address 50:50:50:50::1/64
!
vrrp-ipv6-group 50
priority 200
virtual-address 50:50:50:50::5
virtual-address fe80::50:50:50:5
no shutdown
VLT-100-PEER-1#show vrrp ipv6 brief
Interface Group Pri Pre State Master addr Virtual
addr(s) Description
---------------------------------------------------------------------
-----------------------------
Vl 50 IPv6 50 200 Y Master fe80::201:e8f...50:50:50:50::5
fe80::50:50:50:5
VLT-100-PEER-2#show running-config interface vlan 50
!
interface Vlan 50
ip address 50.50.50.2/24
ipv6 address 50:50:50:50::2/64
ip ospf priority 90
!
vrrp-ipv6-group 50
virtual-address 50:50:50:50::5
virtual-address fe80::50:50:50:5
no shutdown
VLT-100-PEER-2#show vrrp ipv6 brief
Interface Group Pri Pre State Master addr Virtual
addr(s) Description
---------------------------------------------------------------------
-----------------------------
Vl 50 IPv6 50 100 Y Backup fe80::201:e8f...50:50:50:50::5
fe80::50:50:50:5

6.3 mVLT in MXL Switches
Figure 9: mVLT in MXL Switch
The above figure illustrates configuring mVLT between M1000 systems. With FTOS
9.2(0.0) release, VLT is supported between MXL switches within a chassis (intra chassis)
and mVLT is supported between M1000 chassis (inter chassis). With this mVLT
architecture, the VMs in the blade servers within the chassis can migrate to other chassis
blade servers. This flexible and scalable architecture predominantly serves east-west traffic
within the data centers. The upstream traffic extends through the uplink to the ToR/Core
switch with layer-3 connectivity.
This architecture focuses on handling multiple VMs in the Blade servers within the same
VLAN’s across multiple M1000 chassis. VM migration within the rack and across the racks
within the data center is deployed. High availability is incorporated at all layers from the
active servers to the network connectivity.
Active System Infrastructure with appropriate profiles is deployed to aggregate the access
traffic matching the customer workloads.
VLT between
MXL Switches
mVLT between
M1000 Chassis
WAN
Layer-3
OSPF/IS-IS /BGP

6.4 Auto VLT in IOM
Auto VLT deployment in IOM is supported in the FTOS 9.2(0.0) release. The VLT domain
with dual IOM in the M1000 chassis is enabled with the CLI command:
stack-unit 0 iom-mode vlt
Figure 10: Auto VLT in IOA
Prerequisites:
1. Connect the integrated base module 40gig ports between the two IOMs.
2. Configure the iom-mode as “VLT” using the CLI “stack-unit 0 iom-mode vlt”
3. Reload the IOMs for the VLT mode to take effect.
Important Information for IOA VLT
 With VLT in IOA, VLTi (ICL) port-channel is always configured as Po 127 (static port-
channel).
 Uplink VLT port-channel is Po 128 dynamic (with LACP)
 Server-side VLT port-channel is auto-assigned (LACP) between the port-channel
ranges (1-126). ICL is established between base module forty GIG ports (Fo 33 and 37)
 Uplink VLT port-channel is established between ports 41 – 56 (optional module ports)
 Server-side VLT LAG is established between ports 1 – 32.
 Back-up link is established between the IOM’s IP address (169.254.31.X)
Auto-VLT between
IOM Switches
1) Connect the integrated
base module 40gig ports
between the two IOM’s.
2) Configure the iom-mode
as “VLT” using the CLI
“stack-unit 0 iom-mode vlt”
3) Reload the IOM’s for the
VLT mode to take effect.

FTOS(conf)#
FTOS(conf)#stack-unit 0 iom-mode ?
programmable-mux Programmable Mux mode
stack Stack mode
standalone Standalone mode
vlt Vlt mode
FTOS(conf)#stack-unit 0 iom-mode vlt
% You are about to configure Auto VLT to your IOA module, please reload the
IOA and then plug in the ICL cable for the changes to take effect.
FTOS#show run int po 127
!
interface Port-channel 127
mtu 12000
channel-member fortyGigE
0/33,37
no shutdown
FTOS#
FTOS#show run int po 128
!
mtu 12000
portmode hybrid
switchport
fip-snooping port-mode fcf
vlt-peer-lag port-channel 128
no shutdown
FTOS#
FTOS#show vlt brief
VLT Domain Brief
------------------
Domain ID: 1
Role: Primary
Role Priority: 32768
ICL Link Status: Up
VLT Peer Status: Up
Local Unit Id: 0
Version: 6(1)
Local System MAC address: 00:01:e8:43:00:08
Remote System MAC address: 00:1e:c9:f1:02:0a
Configured System MAC address: 00:01:00:03:00:02
Peer-Routing : Disabled
FTOS#
FTOS#show vlt backup-link
VLT Backup Link
-----------------
Destination: 169.254.31.25
Peer HeartBeat status: Up
HeartBeat Timer Interval: 1
HeartBeat Timeout: 3
UDP Port: 34998
HeartBeat Messages Sent: 449505
HeartBeat Messages Received: 449496

FTOS#
FTOS#show vlt detail
Local LAG Id Peer LAG Id Local Status Peer Status Active VLANs
------------ ----------- ------------ ----------- -------------
1 1 UP UP 1
128 128 UP UP 1
FTOS#
FTOS#show int po brief
Codes: L - LACP Port-channel
LAG Mode Status Uptime Ports
L 1 L2 up 5d6h5m Te 0/6 (Up)
127 L2 up 5d6h5m Fo 0/33 (Up)
Fo 0/37 (Up)
L 128 L2 up 5d6h5m Te 0/41 (Up)
Te 0/42 (Up)
FTOS#
FTOS#show run int po 1 (auto config)
!
mtu 12000
portmode hybrid
switchport
no shutdown
FTOS#
FTOS#show vlan
Codes: * - Default VLAN, G - GVRP VLANs, R - Remote Port Mirroring VLANs, P
- Primary, C - Community, I - Isolated
O - Openflow
Q: U - Untagged, T - Tagged
x - Dot1x untagged, X - Dot1x tagged
o - OpenFlow untagged, O - OpenFlow tagged
G - GVRP tagged, M - Vlan-stack, H - VSN tagged
i - Internal untagged, I - Internal tagged, v - VLT untagged, V - VLT
tagged
NUM Status Description Q Ports
* 1 Active U Po1(Te 0/6)
U Po127(Fo 0/33,37)
U Po128(Te 0/41-42,44)
U Te 0/1-5,7-32
FTOS(conf)#int ten 0/6
FTOS(conf-if-te-0/6)#vlan tagged ?
VLAN-RANGE Comma/Hyphen separated VLAN ID set
FTOS(conf-if-te-0/6)#vlan tagged 4 >>>>>>> Configure VLAN
FTOS(conf-if-te-0/6)#do show vlan
Codes: * - Default VLAN, G - GVRP VLANs, R - Remote Port Mirroring VLANs, P
- Primary, C - Community, I - Isolated
O - Openflow
Q: U - Untagged, T - Tagged

x - Dot1x untagged, X - Dot1x tagged
o - OpenFlow untagged, O - OpenFlow tagged
G - GVRP tagged, M - Vlan-stack, H - VSN tagged
i - Internal untagged, I - Internal tagged, v - VLT untagged, V - VLT
tagged
NUM Status Description Q Ports
* 1 Active U Po1(Te 0/6)
U Po127(Fo 0/33,37)
U Po128(Te 0/41-42,44)
U Te 0/1-5,7-32
4 Active T Po1(Te 0/6)
T Po128(Te 0/41-42,44)
V Po127(Fo 0/33,37)
FTOS(conf-if-te-0/6)#

6.5 LAG Features
The FTOS 9.2(0.0) release supports 16-member LAG. This enhances the bundling capacity
of the VLT nodes handling the traffic. Similarly, a 16-member port-channel LAG from the
ToR, with 8 links connecting to VLT peer-1 and another 8 links connecting to VLT peer-2
handles less oversubscribed fabric requirement for specific scenarios. The following figure
represents a 16-port 10G link from a S4810 switch connected to a 40G breakout of a
Z9000 switch. This LAG support generally assists the large east-west traffic pattern based
on unique workloads.
Figure 11: 16-Members LAG in VLT
128 VLT LAG port-channels per node is supported in FTOS 9.2. With Z9000 and S6000
switches (32x40G or 128x10G ports), each port is formed as VLT LAG connecting to its
peer for highly over-subscribed environments such as campus networks, small/medium
enterprise networks etc.,
Depending upon the specific requirement of fabric subscription, the number of user ports
can scale from 720x10G ports to 3600x10G and beyond.

7 VLT Peers Connected with 10G VLTi
Figure 12: VLT with 10G VLTi Links
This topology represents the oversubscribed ToR to VLT aggregation switch. Any
failure in the VLT links forces the traffic through the other VLT peer and using VLTi
depending upon the topology. This setup is deployed in small and medium enterprise
networks, data centers, and campus networks connecting server and storage clusters.
Generally, north to the VLT domain, the aggregator is connected to the core routers
for north bound external traffic. Being layer 3 at the core, the IGP is extended up to
the VLTI link with ‘peer-routing’. IGP (OSPF/IS-IS) is configured in the VLAN interfaces,
which forms adjacency between the VLT peers,
With the MAC, ARP tables, IGMP state synchronization within the VLT peers, the VLT
switches represent a single logical switch to the underlying layer-2 ToR switches
ensuring high availability and resiliency in operation.

8 VLT Peers Connected with 40G VLTi
Figure 13: VLT with 40G VLTi Links
This topology is used for non-blocking and oversubscribed ToR to VLT aggregation. Any
failure in the VLT link forces the traffic through the diverse path of the other VLT peer and
using VLTi depending upon the topology. This VLTi with two or more 40G can handle major
traffic flow during failure scenarios. This setup is deployed in enterprise networks with
multicast traffic streams and high performance computing (HPC) data centers connecting to
server and storage clusters.
Seamless VM migration within the rack and between the racks is ensured with this topology.
As all the port-channel links are effectively utilized, latency and congestion in traffic is
considerably less compared to the conventional dual or collapsed core network model.

9 VLT Peers Connected to Standalone Switches
Figure 14: VLT Domain Connected to Standalone Switches
This simple topology represents the oversubscribed/non-blocking Top-of-Rack (ToR)
to VLT aggregation. All the ToR switches are standalone rack mounted switches with
port-channels extending to VLT domains. Although this is a simple and cost-effective
setup, the redundancy at the ToR layer switch must match the NIC teaming from the
servers connected to the adjoining ToR switches in the other racks. The physical
implementation of this topology mandates that the adjoining racks, ToR switches, and
servers have the same VLAN. With dual NICs in the servers, the Twinax DAC Copper
cables are used to connect both the ToR switches. This setup is deployed in
campuses, enterprise networks connecting to server and storage clusters.

10 VLT Peers Connected to Stacked Switches
Figure 15: VLT Peers Connected to Stacked Switches
This simple topology represents the oversubscribed ToR to VLT aggregation. All the
ToR switches are stacked switches with port-channels from stacked members
extending to VLT domains. This setup is deployed in campuses and enterprise
networks connecting to server and storage clusters.
Stacked ToR switches offer redundancy at multiple levels for the server clusters. Port-
channel interface is connected from the stacked members to the VLT peers. Stacked
ToR and the dual NICs in the server form the extended access redundancy while
offering high availability for the network.
This topology offers over-subscribed ToR switches connected to the VLT domains.
Based on the requirements, and for the high port density requirement, stacked ToRs
are deployed connecting to VLT domains. However, the VLT nodes cannot be
stacked.

11 Single VLAN across Two ToRs
Figure 16: Single VLAN across ToR Switch
This is an idealistic topology, where a single VLAN is configured for each ToR switch.
Not many scenarios need this topology except a few that have multiple replication
server/storage for specific applications, workload aware VM migrations, web-services
with multi proxy web servers, etc.

12 Multi VLANs across Multiple ToRs
Figure 17: Multi VLAN across ToR
This topology is widely deployed with multi VLANs spanning across multiple ToR switches
connected to the VLT domain. Redundancy at the server level is achieved by deploying multi
services at multiple ToR switches. Server load balancing is deployed in these scenarios.
For all the VLANs created at the ToR switch and terminated at the VLT peers, the VLT links need
to be configured as tagged port. The VLANs at the VLT peer are configured as L3-VLAN with the
corresponding IP addresses configured on both the VLT peers.

13 Inter-VLAN Routing between ToRs
Figure 18: Inter VLAN Routing in VLT
With the VLT links connected to the VLT domain, the inter-VLAN routing between
multiple layer-3 VLANs in the VLT domain is through the receiving VLT peer and the
VLT port-channel of the destination VLAN. Due to the ARP and MAC synchronization,
the egress port details are synchronized in both the VLT peers. Any failure in the
node/link is seamlessly handled.
With ‘peer-routing’ feature, OSPF/IS-IS/BGP is supported for routing across VLAN
subnets.

14 VLT in Distributed Core
Figure 19: VLT in Distributed Core
In a distributed core with multi spine and leaf nodes, VLT has to be implemented in
the leaf nodes. VLT in the spine layer may not scale well in a distributed core network
(CLOS) architecture. If the multiple nodes in the spine layer has VLT configured, with
the leaf node VLT connected to the spine node VLT (as in mVLT) without any
redundant links, this is a normal multi VLT topology. However if the redundant links,
as in a distributed core network, connect to the other leaf VLT in domain-2 and
spanning the same VLANs, then there is a possibility of loop in the fabric for a layer-2
topology. Deploying STP to avoid loops can block such multi-links in the distributed
core. Further, implementing VLT in the spine layer deprives the available capacity of
the fabric as few ports are used for VLTi.
With layer-3 in the fabric, there is not much advantage of VLT implemented in the
spine layer. However, with VLT in the leaf nodes, there is NIC teaming or Server load
balancing as well as redundancy in the ToR level as the access is of layer-2.
Besides managing the distributed core, the Active Fabric Manager (AFM 1.5) has the
VLT features incorporated for auto configuring the VLT peers and the ToR switches
based on requirements. With auto configured VLT domains, VLTI ports and the VLT
port-channels, it easy to manage the VLT domain through AFM.

14.1 VLT in Leaf layer
Figure 20: VLT in Leaf Layer
14.2 VLT in Spine Layer
AFM 1.5 uses distributed core with VLT nodes in the spine layer and multiple VLT
domains in the leaf layer as shown in the figure below. The uplinks are terminated at
the spine layer unlike the leaf layer in the conventional distributed core. The dual
spine nodes are Z9000 switches and the leaf nodes are S4810 switches. The 40G links
are interlinked between the Z9000 and S4810 switches in this architecture.
Figure 21: VLT in Spine and Leaf Layer
The summary view of AFM1.5 for design and deployment in VLT domains is shown in
the figure below.
Figure 22: AFM Fabric Design Summary View

15 STP Flavors across ToR
Figure 23: STP in VLT
Current VLT feature supports only RSTP in the VLT domain. No other STP flavors are
supported in the VLT domain. Deploying RSTP in the VLT domain improves the
convergence during the coming up of VLT and ToR nodes. Moreover, it eliminates
any loop due to misconfigured ports. RSTP states are synchronized in both the VLT
peers; any RSTP BPDU received on the secondary VLT peer is tunneled to the primary
VLT peer for processing. Only the primary VLT peer generates and forwards the BPDU
on the VLT Port.
RSTP in this VLT domain keeps all the ports in the FWD state, as there is no loop in the
topology. Primarily RSTP is required to isolate network loop in case of incorrect
cabling in the VLT domain. As a best practice, always keep the primary VLT member
as the Root Bridge, VLT secondary node with bridge priority higher than primary and
lower than ToR, and do not change the bridge-priorities in the ToR/Access switches.
Root-guard is enabled on VLT port-channels. Also, configure the host-connected
ports as edge-ports in the ToR to avoid TCN when servers are booted.
F10S4810AGGR2#show spanning-tree rstp interface port-channel 40
Port-channel 40 is designated Forwarding
Port is a Virtual Link Trunk port
Edge port:no (default) port guard :none (default)
Link type: point-to-point (auto) bpdu filter:disable (default)
Bpdu guard :disable bpduguard shutdown-on-violation : disable RootGuard: disable
LoopGuard disable
Bpdus sent 29, received 0
Interface Designated
Name PortID Prio Cost Sts Cost Bridge ID PortID
--------- -------- ---- ------- ----------- ------- -------------------- --------
Po 40 128.41 128 1800 FWD(vlt) 600 32768 0001.e88b.1c9a 128.41

F10S4810AGGR2#
F10S4810AGGR2#show spanning-tree rstp interface port-channel 90
Port-channel 90 is root forwarding
Port is a Virtual Link Trunk Interconnect port
Edge port:no (default) port guard :none (default)
Link type: point-to-point (auto) bpdu filter:disable (default)
Bpdu guard :disable bpduguard shutdown-on-violation : disable RootGuard: disable
LoopGuard disable
Bpdus sent 1, received 31
--------- -------- ---- ------- ----------- ------- -------------------- --------
Po 90 128.91 128 600 FWD(vltI) 600 4096 0001.e88b.1d3f 128.91
F10S4810AGGR2#
F10S4810AGGR2#
The VLTi link is always in the STP forward state. The Interface status is indicated as
“Forward” with “VLT” and “VLTi “keyword as indicated in the above CLI snapshot.
RSTP Configuration
protocol spanning-tree rstp
no disable
bridge-priority 0 <<< Apply “0” for Primary and “ 4096” for secondary VLT node
forward-delay 4
hello-time 1

16 RSTP in VLT Peers and MSTP in the ToR
Figure 24: RSTP and MSTP in VLT
With MSTP configured in the ToR switches, they fall back to RSTP, once they receive
the RSTP BPDU from the VLT primary switch and converge with RSTP. As a best
practice, ensure that the VLT primary node is configured as root bridge and the
secondary node as secondary root bridge.
F10S4810AGGR1(conf-rstp)#do show spanning-tree rstp brief | except DIS
Executing IEEE compatible Spanning Tree Protocol
Root ID Priority 4096, Address 0001.e88b.1d3f
Root Bridge hello time 2, max age 20, forward delay 15
Bridge ID Priority 4096, Address 0001.e88b.1d3f
We are the root
Configured hello time 2, max age 20, forward delay 15
---------- -------- ---- ------- ----------- ------- -------------------- --------
Po 40 128.41 128 1800 FWD(vlt) 0 4096 0001.e88b.1d3f 128.41
Po 90 128.91 128 600 FWD(vltI) 0 4096 0001.e88b.1d3f 128.91
Interface
Name Role PortID Prio Cost Sts Cost Link-type Edge
---------- ------ -------- ---- ------- ----------- ------- --------- ----
Po 40 Desg 128.41 128 1800 FWD 0 (vlt) P2P No
Po 90 Desg 128.91 128 600 FWD 0 (vltI)P2P No
F10S4810AGGR1(conf-rstp)#

17 RSTP in VLT peer and PVST+ within the ToR's
Figure 25: RSTP and PVST+ in VLT
When the VLT domain with RSTP is connected to the ToRs configured with PVST+,
the PVST + BPDUs received on the VLT peer get tunneled into the VLT domain for the
non-default VLANs.
A few scenarios with multi-vendor devices might have PVST+ configured in the ToR
switches. As a best practice for this scenario, tag the VLT ports for the respective non-
default VLANs so that the PVST BPDs pass through the VLT domain for convergence.
Configuration example:
ToR1 with RPVST+ enabled
spanning-tree mode rapid-pvst
spanning-tree etherchannel guard misconfig
spanning-tree extend system-id
!
VLAN0002
Spanning tree enabled protocol rstp
Root ID Priority 32770
Address 001d.a23e.0d00
This bridge is the root <<<<<<<<<<<<< (TOR 1) is root for vlan 2
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32770 (priority 32768 sys-id-ext 2)
Aging Time 300 sec
Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------

Po10 Desg FWD 4 128.560 P2p
VLAN0003
Address 0001.e8a4.fc7e <<<<<<<<<<< bridge id of TOR2 (Z9000)
Cost 4
Port 560 (Port-channel10)
Aging Time 300 sec
------------------- ---- --- --------- -------- --------------------------------
Po10 Root FWD 4 128.560 P2p
VLAN0004
Address 0001.e8a4.fc7e <<<<<<<<<< TOR 2 is root for vlan 4
Cost 4
Port 560 (Port-channel10)
Aging Time 300 sec
------------------- ---- --- --------- -------- --------------------------------
Po10 Root FWD 4 128.560 P2p
Z9000 (ToR2) Output :
FTOS(conf-pvst)#do show span pvst vlan 2 brief
VLAN 2
Root ID Priority 32770, Address 001d.a23e.0d00
Bridge ID Priority 61440, Address 0001.e8a4.fc7e
---------- -------- ---- ------ ----------- ------- -------------------- --------
Po 100 128.101 128 1800 FWD 1800 32770 001d.a23e.0d00 128.560
Po 128 128.129 128 200000 DIS 1800 32768 0001.e8a4.fc7e 128.129
Interface
---------- ------ -------- ---- ------- ----------- ------- --------- ----

Po 100 Root 128.101 128 1800 FWD 1800 P2P No
Po 128 Dis 128.129 128 200000 DIS 1800 P2P No
FTOS(conf-pvst)#
FTOS(conf-pvst)#do show span pvst vlan 3 brief
VLAN 3
Root ID Priority 32768, Address 0001.e8a4.fc7e
We are the root of Vlan 3
---------- -------- ---- ------ ----------- ------- -------------------- --------
Po 100 128.101 128 1800 FWD 0 32768 0001.e8a4.fc7e 128.101
Po 128 128.129 128 200000 DIS 0 32768 0001.e8a4.fc7e 128.129
Interface
---------- ------ -------- ---- ------- ----------- ------- --------- ----
Po 100 Desg 128.101 128 1800 FWD 0 P2P No
Po 128 Dis 128.129 128 200000 DIS 0 P2P No
FTOS(conf-pvst)#
FTOS(conf-pvst)#do show span pvst vlan 4 br
VLAN 4
Root ID Priority 32768, Address 0001.e8a4.fc7e
We are the root of Vlan 4
---------- -------- ---- ------ ----------- ------- -------------------- --------
Po 100 128.101 128 1800 FWD 0 32768 0001.e8a4.fc7e 128.101
Po 128 128.129 128 200000 DIS 0 32768 0001.e8a4.fc7e 128.129
Interface
---------- ------ -------- ---- ------- ----------- ------- --------- ----
Po 100 Desg 128.101 128 1800 FWD 0 P2P No
Po 128 Dis 128.129 128 200000 DIS 0 P2P No

18 BGP on the Upstream Links from VLT Peers
Figure 26: BGP on VLT Uplinks
Depending on the number of BGP peers and the routes the VLT nodes can peer directly to the
upstream ISP's, or through core routers peering to ISP's, having IBGP to the VLT nodes. With BGP
policies on the ingress route-maps, preference for specific NLRI is implemented. If the core
router needs to have complete Internet routes, then the route-maps with policies for more
specific routes along with default-route is advertised to the VLT peers. As a best practice,
configure the same ASN in both the VLT peers. IBGP peering has to be established between VLT
peers.
In case of hierarchical VLT domains and if IBGP is configured across VLT domains with ‘peer-
routing’ enabled, the spine/core layer has to be configured as route-reflectors.

19 Scenarios with non-VLT Ports
Figure 27: VLAN’s with non-VLT Port
Firewall and load balancers connected to non-VLT ports might be required in certain
networks, where policy based routing directs matching traffic to firewall for deep
packet inspection. For the hosts that are connected to the non-VLT ports, the traffic
may pass through the VLTi links. Hence, the bandwidth required has to be defined for
such scenarios. With FTOS 9.2(0.0) release, the MAC and ARP tables for the orphan
ports are synchronized between the VLT peers.
For networks spread across different geographic regions, extending the network is
achieved in multiple ways. With service providers offering the backbone, MPLS or
BGP with core routers connecting to its peers can extend the VPN services.
Alternately, if the dark fiber is laid across all these locations, layer-2 or layer-3 services
can be extended as explained in the layer-2 domain extension section.

20 VRRP in VLT
In a VLT domain with MAC and ARP synchronized, the VRRP hello messages are sent
to the other peer and the domain offers active-active load balancing unlike the
normal master-standby operation. Any of the upstream packets hitting the VRRP
standby node in the VLT domain need not be sent to the VRRP master node. The
received packet is routed to its destination from the VLT peer. See the figure below.
NOTE: As a best practice, configure identical VRRP configurations on both VLT peers in a
domain. Track interface configuration has no significance in the VLT domain.
NOTE: With FTOS 9.2(0.0), VRRP for IPv4 and IPv6 dual stack is supported
Figure 28: VRRP in VLT
20.1 VRRP Configuration
VLT Peer-1
interface Vlan 200
ip address 10.0.0.252/24
tagged Port-channel 1-2
!
vrrp-group 1
no preempt
virtual-address 10.0.0.254
no shutdown
!
interface Vlan 201
ip address 11.0.0.252/24
!
vrrp-group 1
no preempt
no shutdown
VLT Peer-2
interface Vlan 200
ip address 10.0.0.253/24
!
vrrp-group 1
no preempt
no shutdown
!
interface Vlan 201
ip address 11.0.0.253/24
!
vrrp-group 1
no preempt
no shutdown
VRRP
Active-Active
VRRP
Master

21 Multicast in VLT
VLT nodes have IGMP messages synchronized across their peers through the VLTi
messages. Any of the IGMP joins received at the VLT nodes are synchronized with
their peer, so the IGMP tables are identical in the VLT domain.
VLT nodes learn the mrouter ports from the upstream IGMP querier messages. If
layer-3 VLANs are enabled with PIM-SM, then VLTi ports is statically configured as
mrouter ports. However, it is not required if the VLT domain has only IGMP-Snooping
enabled.
Layer-3 VLANs in the VLT nodes is enabled with PIM-Sparse mode which offers
Active-Active PIM-SM support. Either of the VLT peer act as DR or BDR. However, as a
best practice configure the non-VLT Core router to act as RP for the multicast group.
Any multicast source traffic arriving at the VLT peer is sent to the receiver and also to
the VLTi link. However with the port block at the VLT link of the peer node the
multicast traffic is dropped and not duplicated (Figure 18.0). However, if the receiver is
connected in a non-VLT port on that peer, the multicast traffic would reach the
receiver through the VLTi.
VLTi link is added as mrouter port if the IGMP Query is learnt through VLTi. If the
multicast groups are more with multiple receivers and Source, then the VLTi link
bandwidth has to be increased to handle the multicast flooding on the VLTi links.
Figure 29: Multicast in VLT

With FTOS 9.2(0.0) release, the following features are implemented for multicast:
 PIM routes synchronization between VLT nodes
 Outgoing Interface (OIF) Sync for PIM routes
 PIM First hop (FHR) and Last hop (LHR)
 PIM-SM Intermediate Router support
 Show ip mroute on VLT context
 Multicast peer-routing timeout
 VLT mismatch for multicast
21.1 Multicast in Hierarchical VLT Topology
Figure 30: Multicast in Hierarchical Layer-2 VLT Domain
The Leaf VLT domains configured with layer-2 VLAN’s and has IGMP-snooping
enabled. With IGMP-Snooping on both the VLT nodes, the multicast forwarding states
are synchronized. PIM-Sparse mode has to be enabled on the layer-3 VLANs
participating in the multicast traffic for the Spine VLT domain.
The topology shown in the figure 18.1 meets the scalability needs for large multicast
groups. The following points are to be ensured while deploying multicast service
through VLT domain

 With VLT in Spine and leaf nodes, the RP (Rendezvous Point) for the multicast
group shall be the non-VLT node preferably the core node in upstream.
 Currently VLT node cannot be configured as RP. As a workaround, RP shall be
configured with single multicast enabled Layer-3 VLAN, connected to Spine VLT
domains.
 Disable IGMP-snooping on the upstream links.
For various other VLT topologies with layer-3 between VLT domains, only a single
multicast enabled Layer-3 VLAN has to be configured for the RP reachability.
However other L3 VLANs not participating in multicast is spanned between the Spine
and Leaf VLT domains.
Figure 31: Multicast in Hierarchical VLT with Layer-3 VLT Domain
In a hierarchical VLT domain with Spine and Leaf VLT domains configured for peer-
routing, optimal flow for multicast traffic is ensured by implementing a single
multicast enabled layer-3 VLAN between the VLT domains for the reachability of the
RP or Multicast Source. In this topology (Figure 18.2), the Spine VLT domain acts as
Intermediate PIM-SM Router.
As a workaround for this limitation, the Leaf node VLT domains have to be configured
with Layer-2 VLANs, enabled with IGMP-Snooping, and the Spine VLT domain
configured with PIM-SM as shown in figure 18.1

Multicast peer-routing timeout
By default the multicast peer-routing timeout is configured for 150 seconds. In case of
VLT peer node failure, the alternate node forwards the multicast traffic on behalf of its
VLT peer for the configured timeout period in seconds. As a best practice, the
duration shall be configured more than 105 seconds (matching PIM hold timer value).
VLT Mismatch for multicast
%STKUNIT0-M:CP %PIM-3-PIM_VLT_MRT_IIF_MISMATCH: VLT peers failed to choose
the same interface as Incoming Interface for the Route (176.16.1.10, 225.0.0.2)
VLT-100-SPINE1-FTOS#show vlt mismatch
Global IP Multicast Status
---------------------------
IP Version Local Status Peer Status
---------- ------------- ------------
IPV4 Disabled Enabled
VLT-100-SPINE1-FTOS#
VLT-100-SPINE2-FTOS# show vlt inconsistency ip mroute
Spanned Multicast Routing IIF Inconsistency
Multicast Route LocalIIF PeerIIF
--------------- ---------- ---------
(0.0.0.0 , 225.0.0.0) Vl 18 Vl 4001
VLT-100-SPINE1-FTOS#show vlt statistics multicast
VLT Multicast Statistics
-------------------------------
Info Pkts Sent: 13
Info Pkts Rcvd: 6
Reg Request Sent: 2
Reg Request Rcvd: 4
Reg Response Sent: 2
Reg Response Rcvd: 2
Route updates sent to Peer: 4
Route updates rcvd from Peer: 10
Route update pkts sent to Peer: 3
Route update pkts rcvd from Peer: 7
VLT-100-SPINE1-FTOS#show ip mroute vlt count
IP Multicast Statistics
1 routes using 648 bytes of memory
1 groups, 1.00 average sources per group

Group: 225.0.0.0, Source count: 1
Source: 90.114.112.11
Syslog for multicast
For any misconfigured static routes, if the Incoming interface (VLAN) for (*,G) or (S,G)
route differs in both peers, the following syslog is displayed.
Aug 6 09:56:00: %STKUNIT0-M:CP %PIM-3-PIM_VLT_MRT_IIF_MISMATCH:
Route (0.0.0.0, 225.0.0.0) - Detected IIF mismatch between VLT
peers. Local IIF is Vl 18, Remote IIF is Vl 4001
Aug 6 09:57:03: %STKUNIT0-M:CP %PIM-3-
PIM_VLT_MRT_CLEAR_IIF_MISMATCH: Route (0.0.0.0, 225.0.0.0) - Cleared
IIF mismatch between VLT Peers. IIF chosen is Vl 4001
Aug 6 10:01:27: %STKUNIT0-M:CP %PIM-3-PIM_VLT_MRT_IIF_MISMATCH:
Route (0.0.0.0, 225.0.0.0) - Detected IIF mismatch between VLT
peers. Local IIF is Vl 18, Remote IIF is Vl 4001

21.2 Multicast configuration and Outputs from Leaf and Spine
VLT Nodes
VLT-10-LEAF-1#show vlt statistics igmp-snoop
VLT Igmp-Snooping Statistics
-------------------------------
IGMP Info Pkts sent: 5
IGMP Info Pkts Rcvd: 7
IGMP Reg Request sent: 2
IGMP Reg Request rcvd: 1
IGMP Reg Response sent: 1
IGMP Reg Response rcvd: 1
IGMP PDU Tunnel Pkt sent:83
IGMP PDU Tunnel Pkt rcvd:58
IGMP Tunnel PDUs sent: 257
IGMP Tunnel PDUs rcvd: 320
VLT-10-LEAF-1#show running-config igmp
!
ip igmp snooping enable
-------------------------------
VLT-10-LEAF-1#show ip igmp groups
Total Number of Groups: 5
IGMP Connected Group Membership
Group
Address Interface Mode Uptime Expires Last
Reporter
225.1.1.1 Vlan
10 IGMPv2 00:05:59 00:01:21 192.168.20.20
Member Ports: Po 11
225.1.1.2 Vlan
10 IGMPv2 00:05:59 00:01:21 192.168.20.20
Member Ports: Po 11
225.1.1.3 Vlan
10 IGMPv2 00:05:59 00:01:21 192.168.20.20
Member Ports: Po 11
225.1.1.4 Vlan
10 IGMPv2 00:05:59 00:01:21 192.168.20.20
Member Ports: Po 11
225.1.1.5 Vlan
10 IGMPv2 00:05:59 00:01:21 192.168.20.20
Member Ports: Po 11

VLT-10-LEAF-2#show running-config igmp
!
ip igmp snooping enable
-------------------------------
VLT-10-LEAF-2#show ip igmp groups
Total Number of Groups: 5
IGMP Connected Group Membership
Group
Address Interface Mode Uptime Expires Last
Reporter
225.1.1.1 Vlan
10 IGMPv2 00:07:09 00:02:06 192.168.20.20
Member Ports: Po 11
225.1.1.2 Vlan
10 IGMPv2 00:07:09 00:02:06 192.168.20.20
Member Ports: Po 11
225.1.1.3 Vlan
10 IGMPv2 00:07:09 00:02:06 192.168.20.20
Member Ports: Po 11
225.1.1.4 Vlan
10 IGMPv2 00:07:09 00:02:06 192.168.20.20
Member Ports: Po 11
225.1.1.5 Vlan
10 IGMPv2 00:07:09 00:02:06 192.168.20.20
Member Ports: Po 11
VLT-100-SPINE-1#show running-config pim
!
ip pim rp-address 90.114.112.1 group-address 224.0.0.0/4
VLT-100-SPINE-1(conf-if-range-vl-10,vl-20,vl-100)#show config
!
interface Vlan 10
ip address 192.168.0.3/16
ip pim sparse-mode
no shutdown
!
interface Vlan 20

ip address 192.169.0.3/16
ip pim sparse-mode
no shutdown
!
interface Vlan 100
ip address 100.100.100.1/24
untagged Port-channel 55
ip pim sparse-mode
no shutdown
VLT-100-SPINE-1#show running-config | grep multi
ip multicast-routing
VLT-100-SPINE-1#show ip pim summary
PIM TIB version 110
Uptime 01:37:07
Entries in PIM-TIB/MFC : 5/0
Active Modes :
PIM-SM
Interface summary:
3 active PIM interfaces
0 passive PIM interfaces
4 active PIM neighbors
TIB summary:
5/0 (*,G) entries in PIM-TIB/MFC
0/0 (S,G) entries in PIM-TIB/MFC
0/0 (S,G,Rpt) entries in PIM-TIB/MFC
2 PIM nexthops
1 RPs
0 sources
0 Register states
Message summary:
20/885 Joins sent/received
20/110 Prunes sent/received
0/0 Candidate-RP advertisements sent/received
0/0 BSR messages sent/received
0/0 State-Refresh messages sent/received
0/0 MSDP updates sent/received
0/0 Null Register messages sent/received
0/0 Register-stop messages sent/received
Data path event summary:
10 no-cache messages received
24 last-hop switchover messages received
0/0 pim-assert messages sent/received
0/0 register messages sent/received
Memory usage:
TIB : 2440 bytes

Nexthop cache : 232 bytes
Interface table : 5000 bytes
Neighbor table : 672 bytes
RP Mapping : 180 bytes
VLT-100-SPINE-1#show ip pim tib
PIM Multicast Routing Table
Flags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned,
R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPT,
M - MSDP created entry, A - Candidate for MSDP Advertisement
K - Ack-Pending State
Timers: Uptime/Expires
Interface state: Interface, next-Hop, State/Mode
(*, 225.1.1.1), uptime 00:09:38, expires 00:00:00, RP 90.114.112.1, flags:
SCPJ
Incoming interface: Vlan 100, RPF neighbor 100.100.100.3
Outgoing interface list:
SCPJ
SCPJ
SCPJ
SCPJ
VLT-100-SPINE-1#show ip mroute vlt
IP Multicast Routing Table
Flags: S - Synced
(*, 225.1.1.1), uptime 00:09:51 flags: S
Incoming interface: Vlan 100
Spanned outgoing interface list:
Vlan 10 (S)
Vlan 20 (S)
(90.114.112.2, 225.1.1.1), uptime 00:09:06 flags: S
Vlan 10 (S)
Vlan 20 (S)

(*, 225.1.1.2), uptime 00:09:51 flags: S
Vlan 10 (S)
Vlan 20 (S)
(90.114.112.2, 225.1.1.2), uptime 00:09:06 flags: S
Vlan 10 (S)
Vlan 20 (S)
(*, 225.1.1.3), uptime 00:09:53 flags: S
Vlan 10 (S)
Vlan 20 (S)
(90.114.112.2, 225.1.1.3), uptime 00:09:08 flags: S
Vlan 10 (S)
Vlan 20 (S)
(*, 225.1.1.4), uptime 00:09:53 flags: S
Vlan 10 (S)
Vlan 20 (S)
(90.114.112.2, 225.1.1.4), uptime 00:09:08 flags: S
Vlan 10 (S)
Vlan 20 (S)
(*, 225.1.1.5), uptime 00:09:55 flags: S
Vlan 10 (S)
Vlan 20 (S)
(90.114.112.2, 225.1.1.5), uptime 00:09:09 flags: S
Vlan 10 (S)
Vlan 20 (S)
VLT-100-SPINE-1#show ip pim neighbor
Neighbor Interface Uptime/Expires Ver DR
Prio/Mode GR
Address
192.168.0.4 Vl 10 00:51:00/00:01:21 v2 1 / DR S
192.169.0.4 Vl 20 00:14:06/00:01:40 v2 1 / DR S
100.100.100.2 Vl 100 01:34:44/00:01:42 v2 1 / S
100.100.100.3 Vl 100 01:35:00/00:01:25 v2 1 / DR S

VLT-100-SPINE-2#show ip pim neighbor
Neighbor Interface Uptime/Expires Ver DR
Prio/Mode GR
Address
192.168.0.3 Vl 10 00:51:37/00:01:25 v2 1 / S
192.169.0.3 Vl 20 00:14:44/00:01:34 v2 1 / S
100.100.100.1 Vl 100 01:35:21/00:01:18 v2 1 / S
100.100.100.3 Vl 100 01:35:22/00:01:18 v2 1 / DR S
VLT-100-SPINE-2#show ip mroute vlt
IP Multicast Routing Table
Flags: S - Synced
(*, 225.1.1.1), uptime 00:12:27
Vlan 10
Vlan 20
(90.114.112.2, 225.1.1.1), uptime 00:11:41
Vlan 10
Vlan 20
(*, 225.1.1.2), uptime 00:12:27
Vlan 10
Vlan 20
(90.114.112.2, 225.1.1.2), uptime 00:11:41
Vlan 10
Vlan 20
(*, 225.1.1.3), uptime 00:12:28
Vlan 10
Vlan 20
(90.114.112.2, 225.1.1.3), uptime 00:11:42
Vlan 10
Vlan 20
(*, 225.1.1.4), uptime 00:12:28
Vlan 10
Vlan 20

(90.114.112.2, 225.1.1.4), uptime 00:11:43
Vlan 10
Vlan 20
(*, 225.1.1.5), uptime 00:12:29
Vlan 10
Vlan 20
(90.114.112.2, 225.1.1.5), uptime 00:11:44
Vlan 10
Vlan 20
VLT-100-SPINE-2# show ip pim tib
PIM Multicast Routing Table
Flags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned,
R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPT,
M - MSDP created entry, A - Candidate for MSDP Advertisement
K - Ack-Pending State
Timers: Uptime/Expires
Interface state: Interface, next-Hop, State/Mode
SCJ
Vlan 10 Forward/Sparse 00:12:42/Never
(90.114.112.2, 225.1.1.1), uptime 00:11:56, expires 00:01:56, flags: CT
SCJ
SCJ

SCJ
SCJ

22 S5000 Switches in VLT
Figure 32: S5000 Switches in VLT for Converged Infrastructure
Currently with S5000 Switches in a VLT domain, FCoE is not supported over VLT.
However, with unique FCoE VLAN mapped to specific ports, FCoE traffic is separately
taken from MXL Switchport to S5000 VLT nodes. MXL act as FSB (FIP Snooping
Bridge). The N_Port from the S5000 Switches are connected directly to FC Switch. In
this topology the VLT redundancy benefits for the LAN traffic only.
For further deployment models with S5000 in a converged infrastructure please go
through the following links:
Dell Networking S5000: The Building Blocks of Unified Fabric and LAN/SAN
Convergence
Dell Networking S5000: Deployment of a Converged Infrastructure with FCoE
However, the S5000 Switches with modular ports isnefit the Small/Medium Enterprise
or Campus Networks which could populate with Ethernet ports and offer VLT
features for LAN traffic.

23 Data Centre interconnect through VLTi with
DWDM optics
Extending Layer 2 Data centers through DWDM rings
Figure 33: DCI with VLT
This topology is to extend the layer-2 domain over different regions. For an enterprise
network having offices collocated at geographically different regions, extending the
layer-2 domain of their network is achieved through the DWDM connectivity
depending on their extent.
The VLT domain-1 in the Hub site (Site-1) connects to VLT domain-2 in Remote Site
(Site-2). The VLT Port-Channels between the multi VLT domains is configured with
long-timeout for LACP. The VLT links pass through the DWDM Add-drop multiplexer.
The Non-VLT ports connect to the upstream connectivity.
VLAN's are tagged on the VLT links connecting to DWDM links at both locations.
Depending upon the VLAN's configured on the respective ToR switch, the VLT links
connecting to the ToR is tagged or untagged. If multiple VLAN's are deployed on the
same ToR switch, then tagging is necessary.
Virtual servers deployed in Site-1 is live-migrated to Site-2 with this topology.
Application agility with varying workloads need for VM live motion across different
sites is deployed with this architecture.

23.1 DWDM Interconnect multiple VLT domains aggregated
with hierarchical VLT
Figure 34: Hierarchical VLT Interconnect with DWDM/CWDM
This topology depicts extending multi VLT thro DWDM Optical transport. Also the
extended mVLT’s could have their "local" mVLT's as well, extending the layer-2 links
within the data center. Based on the over subscription rate and the server density, the
mVLT is extended within the limitation. The above diagram depicts the hierarchical
setup of VLT domains. Multiple Remote sites connect to the central Hub site. With
DWDM add drop multiplexer, tuned lambda carries the remote-site traffic to the
respective hub sites.
CAPEX for this network architecture is less comparing to other contemporary layer-2
extension feature (Fabric path, l2vpn, VPLS etc.,) High availability and resiliency across
all layers are built in this topology, covering from N:1 protection from the DWDM ring
to the optical ports linking to the VLT port-channels.
Albeit the North bound traffic is located at Central site, depending on the unique
customers requirement, another core connectivity is made at Remote site as well.
Alternately, if the distance is within 10KM, then 40G –LR4 CWDM optics is used to
directly connect 40G QSFP+ ports interconnecting VLT domains without any DWDM
mux.
For the appropriate optics, please refer
http://salesedge.dell.com/doc?id=0901bc82806791dc&ll=sr

23.2 VLT Scalability
VLT Switch port capacity
S4810 Switch
48x 10G ports and 4 x40G ports
For any of the VLT setup, with non-blocking capacity
30 x10G for downlink (VLT) and 30x10G for upstream
4x10G for VLTi links
This non-blocking is for the VLT peers only.
Z9000 Switch
32x 40G ports
For any of the VLT setup, with non-blocking capacity
15x 40G for downlink (VLT) and 15x40G for upstream
2x 40G for VLTi links
This non-blocking is for the VLT peers only, however with multi VLT, the Over-
subscription ratio increments with the other VLT domain as well.
For the ToR switches connecting to S4810 VLT peers with port-channel of 16
members (8 in each VLT peer) only 7 ToR nodes is connected for each VLT domain.
This scenario suits well for predominant north-south traffic in the network. This is
changed if the user requires over subscription at different levels/layers. The ToR is
S55/S60 switches and the number of port-channel members is less than 8 - 6/4/2
members. If the significance is for East-West traffic in the network, then over-
subscription in the VLT links is implemented.
23.3 Use cases
1. In the data center environment where multiple VM's hosted in Single Server scaling
beyond this range, meticulous planning has to be made for positioning servers under
each VLT domain with the scalability limits in mind. Not many ToR nodes aggregate
to the same VLT domain. In the scalable data center network, multiple VLT domains
connect to group of ToR nodes spanning multiple VLAN’s, while the upstream of VLT
domains connect to the Core router.
2. For enterprise networks with server-to-server and servers to upstream traffic present
in the networks, over-subscription is the ideal solution. With predominant east-west
traffic pattern, multiple VLT domains to multiple ToR nodes could match server
clusters to storage, with users in various VLAN groups.

3. For HPC networks, depending on the baseline, the over-subscription is derived. The major
advantage over the existing topology is of eliminating STP in the access links. Faster
convergence and reduced latency offer maximum advantage to the High performance
computing networks.
23.4 Example Configuration
VLT Peer-1
F10S4810AGGR1#show run vlt
!
vlt domain 10
peer-link port-channel 90
back-up destination 10.16.151.102
primary-priority 1
system-mac mac-address
00:01:e8:8b:11:11
unit-id 0
delay-restore 10
peer-routing
-----------------------------------
-----------
F10S4810AGGR1#show vlt brief
VLT Domain Brief
------------------
Domain ID: 10
Role:
Primary
Role Priority: 1
ICL Link Status: Up
VLT Peer Status: Up
Local Unit Id: 0
Version: 6(1)
Local System MAC address:
00:01:e8:8b:1d:3f
Remote System MAC address:
00:01:e8:8b:1c:9a
Configured System MAC
address:00:01:e8:8b:11:11
Delay-Restore timer: 10
seconds
Peer-Routing :
Enabled
Peer-Routing-timeout timer: 0
seconds
Multicast peer-routing timeout: 150
seconds
F10S4810AGGR1#
-----------------------------------
-----------
F10S4810AGGR1#show vlt detail
Local LAG Id Peer LAG Id Local
VLT Peer-2
F10S4810AGGR2#show run vlt
!
vlt domain 10
primary-priority 65535
system-mac mac-address
00:01:e8:8b:11:11
unit-id 1
delay-restore 10
peer-routing
-----------------------------------
-----------
F10S4810AGGR2#show vlt brief
VLT Domain Brief
------------------
Domain ID: 10
Role:
Secondary
Role Priority:
65535
ICL Link Status: Up
VLT Peer Status: Up
Local Unit Id: 1
Version: 6(1)
00:01:e8:8b:1c:9a
Remote System MAC address:
00:01:e8:8b:1d:3f
Configured System MAC
address:00:01:e8:8b:11:11
Delay-Restore timer: 10
seconds
Peer-Routing :
Enabled
Peer-Routing-timeout timer: 0
seconds
Multicast peer-routing timeout: 150
seconds
F10S4810AGGR2#
-----------------------------------
-----------
F10S4810AGGR2#show vlt detail

Status Peer Status Active VLANs
------------ ----------- --------
---- ----------- -------------
30 30 DOWN
DOWN 10, 20, 30, 60, 203
40 40 UP
UP 10, 110, 120, 201-300
50 50 DOWN
DOWN 10, 110, 120, 201-300
60 60 DOWN
DOWN 80
80 80 DOWN
DOWN 90
-----------------------------------
-----------
F10S4810AGGR1#show vlt role
VLT Role
----------
VLT Role:
Primary
System MAC address:
00:01:e8:8b:11:11
Primary Role Priority: 1
00:01:e8:8b:1d:3f
Local System Role Priority: 1
Local Unit Id: 0
-----------------------------------
-----------
F10S4810AGGR1#show run int po 40
!
description F10S4810ToR1
no ip address
switchport
no shutdown
-----------------------------------
-----------
F10S4810AGGR1#show run int po 90
!
description VLT_CHANNEL
no ip address
no shutdown
F10S4810AGGR1#
Local LAG Id Peer LAG Id Local
Status Peer Status Active VLANs
------------ ----------- --------
---- ----------- -------------
30 30 DOWN
DOWN 10, 20, 30, 60
40 40 UP
UP 110, 120, 201-300
50 50 DOWN
DOWN 10, 110, 120, 201-300
60 60 DOWN
DOWN 80
80 80 DOWN
DOWN 90
-----------------------------------
-----------
F10S4810AGGR2#show vlt role
VLT Role
----------
VLT Role:
Secondary
System MAC address:
00:01:e8:8b:11:11
00:01:e8:8b:1c:9a
Local System Role Priority:
65535
Local Unit Id: 1
-----------------------------------
---
F10S4810AGGR2#show vlt statistics
VLT Domain Statistics
HeartBeat Messages Sent:
452309
HeartBeat Messages Received:
452306
ICL Hello's Sent:
150769
ICL Hello's Received:
150768
Domain Mismatch Errors: 0
Version Mismatch Errors: 0
Config Mismatch Errors: 19
VLT MAC Statistics
----------------
L2 Info Pkts sent:36, L2 Mac-
sync Pkts Sent:325
L2 Info Pkts Rcvd:35, L2 Mac-
sync Pkts Rcvd:461
L2 Reg Request sent:1
L2 Reg Request rcvd:2
L2 Reg Response sent:1
L2 Reg Response rcvd:1

VLT Igmp-Snooping Not Enabled
VLT ARP Statistics
----------------
F10S4810AGGR2#
POD2-AGG1#show running-config | n
Current Configuration ...
! Version 1-0(0-298)
! Last configuration change at Fri May 10 13:16:28 2013 by default
! Startup-config last updated at Mon May 13 13:43:13 2013 by default
!
boot system stack-unit 0 primary system: B:
boot system stack-unit 0 secondary system: B:
boot system stack-unit 0 default system: A:
boot system gateway 10.11.200.254
!
redundancy auto-synchronize full
redundancy disable-auto-reboot stack-unit
!
redundancy disable-auto-reboot stack-unit 0
!
hardware watchdog
!
logging console errors
logging coredump stack-unit all
!
hostname POD2-AGG1
!
enable password 7 b125455cf679b208e79b910e85789edf
!
username admin password 7 1d28e9f33f99cf5c privilege 15
!
default vlan-id 4094
!
vlt domain 22
primary-priority 1
system-mac mac-address 00:00:22:22:22:22
unit-id 0
peer-routing
peer-routing-timeout 300
!
stack-unit 0 provision Z9000
!

interface fortyGigE 0/0
description CORE1
no ip address
!
port-channel-protocol LACP
port-channel 102 mode active
no shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
description CORE2
no ip address
!
no shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
no ip address
shutdown

!
no ip address
shutdown
!
no ip address
shutdown
!
description TOR1
no ip address
!
no shutdown
!
description TOR2
no ip address
!
no shutdown
!
no ip address
no shutdown
!
no ip address
no shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
no ip address

shutdown
!
no ip address
shutdown
!
no ip address
shutdown
!
description VLTi
no ip address
no shutdown
!
description VLTi
no ip address
no shutdown
!
interface ManagementEthernet 0/0
ip address 13.1.1.14/24
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
!
no shutdown
!
interface Loopback 0
ip address 11.1.1.5/32
no shutdown
!
description TOR
no ip address
switchport
rate-interval 30
no shutdown
!
no ip address
switchport
rate-interval 30
no shutdown

!
description VLTi
no ip address
channel-member fortyGigE 0/120,124
rate-interval 30
no shutdown
!
interface Vlan 1
!
vrrp-group 1
priority 110
no shutdown
!
interface Vlan 2
!
vrrp-group 1
priority 110
no shutdown
!
interface Vlan 3
!
vrrp-group 1
priority 110
no shutdown
!
interface Vlan 4
!
vrrp-group 1
priority 110
no shutdown
!
interface Vlan 5
!
vrrp-group 1
priority 110
no shutdown
!
interface Vlan 6
!
vrrp-group 1
priority 110
no shutdown
!
interface Vlan 7

!
vrrp-group 1
priority 110
no shutdown
!
interface Vlan 8
!
vrrp-group 1
priority 110
no shutdown
!
interface Vlan 100
ip address 10.100.1.5/16
tagged Port-channel 1,102
!
vrrp-group 1
authentication-type simple 7 00d8d7903b20fd98
priority 110
no shutdown
!
interface Vlan 2001
ip address 10.20.1.3/24
no shutdown
!
interface Vlan 4094
!untagged Port-channel 127
!
router ospf 1
log-adjacency-changes
graceful-restart mode planned-only
network 10.20.0.0/16 area 0
network 10.2.0.0/16 area 0
auto-cost reference-bandwidth 40000
maximum-paths 64
default-information originate
fast-converge 1
!
management route 0.0.0.0/0 13.1.1.1
!
logging 10.11.140.111
!
snmp-server community private rw
snmp-server community public ro
snmp-server enable traps bgp
snmp-server enable traps snmp authentication coldstart linkdown linkup
snmp-server enable traps vrrp
snmp-server enable traps lacp
snmp-server enable traps stp
snmp-server enable traps ecfm
snmp-server enable traps vlt
snmp-server enable traps xstp
snmp-server enable traps isis
snmp-server enable traps config
snmp-server enable traps envmon cam-utilization fan supply temperature
snmp-server enable traps eoam

snmp-server enable traps ecmp
snmp-server host 10.11.140.111 traps version 2c public udp-port 162
!
ntp server 10.11.140.111
!
clock timezone UTC 00 00
!
protocol lldp
advertise management-tlv system-name
no disable
!
script username admin /f10/flash/scripts/dfmsetup.sh
script username admin /usr/pkg/home/dfmcron.py
!
line console 0
exec-timeout 0 0
line vty 0
line vty 1
line vty 2
line vty 3
line vty 4
line vty 5
line vty 6
line vty 7
line vty 8
line vty 9
!
ecmp-group 1
link-bundle-monitor enable
!
reload-type normal-reload
!
end
POD2-AGG1#

24 VLT Troubleshooting
There are specific “show vlt” commands, which display the detailed output of VLT
parameters and its health status. The following CLI commands are useful to identify
the basic VLT issues.
FTOS#show vlt?
backup-link Backup-link information
brief Brief information on the VLT
counter Counter information on the VLT
detail Detail information on the VLT
inconsistency Inconsistent routes
mismatch Vlt config mismatch information on the VLT
role Role of VLT peer
statistics Statistics of the VLT
FTOS#show vlt brief
VLT Domain Brief
------------------
Domain ID: 10
Role: Primary
Role Priority: 100
ICL Link Status: Up
VLT Peer Status: Up
Local Unit Id: 0
Version: 6(1)
Local System MAC address: 00:01:e8:8b:24:2c
Remote System MAC address: 00:01:e8:8b:24:62
FTOS#show vlt detail
Local LAG Id Peer LAG Id Local Status Peer Status Active VLANs
------------ ----------- ------------ ----------- -------------
10 10 UP UP 2-4, 1000
100 100 UP UP 2-4, 1000
FTOS#show vlt role
VLT Role
----------
VLT Role: Primary

System MAC address: a0:10:10:aa:aa:aa
 Local System Role Priority: 100

 FTOS#show vlt backup-link
 VLT Backup Link
 -----------------
 Destination: 10.16.130.118
 Peer HeartBeat status: Up
 HeartBeat Timer Interval: 1
 HeartBeat Timeout: 3
 UDP Port: 34998
 HeartBeat Messages Sent: 14882
 HeartBeat Messages Received: 14825
VLT mismatch
VLT-10-PEER-1#show vlt mismatch
Domain
-----------
Parameters Local Peer
------------ ----------- ---------
Domain-ID 11 10
Unit-ID 1 1
System-Mac 00:a1:bb:bb:bb:bb a0:10:10:aa:aa:aa
For the VLT peering to be successful, the VLT ‘domain-id ‘should be configured with the
same value in both VLT nodes. Similarly ‘system-mac’ should be configured alike in both
the VLT nodes. However ‘unit-id’ should be unique in the VLT nodes. Any configuration
error on these parameter leads to VLT mismatch and do not form VLT peering. Also FTOS
logs with relevant VLT error messages:
Jun 25 09:24:45: %STKUNIT0-M:CP %VLTMGR-3-VLT_PEER_DOMAINID_MISMATCH:
VLT peer link will not be formed: Local VLT Domain-Id 11, Remote VLT Domain-Id 10 .
Jun 25 09:24:14: %STKUNIT0-M:CP %VLTMGR-6-VLT_HBEAT_DOWN: Heart beat link is
down (down).
Jun 25 09:24:14: %STKUNIT0-M:CP %VLTMGR-6-VLT_PEER_STATUS: Peer chassis is
down.
Jun 25 09:24:13: %STKUNIT0-M:CP %VLTMGR-6-VLT_ELECTION_ROLE: Chassis is
transitioning to standalone role.
Jun 25 09:19:40: %STKUNIT0-M:CP %VLTMGR-3-
VLT_PEER_SYSTEM_MAC_MISMATCH: VLT peer link will not be formed: Local VLT
System-Mac 00:a1:bb:bb:bb:bb, Remote VLT System-Mac a0:10:10:aa:aa:aa
Jun 25 09:17:45: %STKUNIT0-M:CP %VLTMGR-3-VLT_PEER_UNITID_IDENTICAL: VLT
peer link will not be formed: Local VLT UNIT-Id 1, Remote VLT UNIT-Id 1 .
VLT_PEER_CONFIG_MISMATCH_CLEARED: VLT peer link will be formed: No
mismatches found
Jun 25 06:16:18: %STKUNIT0-M:CP %VLTMGR-3-VLT_PEER_UNITID_MISMATCH: Local

VLT UNIT-Id 0, Remote VLT UNIT-Id <UNKNOWN> .
VLT_PEER_CONFIG_MISMATCH_CLEARED: No mismatches found
Jun 25 06:13:58: %STKUNIT0-M:CP %VLTMGR-3-VLT_PEER_UNITID_MISMATCH: Local
VLT UNIT-Id <UNKNOWN>, Remote VLT UNIT-Id 1 .
Jun 25 06:13:34: %STKUNIT0-M:CP %VLTMGR-3-VLT_PEER_SYSTEM_MAC_MISMATCH:
Local VLT System-Mac 00:00:00:00:00:00, Remote VLT System-Mac a0:aa:aa:aa:aa:aa
Jun 25 06:13:04: %STKUNIT0-M:CP %VLTMGR-6-VLT_HBEAT_UP: Heart beat link is up.
Jun 25 06:12:08: %STKUNIT0-M:CP %VLTMGR-6-VLT_PEER_STATUS: Peer chassis is up.
Jun 25 06:12:08: %STKUNIT0-M:CP %VLTMGR-6-VLT_ICL_UP: InterChassis Link is up.
Jun 25 06:12:08: %STKUNIT0-M:CP %VLTMGR-6-VLT_ELECTION_ROLE: Chassis is
transitioning to primary role.
‘VLT statistics’ indicates the probable error conditions for not forming VLT peering
between the VLT nodes. To find the delta values (incrementing or decrementing) from
initial state, clear the VLT statistics by ‘clear vlt statistics’. “vlt brief” indicates the status of
VLT peers, ICL Link and heartbeat between the VLT nodes along with the node roles and
priority.
VLT-100-PEER-1#show vlt statistics
VLT Domain Statistics
-----------------------
HeartBeat Messages Sent: 96882
HeartBeat Messages Received: 96885
ICL Hello's Sent: 32275
ICL Hello's Received: 32279
Domain Mismatch Errors: 0
Version Mismatch Errors: 0
Config Mismatch Errors: 0
VLT-10-PEER-1#show vlt brief
VLT Domain Brief
------------------
Domain ID: 10
Role: Primary
Role Priority: 100
ICL Link Status: Up
VLT Peer Status: Up
Local Unit Id: 0
Version: 6(1)
Remote System MAC address: 00:01:e8:8b:24:62

24.1 Debug outputs
VLT-PEER1#debug vlt ?
backup VLT Backup debugs
election VLT Election debugs
errors VLT Errors debugs
events VLT interested IFM events
hello VLT Hello debugs
VLT-PEER1#debug vlt backup
VLT backup debugging is ON
04:43:08 : mlagMgr: mlagMgrSendHeartBeatPkt
04:43:08 : mlagMgr: mlagMgrSendHeartBeatPacketOut
04:43:08 : mlagMgr: mlagMgrValidateHeartBeatPkt
04:43:08 : mlagMgr: mlagMgrValidateHeartBeatPkt:Valid heartbeat message. ack:1
<snip>
VLT-PEER1#debug vlt events
VLT IFM events debugging is ON
04:45:08 : mlagMgr: mlagElecHandleHelloInPrimState - Entered remote-role:2
<snip>
VLT-PEER1#debug vlt hello
VLT Hello debugging is ON
04:45:53 : mlagMgr: Received Hello PDU
04:45:53 : mlagMgr: mlagElecHandleHelloInPrimState - Entered remote-role:2
04:45:54 : mlagMgr: Received Hello processed successfully
04:45:55 : mlagMgr: mlagMgrIclSend 263 pduType:1
04:45:55 : mlagMgr: Hello packet sent!
04:45:55 : mlagMgr: Hello Timer started 3 secs

24.2 Syslog messages
If the VLT nodes are configured with mismatched domain-id’s
Nov 27 11:04:48: %STKUNIT0-M:CP %VLTMGR-3-VLT_PEER_DOMAINID_MISMATCH: VLT peer
link will not be formed: Local VLT Domain-Id 1, Remote VLT Domain-Id 1000 .
If the VLT unit-id is configured only in one of the peer then we get the following
syslog message:
Nov 27 11:00:43: %STKUNIT0-M:CP %VLTMGR-3-VLT_PEER_UNITID_MISMATCH: Local VLT
UNIT-Id <UNKNOWN>, Remote VLT UNIT-Id 0 .
If the mismatch is due to system MAC address
Nov 27 11:02:29: %STKUNIT0-M:CP %VLTMGR-3-VLT_PEER_SYSTEM_MAC_MISMATCH: Local
VLT System-Mac 00:00:00:00:00:00, Remote VLT System-Mac 00:00:00:00:00:11
When all the mismatches in the VLT domain are cleared, FTOS displays the following
syslog:
Nov 27 11:02:54: %STKUNIT0-M:CP %VLTMGR-3-VLT_PEER_CONFIG_MISMATCH_CLEARED: No
mismatches found
If the backup link heartbeat is disrupted due to backup link failure or mismatch in
backup link hello time
Nov 27 11:03:56: %STKUNIT0-M:CP %VLTMGR-6-VLT_HBEAT_DOWN: Heart beat link is down
(down).
When the ICL (VLTi) link connecting VLT peers is down:
Nov 27 11:04:08: %STKUNIT0-M:CP %VLTMGR-6-VLT_ICL_DOWN: InterChassis Link is down
(link down).
Nov 27 11:04:08: %STKUNIT0-M:CP %VLTMGR-6-VLT_PEER_STATUS: Peer chassis is down.
When the entire mismatches are cleared and all the links are UP, FTOS display the
following:
Nov 27 11:05:32: %STKUNIT0-M:CP %VLTMGR-6-VLT_HBEAT_UP: Heart beat link is up.
Nov 27 11:05:34: %STKUNIT0-M:CP %VLTMGR-3-VLT_PEER_CONFIG_MISMATCH_CLEARED:
VLT peer link will be formed: No mismatches found
Nov 27 11:05:34: %STKUNIT0-M:CP %VLTMGR-6-VLT_ICL_UP: InterChassis Link is up.
Nov 27 11:05:34: %STKUNIT0-M:CP %VLTMGR-6-VLT_PEER_STATUS: Peer chassis is up.

24.3 SNMP Traps
FTOS generates SNMP Trap for the various VLT mismatch objects. Also trap is
generated when the VLTi link bandwidth exceeds 80% of its maximum capacity. With
FTOS 9.2(0.0) release, multiple VLT MIB objects have been incorporated into FTOS for
easier manageability and troubleshooting
FTOS(conf)#snmp-server enable traps vlt
Active Fabric Manager (AFM) polling for the VLT MIB and SNMP Traps from the VLT
domain alerts the users for necessary corrective action.

25 FTOS Upgrade in VLT topology
FTOS upgrading procedure is almost the same as in other topology, except that certain
preliminary steps to be taken care before upgrading the VLT nodes.
1. Upgrade Boot-flash partition A or B with the new image using upgrade command on
both VLT peers.
On both VLT peers, set Primary boot parameter to boot system from upgraded
bootflash partition [A or B]
2. Reload or Power-cycle one of the VLT Peer [Preferably Secondary VLT peer]. With
FTOS 9.2(0.0) release, ‘graceful lacp’ feature ensures traffic flow through the other
port-channel member.
3. Wait for Peer 2 to come up; VLT adjacency will be established.
(Peer 2 – new image and Peer 1 - old image)
4. Peer 2 will start bringing up all the VLT LAG ports after a minute. Wait for all VLT LAG
ports to come up.
5. Use the command ‘show vlt detail’ to confirm that all VLT ports in the local chassis
are up.
6. After upgrade, save the running-configuration to memory using the "write memory"
CLI command.
7. Ensure both Nodes are forwarding traffic.
8. As soon as all the VLT ports are Up on Peer 2, repeat Step2 to Step5 for Peer1.

26 Conclusion
VLT offers efficient and optimal connectivity, cost effectiveness and serves as an
optimum solution for the conventional layer-2 domain. As illustrated in various
topologies, implementing VLT and migrating the existing legacy layer-2 networks to
the VLT is done without any complexity. VLT improves the high availability for any
network and enables seamless traffic flow during failures at any layer. VLT enables
convergence and network agility.
Dell Networking offers modular and stable FTOS for its high performance Ethernet
Switches combined with low power consumption and low latency for the traffic,
redefining the way the data center, enterprise, and campus networks are being built.
Dell Networking champions a green solution while maximizing customer satisfaction.

Dell VLT reference architecture v2 0

More Related Content

What's hot (20)

Similar to Dell VLT reference architecture v2 0 (20)

More from David Pasek (20)

Recently uploaded (20)

Dell VLT reference architecture v2 0