brocade-ip-fabric-bvd-published

BROCADE VALIDATED DESIGN
Brocade IP Fabric and Network Virtualization
with BGP EVPN
53-1004308-03
12 August 2016

© 2016, Brocade Communications Systems, Inc. All Rights Reserved.
Brocade, Brocade Assurance, the B-wing symbol, ClearLink, DCX, Fabric OS, HyperEdge, ICX, MLX, MyBrocade, OpenScript, VCS, VDX, Vplane, and
Vyatta are registered trademarks, and Fabric Vision is a trademark of Brocade Communications Systems, Inc., in the United States and/or in other
countries. Other brands, products, or service names mentioned may be trademarks of others.
Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment,
equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without
notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade
sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the
United States government.
The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to the accuracy of this
document or any loss, cost, liability, or damages arising from the information contained herein or the computer programs that accompany it.
The product described by this document may contain open source software covered by the GNU General Public License or other open source license
agreements. To find out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, and
obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.
Brocade IP Fabric and Network Virtualization with BGP EVPN
2 53-1004308-03

Contents
List of Figures...........................................................................................................................................................................................................................................................................5
Preface...........................................................................................................................................................................................................................................................................................7
Brocade Validated Designs..................................................................................................................................................................................................................................... 7
Purpose of This Document.....................................................................................................................................................................................................................................7
Target Audience..............................................................................................................................................................................................................................................................7
About the Authors.........................................................................................................................................................................................................................................................7
Document History........................................................................................................................................................................................................................................................8
About Brocade...............................................................................................................................................................................................................................................................8
Introduction ............................................................................................................................................................................................................................................................................... 9
Brocade IP Fabric Technology Overview..................................................................................................................................................................................................................11
Benefits...............................................................................................................................................................................................................................................................................11
Terminology......................................................................................................................................................................................................................................................................11
Functional Components of Brocade IP Fabric...........................................................................................................................................................................................12
Leaf-Spine Layer 3 Clos Topology (Two-Tier).................................................................................................................................................................................12
Optimized 5-Stage Layer 3 Clos Topology (Three-Tier)...........................................................................................................................................................14
Edge Services and Border Leafs............................................................................................................................................................................................................. 15
Brocade IP Fabric Underlay Routing..................................................................................................................................................................................................... 15
Network Virtualization with BGP EVPN................................................................................................................................................................................................................... 19
VXLAN Layer 2 Extension Using Flood and Learn..............................................................................................................................................................................20
BGP EVPN for VXLAN..........................................................................................................................................................................................................................................22
VTEP...................................................................................................................................................................................................................................................................... 23
Static Anycast Gateway................................................................................................................................................................................................................................23
Overlay Gateway..............................................................................................................................................................................................................................................23
BGP EVPN Control Plane..........................................................................................................................................................................................................................24
ARP Suppression............................................................................................................................................................................................................................................25
VLAN Scoping..................................................................................................................................................................................................................................................26
Conversational Learning..............................................................................................................................................................................................................................27
Integrated Routing and Bridging............................................................................................................................................................................................................ 28
Multitenancy....................................................................................................................................................................................................................................................... 29
Ingress Replication......................................................................................................................................................................................................................................... 30
vLAG Pair............................................................................................................................................................................................................................................................ 30
IP Fabric Validated Designs...........................................................................................................................................................................................................................................33
Pervasive eBGP.........................................................................................................................................................................................................................................................33
iBGP Within a PoD and eBGP Between PoDs........................................................................................................................................................................................34
Hardware and Software Matrix...........................................................................................................................................................................................................................35
Brocade IP Fabric Configuration...................................................................................................................................................................................................................... 35
Node ID Configuration................................................................................................................................................................................................................................. 35
IP Fabric Infrastructure Links.....................................................................................................................................................................................................................37
Loopback Interfaces.......................................................................................................................................................................................................................................37
Server-Facing Links.......................................................................................................................................................................................................................................38
Deployment Model-1: eBGP Underlay Configuration for Optimized 5-Stage Clos Fabric.................................................................................40
Deployment Model-1: eBGP Underlay Configuration for 3-Stage Clos Fabric.......................................................................................................... 46
Deployment Model-2: iBGP Underlay Configuration for Optimized 5-Stage Clos Fabric...................................................................................51
Network Virtualization with BGP EVPN........................................................................................................................................................................................................57
53-1004308-03 3

Overlay Gateway Configuration...............................................................................................................................................................................................................57
Deployment Model-1: eBGP EVPN Configuration for Optimized 5-Stage Clos Fabric........................................................................................57
Deployment Model-1: eBGP EVPN Configuration for 3-Stage Clos Fabric.................................................................................................................67
Deployment Model-2: iBGP EVPN Configuration for Optimized 5-Stage Clos Fabric........................................................................................72
Tenant Provisioning........................................................................................................................................................................................................................................83
vLAG Pair Configuration..............................................................................................................................................................................................................................87
Illustration Examples................................................................................................................................................................................................................................................ 87
Example-1: Tenant and L2 Extension Between Racks in a 3-Stage Clos Fabric........................................................................................................87
Example-2: Tenant and L2 Extension Between PoDs in an Optimized 5-Stage Clos Fabric...........................................................................101
Example-3: Tenant Extension Outside the Fabric........................................................................................................................................................................ 116
Example-4: VLAN Scoping at the ToR Level................................................................................................................................................................................126
Example-5: VLAN Scoping at the Port Level Within a ToR..................................................................................................................................................135
Example-6: Route Leaking for the Service VRF..........................................................................................................................................................................144
Design Considerations....................................................................................................................................................................................................................................................163
Appendix—Configuration of the Nodes.................................................................................................................................................................................................................167
vLAG Active/Active Pair Leaf............................................................................................................................................................................................................................167
Individual Non-Redundant Leaf.......................................................................................................................................................................................................................173
Spine Designated to Exchange Only Underlay Routes...................................................................................................................................................................... 177
Spine Designated to Exchange Both Underlay and Overlay Routes......................................................................................................................................... 179
Super-Spine Designated to Exchange Only Underlay Routes....................................................................................................................................................... 181
Super-Spine Designated to Exchange Both Underlay and Overlay Routes.........................................................................................................................183
Edge Leaf..................................................................................................................................................................................................................................................................... 185
References............................................................................................................................................................................................................................................................................. 189
4 53-1004308-03

List of Figures
Figure 1 on page 14—Leaf-Spine L3 Clos Topology
Figure 2 on page 15—Optimized 5-Stage L3 Clos Topology
Figure 3 on page 17—eBGP for Underlay
Figure 4 on page 18—iBGP for Underlay
Figure 5 on page 20—VTEPs and L2 Extension with Flood and Learn
Figure 6 on page 22—Routing Between VXLANs in a Flood-and-Learn Topology
Figure 7 on page 23—VTEPs and L2 Extension with the BGP EVPN Control Plane
Figure 8 on page 26—ARP Suppression
Figure 9 on page 27—VLAN Scoping at the Leaf Level
Figure 10 on page 27—VLAN Scoping at the Port Level Within a ToR
Figure 11 on page 28—Asymmetric IRB
Figure 12 on page 29—Symmetric IRB
Figure 13 on page 30—Multitenancy
Figure 14 on page 31—Active-Active vLAG
Figure 15 on page 33—Pervasive eBGP in an Optimized 5-Stage IP Fabric
Figure 16 on page 34—Pervasive eBGP in a 3-Stage IP Fabric
Figure 17 on page 34—iBGP Within a PoD and eBGP Between PoDs in an Optimized 5-Stage IP Fabric
Figure 18 on page 88—Tenant and Layer 2 Extension Between Two Racks
Figure 19 on page 102—Tenant and Layer 2 Extension Between Two PoDs Connected by Super-Spines
Figure 20 on page 117—Tenant Extension Outside the Fabric Through Edge Leafs
Figure 21 on page 126—VLAN Scoping at the ToR Level
Figure 22 on page 136—VLAN Scoping at the Port Level Within a ToR
Figure 23 on page 145—Services Provisioning on the Border Leaf
Figure 24 on page 145—Service VRF with Route Leaking on the Border Leaf
Figure 25 on page 146—Topology of the Service VRF with Route Leaking from Tenants
53-1004308-03 5

List of Figures
6 53-1004308-03

Preface
∙ Brocade Validated Designs.............................................................................................................................................................................................7
∙ Purpose of This Document.............................................................................................................................................................................................7
∙ Target Audience.....................................................................................................................................................................................................................7
∙ About the Authors.................................................................................................................................................................................................................7
∙ Document History................................................................................................................................................................................................................8
∙ About Brocade.......................................................................................................................................................................................................................8
Brocade Validated Designs
Brocade validated designs are reference architectures that are created and validated by Brocade engineers to address various customer
deployment scenarios and use cases. These validated designs provide a well-defined and standardized architecture for each
deployment scenario, and they incorporate a broad set of technologies and feature sets across Brocade's product range that address
customer-unique requirements. These designs are comprehensively validated end-to-end so that the design solutions and
configurations can be deployed more quickly, more reliably, and more predictably. Brocade validated designs are continuously validated
using a test automation framework to ensure that once a design has been validated, it remains validated on new software releases and
products.
Purpose of This Document
This Brocade validated design provides guidance for designing and implementing IP fabric in a data center network using Brocade
hardware and software. It details the Brocade reference architecture for deploying IP fabric and EVPN-based VXLAN overlay.
It should be noted that not all features such as automation practices, zero-touch provisioning, and monitoring of the Brocade IP fabric
are included in this document. Future versions of this document are planned to include these aspects of the Brocade IP fabric solution.
The design practices documented here follow the best-practice recommendations, but there are variations to the design that are
supported as well.
Target Audience
This document is written for Brocade systems engineers, partners, and customers who design, implement, and support data center
networks. This document is intended for experienced data center architects and engineers. This document assumes that the reader has a
good understanding of data center switching and routing features and of Multi-Protocol BGP/MPLS VPN[5] for understanding
multitenancy in VXLAN EVPN networks.
About the Authors
Krish Padmanabhan is a Principal Engineer on the IP SQA team at Brocade. Krish has extensive experience in the networking industry
and in particular the data-center switching and routing, with roles ranging from product development, testing, systems and solution
validation, to customer-centric testing. At Brocade, he is focused on developing and validating solution architectures that customers can
use in deployments. He holds a CCIE certification in Routing and Switching.
Anuj Dewangan is the lead Technical Marketing Engineer (TME) for Brocade's data center switching products. He holds a CCIE in
Routing and Switching and has several years of experience in the networking industry with roles in software development, solution
validation, and technical marketing. At Brocade, his focus is creating reference architectures, working with customers and account teams
53-1004308-03 7

to address their challenges with data center networks, and creating product and solution collateral. He speaks at industry events and has
authored several white papers on data center networking.
Poorani Arthanari is a Staff Engineer on the IP SQA team at Brocade. Poorani has extensive experience testing data center fabric and IP
routing technologies. She has been involved in validating solution architectures.
The authors would like to acknowledge the following Brocadians for their technical guidance in developing this validated design:
∙ Mangesh Shingane: Principal Engineer
∙ Syed Hasan Raza Naqvi: Technical Leader
∙ Venugopal Mundathaya: Senior Staff Engineer
Document History
Date Part Number Description
March 23, 2016 53-1004308-01 Initial release.
March 30, 2016 53-1004308-02 Minor formatting changes.
August 12, 2016 53-1004308-03 IP unnumbered interface support for 3-stage
fabric.
Illustration examples for:
∙ VLAN scoping at the ToR level and
within the ToR
∙ Route leaking with the service VRF on
the edge leaf
Additional design considerations.
About Brocade
Brocade® (NASDAQ: BRCD) networking solutions help the world's leading organizations transition smoothly to a world where
applications and information reside anywhere. This vision is designed to deliver key business benefits such as unmatched simplicity,
non-stop networking, application optimization, and investment protection.
Innovative Ethernet and storage networking solutions for data center, campus, and service provider networks help reduce complexity and
cost while enabling virtualization and cloud computing to increase business agility.
To help ensure a complete solution, Brocade partners with world-class IT companies and provides comprehensive education, support,
and professional services offerings (www.brocade.com).
Preface
8 53-1004308-03

Introduction
Based on the principles of the New IP, Brocade is building on the proven success of the VDX platform by expanding our cloud-
optimized network and network virtualization architectures to meet customer demand for higher levels of scale, agility, and operational
efficiency.
This document describes cloud-optimized network designs using Brocade IP fabrics for building data-center sites. The configurations
and design practices documented here are fully validated and conform to the Brocade IP fabric reference architectures. The intention of
this Brocade validated design document is to provide reference configurations and document best practices for building cloud-scale
data-center networks using Brocade VDX switches and Brocade IP fabric architectures.
This document describes the following architectures:
∙ Brocade IP fabric deployed in 3-stage and optimized 5-stage folded Clos topologies
∙ Brocade IP fabric with network virtualization using BGP EVPN deployed in 3-stage and optimized 5-stage folded Clos
topologies
We highly recommend reviewing the data-center fabric architectures described in the Brocade Data Center Fabric Architectures[7] white
paper for a detailed discussion on data-center architectures for building data-center sites.
53-1004308-03 9

Introduction
10 53-1004308-03

Brocade IP Fabric Technology Overview
∙ Benefits.......................................................................................................................................................................................................................................11
∙ Terminology............................................................................................................................................................................................................................. 11
∙ Functional Components of Brocade IP Fabric.................................................................................................................................................. 12
Brocade IP fabric provides a Layer 3 Clos deployment architecture for data center sites. With Brocade IP fabric, all links in the Clos
topology are Layer 3 links. The Brocade IP fabric includes the networking architecture; the protocols used to build the network; turnkey
automation features used to provision, manage, and monitor the networking infrastructure; and the hardware differentiation with Brocade
VDX switches. The following sections describe the validated design for data center sites with Brocade IP fabrics. Because the
infrastructure is built on IP, advantages like the following are leveraged: loop-free communication using industry-standard routing
protocols, ECMP, very high solution scale, and standards-based interoperability.
Benefits
Some of the key benefits of deploying data center sites with Brocade IP fabrics:
Highly scalable infrastructure—Because the Clos topology is built with IP protocols, the scale of the infrastructure is very high. The port
and rack scales are documented with descriptions of the Brocade IP fabric deployment topologies.
Standards-based and interoperable protocols—The Brocade IP fabric is built with industry-standard protocols like Border Gateway
Protocol (BGP) and Open Shortest Path First (OSPF). These protocols are well understood and provide a solid foundation for a highly
scalable solution. In addition, industry-standard overlay control- and data-plane protocols like BGP-EVPN and Virtual Extensible Local
Area Network (VXLAN) are used to extend the Layer 2 domain and extend tenancy domains by enabling Layer 2 communications and
VM mobility.
Active-active vLAG pairs—By supporting vLAG pairs on leaf switches, dual-homing of the networking endpoints is supported. This
provides higher redundancy. Also, because the links are active-active, vLAG pairs provide higher throughput to the endpoints. vLAG
pairs are supported for all 10-GbE, 40-GbE, and 100-GbE interface speeds, and up to 32 links can participate in a vLAG.
Support for unnumbered interfaces—Using Brocade Network OS support for IP unnumbered interfaces, only one IP address per switch
is required to configure the routing protocol peering. This support significantly reduces the planning and use of IP addresses, and it
simplifies operations.
Programmable automation—Brocade server-based automation provides support for common industry automation tools such as Python
Ansible, Puppet, and YANG model based REST and NETCONF APIs. The prepackaged PyNOS scripting library and editable
automation scripts execute predefined provisioning tasks, while allowing customization for addressing unique requirements to meet
technical or business objectives when the enterprise is ready.
Ecosystem integration—The Brocade IP fabric integrates with leading industry solutions and products like VMware vSphere, NSX, and
vRealize. Cloud orchestration and control are provided through OpenStack and OpenDaylight based Brocade SDN Controller support.
Terminology
Term Description
ARP Address Resolution Protocol
AS Autonomous System
ASN Autonomous System Number
BFD Bidirectional Forwarding Detection
BGP Border Gateway Protocol
53-1004308-03 11

Term Description
BUM Broadcast, Unknown unicast, and Multicast
DCI Data Center Interconnect
eBGP External Border Gateway Protocol
This refers to BGP peering between two nodes in two different autonomous systems.
ECMP Equal Cost Multi-Path
EVPN Ethernet Virtual Private Network
iBGP Internal Border Gateway Protocol
This refers to BGP peering between two nodes in the same autonomous system.
IP Internet Protocol
IRB Integrated Routing and Bridging
MAC Media Access Control
MP-BGP Multi-Protocol Border Gateway Protocol
MPLS Multi-Protocol Label Switching
ND Neighbor Discovery
NLRI Network Layer Reachability Information
PoD Point of Delivery
RD Route Distinguisher
RT Route Target
ToR Top of Rack switch
Also leaf or VTEP in an IP fabric context.
UDP User Datagram Protocol
vLAG Virtual Link Aggregation Group
VLAN Virtual Local Area Network
VM Virtual Machine
VNI VXLAN Network Identifier
VPN Virtual Private Network
VRF VPN Routing and Forwarding instance
An instance of the routing/forwarding table with a set of networks and hosts in a router. A router may have multiple such
instances isolated from each other. Also referred to as a tenant. In IP fabric, this may be localized to one VTEP/leaf or may be
spread across multiple VTEPs across the IP fabric and beyond the border leaf.
VTEP VXLAN Tunnel End Point
In IP fabric, leaf and VTEP are used interchangeably.
VXLAN Virtual Extensible Local Area Network
Functional Components of Brocade IP Fabric
Leaf-Spine Layer 3 Clos Topology (Two-Tier)
The leaf-spine topology has become the de facto standard for networking topologies when building medium- to large-scale data center
infrastructures. The leaf-spine topology is adapted from Clos telecommunications networks. The Brocade IP fabric within a PoD
resembles a two-tier or 3-stage folded Clos fabric. The two-tier leaf-spine topology is shown in Figure 1. The bottom layer of the IP
fabric has the leaf devices (top-of-rack switches), and the top layer has spines. The role of the leaf is to provide connectivity to the
12 53-1004308-03

endpoints in the data center network. These endpoints include compute servers and storage devices as well as other networking devices
like routers, switches, load balancers, firewalls, and any other physical or virtual networking endpoints. Because all endpoints connect
only to the leaf, policy enforcement, including security, traffic-path selection, QoS marking, traffic policing, and shaping, is implemented
at the leaf.
More importantly, the leafs act as the anycast gateways for the server segments to facilitate mobility with the VXLAN overlay.
The role of the spine is to provide connectivity between leafs. The major role of the spine is to participate in the control-plane and data-
plane operations for traffic forwarding between leafs. The spine devices serve two purposes: BGP control plane (route reflectors for leaf
or eBGP peering with leaf) and IP forwarding based on the outer IP header in the underlay network. Since there are no network
endpoints connected to the spine, tenant VRFs or VXLAN segments are not created on spines. Their routing table size requirements are
also very light to accommodate just the underlay reachability. Note that all spine devices need not act as BGP route reflectors; only
selected spines in the spine layer can act as BGP route reflectors in the overlay design. More details are provided in the "BGP EVPN
Control Plane" section of the "Network Virtualization with BGP EVPN" chapter.
As a design principle, the following requirements apply to the leaf-spine topology:
∙ Each leaf connects to all spines in the network through 40-GbE links.
∙ Spines are not interconnected with each other.
∙ Leafs are not interconnected with each other for data-plane purposes. (The leafs may be interconnected for control-plane
operations such as forming a server-facing vLAG.)
∙ The network endpoints do not connect to the spines.
This type of topology has the predictable latency and also provides the ECMP forwarding in the underlay network. The number of hops
between two leaf devices is always two within the fabric. This topology also enables easier scale out in the horizontal direction as the data
center expands and is limited by the port density and bandwidth supported by the spine devices.
This validated design recommends the same hardware in the spine layer. Mixing different hardware is not recommended.
IP Fabric Infrastructure Links
All fabric nodes—leafs, spines, and super-spines—are interconnected with Layer 3 interfaces. In the validated design,
∙ 40-GbE links are used between the fabric nodes.
∙ All these links are configured as Layer 3 interfaces with /31 IPv4 address. For a simple 3-stage fabric, IP unnumbered
interfaces can be used. We do not recommend a mix of unnumbered and numbered interfaces within a fabric. Also, for a
5-stage IP fabric, numbered interfaces are highly recommended.
∙ The MTU for these links is set to jumbo MTU. This is a requirement to handle the VXLAN encapsulation of Ethernet frames.
Server-Facing Links
The server-facing or access links are on the leaf nodes. In the validated design,
∙ 10-GbE links are used for server-facing VLANs.
∙ These links are configured as Layer 2 trunks with associated VLANs.
∙ The MTU for these links is set to the default: 1500 bytes.
∙ Spanning tree is disabled.1
1 Spanning tree must be enabled if there are Layer 2 switches/bridges between a leaf and servers.
53-1004308-03 13

FIGURE 1 Leaf-Spine L3 Clos Topology
Optimized 5-Stage Layer 3 Clos Topology (Three-Tier)
Multiple PoDs based on leaf-spine topologies can be connected for higher scale in an optimized 5-stage folded Clos (three-tier)
topology. This topology adds a new tier to the network, known as a super-spine. This architecture is recommended for interconnecting
several EVPN VXLAN PoDs. Super-spines function similar to spines: BGP control-plane and IP forwarding based on the outer IP
header in the underlay network. No endpoints are connected to the super-spine. Figure 2 shows four super-spine switches connecting
the spine switches across multiple data center PoDs.
The connection between the spines and the super-spines follows the Clos principles:
∙ Each spine connects to all super-spines in the network.
∙ Neither spines nor super-spines are interconnected with each other.
14 53-1004308-03

FIGURE 2 Optimized 5-Stage L3 Clos Topology
Edge Services and Border Leafs
For two-tier and three-tier data center topologies, the role of the border leaf in the network is to provide external connectivity to the data
center site. In addition, since all traffic enters and exits the data center through the border leaf switches, they present the ideal location in
the network to connect network services like firewalls, load balancers, and edge VPN routers. The border leaf switches connect to the
WAN edge devices in the network to provide external connectivity to the data center site. As a design principle, two border leaf switches
are recommended for redundancy. The WAN edge devices provide the interfaces to the Internet and DCI solutions. For DCI, these
devices function as the Provide Edge (PE) routers, enabling connections to other data center sites through WAN technologies like
Multiprotocol Label Switching (MPLS) VPN and Virtual Private LAN Services (VPLS). The Brocade validated design for DCI solutions is
discussed in a separate validated design document.
There are several ways that the border leafs connect to the data center site. In three-tier (super-spine) architectures, the border leafs are
typically connected to the super-spines as depicted in Figure 2. In two-tier topologies, the border leafs are connected to the spines as
depicted in Figure 1. Certain topologies may use the spine as border leafs (known as a border spine), overloading two functions into one.
This topology adds additional forwarding requirements to spines—they need to be aware of the tenants, VNIs, and VXLAN tunnel
encapsulation and de-encapsulation functions.
Brocade IP Fabric Underlay Routing
IP fabric collectively refers to the following:
∙ IPv4 network address assignments to the links connecting the nodes in the fabric: spines, leafs, super-spines, and border leafs.
∙ Control-plane protocol used for reachability between the nodes. A smaller scale topology might benefit from a link-state
protocol such as OSPF. Large scale topologies, however, typically use BGP. Brocade validated design recommends BGP as
the protocol for underlay network reachability.
∙ Resiliency feature such as BFD.
53-1004308-03 15

There are several underlay deployment options. In the validated design, we recommend two deployment models based on how the BGP
protocol is deployed in the IP fabric:
∙ eBGP for Underlay—eBGP peering between each tier of nodes: between the leaf and the spine; between the spine and the
super-spine; and between the super-spine and the border leaf.
∙ iBGP for Underlay—iBGP peering between the leaf and the spine within the PoD and spines as BGP route reflectors. eBGP
peering between the PoDs through the super-spine layer for inter-PoD reachability.
eBGP for Underlay
This deployment model refers to the usage of eBGP peering between the leaf and the spine in the fabric. In this model, each leaf node is
assigned its own autonomous system (AS) number. The other nodes are grouped based on their role in the fabric, and each of these
groups is assigned a separate AS number, as shown in Figure 3. Using eBGP in an IP fabric is simple and also provides the ability to
apply BGP policies for traffic engineering on a per-leaf or per-rack basis since each leaf or rack in a PoD is assigned a unique AS
number. Private AS numbers should be used in the fabric. One design consideration for the AS number assignment is that a 2-byte AS
number provides a maximum of 1023 private AS numbers (ASN 64512 to ASN 65534); if the IP fabric is larger than 1023 devices, we
recommend using 4-byte private AS numbers (ASN 4,200,000,000 to 4,294,967,294).
∙ Each leaf in a PoD is assigned its own AS number.
∙ All spines inside a PoD belong to one AS.
∙ All super-spines are configured in one AS.
∙ Edge or border leafs belong to a separate AS.
∙ Each leaf peers with all spines using eBGP.
∙ Each spine peers with all super-spines using eBGP.
∙ There is no eBGP peering between leafs.
∙ There is no eBGP peering between spines.
∙ There is no eBGP peering between super-spines.
16 53-1004308-03

FIGURE 3 eBGP for Underlay
iBGP for Underlay
In this deployment model, each PoD and edge services PoD is configured with a unique AS number, as shown in Figure 4. The spines
and leafs in a PoD are configured with the same AS number. The iBGP design is different than the eBGP design because iBGP must be
fully meshed with all BGP-enabled devices in an IP fabric. In order to avoid the complexities of a full mesh, route reflectors must be used
in the fabric. iBGP peering is between the spine and the leaf in a PoD, and all spines in a PoD act as BGP route reflectors to the leafs for
the underlay.
eBGP is used to peer between spines and super-spines. The super-spine layer is configured with a unique AS number; all super-spines
use the same AS number.
When an EVPN Address-Family is enabled for overlay,
∙ Two spines in each PoD are enabled with EVPN AFI, and they act as the RR to the leaf.
∙ Leafs exchange EVPN routes to the spine RRs.
∙ These spines also exchange EVPN routes with super-spines.
∙ Edge leafs exchange EVPN routes with super-spines.
53-1004308-03 17

FIGURE 4 iBGP for Underlay
18 53-1004308-03

Network Virtualization with BGP EVPN
∙ VXLAN Layer 2 Extension Using Flood and Learn......................................................................................................................................20
∙ BGP EVPN for VXLAN.................................................................................................................................................................................................22
Network virtualization is the process of creating virtual, logical networks on physical infrastructures. With network virtualization, multiple
physical networks can be consolidated to form a logical network. Conversely, a physical network can be segregated to form multiple
virtual networks. Virtual networks are created through a combination of hardware and software elements spanning the networking,
storage, and computing infrastructure. Network virtualization solutions leverage the benefits of software in terms of agility and
programmability, along with the performance acceleration and scale of application-specific hardware.
Virtual Extensible LAN (VXLAN) is an overlay technology that provides Layer 2 connectivity for workloads residing across the data
center network. VXLAN creates a logical network overlay on top of physical networks, extending Layer 2 domains across Layer 3
boundaries. VXLAN provides decoupling of the virtual topology provided by the VXLAN tunnels from the physical topology of the
network. It leverages Layer 3 benefits in the underlay, such as load balancing on redundant links, which leads to higher network
utilization. In addition, VXLAN provides a large number of logical network segments, allowing for large-scale multitenancy in the network.
VXLAN is based on the IETF RFC 7348 standard. VXLAN has a 24-bit Virtual Network ID (VNID) space, which allows for 16 million
logical networks compared to a traditional VLAN, which supports a maximum of 4096 logical segments. VXLAN eliminates the need for
Spanning Tree Protocol (STP) in the data center network, and it provides increased scalability and improved resiliency. VXLAN has
become the de facto standard for overlays that are terminated on physical switches or virtual network elements.
The traditional Layer 2 extension mechanisms using VXLAN rely on "Flood and Learn" mechanisms. These mechanisms are very
inefficient, delaying MAC address convergence and resulting in unnecessary flooding. Also, in a data center environment with VXLAN-
based Layer 2 extension mechanisms, a Layer 2 domain and an associated subnet might exist across multiple racks and even across all
racks in a data center site. With traditional underlay routing mechanisms, routed traffic destined to a VM or a host belonging to the
subnet follows an inefficient path in the network, because the network infrastructure is aware only of the existence of the distributed
Layer 3 subnet, but it is not aware of the exact location of the hosts behind a leaf switch.
With Brocade BGP-EVPN network virtualization, network virtualization is achieved by creating a VXLAN-based overlay network.
Brocade BGP-EVPN network virtualization leverages BGP-EVPN to provide a control plane for the virtual overlay network. BGP-EVPN
enables control-plane learning for end hosts behind remote VXLAN tunnel end points (VTEPs). This learning includes reachability for
Layer 2 MAC addresses and Layer 3 host routes.
Some key features and benefits of Brocade BGP-EVPN network virtualization are summarized as follows:
Active-active vLAG pairs—vLAG pairs for a multiswitch port channel for dual homing of network endpoints are supported at the leaf.
Both switches in the vLAG pair participate in the BGP-EVPN operations and are capable of actively forwarding traffic.
Static anycast gateway—With static anycast gateway technology, each leaf is assigned the same default gateway IP and MAC addresses
for all connected subnets. This ensures that local traffic is terminated and routed at Layer 3 at the leaf. This also eliminates any
suboptimal inefficiencies found with centralized gateways. All leafs are simultaneously active forwarders for all default traffic for which
they are enabled. Also, because the static anycast gateway does not rely on any control-plane protocol, it can scale to large deployments.
Efficient VXLAN routing—With the existence of active-active vLAG pairs and the static anycast gateway, all traffic is routed and switched
at the leaf. Routed traffic from the network endpoints is terminated in the leaf and is then encapsulated in the VXLAN header to be sent
to the remote site. Similarly, traffic from the remote leaf node is VXLAN-encapsulated and must be decapsulated and routed to the
destination. This VXLAN routing operation in to and out of the tunnel on the leaf switches is enabled in the Brocade VDX 6740 and
6940 platform ASICs. VXLAN routing performed in a single pass is more efficient than competitive ASICs.
Data-plane IP and MAC learning—With IP host routes and MAC addresses learned from the data plane and advertised with BGP-EVPN,
the leaf switches are aware of the reachability of the hosts in the network. Any traffic destined to the hosts takes the most efficient route
in the network.
53-1004308-03 19

Layer 2 and Layer 3 multitenancy—BGP-EVPN provides the control plane for VRF routing and for Layer 2 VXLAN extension. BGP-
EVPN enables a multitenant infrastructure and extends it across the data center to enable traffic isolation between the Layer 2 and Layer
3 domains, while providing efficient routing and switching between the tenant endpoints.
Dynamic tunnel discovery—With BGP-EVPN, the remote VTEPs are automatically discovered. The resulting VXLAN tunnels are also
automatically created. This significantly reduces operational expense (OpEx) and eliminates errors in configuration.
ARP/ND suppression—The BGP-EVPN EVI leafs discover remote IP and MAC addresses and use this information to populate their
local ARP tables. Using these entries, the leaf switches respond to any local ARP queries. This eliminates the need for flooding ARP
requests in the network infrastructure.
Conversational ARP/ND learning—Conversational ARP/ND reduces the number of cached ARP/ND entries by programming only
active flows into the forwarding plane. This helps to optimize utilization of hardware resources. In many scenarios, there are software
requirements for ARP and ND entries beyond the hardware capacity. Conversational ARP/ND limits storage-in-hardware to active
ARP/ND entries; aged-out entries are deleted automatically.
VM mobility support—If a VM moves behind a leaf switch, with data-plane learning, the leaf switch discovers the VM and learns its
addressing information. It advertises the reachability to its peers, and when the peers receive the updated information for the reachability
of the VM, they update their forwarding tables accordingly. BGP-EVPN-assisted VM mobility leads to faster convergence in the
network.
Open standards and interoperability—BGP-EVPN is based on the open standard protocol and is interoperable with implementations
from other vendors. This allows the BGP-EVPN-based solution to fit seamlessly in a multivendor environment.
VXLAN Layer 2 Extension Using Flood and Learn
Let's consider the simple topology shown in Figure 5, which represents VXLAN extension, to understand how VXLAN flood and learn
works before going into the details of control-based VXLAN using BGP EVPN and the various network functions that the EVPN control
plane enables.
FIGURE 5 VTEPs and L2 Extension with Flood and Learn
20 53-1004308-03

VXLAN tunnel end point (VTEP) may be implemented in hardware (leaf or ToR switch) or in virtualized environments. Each VTEP has a
unique IP address and MAC address. Each VTEP can reach other VTEPs over the underlay IP network.
Each VTEP has its own end host/server segment connected to it. In this topology, all hosts belong to one Layer 2 broadcast domain or,
in simple terms, one VLAN and one IP subnet. The local VLAN numbers may be different in each VTEP, but they are bound to one VNI
number, which is common on all VTEPs. So for all practical purposes, the LAN segment is now identified by a VXLAN VNI, and the
VLAN numbers are only locally significant.
The logical dashed lines shown inside the IP network between the VTEPs represent the head-end or ingress replication paths. This is
used to send what is known as the BUM traffic: Broadcast, Unknown Unicast, and Multicast frames on the Layer 2 segment. The VTEP
unicasts these packets to all other VTEPS connected to a VXLAN segment. This may require additional configuration or provisioning of
tunnels on each VTEP device to all other devices.
Let's consider that H1 wants to communicate with H2:
∙ H1 sends an ARP request.
∙ VTEP-A learns H1 as a local MAC and also maps this host to the VNI, and because the packet is a broadcast packet, it is
encapsulated into the VXLAN packet and replicated; it is then unicast to each of the remote VTEPs participating in this VNI
segment. The outer-src-ip is set to 10.10.10.1, and the outer-dst-ip is the remote VTEP IP.
∙ This packet is sent to every VTEP.
∙ VTEP-B and VTEP-C decapsulate the packet and flood it into their local VXLAN network.
∙ They also learn three pieces of information: the source-ip of VTEP-A, the inner-src-mac of H1, and the VNI. This creates an
L2-MAC-to-VTEP-IP binding: {mac H1, VTEP-ip 10.10.10.1, VNI 10}.
∙ When H2 responds to the ARP request, the packet is unicast to H1. This packet is encapsulated in a VXLAN packet by VTEP-B
and sent as a unicast IP packet based on its routing table:
– outer-ip header - dst: 10.10.10.1, src 10.10.10.2
∙ VTEP-A decapsulates the packet and sends it to H1. It also creates an L2-MAC-to-VTEP-IP binding: {MAC H2, VTEP-ip
10.10.10.2, VNI 10}
∙ Now the communication between H1 and H2 will be unicast. VTEP-A and VTEP-B now know sufficient information to
encapsulate the packets between them. The multicast tree is not used.
When the hosts are in different subnets, we need a Layer 3 gateway in the network to connect to all VNI segments. As seen in Figure 6,
VTEP-C is configured with all VNI numbers in the network and acts as the router or gateway between these VNI segments (see the blue
and red dotted arrows routing between VLAN10 and VLAN20). When hosts send ARP messages for the gateway in their respective
VLANs, VTEP-C will respond.
53-1004308-03 21

FIGURE 6 Routing Between VXLANs in a Flood-and-Learn Topology
For first-hop router redundancy, multiple VTEPs may be configured with all VNIs, and they may run an FHRP protocol between them.
BGP EVPN for VXLAN
As we have seen in the VXLAN flood and learn case, the MAC learning is data frame-driven and flooding of broadcast or unknown
unicast frames depends on ingress replication by VTEPs in the network.
With the BGP EVPN control plane, the MAC learning happens via BGP similar to IPv4/IPv6 route learning in a Layer 3 network. This
reduces flooding in the underlay network except for remarkably silent hosts. This control-plane-based MAC learning enables several
additional functions with BGP as the unified control plane for both Layer 2 and Layer 3 forwarding in the overlay network.
In Figure 7, each VTEP, being a BGP speaker, advertises the MAC and IP addresses of its local hosts to other VTEPs using the BGP
EVPN control plane. A BGP route-reflector may be used for distribution of this information to the VTEPs. Both VTEP discovery and
MAC/IP or MAC/IPv6 host learning happen through the control plane.
Since IPv4/IPv6 addresses are also exchanged in the control plane, each VTEP may act as a gateway for the VNI subnets configured on
it. A centralized Layer 3 gateway is not required. This feature is also referred to as distributed gateway. Also, since each VTEP is aware of
MAC/IP or MAC/IPv6 host bindings, ARP requests need not be flooded between the VTEPS. The VTEP may respond to the ARP
requests on behalf of the target host, if the host address has already been learned. This is referred to as ARP/ND suppression in the
fabric.
22 53-1004308-03

FIGURE 7 VTEPs and L2 Extension with the BGP EVPN Control Plane
BGP EVPN control-plane-based learning allows more flexibility to control the information flow between the VTEPs. It also enables
multitenancy using VRFs similar to MPLS-VPN. Each VTEP may host several tenants and each tenant with a set of VXLAN segments.
Depending on the interest, other VTEPs may import the tenant-specific information. This way both Layer 2 and Layer 3 extensions can
be provisioned on a tenant basis.
BUM traffic may be accommodated either with ingress replication or a multicast tree. Since VTEP discovery also happens through the
control plane, setting up ingress replication does not require additional provisioning or configuration about remote VTEPs. Brocade
EVPN implementation supports ingress replication.
VTEP
In IP fabric, the leaf and border leaf act as VTEPs. Note that only one VTEP is allowed per device. Every VTEP has an overlay interface,
which identifies the VTEP IP address. The VTEP info is exchanged, and remote VTEPs are discovered over BGP EVPN.
Static Anycast Gateway
Each leaf or VTEP has a set of server-facing VLANs that are mapped to VXLAN segments by a VNI number. These VLAN segments
have an associated VE interface (a Layer 3 interface for the VLAN). Each tenant VLAN has anycast gateway IPv4/IPv6 addresses and
associated anycast gateway MAC addresses. These gateway IP/IPv6 addresses and gateway MAC address are consistent for the VLAN
segments shared on all leafs in the fabric.
Overlay Gateway
Each VTEP or leaf is configured with an overlay gateway. This defines the VTEP IP address, which is used as the source IP when
encapsulating packets and is used as the next-hop IP in the EVPN NLRIs. In this validated design, we are using an IPv4 underlay; hence
the overlay interface is associated with the IPv4 address of a loopback interface on the leaf.
53-1004308-03 23

BGP EVPN Control Plane
The BGP EVPN control plane is used for VTEP discovery to learn MAC/IP routes from other VTEPs. The exchange of this information
takes place using EVPN NLRIs. The NLRI uses the existing AFI of 25 (L2VPN). IANA has assigned BGP EVPNs a SAFI value of 70.
The NLRI also carries a tunnel encapsulation attribute. For IP fabric using VXLAN encapsulation, the attribute is set to VXLAN.
In the leaf-spine topology (3-stage Clos or 5-stage Clos), all leafs and border leafs should be enabled with the BGP EVPN Address-
Family to exchange EVPN routes (NLRI) and participate in VTEP discovery. Spine and super-spines do not participate in the VTEP
functionality. However, selected spines in the spine layer should be enabled with the BGP EVPN Address-Family, and all leafs including
border leafs must be peered with the spines who have the BGP EVPN Address-Family enabled.
In the deployment model where eBGP is used, a minimum of two spines in the PoD should be enabled with the EVPN Address-Family.
Note that all spines participate in the eBGP underlay, but only a few designated spines participate in the EVPN.
In the deployment model where iBGP is used, two spines are selected as route-reflectors for the EVPN Address-Family, and each
VTEP leaf has two iBGP neighbors that are the two spine BGP route reflectors. Each spine BGP route reflector has all VTEP leaf nodes
as route-reflector clients and reflects EVPN routes for the VTEP leaf nodes.
In the 5-stage Clos topology, a minimum of two super-spines should be enabled with the EVPN Address-Family, and only the spines
that are enabled with EVPN are peered with these super-spines. More detailed design is discussed in the "Network Virtualization with
BGP EVPN" section of the "IP Fabric Validated Designs" chapter.
EVPN Route Types
EVPN uses different route types to carry various network-layer reachability information. The following are the well-known route types
defined in BGP EVPN:
∙ Route Type-1—Ethernet Auto Discovery. This route is used for remote VTEP discovery and association to the VLAN/VNI.
∙ Route Type-2: MAC/IP advertisement route:
– MAC-only route that carries {MAC address of the host, L2VNI of the VXLAN segment}. This route carries only the Layer 2
information of a host. Whenever a VTEP learns a MAC from its server-facing subnets, it advertises this route into BGP.
– MAC/IP route that carries {MAC address of the host, IPv4/IPv6 address of the host, L2VNI of the VXLAN segment,
L3VNI of the tenant VRF of the host}. This route carries both the Layer 2 and Layer 3 information of the hosts. This route
is advertised by the VTEP when it learns the IPv4/IPv6 host addresses via ARP or ND from the server-facing subnets.
This information enables ARP/ND suppression on other VTEPs.
∙ Route Type-3—Inclusive Multicast Ethernet Tag route. This route is required for sending BUM traffic to all VTEPs interested for
a given bridge domain or VXLAN segment.
∙ Route Type-4—Ethernet Segment route is used for multi-homing of server vlan segments to two ToRs. Only VLAG based
multi-homing is supported.
∙ Route Type-5— IPv4/IPv6 prefix advertisement route {IPv4/IPv6 route, L3VNI, Router-MAC}. This route is advertised for
every Layer 3 server-facing subnet behind a VTEP or external routes.
Tunnel Attribute
Extended community type 0x3, sub-type 0x0c, and tunnel encapsulation type 0x8 (VXLAN). This is included with all EVPN routes.
Layer 3 VNI or Tenant VRF
Each tenant VRF is configured with a unique Layer 3 VNI. This is required for inter-subnet routing. This VNI must be the same for a
tenant VRF on all VTEPs including the border leaf. Both Type-2 and Type-5 routes carry this Layer 3 VNI.
24 53-1004308-03

Router-MAC Extended Community
Extended community type EVPN (0x06) and sub-type 0x03.
The router-mac is the MAC address of the VTEP advertising a route. This is also required along with the Layer 3 VNI for inter-subnet
routing as explained in the "Integrated Routing and Bridging (IRB)" section of this chapter, and it is carried in both Type-2 MAC/IP routes
and Type-5 prefix routes. In the data plane, this MAC address is used as the inner destination MAC address when a packet is routed.
MAC-Mobility Attribute
Extended community type EVPN (0x06) and sub-type 0x00. Carries a 32-bit sequence number.
This enables MAC or station moves between the VTEPs. When a MAC moves, for example, from VTEP-1 to VTEP-2, VTEP-2
advertises a MAC (or MAC/IP) route with a higher sequence number. This update triggers a best-path calculation on other VTEPs,
thereby detecting the host move to VTEP-2.
ARP Suppression
Control-plane distribution of MAC/IP addresses enables ARP suppression in the fabric for Layer 2 extensions between racks. A portion
of the fabric is shown in Figure 8 to illustrate the ARP suppression functionality in the fabric.
When the hosts come up, they typically ARP for the gateway IP that is hosted by leafs. Let's consider the case where H2 ARPs for the
gateway address. Note that both leafs have the same anycast gateway address for the host VXLAN segment.
∙ Leaf2 learns the MAC/IP (or ARP) binding for H2.
∙ Leaf2 will advertise the MAC/IP route into the BGP EVPN Address-Family.
∙ Leaf1 will learn this route and populate it in its MAC/IP binding table.
∙ H1 sends an ARP request to H2. Leaf1 will respond on behalf of H2.
∙ Extending the same information flow for H1, when Leaf2 learns H1's MAC/IP route, it will respond to ARP requests on behalf of
H1.
Compared to the data-plane-based learning in Layer 2 extension technologies such as VPLS or VXLAN flood and learn, where ARP
traffic is also sent over an overlay network, VXLAN EVPN significantly reduces ARP/ND flooding in the fabric.
53-1004308-03 25

FIGURE 8 ARP Suppression
VLAN Scoping
As discussed earlier, in VXLAN networks, each VLAN is mapped to a VNI number of a VXLAN segment. This provides an interesting
option to break the 4K limit of the 802.1Q VLAN space. The VLAN tag (or c-tag) on the wire or the port VLAN membership may be
locally scoped or locally significant at the leaf level or at the port level within a leaf.
VLAN Scoping at the Leaf Level
In this case, the VLANs are scoped at the leaf or ToR level. Refer to Figure 9.
In this example, VLAN 10 is mapped to VNI 10 on Leaf1, and VLAN 20 is mapped to VNI 10 on Leaf2. By mapping to the same VNI, the
two VLAN segments (VLAN 10 and VLAN 20) are on the same bridge domain. With this mapping, hosts on these VLANs have Layer 2
extension between them, and they belong to one VXLAN segment identified by the VNI 10.
26 53-1004308-03

FIGURE 9 VLAN Scoping at the Leaf Level
VLAN Scoping at the Port Level Within a Leaf
VLAN scoping at the port level can be accomplished using the Virtual-Fabric feature on Brocade switches. The Virtual-Fabric feature
basically abstracts a VLAN or bridge domain and decouples the VLAN tag (or c-tag) on the wire.
Refer to Figure 10. In this example, Port1, VLAN tag 10, and Port2, VLAN tag 20, are mapped to a VLAN 5001, and VLAN 5001 is
mapped to VNI 5001. With this mapping, the hosts H1 (VLAN 10), H2 (VLAN 20), and H3 (VLAN 501) are bound to one VXLAN
segment identified by the VNI 5001.
FIGURE 10 VLAN Scoping at the Port Level Within a ToR
Conversational Learning
Conversational learning helps conserve the hardware forwarding table by programming only those ARP/ND or MAC entries for which
there are active conversations or traffic flows. With this feature, the control plane may hold more host entries than what the hardware
53-1004308-03 27

table can support. When there is sufficient space in hardware, all host entries are programmed. When there is no space, conversational
learning kicks in and starts aging out the inactive entries. Note that the host subnets are inserted into the hardware (LPM table) regardless
of the activity. The host entries are inserted in the hardware (/32 IPv4 or /128 IPv6 host route table) based on the traffic.
Integrated Routing and Bridging
With the anycast gateway function, each VTEP or leaf acts as an Integrated Routing and Bridging (IRB) device providing Layer 2
extension as well Layer 3 routing between the VXLAN segments in a tenant. Note that the tenant may span multiple leafs. There are two
variations of IRB implementation in the IP fabric: asymmetric IRB and symmetric IRB.
Asymmetric IRB
FIGURE 11 Asymmetric IRB
In Figure 11, a tenant, SALES, is provisioned in the fabric with two VNI segments, VNI 10 and VNI 20. Leaf1 has servers connected to it
on VNI 10 only. Yet it is provisioned with both VXLAN segment VNI 10 and VNI 20. If H1 in VNI 10 needs to communicate with H3 in
VNI 20, Leaf1 routes the packet first between the segments and then bridges the packet on VNI 20 and the packet is sent on the
overlay. Leaf2 will decapsulate the VXLAN headers and send the packet to H3.
Essentially, the ingress VTEP both routes and bridges the packet; this method is referred as asymmetric IRB. This also means that every
VTEP must be configured with all VXLAN segments in a given tenant regardless of any local servers connected to the VNI segment.
Symmetric IRB
Figure 12 depicts symmetric IRB. Here, every tenant is assigned a Layer 3 VNI. This is analogous to a Layer 3 routing interface between
two switches. This VNI must be the same for a given tenant on all leafs where it is provisioned.
The MAC/IP host routes are advertised by the VTEP with the L2 VNI as well as an L3 VNI and the router-mac address of the VTEP.
When a packet is routed over the L3 VNI, the dst-mac of the inner Ethernet payload is set to the router-mac of the remote VTEP. In
28 53-1004308-03

Figure 12, routing from H1 to H3 always occurs over this L3 VNI. That is, both leaf devices route the packet once: by the ingress leaf
from the server VLAN/VNI to the L3 VNI and by the egress leaf from the L3 VNI to the server VLAN/VNI.
A significant advantage of this method is that all VNIs of a given tenant need not be created on all leafs. They are created only when
there is server connectivity to those VNIs. In Figure 12, Leaf1 is not configured with VNI 20. Also note that on Leaf2, even though VNI 10
is present, a packet from H3 to H1 will be routed directly on to the L3 VNI of the tenant. This adds the additional requirement that the
host routes on all VXLAN segments in a given tenant need to be downloaded to the Leaf's forwarding table.
FIGURE 12 Symmetric IRB
Brocade IRB Implementation
Both symmetric and asymmetric IRB methods are implemented on Brocade switches. If the target VNI segment is configured on a
VTEP, asymmetric IRB is performed. Otherwise, the packet is routed over the L3 VNI or symmetric routing occurs. Every tenant VRF is
assigned with an L3 VNI.
In the Brocade implementation, we get the best of both schemes:
∙ No need to create all server VNIs on all leafs for a tenant.
∙ If a target VNI segment is not local and is extended behind one or more remote VTEPs, download the host routes on that target
segment into hardware based on traffic activity. Traffic to these hosts will be routed over the L3 VNI.
Multitenancy
In BGP EVPN, multiple tenants can co-exist and share a common IP transport network while having their own separate routing domain
in the VXLAN overlay network. Every tenant in the EVPN network is identified by a VRF (VPN routing and forwarding instance), and
VRFs can span multiple leafs in a data center. (Similar to Layer 3 MPLS VPNs with tenant VRFs on multiple PE devices). Each VRF can
have a set of server-facing VLANs and a Layer 3 VLAN interface with a unique VNI used for symmetric routing purposes. This VNI
should be the same if the same tenant VRF is provisioned on other leafs including a border leaf.
53-1004308-03 29

FIGURE 13 Multitenancy
Ingress Replication
Although host reachability information is exchanged over the control plane to drastically reduce flooding in a VLAN, certain situations
require the flooding of frames, as in traditional Ethernet networks such as but not limited to:
∙ MAC aging
∙ Silent hosts
∙ L2 multicast or broadcast
Ingress replication is a technique used to accommodate flooding in such cases by the VTEPs in IP fabric. Each VTEP for a given
VXLAN segment (or server VLAN) computes the list of VTEPs having the same segment using the IMR (Inclusive Multicast Route)
routes. Whenever the VTEP must flood a frame in a VXLAN segment, it replicates the frame in hardware and unicasts the frame to each
of the VTEPs in the IMR list for that segment.
vLAG Pair
vLAG is the solution recommended for leaf-level redundancy. Server multihoming is supported only through vLAG behind two VTEPs.
Multihoming to two separate VTEPs is not supported. In the validated design, we have two pairs of VTEPs in each PoD operating in
vLAG mode, and servers are dual-homed to these VTEPs with a port channel.
When the two leafs are in vLAG mode, they act as one logical VTEP or end point. As shown in Figure 14, both leafs are configured with
the same VTEP IP address. From other VTEPs in the network, this pair appears as a single VTEP. This is very important because having
two physical switches in this mode on each rack does not result in an increased number of VTEPs or additional tunneling requirements
on other VTEPs in the network.
30 53-1004308-03

FIGURE 14 Active-Active vLAG
53-1004308-03 31

32 53-1004308-03

IP Fabric Validated Designs
∙ Pervasive eBGP.................................................................................................................................................................................................................33
∙ iBGP Within a PoD and eBGP Between PoDs...............................................................................................................................................34
∙ Hardware and Software Matrix..................................................................................................................................................................................35
∙ Brocade IP Fabric Configuration..............................................................................................................................................................................35
∙ Network Virtualization with BGP EVPN................................................................................................................................................................57
∙ Illustration Examples........................................................................................................................................................................................................87
This section provides the details of key deployment models with the validated configuration templates. Brocade validated design
recommends two models for the IP fabric deployment; these deployment models are categorized based on how the underlay is
designed for interconnecting leaf, spine, super-spine, and border-leaf nodes. The first deployment model uses pervasive eBGP for the
IPv4 underlay and EVPN peering. The second deployment model uses iBGP for the IPv4 underlay and EVPN peering within the PoD
with two spines as route-reflectors and eBGP for interconnecting the PoDs.
Pervasive eBGP
The design shown in Figure 15 uses eBGP as the control plane protocol between the layers of nodes, and each leaf is in its own
autonomous system. This design using eBGP as a routing protocol within the data center is based on the IETF draft: Use of BGP for
routing in large-scale data centers.[2] By adding the VXLAN EVPN control plane, this design is extended to support Layer 2 extension
and Layer 3 multitenancy in the fabric.
Figure 16 shows the design for a 3-stage IP fabric using eBGP as the control protocol. Note that the border leafs are connected to the
spines in this design.
FIGURE 15 Pervasive eBGP in an Optimized 5-Stage IP Fabric
53-1004308-03 33

FIGURE 16 Pervasive eBGP in a 3-Stage IP Fabric
iBGP Within a PoD and eBGP Between PoDs
The design shown in Figure 17 uses iBGP as the control plane protocol within a PoD and eBGP between PoDs and super-spines.
FIGURE 17 iBGP Within a PoD and eBGP Between PoDs in an Optimized 5-Stage IP Fabric
34 53-1004308-03

Hardware and Software Matrix
TABLE 1 Platforms Used in This Validated Design
Places in the Network Brocade Platform Software Version
Leaf Nodes VDX 6740
VDX 6940-144S
Network OS 7.0.1
Spine Nodes VDX 6940-36Q Network OS 7.0.1
Super-Spine Nodes VDX 8770-4 Network OS 7.0.1
Edge or Border Leaf VDX 6940-36Q Network OS 7.0.1
WAN Edge Router MLXe-8 NetIron 5.9ba
TABLE 2 All Brocade Switch Platforms That Support IP Fabric
Places in the Network Brocade Platform Software Version
Leaf Nodes VDX 6740
VDX 6940-36Q
VDX 6940-144S
Network OS 7.0.1
Spine Nodes VDX 6940-36Q
VDX 8770-4
VDX 8770-8
Network OS 7.0.1
Super-Spine Nodes VDX 6940-36Q
VDX 8770-4
VDX 8770-8
Network OS 7.0.1
Edge or Border Leaf VDX 6940-36Q Network OS 7.0.1
WAN Edge Router MLXe-8 NetIron 5.9ba
Brocade IP Fabric Configuration
This section covers the aspects of provisioning and validation of the IP fabric network topology. The IPv4 fabric underlay alone is
sufficient for data centers where multitenancy or Layer 2 extension is not a requirement. In this case, the server VLANs or subnets may
be advertised directly into BGP to establish connectivity between the racks and PoDs in the data center and to external networks.
Node ID Configuration
The VDX platforms used as leaf, spine, and super-spine nodes are enabled with VCS ID 1 by default. Since these nodes will be
independent in IP fabric, we must ensure that they do not form a VCS fabric between them. This is achieved by configuring a unique
VCS ID on each node.
In the validated design, each node—spine, leaf, super-spine, and edge leaf—is configured with a unique VCS ID. The RBridge ID may be
re-used. We recommend using RBridge ID 1 for individual leafs and using RBridge IDs 1 and 2 for the vLAG pair.
53-1004308-03 35

Enable Virtual-Fabric on all leafs and edge leafs:
The vLAG pair is assigned its own unique VCS ID, and each node in the vLAG pair has a separate RBridge ID. For example, in the
validated design, Leaf1 is a 2-node vLAG pair.
vLAG peer 1:
vLAG peer 2:
Verify the configuration:
From the primary node of the vLAG pair, enable virtual fabric. For instance, as shown above, RBridge 2 is the primary node in the Leaf1
vLAG pair.
36 53-1004308-03

IP Fabric Infrastructure Links
All nodes in the IP fabric—leafs, spines, and super-spines—are interconnected with Layer 3 interfaces. In the validated design,
∙ 40-G links are used between the nodes.
∙ All these links are configured as Layer 3 interfaces with /31 IPv4 address.2
For a simple 3-stage fabric, IP unnumbered
interfaces can be used. We do not recommend a mix of unnumbered and numbered interfaces within a fabric. Also for a 5-
stage IP fabric, numbered interfaces are highly recommended.
∙ The MTU for these links is set to Jumbo MTU. This is a requirement to handle the VXLAN encapsulation of Ethernet frames.
∙ Disable the fabric ISL and trunk features.
Loopback Interfaces
Each leaf and border leaf needs a loopback interface with a unique IPv4 address to use as the VTEP IP. This is not required on spines
and super-spines. This step may be skipped if VXLAN EVPN overlay is not used in the IP fabric.
Each device in the fabric needs one loopback interface with a unique IPv4 address for the purpose of router ID.
2 An IP unnumbered interface is another variation that can be used for the fabric links. This interface may be used in a 3-stage fabric. Refer to the
"Deployment Model-1: eBGP Underlay Configuration for 3-Stage Clos Fabric" section.
53-1004308-03 37

Configure the IP router ID using the IP address of the loopback 2 interface.
Server-Facing Links
Individual Leaf/ToR
The server-facing or access links are on the leaf nodes. In the validated design:
∙ 10-G links are used for server-facing VLANs.
∙ These links are configured as Layer 2 trunks with VLANs associated.
∙ The MTU for these links is set to the default: 1500 bytes.
∙ Disable fabric ISL and trunk features.
∙ Spanning tree is disabled.3
vLAG Pair/ToR
vLAG configuration involves three steps:
∙ Node ID configuration on the pair of devices.
∙ Inter-switch links or ISL configuration on both devices.
∙ Configuring the server-facing port channels and adding the required VLANs on them.
Node ID Configuration on the vLAG Pair
Refer to the "Node ID Configuration" section earlier in this chapter for assigning the node ID to the vLAG pair.
∙ Pod1-Leaf1-1, rbridge-id 1
3 If there are L2 switches or bridges between a leaf and servers, spanning tree must be enabled. If there is a possibility of enabling bridges inadvertently
under the leaf nodes, we recommend enabling spanning tree and configuring the server ports as edge ports.
POD1-Leaf3(conf-if-te-1/0/4)# spanning-tree autoedge
38 53-1004308-03

∙ Pod1-Leaf1-2, rbridge-id 2
ISL Configuration
As shown in the illustration below, the vLAG pair is interconnected by two 40-G Ethernet ports for ISL.
Server Port-Channel Configuration
In the configuration shown below, port channel 113 is configured as a vLAG.
53-1004308-03 39

Deployment Model-1: eBGP Underlay Configuration for Optimized 5-Stage Clos Fabric
Key points to consider as design principle for eBGP as IPv4 underlay. Refer to Figure 15 for the topology information.
∙ Each leaf is in a private AS.
∙ The vLAG pair (Dual-Leaf) is considered as one leaf; both devices in the pair are in the same private AS.
∙ All spines within a PoD are in one private AS.
∙ All super-spines are in one private AS.
∙ All border leafs are in one private AS.
∙ eBGP peering with MD5 authentication is used between the layers of nodes.
∙ BFD is enabled on each link with BGP as the client installing the BFD session between the neighbors. We recommend that you
use the default BFD timers.
∙ Two spines are designated to advertise the EVPN Address Family.
∙ Two super-spines are designated to advertise the EVPN Address Family.
40 53-1004308-03

Spine Configuration
All spines within a PoD have a similar configuration for IPv4 underlay. Peer groups are used to simplify the configurations and also for
efficiency in BGP update processing.
∙ Configure the directly connected leafs' IP addresses in one peer group: leaf-group.
∙ Configure the directly connected super-spine IPs into another peer group: super-spine-group.
∙ Enable MD5 authentication and BFD to all peers.
Each spine should establish IPv4 Address Family peering with all leafs inside the PoD and super-spines. (Note that when verifying the
peerings, leafs in a vLAG pair share one common AS number between them, and super-spines belong to one AS number.)
53-1004308-03 41

Check the BFD adjacency with every connected device.
Leaf Configuration
All leafs within a PoD have a similar configuration for IPv4 underlay. Peer groups are used to simplify the configuration and also for
∙ Configure the directly connected IP addresses of the spines into a peer group: spine-group.
∙ Enable MD5 authentication to the peer group.
∙ Enable BFD to the peer group.
∙ Advertise the VTEP IP address if EVPN overlay needs to be provisioned.
∙ For IP fabric implementations without overlay EVPN, advertise server subnets as appropriate using either a network statement
or a redistribute connected statement under the IPv4 Unicast Address Family.
42 53-1004308-03

Check the BGP neighbors. The leaf must be peering with all spines within the PoD for IPv4 Address Family route exchange.
BFD neighbors.
53-1004308-03 43

Check the route table to see the paths to other VTEP IPs in the fabric. For instance, in the table below taken from a leaf, it sees 4 paths
(due to 4 spines) to every other VTEP IP in the fabric—both inside the PoD and the VTEPs in another PoD.
Super-Spine Configuration
This is applicable to all super-spines to exchange only IPv4 underlay routes. Peer groups are used to simplify the configuration.
∙ Create a peer group for each PoD:
– pod1_spine-group—Add the directly connected neighbor addresses of all spines in PoD1 to this group.
∙ Create a separate peer group for the edge leafs: edge-group. Add the directly connected neighbor addresses of edge leafs to
this group.
∙ Enable MD5 authentication to all peer groups.
∙ Enable BFD to all peer groups.
44 53-1004308-03

Each super-spine should be peering with four spines per PoD and two edge leafs for IPv4 Address Family route exchange.
53-1004308-03 45

BFD session with each BGP peer.
Border/Edge Leaf Configuration
The configuration of edge or border leafs is similar to that of leafs. They peer with the super-spines instead of spines.
∙ Configure a peer group superspine-group. Add the directly connected neighbor addresses of the super-spines to the group.
These super-spines exchange only IPv4 routes.
∙ Enable MD5 authentication.
∙ Enable BFD.
∙ Optionally, advertise external networks directly into IPv4 underlay routing (for an IP fabric without EVPN overlays).
Deployment Model-1: eBGP Underlay Configuration for 3-Stage Clos Fabric
Refer to Figure 16 for the topology information. The underlay routing configuration for a 3-stage fabric is very similar to that of the
5-stage fabric with the exception of peering to super-spines by spines and border leafs. Border leafs are directly connected to spines.
46 53-1004308-03

A 3-stage fabric may be built using either numbered or unnumbered fabric interfaces. This section explains building a 3-stage fabric
with unnumbered interfaces. (For numbered interfaces, refer to the "IP Fabric Infrastructure Links" section and the 5-stage deployment
model.)
Key points to note:
∙ Fabric links are configured as unnumbered interfaces.
∙ Each leaf is in a private AS.
∙ The vLAG pair is considered as one leaf; both devices in the pair are in the same private AS.
∙ All spines within the PoD are in one private AS.
∙ All border leafs are in one private AS.
∙ eBGP multihop peering is established over loopback interface IP addresses with MD5 authentication.
∙ BFD sessions are established on the links between the layers of nodes.
Fabric Infrastructure Links—Unnumbered
The IP unnumbered option for fabric interfaces4
significantly simplifies the fabric provisioning for a 3-stage fabric.
∙ No IP addressing scheme is needed for the links between the nodes. Each node is represented by just one IP address or router
ID.
∙ The unnumbered interfaces are associated with a numbered loopback interface on the switch. This loopback interface’s IP
address is used as the source address for BGP peering. This IP address is exchanged over LLDP between the nodes. This
eliminates the need to run an IGP or static routing to reach the neighbor’s loopback address for BGP peering.
In the “Loopback Interfaces” section, we configured two loopback interfaces on each node to be used as the router ID. One of them is
used as the router ID. The unnumbered interfaces are associated with this loopback interface, i.e. Loopback 2. For example, on Leaf1:
Verify the neighbor discovery over the link using LLDP. Also verify the reachability to the loopback interface of the neighbor connected
over this unnumbered link. For instance, a link between the nodes Leaf1-1 and Spine2:
4 Note that a 3-stage fabric can also be built using numbered fabric interfaces. We do not recommend having a mix of both numbered and
unnumbered interfaces within a fabric. For a 5-stage IP fabric, we highly recommend using numbered interfaces.
53-1004308-03 47

Spine Configuration
All spines within a PoD have a similar configuration for IPv4 underlay. Peer groups are used to simplify the configurations and also for
∙ Configure the leafs' router IDs in one peer group: leaf-group.
∙ Configure the edge leafs' router IDs in one peer group: edge-group.
∙ Enable eBGP multihop, MD5 authentication, and BFD to both peer groups.
∙ Set the BGP peering source interface to the loopback interface (used as router ID).
48 53-1004308-03

Leaf Configuration
All leafs within a PoD have a similar configuration for IPv4 underlay. Peer groups are used to simplify the configuration and also for
∙ Configure the spines’ router ID loopback IP addresses into a peer group: spine-group.
∙ Set the BGP peering source interface to the loopback interface (used as the router ID).
∙ For IP fabric implementations without overlay EVPN, advertise server subnets as appropriate using either a network statement
or a redistribute connected statement under IPv4 Unicast Address Family.
53-1004308-03 49

Edge or border leafs peer with the spines and exchange both IPv4 and EVPN routes.
∙ Configure the spines’ router ID loopback IP addresses into a peer group: spine-group.
∙ Enable eBGP multihop, MD5 authentication, and BFD to the peer group.
∙ Set the BGP peering source interface to the loopback interface (used as the router ID).
∙ Advertise the VTEP IP if EVPN overlay needs to be provisioned.
∙ Optionally, advertise external networks directly into IPv4 underlay routing (for an IP fabric without EVPN overlays).
50 53-1004308-03

Deployment Model-2: iBGP Underlay Configuration for Optimized 5-Stage Clos Fabric
Key points to consider as a design principle for iBGP as IPv4 underlay (refer to Figure 16 for topology information):
∙ Each PoD is in one private AS.
∙ iBGP is used as the underlay within a PoD.
∙ eBGP routes are exchanged between the PoDs and border leafs through super-spines.
∙ In each PoD, all four spines act as the IPv4 RR to leafs.
∙ In each PoD, only two spines act as the EVPN RR to leafs.
∙ Use peer groups to group neighbors into IPv4 only and IPv4+EVPN speakers.
Spine Configuration
All spines within a PoD have a similar configuration for IPv4 underlay. Peer groups are used to simplify configuration and also for
∙ Configure the directly connected leafs' IP addresses in one peer group: leaf-group.
∙ Configure the directly connected super-spine IPs into another peer group: super-spine-group.
∙ All spines should have one cluster ID since they are IPv4 route reflectors to leafs.
53-1004308-03 51

Each spine should establish IPv4 Address-Family peering with all leafs inside the PoD and all super-spines.
52 53-1004308-03

Leaf Configuration
All leafs within a PoD have a similar configuration for IPv4 underlay. Peer groups are used to simplify the configuration.
∙ Configure the directly connected IP addresses of the spines into a peer-group spine-group.
∙ Enable MD5 authentication to the peer group.
∙ Enable BFD to the peer group.
∙ Advertise the connected networks.
Each leaf should establish IPv4 Address-Family peering with four spines inside the PoD.
53-1004308-03 53

Super-spines have a similar configuration for IPv4 underlay. Peer groups are used to simplify the configuration.
∙ Create two peer groups for each PoD, one group to exchange IPv4 routes and another group to exchange both IPv4 and
EVPN routes:
– pod1-spine-ip-group—Two spines in each PoD support only IPv4 routes. Add the directly connected neighbor addresses
of these two spines to this group.
– pod1-spine-evpn-group—Two spines in each PoD support both IPv4 and EVPN routes. Add the directly connected
neighbor addresses of these two spines to this group.
– Similar configuration for PoD2 and other PoDs.
∙ Create a separate peer group to the edge PoD. Add the directly connected neighbor addresses of edge leafs to this group.
54 53-1004308-03

Each super-spine should be peering with four spines per PoD and two edge leafs for the IPv4 Address Family.
53-1004308-03 55

Edge leafs peer with the super-spines and exchange both IPv4 and EVPN routes. So one peer group is sufficient.
∙ Configure a peer group, and add the directly connected neighbor addresses of the super-spines to the group.
– Enable MD5 authentication.
– Enable BFD.
∙ Activate the peer group for the IPv4 Address Family.
56 53-1004308-03

Overlay Gateway Configuration
Following are the steps involved in configuring the overlay gateway or VTEP on a leaf and border leaf.
∙ Create an overlay gateway, and assign it a name.
∙ Enable Layer 2 extension.
∙ Associate the loopback interface whose IPv4 address is used as the VTEP IP.
∙ Associate the rbridge-id of the leaf switch.
∙ Map the VLANs to the VNI number. In this validated design, we're using the auto mapping of VLAN to a VNI. For instance,
VLAN 2001 is mapped to VNI 2001. (This simplified mapping option should work for most implementations unless there is a
specific requirement to map the server VLAN range to a specific VNI range in the VXLAN domain.)
Deployment Model-1: eBGP EVPN Configuration for Optimized 5-Stage Clos Fabric
This configuration is applicable to the model shown in Figure 15, where eBGP is used as the control protocol for underlay.
53-1004308-03 57

BGP Underlay Configuration
When enabling network virtualization with EVPN overlay, the underlay configuration needs a few changes to accommodate the BGP
peers that exchange only IPv4 routes and the BGP peers that exchange both IPv4 and EVPN routes. This is accomplished by using
BGP peer-groups. In the 5-stage fabric:
∙ Two spines in each PoD exchange only IPv4 Address-Family routes.
∙ Two spines in each PoD exchange both IPv4 and EVPN Address-Family routes—referred to as EVPN spines.
∙ Two super-spines exchange only IPv4 Address-Family routes.
∙ Two super-spines exchange both IPv4 and EVPN Address-Family routes—referred to as EVPN super-spines.
Leaf Configuration
This is applicable to all Leafs. With EVPN control-plane, the configuration needs to accommodate the exchange of EVPN routes only
with two designated spines. Peer-groups are used to simplify the configuration and also for efficiency in BGP update processing.
∙ Configure the directly connected IP addresses of the spines into two peer-groups—spine-evpn-group and spine-ip-group. This
is required because only 2 spines exchange EVPN routes, but all 4 spines exchange ipv4 routes. (Refer to the "Network
Virtualization with BGP EVPN" for EVPN Address-Family configuration.) For simple IP fabric implementation, this may be
ignored and all spines can be added to one peer group.
∙ Enable MD5 authentication to both peer groups.
∙ Enable BFD to both peer groups.
∙ Enable the IPv4 Address-Family, and advertise the VTEP IP address.
Spine Configuration
This is applicable to the two spines designated to exchange only IPv4 routes with leafs and super-spines.
58 53-1004308-03

∙ Configure the directly connected leafs IP addresses in one peer group leaf-group.
∙ Configure the directly connected super-spine IPs into another peer group super-spine-group.
∙ Enable the IPv4 Address-Family.
EVPN Spine Configuration
This is applicable only on the two spines designated to exchange IPv4 and EVPN routes.
∙ Configure the directly connected leafs IP addresses in one peer-group leaf-group.
∙ Configure the directly connected super-spine IPs into two peer-groups superspine-ip-group and superspine-evpn-group. The
second group will contain only the two super-spines designated to exchange IPv4 and EVPN routes.
∙ Enable MD5 authentication to all peers.
∙ Enable BFD to all peers with default timer values.
53-1004308-03 59

This is applicable to two super-spines designated to exchange only IPv4 underlay routes. Peer-groups are used to simplify the
configuration.
∙ Create a peer-group for each PoD:
∙ Create a separate peer-group for the Edge leafs—edge-group. Add the directly connected neighbor addresses of edge leafs to
this group.
60 53-1004308-03

EVPN Super-Spine Configuration
This is applicable only on the super-spines designated to exchange both IPv4 and EVPN routes. This can be skipped for the IP fabric
implementation without the EVPN control-plane.
∙ Create two peer-groups for each PoD, one group to exchange only IPv4 routes and the other group to exchange both IPv4 and
EVPN routes. For simple IP fabric implementation, this may be ignored and all spines in a PoD can be added to one peer-
group.
– pod1_spine-ip-group—Two spines in each PoD support only IPv4 routes. Add the directly connected neighbor addresses
– pod1_spine-evpn-group—Two spines designated in each PoD support both IPv4 and EVPN routes. Add the directly
connected neighbor addresses of these two spines to this group.
∙ Create a separate peer-group for the Edge leafs—edge-group. Add the directly connected neighbor addresses of Edge leafs to
this group.
∙ Enable MD5 authentication and BFD to all peer-groups.
53-1004308-03 61

The configuration of edge or border leafs is similar to that of leafs. They peer with the super-spines. They exchange IPv4 routes with all
super-spines and EVPN routes with two designated super-spines.
∙ Configure a peer group superspine-ip-group. Add the two directly connected neighbor addresses of the two super-spines to
the group. These super-spines exchange only IPv4 routes.
∙ Configure another peer group superspine-evpn-group. Add the two designated super-spine addresses to this group. These
super-spines exchange both IPv4 and EVPN routes. For simple IP fabric implementation, this step may be skipped and all
super-spine neighbors may be added to just one peer group.
62 53-1004308-03

∙ Enable MD5 authentication and BFD to all peer groups.
∙ Enable the IPv4 Address-Family, and advertise the VTEP IP address.
BGP Overlay Configuration
Leaf Configuration
This configuration is applicable to all leafs in each of the PoDs. They exchange EVPN routes with two designated spines in their
respective PoDs.
∙ Enable the EVPN Address-Family.
∙ Activate the designated EVPN spines under EVPN Address-Family. (Use the peer-group already configured in the underlay
configuration.)
∙ Enable the "allowas-in 1" feature on vLAG leafs to facilitate learning of the routes between the vLAG peers. This is a requirement
because the vLAG pair is in the same AS number. This is the case in the pervasive eBGP model of underlay.
∙ When EVPN routes are advertised into eBGP by a node, the next hop is set to its peering address. This follows the standard
BGP behavior. The next hop should always point to the IP address of the VTEP that originated these routes. Enable the "next-
hop unchanged" configuration to the peers.
53-1004308-03 63

All leafs should see two EVPN neighbors. (Two spines participate in EVPN route exchange.)
This is applicable only to the two spines in each PoD designated to exchange the EVPN routes with leafs and super-spines.
∙ Activate the leaf group already created in the underlay configuration into the EVPN Address-Family.
∙ Activate the superspine-evpn-group into the EVPN Address-Family.
64 53-1004308-03

Each EVPN spine will establish EVPN Address-Family adjacency with all leafs inside the PoD and two designated super-spines.
This is applicable to the super-spines designated for the EVPN route exchange in the fabric with spines and edge leafs.
∙ Activate the spine-evpn-group peer groups of each PoD into the EVPN Address-Family.
∙ Activate the edge leafs peer group into the EVPN Address-Family.
53-1004308-03 65

Each super-spine has two spines in each of the PoDs and two border leafs as EVPN Address-Family neighbors.
This is applicable to all border leafs in the fabric.
∙ Activate the superspine-evpn-group peer groups into the EVPN Address-Family.
∙ When EVPN routes are advertised into eBGP by a node, the next hop is set to its peering address. This follows standard BGP
behavior. The next hop should always point to the IP address of the VTEP that originated these routes. Enable the "next-hop
unchanged" configuration to the peers.
66 53-1004308-03

Each border leaf establishes EVPN peering with two super-spines.
Deployment Model-1: eBGP EVPN Configuration for 3-Stage Clos Fabric
This configuration is application to the deployment model shown in Figure 16, where eBGP is used as the underlay routing protocol in a
3-stage Clos fabric.
BGP peer groups.
∙ Two spines exchange only IPv4 Address-Family routes.
∙ Two spines exchange both IPv4 and EVPN Address-Family routes.
Leaf Configuration
This is applicable to all leafs. With the EVPN control plane, the configuration needs to accommodate the exchange of EVPN routes only
with two designated spines. Peer groups are used to simplify the configuration and also for efficiency in BGP update processing.
∙ Configure the router ID loopback IP addresses of the spines into two peer groups: spine-evpn-group and spine-ip-group. This
is required because only two spines exchange EVPN routes, but all four spines exchange IPv4 routes.
∙ BGP peering source interface set to loopback interface (used as the router ID).
53-1004308-03 67

∙ Enable the IPv4 Address Family, and advertise the VTEP IP address.
Spine Configuration
This is applicable to all spines inside a PoD.
∙ Configure the router ID loopback IP addresses of the leafs in one peer-group leaf-group.
∙ Configure the router ID loopback IP addresses of the edge leafs' IPs into a peer-group edge-group.
∙ Enable the IPv4 Address Family.
68 53-1004308-03

The configuration of edge or border leafs is similar to that of leafs. They peer with the spines. They exchange IPv4 routes with all spines
and EVPN routes with two designated spines.
∙ Configure a peer group spine-ip-group. This group consists of the router IDs of spines that exchange only IPv4 routes.
∙ Configure another peer group spine-evpn-group. This group consists of router IDs of spines that exchange both IPv4 and
EVPN routes.
53-1004308-03 69

Leaf Configuration
This is applicable to all leafs.
∙ Activate the designated EVPN spines under the EVPN Address Family (use the peer group already configured in the underlay
configuration).
because the vLAG pair is in the same AS number. This is the case in the pervasive eBGP model of underlay.
∙ When EVPN routes are advertised into eBGP by a node, the next hop is set to its peering address. This follows standard BGP
behavior. The next hop should always point to the IP address of the VTEP that originated these routes. Enable the next-hop
unchanged configuration to the peers.
70 53-1004308-03

As shown below (with the show ip bgp summary command), there are four neighbors for IPv4 AFI. Of these four neighbors, two are
listed as neighbors for EVPN AFI (show bgp evpn summary). In other words, all four spines exchange IPv4 routes, and only two
exchange EVPN routes.
This is applicable only to the two spines designated to exchange EVPN routes with leafs and edge leafs.
∙ Enable the EVPN Address Family.
∙ Activate the leaf-group peer group into the EVPN Address Family.
∙ Activate the edge-leaf's peer group into the EVPN Address Family.
53-1004308-03 71

This is applicable to all edge leafs. Activate the EVPN route exchange with the designated spines for EVPN.
Deployment Model-2: iBGP EVPN Configuration for Optimized 5-Stage Clos Fabric
This configuration is applicable to the deployment model shown in Figure 16, where iBGP is used as the underlay routing protocol within
a PoD.
BGP peer groups. In the 5-stage fabric using iBGP inside a PoD:
∙ All spines exchange IPv4 routes with leafs and super-spines.
∙ All spines act as the route reflector to all leafs inside their PoD for IPv4 Address-Family routes.
∙ Two spines are designated to exchange EVPN routes with leafs and super-spines. These are referred to as EVPN spines.
∙ EVPN spines act as the route reflector to all leafs inside their PoD for EVPN Address-Family routes.
∙ Two super-spines are designated to exchange EVPN routes with spines in each PoD and border leafs. These are referred to as
EVPN super-spines.
72 53-1004308-03

Spine Configuration
This configuration is applicable to the spines in each POD that exchange only IPv4 routes with leafs and super-spines. Peer groups are
used to simplify configuration and also for efficiency in BGP update processing.
∙ Configure the directly connected leaf IP addresses in one peer group leaf-group.
∙ Configure the directly connected super-spine IPs into another peer group super-spine-group.
∙ All spines to have one cluster ID.
∙ Enable IPv4 Address-Family, redistribute connected-routes.
∙ Enable IPv4 Address-Family route reflection to all leafs in leaf-group.
Each spine should establish IPv4 Address-Family peering with all leafs inside its PoD and all super-spines.
53-1004308-03 73

This is applicable only on the two spines designated to exchange IPv4 and EVPN routes with leafs and super-spines.
∙ Configure all leafs in a peer group leaf-group.
∙ Configure the directly connected super-spine IPs into two peer groups superspine-ip-group and superspine-evpn-group. The
second group will contain only those two super-spines designated to exchange IPv4 and EVPN routes.
∙ All spines are to have one cluster ID.
∙ Enable IPv4 Address-Family; redistribute connected routes.
∙ Enable IPv4 Address-Family route reflection to all leafs in leaf-group.
74 53-1004308-03

Leaf Configuration
This is applicable to all leafs in a PoD. Peer groups are used to simplify the configuration.
∙ Configure the directly connected IP addresses of the spines into two peer groups: spine-evpn-group and spine-ip-group. This
is required because only two spines exchange EVPN routes, but all four spines exchange IPv4 routes.
∙ Enable MD5 authentication to both peer groups.
∙ Enable BFD to both peer groups.
∙ Advertise the connected networks.
53-1004308-03 75

Each leaf should establish IPv4 Address-Family peering with all inside the PoD.
This is applicable to super-spines that exchange only IPv4 routes with spines in each PoD.
∙ Create one peer group for each PoD.
– pod1_spine-group—All spines in PoD1 and exchange only IPv4 routes. Add the directly connected neighbor addresses of
these two spines to this group.
– pod2_spine-group—All spines in PoD2 and exchange only IPv4 routes. Add the directly connected neighbor addresses of
these two spines to this group.
∙ Create a separate peer group to the Edge PoD—edge-group. Add the directly connected neighbor addresses of the edge leafs
to this group.
76 53-1004308-03

Each super-spine should be peering with four spines per PoD and two edge leafs for the IPv4 Address-Family.
53-1004308-03 77

This is applicable to the super-spines designated to exchange both IPv4 and EVPN routes with spines in each PoD and edge leafs.
∙ Create two peer groups for each PoD: one group to exchange IPv4 routes and another group to exchange both IPv4 and
EVPN routes:
– pod1-spine-ip-group—Two spines in each PoD support only IPv4 routes. Add the directly connected neighbor addresses
– pod1-spine-evpn-group—Two spines in each PoD support both IPv4 and EVPN routes. Add the directly connected
neighbor addresses of these two spines to this group.
∙ Create a separate peer group to the edge PoD—edge-group. Add the directly connected neighbor addresses of edge leafs to
this group.
78 53-1004308-03

Each super-spine should peer with four spines per PoD and two edge leafs for the IPv4 Address-Family.
53-1004308-03 79

Border/Edge-Leaf Configuration
The configuration of border or edge leafs is similar to that of leafs. But they peer with the super-spines. They exchange IPv4 routes with
all super-spines and EVPN routes with two designated super-spines.
∙ Configure a peer group superspine-ip-group. Add two directly connected neighbor address of two super-spines to the group.
These super-spines exchange only IPv4 routes.
∙ Configure another peer-group superspine-evpn-group. Add the two designated super-spine addresses to this group. These
super-spines exchange both IPv4 and EVPN routes. For simple IP fabric implementation, this step may be skipped and all
super-spine neighbors may be added to just one peer group.
∙ Enable the IPv4 Address-Family and advertise the VTEP IP address.
80 53-1004308-03

The border leaf should establish IPv4 peering with all super-spines.
Leaf Configuration
This is applicable to all leafs in each PoD.
∙ Activate the designated EVPN spines under the EVPN Address-Family. (Use the peer group already configured in the underlay
configuration.)
53-1004308-03 81

because the vLAG pair is in the same AS. This is the case in the pervasive eBGP model of underlay.
This is applicable only to the two spines designated to exchange the EVPN routes with leafs and super-spines.
∙ Activate EVPN super-spines under the EVPN Address-Family.
∙ Activate all leafs under the EVPN Address-Family.
∙ Act as the route reflector of the EVPN Address-Family to the leafs peer group.
EVPN Super-spine Configuration
This configuration is applicable to the super-spines designated to exchange both IPv4 and EVPN routes with spines in each PoD.
∙ Activate EVPN spines in each PoD under the EVPN Address-Family.
82 53-1004308-03

∙ Activate all edge leafs under the EVPN Address-Family.
This is applicable to all border leafs. They exchange EVPN routes with two designated super-spines.
∙ Activate EVPN super-spines under the EVPN Address-Family.
Tenant Provisioning
Tenant provisioning refers to the configuration on leafs to enable server VLANs and network connectivity to tenant VRF contexts and
mapping these VLANs and VRFs to the overlay control and forwarding planes to establish Layer 2 extension and multitenancy. This is
applicable to both 3-stage and 5-stage Clos fabrics.
Enable Conversational Learning of MAC Entries
This is applicable to all leafs in the fabric for conservation of L2 forwarding table space.
53-1004308-03 83

Anycast Gateway MAC Configuration
Anycast gateway MAC configuration is applied to all leafs (except edge leafs) in the data center. This is used as the gateway MAC or
router MAC for all server-facing subnets. This enables seamless workload move within and across the PoDs in the data center. We
recommend setting the U/L bit to 1 in the MAC address to indicate that it is a locally administered MAC address and not to conflict with
any real MAC addresses.
The MAC addresses must be different for IPv4 and IPv6, but the OUI portion (first three bytes) must be same.
Enable Conversational Learning of ARP/ND Host Entries
This is required on all leafs and edge leafs.
VRFs, Server VLANs, and Subnets Configuration
Following are the steps involved in tenant VRF configuration.
1. Assign a unique RD. Every tenant must have a unique RD value per leaf/ToR where it is provisioned. In the validated design, we
are using the following format: IPv4_Address:nn where
∙ IPv4_Address is the router ID of the VTEP.
∙ nn is a unique number for the tenant VRF. This value is re-used on other leafs as well where the same tenant is
provisioned.
For example, vrf201 has the following RD values on leafs where it is provisioned.
– On leaf1: 10.121.1.11:201
– On leaf5: 10.121.1.51:201
– On border-leaf1: 10.123.4.1:201
2. Assign a unique L3 VNI number.
3. Assign import and export route targets for IPv4 and IPv6 tenant routes.
In the configuration templates below, the following tenant profile is enabled on a leaf:
Configure Tenant VRF Profile:
∙ Name: vrf101
84 53-1004308-03

∙ L3 VNI: 7101
∙ IPv4 and IPv6: enabled
∙ Route-target 101:101
∙ Server-facing VLAN 2001
Assign Layer 3 Interface for the L3 VNI of the Tenant VRF:
This is the routing interface for the Integrated Routing and Bridging (IRB) operation on the leaf.
Assign Server-Facing VLAN:
Assign VE (L3) Interface for the Server-Facing VLAN:
53-1004308-03 85

Advertise Tenant Layer 3 Routes from the Leaf
IPv4
IPv6
Enable the EVPN Instance for the Tenant VLAN Segments
Once the server-facing VLANs are created and mapped to VNI segments on the leaf, those VNI segments must be enabled into the
control plane. As was done for the tenant VRF, the VNI segments also require an RD (route distinguisher) and an RT (route target). This is
also defined as the MAC-VRF and enables learning remote MAC addresses when the same VLAN segment is extended to other leafs or
VTEPs in the fabric.
The RD and RT configuration is set to auto in this design for simplicity and may be followed for most of the deployments. Advanced
users may define a different scheme of RD and RT. A user-defined RD/RT is not covered in this document.
86 53-1004308-03

vLAG Pair Configuration
A vLAG pair or redundant ToR requires a few additional configuration steps:
∙ Same VTEP IP
∙ Separate or unique router IDs
The configuration of two leafs in a dual-ToR vLAG pair is shown side-by-side for comparison. (Please note that the configuration for
both switches in the vLAG pair can be done from the primary node.)
∙ The Loopback1 interface has the same IP address on both nodes; this is used as the VTEP IP under overlay gateway.
∙ The Loopback2 interface has a unique IP address on each node; this is used as the IP router ID for the node.
∙ Attach both RBridge IDs under the overlay gateway.
Illustration Examples
In this section we illustrate the use cases by using sections of the validated design network topology as appropriate. This will help the
reader to further understand the deployment scenarios.
Example-1: Tenant and L2 Extension Between Racks in a 3-Stage Clos Fabric
Figure 18 shows a section of the topology to illustrate the following with configuration and verification. Two racks are shown in the
diagram.
53-1004308-03 87

∙ Rack1 has a redundant vLAG ToR, leaf1-1 and leaf1-2, referred to as leaf1 collectively.
∙ Rack5 has an individual ToR, leaf5.
∙ A tenant VRF vrf201 is provisioned on both racks.
∙ The tenant has two server VLANs mapped to VNIs 3001 and 3801.
∙ Server VLAN 3001 is extended between these two racks. VLAN/VNI 3001 is provisioned on both racks, and there are hosts on
these racks.
∙ Server VLAN 3801 is a VLAN provisioned on Rack1 only, but it belongs to the same tenant. Routing between VNI 3001 and
3801 is required within this tenant both in the same rack and across the racks.
∙ This example also illustrates the symmetric and asymmetric routing operation.
The configuration on leafs is identical on each of the leafs except for the VTEP IP, router ID, and RD configurations. The vLAG pair is
represented with one VTEP IP address. The use of anycast gateway addresses for the server-facing VLAN interfaces simplifies the
configuration drastically. Please note that the configuration for the vLAG pair is done from the primary node.
FIGURE 18 Tenant and Layer 2 Extension Between Two Racks
Configuration
Check the Node ID on Each ToR
The RBridge ID is required for the Layer 3 and EVPN configuration on each node.
For the vLAG pair, Leaf1-2 is the primary node. The configuration for both devices in the pair is done from Leaf1-2. The RBridge IDs are
45 and 46 for Leaf1-1 and Leaf1-2, respectively. These IDs are used for the ports and for the Layer 3 configuration.
88 53-1004308-03

Leaf5 is an individual ToR with an RBridge ID of 51.
Configuration on the Leaf1 vLAG Pair
The configuration is shown in three parts for clarity. Common configuration such as port channel and VLANs are shown in one block.
The tenant, Layer 3 interfaces, and BGP-EVPN configuration is shown in the second block under each RBridge ID. The common
overlay-gateway configuration is shown in the third block. Please note that the entire configuration is applied from the primary node in
this two-node vLAG pair.
The configuration is pretty much the same except for the router ID and RD of the tenant VRF. This makes it easier to automate the
provisioning on various nodes.
53-1004308-03 89

90 53-1004308-03

Configuration on Leaf5
53-1004308-03 91

92 53-1004308-03

Verification
Verify VLAN Extension Between the Racks
Check the L2 extended VLAN on each node. This should show the local L2 trunk ports and also the tunnels to all remote VTEPs where
the same VLAN segment is extended.
In the following output from the Leaf1 vLAG pair, there are five tunnels for VLAN 3001, which indicates that the same VLAN/VNI
segment is provisioned on five other VTEPs or ToRs. Note that one of the tunnels, Tu 61442, is destined to Leaf5. Also note that there
are four underlay next hops to reach this tunnel destination in the fabric.
53-1004308-03 93

In the following output shown from Leaf5, Tunnel 61441 is destined to the vLAG Leaf1 pair's VTEP IP: 10.121.1.1.
94 53-1004308-03

VLAN Layer 3 Interface State on the vLAG Pair
53-1004308-03 95

VLAN Layer 3 Interface State on the Leaf5 ToR
Local Host Entries on Each Leaf
Depending on the port-channel hashing on server-facing links, the ARP entries may be learned on any of the nodes in the vLAG pair.
Make sure that all host entries are learned collectively in the vLAG pair.
96 53-1004308-03

Remote Host Entries in the Extended VLAN
BGP and ARP Table on Leaf5
The following table shows the BGP and ARP entries of the remote host behind the Leaf1 pair. Note that the next hop is set to 10.121.1.1,
which is a common VTEP IP of the vLAG pair. This causes the redundant leaf to appear as one VTEP in the underlay network, and load
balancing is accomplished.
In the ARP table, both the local and remote entries are indicated with different types. BGP EVPN for remote entries signify that they were
learned over BGP EVPN. The local entries are shown as "Dynamic" entries.
Verify Tenant Extension Between the Racks
Tenant extension ensures routing between the VXLAN segments within the same tenant.
53-1004308-03 97

As shown in Figure 18, VNI segment 3802 is provisioned only on the vLAG ToR but is part of the tenant on both ToRs. Let's go over a
list of verification steps required to ensure that communication between the hosts in VNI 3001 on Leaf5 and hosts in VNI 3802 on
vLAG Leaf1.
RMAC of Each Node
There is one RMAC assigned to every VTEP. This information can be obtained by looking at any of the L3 interfaces or the L3 VNI's
associated VLAN interface. For the vLAG pair, even though they have same VTEP IP, they are assigned a unique router MAC.
L3 VNI State on the Nodes
L3 VNI 7201 is assigned to the tenant VRF. Make sure that the vLAG ToR and Leaf5 have tunnels established to each other and that this
VNI is activated on it.
As seen in the following table for the output from Leaf1, the tunnel source is the VTEP IP of the vLAG, 10.121.1.1, and the destination IP is
the VTEP IP of Leaf5, 10.121.1.5. (Notice additional tunnels in the list; these are destined to other VTEPs where the same tenant is
provisioned.)
98 53-1004308-03

L3 VNI state from Leaf5:
53-1004308-03 99

Verify the Route to the Remote Subnet of the Same Tenant
The following table shows the BGP entry on Leaf5 for the remote subnet of VNI 3802. (Note that the host entries are also advertised
over BGP, but will be ignored by Leaf5 since this VNI is not locally provisioned and only routing is desired.)
There are four entries in the BGP table: two originators in the vLAG pair, and those two entries are learned from two spines exchanging
EVPN routes. Again, the next hop is the same due to the common VTEP IP used by the vLAG pair.
100 53-1004308-03

Example-2: Tenant and L2 Extension Between PoDs in an Optimized 5-Stage Clos
Fabric
In this example, we illustrate the extension of a tenant and a Layer 2 segment between racks in two different PoDs. As shown in
Figure 19, tenant VRF vrf101 is extended between these two racks: POD1-leaf1 and POD2-leaf1 dual or vLAG pair. VXLAN segment 2001
is extended across the PoD. VLAN 3901 is provisioned only on the Leaf1 pair in POD1.
53-1004308-03 101

FIGURE 19 Tenant and Layer 2 Extension Between Two PoDs Connected by Super-Spines
Configuration
Check the Node ID on Each ToR
The RBridge ID is required for the Layer 3 and EVPN configuration on each node.
For the POD1 vLAG pair, Leaf1-2 is the primary node. The configuration for both devices in the pair is done from Leaf1-2. The RBridge
IDs are 45 and 46 for Leaf1-1 and Leaf1-2, respectively. These IDs are used for the ports and for the Layer 3 configuration.
For the POD2 vLAG pair, Leaf1-2 is the primary node. The configuration for both devices in the pair is done from Leaf1-2. The RBridge
IDs are 45 and 46 for Leaf1-1 and Leaf1-2, respectively.
102 53-1004308-03

Configuration on the PoD1-leaf1 vLAG Pair
The configuration is shown in three parts for clarity. Common configuration such as port channel and VLANs is shown in one block. The
tenant, Layer 3 interfaces, and BGP-EVPN configuration is shown in the second block under each RBridge ID. The common overlay-
gateway configuration is shown in the third block. Please note that the entire configuration is applied from the primary node in this two-
node vLAG pair.
53-1004308-03 103

104 53-1004308-03

Configuration on the POD2-leaf1 vLAG Pair
53-1004308-03 105

106 53-1004308-03

Verification
Verify VLAN Extension Between the Nodes
In the output below from the POD1-Leaf1 vLAG ToR, there are six tunnels for VLAN 2001, which indicates that the same VLAN/VNI
segment is provisioned on six other VTEPS or ToRs. Note that one of the tunnels, Tu 61448, is destined to the POD2-Leaf1 vLAG ToR.
Also note that there are four underlay next hops to reach this tunnel destination in the fabric as there are four spines.
53-1004308-03 107

The output below from the POD2-Leaf1 vLAG shows the state of VLAN 2001.
VLAN Layer 3 Interface State on the POD1-Leaf1 vLAG
108 53-1004308-03

VLAN Layer 3 Interface State on the POD2-Leaf1 vLAG
53-1004308-03 109

Local Host Entries on Each Leaf/ToR
Remote Host Entries in the Extended VLAN
BGP and ARP Table on POD1-Leaf1
The following table shows a BGP entry and ARP entries of the remote hosts behind the POD2-leaf1 pair. Note that the next hop is set to
10.122.2.1, which is the common VTEP IP of the vLAG pair. This causes the redundant leaf to appear as one VTEP in the underlay
network, and load balancing is accomplished.
In the ARP table, both local and remote entries are indicated with different types: "Dynamic" for local entries; and BGP-EVPN for remote
entries, signifying that they were learned over BGP EVPN.
10.107.1.20 and 10.107.1.21 are the local hosts. (Even though 10.107.1.21 is shown as remote, the MAC entry lookup makes it a local host in
the vLAG pair).
10.107.1.30 and 10.107.1.31 are the hosts attached to the POD2-Leaf1 pair.
110 53-1004308-03

BGP and ARP Table on POD2-Leaf1
53-1004308-03 111

Verify Tenant Extension Between the Racks
Tenant extension ensures routing between the VXLAN segments within the same tenant.
As shown in Figure 19, VNI segment 3901 is provisioned only on the POD1-Leaf1 vLAG pair, but it is part of the tenant on both leafs.
Let's go over a list of verification steps required to ensure that communication between the hosts in VNI 2001 on POD2-Leaf1 and hosts
on VNI 3901 on POD1-Leaf1.
RMAC of Each Node
There is one RMAC assigned to every VTEP. This information can be obtained by looking at any L3 interface or the Layer 3 VNIs
associated the VLAN interface. For the vLAG pair, even though they have same VTEP IP, they are assigned a unique router MAC
address.
112 53-1004308-03

L3 VNI State on the Nodes
L3 VNI 7101 is assigned to the tenant VRF. Make sure that the vLAG pair and leaf5 have tunnels established to each other and that this
VNI is activated on it.
As seen in the following table for the output taken from POD1-Leaf1, the tunnel source is the VTEP IP of the vLAG (10.121.1.1), and the
destination IP is the vLAG VTEP IP of POD2-Leaf1 (10.122.2.1). (Notice additional tunnels in the list; these are destined to other VTEPs
where the same tenant is provisioned.)
53-1004308-03 113

The L3 VNI state from POD2-Leaf1 is shown below.
114 53-1004308-03

Verify the Route to the Remote Subnet of the Same Tenant
The following table shows the BGP entry on POD2-Leaf1 for the remote subnet of VNI 3901. (Note that the host entries are also
advertised over BGP, but they will be ignored by this leaf as this VNI is not locally provisioned and only routing is desired).
53-1004308-03 115

There are four entries in BGP table: two originators in the vLAG pair, and those two entries are learned from two spines exchanging
EVPN routes. The next hop is the same due to the common VTEP IP used by the vLAG pair.
Example-3: Tenant Extension Outside the Fabric
In "Example-2: Tenant and L2 Extension Between PoDs in an Optimized 5-Stage Clos Fabric," we illustrated extending a tenant VRF
across racks in two PoDs. In this section, let's see the steps involved in extending the same tenant outside the fabric through the border
or edge leafs.
Figure 20 shows a section of the validated design. Here, we're extending tenant vrf vrf101 outside the fabric through the edge leaf. The
edge leaf is connected to a WAN edge router, and the tenant VRF is extended to the WAN edge.
116 53-1004308-03

FIGURE 20 Tenant Extension Outside the Fabric Through Edge Leafs
Configuration
We will skip through the configurations of the POD1-Leaf1 and POD2-Leaf1 vLAG pairs since they have already been covered earlier
and will focus on the configurations of the edge leafs.
Edge-Leaf1 Configuration
On the edge leaf, we do not recommend any server VLAN segments.
For the fabric side, we need only a VNI segment for the purpose of the L3 routing VNI for the tenant VRF. This VNI must be consistent
with other leafs for a given tenant. In this example, we're using VNI 7101 as the L3 VNI for the tenant vrf101.
For the external-facing side, we need another VLAN for peering with external routers.
53-1004308-03 117

118 53-1004308-03

Edge-Leaf2 Configuration
On the edge leaf, we do not recommend any server VLAN segments.
For the fabric side, we need only a VNI segment for the purpose of the L3 routing VNI for the tenant VRF. This VNI must be consistent
with other leafs for a given tenant. In this example, we're using VNI 7101 as the L3 VNI for tenant vrf101.
For the external facing side, we need another VLAN for peering with external routers.
53-1004308-03 119

120 53-1004308-03

Verification
RMAC of Each Node
There is one RMAC assigned to every VTEP. This information can be obtained by looking at any L3 interface or the VLAN interface
associated with the Layer 3 VNI. For the vLAG pair, even though the nodes have the same VTEP IP, they are assigned a unique router
MAC.
POD1-Leaf1 Pair
POD2-Leaf1 Pair
Verify the L3 VNI State on the Nodes
Here we need to make sure that the Layer 3 VNI is associated with tunnels to every other node that has been provisioned with the same
tenant.
For instance, the output from POD1-Leaf1-1 shows three tunnels. Looking at the destination IPs, we can confirm that POD2-Leaf1, Edge-
Leaf1, and Edge-Leaf2 have been associated with the Layer 3 VNI of 7101 of tenant vrf101. (The source IP is the VTEP IP of the POD1-
Leaf1 vLAG pair.)
53-1004308-03 121

The following shows the VNI state from Edge-Leaf1. It is associated with tunnels destined to POD1-Leaf1 (10.121.1.1) and POD2-Leaf1
(10.122.2.1).
On Edge-Leaf2 also, let's ensure that the tunnels to POD1-Leaf1 (10.121.1.1) and POD2-Leaf1 (10.122.2.1) are associated with Layer 3 VNI
7101.
122 53-1004308-03

Verify the Route to a Fabric Segment on the Edge Leaf
Let's look at the route entry to the subnet of VLAN/VNI 2001 (10.107.1.0/24). It is advertised by the vLAG pairs in two PoDs. Effectively,
we should see two equal paths. Since the RMACs are different between vLAG peers within the vLAG pair, we see four paths, as shown
below. Also, note that the route is advertised by the edge leaf to its external BGP peer.
(The "show ip bgp routes <prefix> vrf <vrf-name>" command lists the routes sent to the route-table manager after the best-path
computations are complete. If this output is not correct, check the "show bgp evpn routes type ipv4-prefix <> tag 0" command.)
Similarly, for the route to the VNI 3901 subnet learned from the POD1-Leaf1 vLAG pair whose VTEP IP is 10.121.1.1:
53-1004308-03 123

Verify the Route to an External Network on the Internal Leafs
As shown in Figure 20, external network 172.23.150.0/26 must be reachable from the tenant VRF of the internal leafs. Let us look at the
route verification, step by step, starting from the edge leaf.
First, verify the route on Edge-Leaf1. As shown, the route is installed in the correct VRF and is pointing to the external next hop of the
WAN edge router.
The next step is to verify that this route gets advertised by the edge leafs into the fabric in EVPN Address-Family. The important fields to
look at in this output are L3 VNI, Router MAC, RD, RT, and Next Hop, as highlighted below.
124 53-1004308-03

Now let's look at the BGP entry on one of the internal leafs, say POD1-Leaf1. It should see two paths to the external network as both
edge leafs are advertising that network into the fabric. As you see in the output below, there are four entries—due to the fact that they're
learned from two spines. Essentially, there are two unique entries.
Verify that the routes are sent to the route table by BGP.
53-1004308-03 125

Example-4: VLAN Scoping at the ToR Level
VLAN scoping is briefly discussed in the “VLAN Scoping” section under the technology overview “Network Virtualization with BGP
EVPN" chapter.
Refer to the Figure 21 for the topology used to illustrate the VLAN scoping at the leaf or ToR level. For the purpose of illustration, we’ve
chosen a vLAG pair and an individual leaf. Both ToRs may be vLAG pairs or individual leafs.
As seen in the figure, each leaf has a server VLAN that requires a Layer 2 extension to the other rack. Also note that the VLAN numbers
are different. By mapping these VLANs to the same VNI number—8000 in this case—we achieve bridging or L2 extension between
them. The servers now have L2 adjacency between them. In other words, they are in the same bridge domain or broadcast domain. In
essence, the VLAN tag on the wire between the servers and the leaf is decoupled from the bridge domain. This VLAN tag need not be
identical on both sides to have Layer 2 adjacency or extension. In other words, the VLAN number is relevant only at the ToR level.
FIGURE 21 VLAN Scoping at the ToR Level
Configuration
The configuration steps are similar to the L2 extension illustrated in “Example-1: Tenant and L2 Extension between Racks in a 3-Stage
Clos Fabric.” The difference is in the VLAN-to-VNI mapping under the overlay gateway configuration. A sample configuration is shown
below for a quick reference; as highlighted, a server VLAN is manually mapped to a VNI number.
126 53-1004308-03

The table below summarizes the provisioning of L2 extension on the leafs.
Leaf 1 Leaf 5
∙ Server traffic is tagged with VLAN 100.
– Create VLAN 100.
– Create the VE 100 Layer 3 interface for first-hop routing.
– Assign the anycast GW 10.100.1.254 address to VE 100.
∙ Map VLAN 100 to VNI 8000 under the overlay gateway.
∙ Server traffic is tagged with VLAN 20.
– Create VLAN 20.
– Create the VE 20 Layer 3 interface for first-hop routing.
– Assign the anycast GW 10.100.1.254 address to VE 20.
∙ Map VLAN 20 to VNI 8000 under the overlay gateway.
Complete configurations and verification steps on the leafs in the Figure 21 topology are given in the sections that follow.
The configuration is shown in three parts for clarity:
∙ Common configurations, such as port channel and VLANs, are shown in one block.
∙ The tenant, Layer 3 interfaces, and BGP EVPN configurations are shown in the second block under each RBridge ID.
∙ The common overlay-gateway configuration is shown in the third block.
Please note that the entire configuration is applied from the primary node in this two-node vLAG pair.
53-1004308-03 127

128 53-1004308-03

53-1004308-03 129

130 53-1004308-03

Verification
Check the L2-extended VLAN on each node. This should show the local L2 trunk ports and also the tunnels to all remote VTEPs where
In the output below from the Leaf1 vLAG pair, there is one tunnel for VLAN 100, which indicates that the same VLAN/VNI segment is
provisioned on one other VTEP or ToR. Note that one of the tunnels, Tu 61445, is destined to Leaf5. Also note that there are four
underlay next hops to reach this tunnel destination in the fabric.
53-1004308-03 131

In the output below from Leaf5, Tunnel 61442 is destined to the vLAG Leaf1 pair's VTEP IP 10.121.1.1.
132 53-1004308-03

VLAN Layer 3 Interfaces State on the vLAG Pair
VLAN Layer 3 Interfaces State on the Leaf5 ToR
53-1004308-03 133

Remote Host Entries in the Extended VLAN BGP and ARP Table on Leaf5
The table below from Leaf5 shows the BGP and ARP entries of a remote host behind the Leaf1 pair. Note that the next hop is set to
10.121.1.1, which is a common VTEP IP of the vLAG pair. There are two entries in BGP since there are two spines exchanging the EVPN
routes.
In the hardware ARP table, both the local and remote entries are indicated with different types. The local host entries are of type
Dynamic, and the remote host entries are of type BGP-EVPN. Note that the remote host entries are shown under the virtual interface of
local VLAN 20 on Leaf5 (not VLAN 100 as in the remote ToR).
134 53-1004308-03

Example-5: VLAN Scoping at the Port Level Within a ToR
VLAN scoping is briefly discussed in the “VLAN Scoping” section under the “Network Virtualization with BGP EVPN” chapter.
Port VLAN scoping enables complete abstraction of a bridge domain where the VLAN tags on the server-side data frame on two ports
can be different and still be bridged between the ports. The VLAN tag is localized at the port level rather than at the ToR level.
Refer to the topology shown in Figure 22.
On the vLAG leaf, there are two port channels or LAG bundles: po111 and po112. Each has server traffic tagged with an 802.1q VLAN tag
of 10 and 30, respectively. From the port VLAN scoping perspective, these tags are referred to as c-tags. The {port,vlan} is added as a
member of a virtual-fabric VLAN. In this case, there is a fabric VLAN ID 6000. (Note that this number is above the 802.1q VLAN range
of 4096.)
In summary, VLAN 6000 comprises two members (port, vlan). (Unlike the ports in traditional VLAN cases.)
∙ (po111, vlan tag 10)
∙ (po112, vlan tag 30)
On Leaf5, VLAN 40 is mapped to VNI 8001. On the Leaf1 pair, VLAN 6000 is mapped to VNI 8001. Thus we're providing Layer 2
extension within and between the leafs for server-side traffic with different dot1q VLAN tags.
53-1004308-03 135

FIGURE 22 VLAN Scoping at the Port Level Within a ToR
Configuration
The configuration steps are similar to the L2 extension illustrated in “Example-4: VLAN Scoping at the ToR Level.” The difference is in
the virtual-fabric port-VLAN scoping on the vLAG pair.
A sample configuration is given below as a quick reference for port-VLAN scoping. In this example, {po111, c-tag 10} and {Te 1/0/3, c-tag
20} are mapped to VLAN 6000.5
With this configuration, it is possible to bridge traffic on these ports with the specified dot1q tags.
The configuration is shown in three parts for clarity:
∙ Common configurations, such as port channel and VLANs, are shown in one block.
∙ The tenant, Layer 3 interfaces, and BGP EVPN configurations are shown in the second block under each RBridge ID.
∙ The common overlay-gateway configuration is shown in the third block.
Please note that the entire configuration is applied from the primary node in this two-node vLAG pair.
5 Multiple c-tags on the same L2 port cannot be mapped to a VLAN.
136 53-1004308-03

53-1004308-03 137

138 53-1004308-03

53-1004308-03 139

140 53-1004308-03

Verification
In the output below from the Leaf1 vLAG pair, there is one tunnel for VLAN 6000, which indicates that the same VLAN/VNI segment is
provisioned on one other VTEP or ToR. Note that one of the tunnels, Tu 61445, is destined to Leaf5. Also note that there are four
underlay next hops to reach this tunnel destination in the fabric.
53-1004308-03 141

In the output below from Leaf5, Tunnel 61442 is destined to the vLAG Leaf1 pair's VTEP IP 10.121.1.1
142 53-1004308-03

Remote Host Entries in the Extended VLAN BGP and ARP Table on Leaf5
53-1004308-03 143

The table below taken on Leaf5 shows the BGP and ARP entries of the remote hosts behind the Leaf1 pair. Note that the next hop is set
to 10.121.1.1, which is a common VTEP IP of the vLAG pair. There are two entries in BGP since there are two spines exchanging the
EVPN routes.
In the ARP table, both the local and remote entries are indicated with different types: BGP-EVPN for remote entries, signifying that they
were learned over BGP-EVPN; Dynamic for local entries. Note that the remote host entries are imported into the virtual interface of local
VLAN 40 on Leaf5.
Example-6: Route Leaking for the Service VRF
With network virtualization for multitenant environments, typically the tenant VRFs are extended to the border leaf and they are
connected to the service VRF through a firewall/NAT/LB appliance to a service VRF. This poses a challenge of VRF and interface
scalability on the border leaf. In these cases, we recommend provisioning multiple border leafs and distributing the tenants across them.
144 53-1004308-03

FIGURE 23 Services Provisioning on the Border Leaf
A service VRF with route leaking addresses the scalability requirements on the border leaf for certain controlled deployments. The routes
to the services are leaked to the tenants in the fabric and vice-versa without the need to extend these tenant VRFs to the border leaf. As
shown in Figure 24, the edge leaf does not have the tenant VRFs provisioned on it. The routes from the tenants are imported into the
service VRF, and the service VRF typically advertises a default route toward the tenants in the fabric. There are other possible variations
with this approach. One may connect the storage directly to the service VRF itself. It is also possible to connect to the Internet directly
from the service VRF if the tenants have globally scoped addresses or if address translation occurs elsewhere.
FIGURE 24 Service VRF with Route Leaking on the Border Leaf
Since the routes between the tenants and the service VRF are leaked between each other, consider the following points:
∙ Unique IP addressing is needed for the tenants.
∙ Provisioning a per-tenant stateful firewall would be a challenge. One device must be able to handle all the transactions. So
carefully consider the scale requirements of the firewall.
∙ Intertenant traffic is possible through the service VRF because all routes are imported there. To prevent this, we recommend
having the necessary safeguards inside the tenants.
53-1004308-03 145

FIGURE 25 Topology of the Service VRF with Route Leaking from Tenants
Figure 25 shows a part of the validated topology to illustrate route leaking between tenant VRFs and the service VRF. As shown, there
are two tenant VRFs in the fabric: VRF202 and VRF203. Also note that VRF202 is also extended to Leaf5 (in other words, the tenant is
provisioned on two racks). These tenants are expected to have access to a common service attached to the border leaf. The border leafs
have been configured with a service VRF. Each VRF has its own L3 VNI for symmetric routing.
The routes from tenants are leaked into the service VRF, and routes from Service are leaked into all tenant VRFs using export/import
route targets, as shown in the table below.
Leaf1 VLAG Pair Leaf5 Edge-Leaf1 Edge-Leaf2
Tenant vrf202 vrf202, L3VNI 7202
Export RT 202:202
Import RT 202:202
Import RT 8190:8190
vrf202, L3VNI 7202
Export RT 202:202
Import RT 202:202
Service, L3VNI 8190
Import RT 202:202
Export RT 8190:8190
Import RT 203:203
Service, L3VNI 8190
Import RT 202:202
Export RT 8190:8190
Import RT 203:203
Tenant vrf203 vrf203, L3VNI 7203
Export RT 203:203
Import RT 203:203
Import RT 8190:8190
Not provisioned
As explained in the earlier sections on routing and in tenant extension illustrations, when the routes are exported or advertised from the
VRF, the L3VNI associated with the VRF is also included with the route. This creates an asymmetry in the L3VNI numbers in this case.
For example, see the table below:
Leaf1 Pair - VRF vrf202 Edge-Leaf1 - VRF Service Edge-Leaf2 - VRF Service
Advertise EVPN type-5 prefix route
10.111.9.0/24 and type-2 host routes
10.111.9.20/32 and 10.111.9.21/32.
∙ Export RT 202:202
The received route 10.111.9.0/24 matches import
RT 202:202.
But the L3VNI is 7202 and not 8190 (of VRF
service).
The received route 10.111.9.0/24 matches import
RT 202:202.
But the L3VNI is 7202 and not 8190 (of VRF
service).
146 53-1004308-03

∙ Next hop 10.121.1.1
∙ L3VNI 7202
∙ Create a VE interface and associate
with VNI 7202.
with VNI 7202.
The received route matches import RT
8190:8190.
But the L3VNI is 8190 and not 7202 (of
vrf202).
with VNI 8190.
Advertise EVPN prefix route 0/0 and
172.161.108.0/24.
∙ RT 8190:8190
∙ Next hop 10.123.3.1
∙ L3VNI 8190
172.161.108.0/24.
∙ RT 8190:8190
∙ Next hop 10.123.3.1
∙ L3VNI 8190
Similarly for the tenant VRF vrf203:
Advertise EVPN type-5 prefix route
10.111.17.0/24 and type-2 host routes
10.111.17.20/32 and 10.111.17.21/32.
∙ RT 203:203
∙ Next hop 10.121.1.1
∙ L3VNI 7203
Received route 10.111.17.0/24 matches import RT
203:203.
But L3VNI is 7203 and not 8190 (of VRF
service).
with VNI 7203.
Received route 10.111.17.0/24 matches import RT
203:203.
But L3VNI is 7203 and not 8190 (of VRF
service).
with VNI 7203.
The received route matches the import RT
8190:8190.
But the L3VNI is 8190 and not 7203 (of
vrf203).
with VNI 8190.
172.16.108.0/24.
∙ Export RT 8190:8190
∙ Next hop 10.123.3.1
∙ L3VNI 8190
Advertise EVPN prefix routes 0/0 and
172.16.108.0/24.
∙ Export RT 8190:8190
∙ Next hop 10.123.3.1
∙ L3VNI 8190
In summary:
∙ On the leafs, we must create one additional VE interface in the default VRF and associate it with a VNI number equal to the
L3VNI of the service VRF.
∙ On the border leaf, for every tenant that is leaked into the service VRF, create a VE interface in the default VRF and associate it
with the VNI number equal to the L3VNI of the tenant.
These additional VNIs must be activated in the EVPN instance by the leafs and border leafs.
Leaf1 Pair Leaf5 Border Leafs
VNI 8190, VLAN/VE 8190 in the default VRF VNI 8190, VLAN/VE 8190 in the default VRF VNI 7202, VLAN/VE 7202 in the default VRF
VNI 7203, VLAN/VE 7203 in the default VRF
Configuration
The following sections provide the incremental configuration relevant to the route leaking between the services and the tenant VRFs. A
default route and a subnet route are injected from the service VRF of the edge leaf into the fabric, and the tenants import it. The tenants'
VLAN subnets and host routes are similarly imported by the service VRF.
The Leaf1 vLAG pair has both vrf202 and vrf203 tenant VRFs.
53-1004308-03 147

148 53-1004308-03

Leaf5 has been provisioned with just the vrf202 tenant VRF.
53-1004308-03 149

Configuration on the Edge Leaf
The edge leaf is provisioned with only the service VRF. In this illustration, the edge leaf advertises two routes: a default route (say to a
service appliance) and a subnet route (say of a VLAN connecting storage network).
150 53-1004308-03

53-1004308-03 151

Verification
Route Learning from the Service VRF into Tenants
In the topology used in this illustration, the Service VRF is advertising a default route and a subnet route toward the tenants in the fabric
as an EVPN type-5 prefix route. The tenants (VRFs) on the leafs import these routes.
Route Origination from the Service VRF of the Edge Leaf :
Service VRF Routing Table
Service VRF BGP Entries
Advertising the Routes into EVPN
152 53-1004308-03

Routes Received by the Leaf1 vLAG Pair from the Service VRF:
EVPN Routes Received from Edge Leafs
There are two entries for the default route from each edge leaf, as there are two EVPN spines in the fabric. Also note that the Leaf1 vLAG
pair has both vrf202 and vrf203 tenants. The routes received from edge leafs are imported into both VRFs. The following output is
taken from one of the nodes in the vLAG pair. Verification steps are the same for the second node also.
53-1004308-03 153

VE Interface States
154 53-1004308-03

Tenant VRF vrf202
Tenant VRF vrf203
53-1004308-03 155

Routes Received by Leaf5 from the Service VRF
Leaf5 receives the routes advertised by the two edge leafs from two EVPN spine neighbors. The CLI output shows the BGP entry for
the default route.
156 53-1004308-03

Leaf5 Tenant VRF vrf202
Leaf5 imports the routes received from the service into tenant VRF vrf202.
53-1004308-03 157

158 53-1004308-03

Route Learning into the Service VRF from Tenants
The service VRF on the edge leaf learns hosts and subnet routes from the tenants in EVPN type-2 and type-5 routes respectively.
Leaf1 advertises subnet and hosts routes from tenants vrf202 and vrf203.
Leaf5 advertises subnet and hosts routes from tenant vrf202.
Tenant vrf202 has the same subnet extended (L2 extension) between Leaf1 and Leaf5. So verification should include the host entries as
well to ensure that they point to the correct VTEP IP of the ToR to which they're connected.
Leaf1 Leaf5 Edge-Leaf
Tenant vrf202
Subnet:
10.111.9.0/24
Hosts:
10.111.9.20
10.111.9.21
Tenant vrf202
Subnet:
10.111.9.0/24
Hosts:
10.111.9.50
10.111.9.51
VRF service
Subnets as trap routes:
10.111.9.0/24
10.111.17.0/24
Hosts routes behind VTEP next hops:
10.111.9.20 --> Leaf1 VTEP IP 10.121.1.1, VE 7202. VNI 7202
Tenant vrf203
Subnet:
10.111.17.0/24
Hosts:
10.111.17.20
10.111.17.21
Tenant vrf202 not
provisioned
Edge-Leaf1
Note that the subnet routes in the route table point to the VTEP next hops, but in hardware they're programmed as trap entries to
facilitate conversational host route download into the hardware.
The EVPN entry for one of the subnets, 10.111.9.0/24, is shown below. This route is advertised by both the Leaf1 vLAG pair (two nodes)
and Leaf5 (individual ToR). In the vLAG pair, both the nodes advertise the routes into BGP EVPN. So we see three BGP entries received
from two EVPN spines; hence a total of six entries.
53-1004308-03 159

VE Interface States
160 53-1004308-03

Routes Received from Tenant vrf202
Routes Received from Tenant vrf203
53-1004308-03 161

162 53-1004308-03

Design Considerations
Scale
The following table gives various scale parameters and platforms used in this validated test topology. Note that this is not a measure of
the maximum scale that can be supported with Brocade switches in IP fabric.
Parameter PoDY1 PoD2 Border Leaf
Platform used as leaf VDX 6940-144S VDX 6740 VDX 6940-36Q
Platform used as spine VDX 6940-36Q VDX 6940-36Q N/A
Number of server racks/leafs 8 8 N/A
Number of spines 4 4 N/A
Number of tenant VRFs per rack 106 20 70
Number of tenants local to the leaf (not extended to
other racks)
4 4 N/A
Number of tenants extended within the PoD to all racks 100 16 N/A
Number of server VLAN segments per rack 507 505 N/A
Number of VLANs used for L3 VNI of tenant VRFs per
rack
106 20 70
Number of L2 VNIs per rack 507 505 N/A
Number of L2 VNIs (server VLAN segments) extended
within the PoD to all leafs/racks
400 400 N/A
ARP-suppressed VLANs per leaf/rack 64 64 N/A
ND-suppressed VLANs per leaf/rack 12 12 N/A
Platform used as super-spine VDX 8770-4
Number of super-spines 4
Number of tenants extended between the PoDs 16
ARP/ND Suppression Guidelines
∙ This feature is enabled on a per-VLAN basis.
∙ Enabling this feature involves the hardware ACL table, and this resource is shared with other ACL features as well.
∙ ARP/ND suppression is needed only on server-facing VLANs.
∙ Enable ARP/ND suppression on both nodes of vLAG pairs.
∙ On individual non-redundant leafs, suppression is required only if the VLAN is L2-extended to other leafs.
∙ Use the DAI TCAM profile. With this profile, the validated scale is 64 and 12 VLANs for IPv4 and IPv6 respectively per leaf/
rack.
53-1004308-03 163

∙ In the case of a vLAG pair, the profile configuration must be set for each RBridge in the pair.
Recommendations for ISL Ports in a vLAG Pair Leaf
∙ We recommend picking ISL ports from the same port group on the switch. Port-group information about the leaf platforms is
given in the Brocade VDX hardware installation guides.
∙ For redundancy, we recommend having a minimum of two ISL ports between the switches in the vLAG pair
∙ The bandwidth requirement for ISL links depends on the number of fabric links and the traffic pattern. The ISL links are
primarily used for routed traffic received over the L3 VNI depending on the router MAC used in the data packet. A good rule of
thumb is to provision links with half the bandwidth of the fabric links. For example, if there are four 40G fabric links on each
switch, provision two 40G links as ISL between the switches.
Fabric Link Tracking on a vLAG Pair
With BGP/EVPN network virtualization, two spines are designated to exchange EVPN AFI routes. Loss of both links connecting these
EVPN spines would result in a traffic black-hole for the tenants. In a vLAG ToR, we can prevent this by tracking the links to EVPN spines
and isolating the node from the fabric if it loses those links by shutting down the remaining fabric links and server port-channel member
ports.
∙ On each node of the vLAG pair, identify the links connected to the spines that exchange EVPN routes.
∙ Track these links under other fabric links and the server-facing port-channel member ports.
The steps are shown in the following captures from one of the nodes in a vLAG leaf. Repeat the steps on the other node as well.
164 53-1004308-03

Track these two links under the remaining fabric ports.
Track under the server-facing port-channel member ports.
53-1004308-03 165

L2 Loop Detection and Prevention
Brocade leaf platforms provide two options for L2 loop detection and prevention.
∙ Detect MAC move and shut the L2 port.
∙ BGP EVPN dampening mechanism for L2 routes or MAC routes.
We recommend the following configuration to make the L2 port-shut take precedence. With this configuration, the L2 port will be shut
down if a MAC moves 5 times within an interval of 10s.
BGP TTL Security
This is applicable for eBGP peering only. This configuration can be applied to a specific neighbor or a peer group.
166 53-1004308-03

Appendix—Configuration of the Nodes
This appendix includes the relevant configurations of a few nodes in the fabric.
vLAG Active/Active Pair Leaf
! 2-node vLAG pair
! Node 1, Rbridge-id 45
! Node 2, Rbridge-id 46
vcs virtual-fabric enable
interface Vlan 701
description VLAN 701, VNI 701, Tenant vrf71;
!
interface Vlan 2001
description VLAN 2001, VNI 2001, Tenant vrf101; extended to POD2
!
interface Vlan 3001
description VLAN 3001, VNI 3001, Tenant vrf101; extended within POD1
!
interface Vlan 3802
description VLAN 3802, VNI 3802, Tenant vrf201;
!
interface Vlan 7071
description VLAN 7071, VNI 7071, Tenant vrf71; Layer 3 VNI
!
interface Vlan 7101
!
interface Vlan 7201
!
! Node 1 in the vLAG pair
! L3, tenant VRFs, BGP, and EVPN-instance configuration
rbridge-id 45
ip anycast-gateway-mac 0201.0101.0101
ip router-id 10.121.1.11
vrf vrf101
rd 10.121.1.11:101
vni 7101
address-family ipv4 unicast
route-target export 101:101 evpn
route-target import 101:101 evpn
!
!
!
vrf vrf201
rd 10.121.1.11:201
vni 7201
!
!
!
vrf vrf71
rd 10.121.1.11:71
vni 7071
53-1004308-03 167

!
!
!
host-table aging-mode conversational
evpn-instance pod1-leaf1
route-target both auto ignore-as
rd auto
duplicate-mac-timer 5 max-count 3
vni add 4-6,701,2001,3001,3802
!
router bgp
local-as 4200000001
capability as4-enable
neighbor spine-evpn-group peer-group
neighbor spine-evpn-group remote-as 4200000000
neighbor spine-evpn-group password 2 $PVNHITJVPWQ=
neighbor spine-evpn-group bfd
neighbor spine-ip-group peer-group
neighbor spine-ip-group remote-as 4200000000
neighbor spine-ip-group password 2 $PVNHITJVPWQ=
neighbor spine-ip-group bfd
neighbor 10.11.1.0 peer-group spine-ip-group
neighbor 10.12.1.0 peer-group spine-evpn-group
network 10.121.1.1/32
maximum-paths 8
graceful-restart
!
address-family ipv4 unicast vrf vrf101
redistribute connected
maximum-paths 8
!
maximum-paths 8
!
maximum-paths 8
!
maximum-paths 8
!
maximum-paths 8
!
maximum-paths 8
!
address-family l2vpn evpn
graceful-restart
neighbor spine-evpn-group activate
neighbor spine-evpn-group allowas-in 1
neighbor spine-evpn-group next-hop-unchanged
!
!
ipv6 anycast-gateway-mac 0201.0102.0202
interface Loopback 1
no shutdown
ip address 10.121.1.1/32
!
no shutdown
168 53-1004308-03

ip address 10.121.1.11/32
!
interface Ve 701
vrf forwarding vrf71
ipv6 anycast-address fd2d:d47f:115:2bd::254/64
ipv6 nd cache expire 270
ip anycast-address 10.115.1.254/24
ip arp-aging-timeout 4
no shutdown
!
interface Ve 2001
ipv6 anycast-address fd2d:d47f:107:1::254/64
no shutdown
!
interface Ve 3001
ipv6 anycast-address fd2d:d47f:111:bb9::254/64
no shutdown
!
interface Ve 7071
ipv6 address use-link-local-only
no shutdown
!
interface Ve 7101
no shutdown
!
interface Ve 7201
no shutdown
!
!
! Node 2 in the vLAG pair
! L3, tenant VRFs, BGP, and EVPN-instance configuration
rbridge-id 46
vrf vrf101
rd 10.121.1.12:101
vni 7101
!
!
!
vrf vrf201
rd 10.121.1.12:201
vni 7201
!
!
!
vrf vrf71
53-1004308-03 169

rd 8052:71
vni 7071
!
!
!
rd auto
vni add 4-6,701,2001,3001,3802
!
router bgp
local-as 4200000001
network 10.121.1.1/32
maximum-paths 8
graceful-restart
!
!
maximum-paths 8
!
maximum-paths 8
!
!
maximum-paths 8
!
maximum-paths 8
!
graceful-restart
neighbor spine-evpn-group allowas-in 1
!
!
no shutdown
ip address 10.121.1.1/32
!
170 53-1004308-03

no shutdown
ip address 10.121.1.12/32
!
interface Ve 701
ipv6 anycast-address fd2d:d47f:115:2bd::254/64
no shutdown
!
interface Ve 2001
no shutdown
!
interface Ve 3001
no shutdown
!
interface Ve 7071
no shutdown
!
interface Ve 7101
no shutdown
!
interface Ve 7201
no shutdown
!
!
! Fabric infrastructure L3 links, server-facing links, and vLAGs
interface TenGigabitEthernet 45/0/5
channel-group 111 mode active type standard
fabric isl enable
fabric trunk enable
lacp timeout long
no shutdown
!
fabric isl enable
fabric trunk enable
lacp timeout long
no shutdown
!
fabric isl enable
fabric trunk enable
lacp timeout long
no shutdown
!
fabric isl enable
fabric trunk enable
lacp timeout long
no shutdown
53-1004308-03 171

!
fabric isl enable
fabric trunk enable
lacp timeout long
no shutdown
!
fabric isl enable
fabric trunk enable
lacp timeout long
no shutdown
!
interface FortyGigabitEthernet 45/0/97
mtu 9216
description Link to spine1
no fabric isl enable
no fabric trunk enable
ip mtu 9018
ip proxy-arp
ip address 10.11.1.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.12.1.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.13.1.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.14.1.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.11.2.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
172 53-1004308-03

ip address 10.12.2.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.13.2.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.14.2.1/31
no shutdown
!
interface Port-channel 111
vlag ignore-split
switchport
switchport mode trunk-no-default-native
switchport trunk allowed vlan add 701,3001
spanning-tree shutdown
no shutdown
!
vlag ignore-split
switchport
switchport trunk allowed vlan add 3802
no shutdown
!
vlag ignore-split
switchport
switchport trunk allowed vlan add 2001
no shutdown
!
mac-address-table learning-mode conversational
overlay-gateway leaf1
type layer2-extension
ip interface Loopback 1
attach rbridge-id add 45-46
map vlan vni auto
activate
!
Individual Non-Redundant Leaf
!Rbridge-id 51
interface Vlan 1101
description VLAN 1101, VNI 1101, Tenant VRF vrf111;
!
interface Vlan 2401
!
interface Vlan 3001
!
53-1004308-03 173

interface Vlan 7109
!
interface Vlan 7111
!
interface Vlan 7201
!
rbridge-id 51
vrf vrf109
rd 10.121.1.51:109
vni 7109
!
!
!
vrf vrf111
rd 10.121.1.51:111
vni 7111
!
!
!
vrf vrf201
rd 10.121.1.51:201
vni 7201
!
!
!
rd auto
vni add 1101,2401,3001
!
router bgp
local-as 4200000005
network 10.121.1.5/32
174 53-1004308-03

maximum-paths 8
graceful-restart
!
maximum-paths 8
!
maximum-paths 8
!
maximum-paths 8
!
maximum-paths 8
!
maximum-paths 8
!
maximum-paths 8
!
graceful-restart
!
!
no shutdown
ip address 10.121.1.5/32
!
no shutdown
ip address 10.121.1.51/32
!
interface Ve 1101
ipv6 anycast-address fd2d:d47f:119:44d::254/64
no shutdown
!
interface Ve 2401
no shutdown
!
interface Ve 3001
no shutdown
!
interface Ve 7109
no shutdown
!
interface Ve 7111
53-1004308-03 175

no shutdown
!
interface Ve 7201
no shutdown
!
switchport
switchport mode trunk
switchport trunk allowed vlan add 1101,2401,3001
switchport trunk tag native-vlan
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.11.7.1/31
no shutdown
!
mtu 9216
fabric isl enable
fabric trunk enable
ip mtu 9018
ip proxy-arp
ip address 10.12.7.1/31
no shutdown
!
mtu 9216
fabric isl enable
fabric trunk enable
ip mtu 9018
ip proxy-arp
ip address 10.13.7.1/31
no shutdown
!
mtu 9216
fabric isl enable
fabric trunk enable
ip mtu 9018
ip proxy-arp
ip address 10.14.7.1/31
no shutdown
!
mac-address-table learning-mode conversational
overlay-gateway leaf5
attach rbridge-id add 51
map vlan vni auto
activate
!
176 53-1004308-03

Spine Designated to Exchange Only Underlay Routes
rbridge-id 41
router bgp
local-as 4200000000
fast-external-fallover
neighbor leaf-group peer-group
neighbor leaf-group password 2 $PVNHITJVPWQ=
neighbor leaf-group bfd
neighbor 10.11.1.1 remote-as 4200000001
neighbor 10.11.1.1 peer-group leaf-group
neighbor 10.41.1.0 peer-group superspine-group
maximum-paths 8
graceful-restart
!
no shutdown
ip address 10.124.11.1/32
!
!
mtu 9216
description Link to leaf1-1 vLAG pair
ip mtu 9018
ip proxy-arp
ip address 10.11.1.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.11.2.0/31
no shutdown
!
mtu 9216
description Link to superspine-4
ip mtu 9018
ip proxy-arp
ip address 10.44.1.1/31
no shutdown
!
53-1004308-03 177

mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.43.1.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.42.1.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.41.1.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.11.3.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip address 10.11.4.0/31
no shutdown
!
mtu 9216
description Link to leaf3
ip mtu 9018
ip address 10.11.5.0/3
no shutdown
!
mtu 9216
ip mtu 9018
ip address 10.11.6.0/31
no shutdown
!
mtu 9216
ip mtu 9018
178 53-1004308-03

ip address 10.11.7.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip address 10.11.8.0/31
no shutdown
!
Spine Designated to Exchange Both Underlay and Overlay Routes
rbridge-id 42
router bgp
local-as 4200000000
neighbor leaf-group peer-group
neighbor leaf-group password 2 $PVNHITJVPWQ=
neighbor leaf-group bfd
neighbor superspine-evpn-group peer-group
neighbor superspine-evpn-group remote-as 4200000020
neighbor superspine-evpn-group password 2 $PVNHITJVPWQ=
neighbor superspine-evpn-group bfd
neighbor superspine-ip-group peer-group
neighbor superspine-ip-group remote-as 4200000020
neighbor superspine-ip-group password 2 $PVNHITJVPWQ=
neighbor superspine-ip-group bfd
neighbor 10.41.2.0 peer-group superspine-ip-group
neighbor 10.42.2.0 peer-group superspine-evpn-group
maximum-paths 8
graceful-restart
!
graceful-restart
retain route-target all
neighbor superspine-evpn-group activate
neighbor superspine-evpn-group next-hop-unchanged
neighbor leaf-group activate
neighbor leaf-group next-hop-unchanged
!
!
no shutdown
ip address 10.124.12.1/32
!
53-1004308-03 179

!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.12.1.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.12.2.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.41.2.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.42.2.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.43.2.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.44.2.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.12.3.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.12.4.0/31
no shutdown
180 53-1004308-03

!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.12.5.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.12.6.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.12.7.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.12.8.0/31
no shutdown
!
Super-Spine Designated to Exchange Only Underlay Routes
rbridge-id 67
router bgp
local-as 4200000020
neighbor edge-group peer-group
neighbor edge-group remote-as 4200000021
neighbor edge-group password 2 $PVNHITJVPWQ=
neighbor edge-group bfd
neighbor pod1_spine-group peer-group
neighbor pod1_spine-group remote-as 4200000000
neighbor pod1_spine-group password 2 $PVNHITJVPWQ=
neighbor pod1_spine-group bfd
neighbor pod2_spine-group peer-group
neighbor pod2_spine-group remote-as 4200000010
neighbor pod2_spine-group password 2 $PVNHITJVPWQ=
neighbor pod2_spine-group bfd
neighbor 10.31.1.1 peer-group edge-group
neighbor 10.41.1.1 peer-group pod1_spine-group
53-1004308-03 181

maximum-paths 8
graceful-restart
!
!
no shutdown
ip address 10.125.5.1/32
!
!
mtu 9216
description Link to pod1-spine1
ip mtu 9018
ip proxy-arp
ip address 10.41.1.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.41.2.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.41.3.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.41.4.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.41.5.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.41.6.0/31
no shutdown
!
mtu 9216
182 53-1004308-03

ip mtu 9018
ip proxy-arp
ip address 10.41.7.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.41.8.0/31
no shutdown
!
mtu 9216
description Link to edge-leaf1
ip mtu 9018
ip proxy-arp
ip address 10.31.1.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.31.2.0/31
no shutdown
!
Super-Spine Designated to Exchange Both Underlay and Overlay Routes
rbridge-id 68
router bgp
local-as 4200000020
neighbor edge-group peer-group
neighbor edge-group remote-as 4200000021
neighbor edge-group password 2 $PVNHITJVPWQ=
neighbor edge-group bfd
neighbor pod1_spine-evpn-group peer-group
neighbor pod1_spine-evpn-group remote-as 4200000000
neighbor pod1_spine-evpn-group password 2 $PVNHITJVPWQ=
neighbor pod1_spine-evpn-group bfd
neighbor pod1_spine-ip-group peer-group
neighbor pod1_spine-ip-group remote-as 4200000000
neighbor pod1_spine-ip-group password 2 $PVNHITJVPWQ=
neighbor pod1_spine-ip-group bfd
neighbor pod2_spine-evpn-group peer-group
neighbor pod2_spine-evpn-group remote-as 4200000010
neighbor pod2_spine-evpn-group password 2 $PVNHITJVPWQ=
neighbor pod2_spine-evpn-group bfd
neighbor pod2_spine-ip-group peer-group
neighbor pod2_spine-ip-group remote-as 4200000010
neighbor pod2_spine-ip-group password 2 $PVNHITJVPWQ=
neighbor pod2_spine-ip-group bfd
neighbor 10.42.1.1 peer-group pod1_spine-ip-group
53-1004308-03 183

neighbor 10.42.2.1 peer-group pod1_spine-evpn-group
maximum-paths 8
graceful-restart
!
graceful-restart
retain route-target all
neighbor pod2_spine-evpn-group activate
neighbor pod2_spine-evpn-group next-hop-unchanged
neighbor pod1_spine-evpn-group activate
neighbor pod1_spine-evpn-group next-hop-unchanged
neighbor edge-group activate
neighbor edge-group next-hop-unchanged
!
!
no shutdown
ip address 10.125.5.2/32
!
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.42.1.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.42.2.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.42.3.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.42.4.0/31
no shutdown
!
mtu 9216
184 53-1004308-03

ip mtu 9018
ip proxy-arp
ip address 10.42.5.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.42.6.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.42.7.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.42.8.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.32.1.0/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.32.2.0/31
no shutdown
!
Edge Leaf
!Rbridge-id 71
interface Vlan 3945
description Connectivity to the external router for vrf71
!
interface Vlan 3957
description Connectivity to the external router for vrf101
!
interface Vlan 7071
description VLAN 7071, VNI 7071, L3 VNI for VRF71
!
53-1004308-03 185

interface Vlan 7101
description VLAN 7101, VNI 7101, L3 VNI for VRF101
!
rbridge-id 71
vrf vrf101
rd 10.123.4.1:101
vni 7101
!
!
!
vrf vrf71
rd 10.123.4.1:71:71
vni 7071
!
!
!
evpn-instance edge-leaf
rd auto
!
router bgp
local-as 4200000021
neighbor superspine-evpn-group peer-group
neighbor superspine-evpn-group remote-as 4200000000
neighbor superspine-evpn-group password 2 $PVNHITJVPWQ=
neighbor superspine-evpn-group bfd
neighbor superspine-ip-group peer-group
neighbor superspine-ip-group remote-as 4200000000
neighbor superspine-ip-group password 2 $PVNHITJVPWQ=
neighbor superspine-ip-group bfd
network 10.123.3.1/32
maximum-paths 8
graceful-restart
!
neighbor 172.16.101.2 password 2 $PVNHITJVPWRNNl5D
neighbor 172.16.101.2 update-source ve-interface 3957
maximum-paths 8
!
neighbor 172.16.71.2 password 2 $PVNHITJVPWRNNl5D
neighbor 172.16.71.2 update-source ve-interface 3945
maximum-paths 8
!
neighbor fd2d:d47a:101:1::2 remote-as 101
186 53-1004308-03

neighbor fd2d:d47a:101:1::2 activate
neighbor fd2d:d47a:101:1::2 password 2 $PVNHITJVPWRNNl5D
neighbor fd2d:d47a:101:1::2 update-source ve-interface 3957
maximum-paths 8
!
neighbor fd2d:d47a:71:1::2 remote-as 101
neighbor fd2d:d47a:71:1::2 activate
neighbor fd2d:d47a:71:1::2 password 2 $PVNHITJVPWRNNl5D
neighbor fd2d:d47a:71:1::2 update-source ve-interface 3945
maximum-paths 8
!
graceful-restart
neighbor superspine-evpn-group activate
neighbor superspine-evpn-group next-hop-unchanged
!
!
no shutdown
ip address 10.123.3.1/32
!
no shutdown
ip address 10.123.4.1/32
!
interface Ve 3945
ipv6 address fd2d:d47a:71:1::1/64
ip proxy-arp
ip address 172.16.71.1/24
no shutdown
!
interface Ve 3957
ipv6 address fd2d:d47a:101:1::1/64
ip proxy-arp
ip address 172.16.101.1/24
no shutdown
!
interface Ve 7071
no shutdown
!
interface Ve 7101
no shutdown
!
interface TenGigabitEthernet 71/0/36:1
switchport
switchport mode trunk
switchport trunk allowed vlan add 3921-3969
switchport trunk tag native-vlan
fabric isl enable
fabric trunk enable
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.31.1.1/31
no shutdown
!
mtu 9216
53-1004308-03 187

description Link to superspine2
ip mtu 9018
ip proxy-arp
ip address 10.32.1.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.33.1.1/31
no shutdown
!
mtu 9216
ip mtu 9018
ip proxy-arp
ip address 10.34.1.1/31
no shutdown
!
overlay-gateway edge-leaf
attach rbridge-id add 71
map vlan vni auto
activate
!
188 53-1004308-03

References
1. BGP MPLS-Based Ethernet VPN
https://tools.ietf.org/html/rfc7432
2. Use of BGP for routing in large-scale data centers
https://datatracker.ietf.org/doc/draft-ietf-rtgwg-bgp-routing-large-dc/
3. Integrated Routing and Bridging in EVPN
https://datatracker.ietf.org/doc/draft-ietf-bess-evpn-inter-subnet-forwarding/
4. RFC 4760: Multiprotocol Extensions for BGP-4
https://datatracker.ietf.org/doc/rfc4760/
5. RFC 4364: BGP/MPLS IP Virtual Private Networks (VPNs)
https://datatracker.ietf.org/doc/rfc4364/
6. A Network Virtualization Overlay Solution using EVPN
https://datatracker.ietf.org/doc/draft-ietf-bess-evpn-overlay/
7. Brocade Data Center Fabric Architectures white paper
http://www.brocade.com/content/dam/common/documents/content-types/whitepaper/brocade-data-center-fabric-
architectures-wp.pdf
8. Brocade VDX hardware installation guides
http://www.brocade.com/content/html/en/hardware-installation-guide/vdx6740-installguide/index.html
http://www.brocade.com/content/html/en/hardware-installation-guide/vdx6940-installguide/index.html
53-1004308-03 189

brocade-ip-fabric-bvd-published

More Related Content

Viewers also liked (14)

Similar to brocade-ip-fabric-bvd-published (20)

brocade-ip-fabric-bvd-published