Designing Multi-tenant Data Centers Using EVPN
- 2. Objectives
- 3. Architecture Objectives – Evolving DC Requirements • Operational simplicity via uniform control, data plane across L2, L3, DC, WAN • Flexible workload placement and mobility within DC and across DCs • Efficient bandwidth utilization within DC – no flood and learn, ECMP • Traffic engineering - traffic steering, ECMP, FRR • Horizontal Scaling • Multi-tenancy with L2 and L3 VPN in DC • Interworking with Legacy L3VPN / L2VPN WAN
- 4. A DC network fabric must ..... Leaf-1 Leaf-2 Leaf-3 Spine Spine Leaf-4 VM BD-1 BD-1 VM BD-2BD-2 VMVM Leaf-5 VM BD-2 ..... be seamless and act like a single switch / router Leaf-x Leaf-x+1 VM VMVM VM
- 5. Why not VPLS? Why not use VPLS in DC? Simply not designed for DC use-case L2 Only No All-Active Redundancy No per-flow ECMP Load-balancing Flood and Learn MAC learning Is Sub-optimal
- 9. Underlay vs. Overlay Underlay = Transport Physical Network IP, MPLS / SR Transport Traffic Steering, ECMP, FRR,..... Overlay = VPN (L2+L3) Control Plane – EVPN Data Plane – MPLS, VXLAN,..... Policy Driven
- 11. BGP EVPN – EVI VMVM VMVMVM EVI 20 EVI 10 EVI extended over BGP-EVPN Fabric to all the Leafs belonging to the EVI Leafs that don’t belong to a specific EVI will not have MAC-VRF for that EVI, providing efficient scalability EVI: An EVPN instance extends Layer 2 between the Leafs Leaf Spine
- 12. BGP EVPN – Host Connectivity Options, ESI • Ethernet Segment Identifier (ESI) ‘0’ • No DF election Single Home Device (SHD) Multi-home (MHD) All-Active (Per- Flow) LB VM VM ESI-0 ESI-0 ESI-1 ESI-1 • Identical ESI on Leafs • Per VLAN DF election VMSingle homed host Multi-homing with Link Bundling Leaf Spine
- 13. BGP EVPN – MAC and IP Learning • MAC/IP addresses are advertised along with L2 and L3 VPN encap (MPLS label or VNID ) to rest of Leafs via MAC+IP RT-2 • IP Prefix routes are advertised via BGP EVPN via RT-5 Leaf Spine Data Plane, ARP, ND learning from the hosts VMVM VMVM RR RR EVPN Route Type 2 carries MAC and IP reachability with L2+L3 VPN encapsulation, L2+L3 RTs RD Ethernet Segment Identifier Ethernet Tag ID MAC Address Length MAC Address IP Address Length IP Address MPLS Label1 MPLS Label2
- 19. VM Mobility – MAC + IP Challenge: How to detect the correct location of MAC after the movement of host from one Ethernet Segment to another also called “MAC move”? 19 VMVM IP-1 MAC-1 Leaf-3Leaf-1 MAC IP ESI Seq. Next-Hop MAC-1 IP-1 0 0 Leaf-1 Host move Leaf-4Leaf-2 Sequence number and Next-Hop value will be changed after the host move 0x00 Reserved Sequence Number 0x06 Mobility EXT-COMM with EVPN RT-2 carries MAC+IP route sequence number to enable MAC mobility
- 20. VMVM IP-1 MAC-1 Leaf-3Leaf-1 MAC IP ESI Seq. Next-Hop MAC-1 IP-1 0 1 Leaf-3 Leaf-4 ESI-1 Leaf-2 Sequence number is incremented and Next-hop is changed to Leaf-3 VM Mobility, continued
- 22. Overlay Routing Architectures • Centralized Routing • Distributed Routing – Asymmetric IRB • Distributed Routing – Symmetric IRB
- 23. Leaf-1 Leaf-2 Leaf-3 Spine Spine Leaf-4 VM VLAN-1 VLAN-1 VM VLAN-2 VLAN-2 VM VM Leaf-5 VM VLAN-2 Bridging on the leaf Centralized Routing • east<->west routed traffic traverses to centralized L3 gateways • Scale bottleneck: • Centralized have full ARP/MAC state in the DC • Centralized GW needs to host all DC subnets IRB-1 GW MAC IRB-2 GW MAC IRB-1 GW MAC IRB-2 GW MAC Centralized Routing on the Spine Bridging on the leaf L3 L2
- 24. Distributed Routing – Asymmetric IRB • Egress subnet is always local • Inter-subnet packets routed directly to destination VM’s DMAC • Scale bottleneck: • All egress subnets needs to be present on ingress leaf • Ingress leaf maintains ARP/ND state every egress leaf Leaf-2 Leaf-3 Spine Spine Leaf-4 VM VLAN-1 VLAN-1 VM VLAN-2 VLAN-2 VM VM Leaf-5 VM VLAN-2 Routed and Bridged to remote VM IRB-1 GW MAC IRB-2 GW MAC IRB-1 GW MAC IRB-2 GW MAC IP or MPLS Transport (underlay routing) Bridging to local VM MAC IRB-2 GW MAC VLAN-2 IRB-2 GW MAC VRF L3 L2
- 25. Leaf-2 Leaf-3 Spine Spine Leaf-4 VM VLAN-1 VLAN-1 VM VLAN-2 VLAN-2 VM VM Leaf-5 VM VLAN-2 Routed to remote leaf Distributed Routing – Symmetric IRB IRB-1 GW MAC IRB-2 GW MAC IRB-1 GW MAC IRB-2 GW MAC IP or MPLS Transport (underlay routing) Routed to local VM IRB-2 GW MAC VRF • Remote VM IP is installed like a VPN IP route recursively over remote leaf next-hop • No adjacencies to remote hosts even if the subnet is local • Subnet does not need to be local on ingress leaf unless there are local hosts L3 L2
- 27. Leaf-2 Leaf-3 Spine Spine Leaf-4 VM VLAN-1 VLAN-1 VM VLAN-2 VLAN-2 VM VM Leaf-5 VM VLAN-2 Any-cast GW IP and MAC for subnet-a Symmetric IRB – Distributed Any-cast GW • Any-cast GW IP and Any-cast GW MAC configured on ALL leafs with local subnet • Essentially, Subnet GW is distributed across ALL leafs with local subnet GW IP-a GW MAC-a GW IP-b GW MAC- b GW IP-a GW MAC-a GW IP-b GW MAC- b GW IP-b GW MAC- b VLAN-2 GW IP-b GW MAC- b VRF VM Any-cast GW IP and MAC for subnet-a Any-cast GW IP and MAC for subnet-b Any-cast GW IP and MAC for subnet-b Any-cast GW IP and MAC for subnet-b
- 29. Leaf-2 Leaf-3 Spine Spine Leaf-4 VM-a VLAN-1 VLAN-1 VM VLAN-2 VLAN-2 VM VM-b Leaf-5 VM VLAN-2 Host Learning - ARP REQUEST contd. 1. IP packet destined to VM-b triggers ARP for VM-b on Leaf-1 from any-cast GW IP-b and any-cast GW MAC-b 2. ARP to VM-b flooded to all remote leafs where VLAN-b is stretched (via EVPN RT-3 enabled IR) 3. Leafs flood on local BD ports GW IP-a GW MAC-a GW IP-b GW MAC- b GW IP-a GW MAC-a GW IP-b GW MAC- b GW IP-b GW MAC- b VLAN-2 GW IP-b GW MAC- b VRF VM DIP: VM-b DIP: VM-b ARP: VM-b ARP: VM-b ARP: VM-b ARP: VM-b ARP: VM-b RT-2: VM-a
- 30. Leaf-2 Leaf-3 Spine-RR Spine Leaf-4 VM-a VLAN-1 VLAN-1 VM VLAN-2 VLAN-2 VM VM-b Leaf-5 VM VLAN-2 Host Learning – ARP REPLY, MAC+IP RT-2 GW IP-a GW MAC-a GW IP-b GW MAC- b GW IP-a GW MAC-a GW IP-b GW MAC- b GW IP-b GW MAC- b VLAN-2 GW IP-b GW MAC- b VRF VM ARP: VM-b ARP REPLY: VM-b VM-b-MAC GW MAC-b ARP: VM-b • ARP REPLY to GW MAC-b consumed on Leaf-4 and installed in ARP table • EVPN MAC+IP RT-2 advertised to remote leafs via RR EVPN RT-2 RD: Leaf-4: IVM-b--MAC VM-b-IP L23VPN LABEL / VNI L2 VPN LABEL / VNI NH-Leaf-4 L3-RT, L2-RT VM-b MAC Reachability installed in MAC-VRF across remote leafs VM-b IP Reachability installed in IP-VRF across remote leafs as BGP L3VPN route independent of subnet being local or not
- 31. Leaf-2 Leaf-3 Spine Spine Leaf-4 VM-a VLAN-1 VLAN-1 VM VLAN-2 VLAN-2 VM VM-b Leaf-5 VM VLAN-2 Routed to remote leaf IP VRF-a: IP-b/32 -> Leaf-4, L3VPN Label Inter-subnet traffic to VM-b IRB-1 GW MAC IRB-2 GW MAC IRB-1 GW MAC IRB-2 GW MAC IP or MPLS Transport (underlay routing) Routed to local VM-b IP VRF-a: IP-b/32 -> BVI ARP adjacency IRB-2 GW MAC VLAN-2 IRB-2 GW MAC VRF-a VM
- 32. Leaf-2 Leaf-3 Spine Spine Leaf-4 VM-a VLAN-1 VLAN-1 VM VLAN-2 VLAN-2 VM VM-b Leaf-5 VM VLAN-2 Bridged to remote leaf next-hop MAC-VRF: MAC-b -> Leaf-4, L2VPN Label Intra-subnet traffic to VM-b IRB-1 GW MAC IRB-2 GW MAC IRB-1 GW MAC IRB-2 GW MAC IP or MPLS Transport (underlay routing) IRB-2 GW MAC VLAN-2 IRB-2 GW MAC VRF VM Bridged to local VM-b MAC MAC-VRF: MAC-b -> BE1.1
- 33. Summary • Unified control, data plane across L2, L3, DC, WAN • Flexible workload placement and mobility across L2 Overlay • Optimal bandwidth utilization – no flood and learn, ECMP in overlay, underlay • Traffic engineering with MPLS fabric - traffic steering, ECMP, FRR • Horizontal Scaling with distributed symmetric IRB • Multi-tenancy with L2 and L3 VPN • Interworking with Legacy L3VPN / L2VPN WAN