Operationalizing VRF in the Data Center

v
Operationalizing VRF in the Data Center
Ivan Pepelnjak, David Ahern, Dinesh G Dutt
June 23, 2016

Agenda
Why Do We Need VRF in the DC
VRF in the Linux Kernel
Operational Workflow of VRF
June 29, 2016 cumulusnetworks.com 2

Key Takeaways
VRF is a first class citizen in Linux
Uniform model across hosts and routers
 Pushing routing all the way to the edge
Cumulus Linux 3.0 release supports all the
familiar constructs around VRF usage
June 29, 2016 cumulsnetworks.com 3

Agenda

5 © ipSpace.net 2016 VRF 101
What Is a Virtual Routing and Forwarding (VRF) Table?
VRF = independent routing instance
• Routing protocols, routing and forwarding table
• Management plane is common
• Exposure to DoS is shared
• Inter-device connectivity with trunking interfaces, overlays or MPLS
VRF-capable router
Management
Trunking interface or MPLS
OSPF topology BGP table
OSPF process BGP process
IP routing table
IP forwarding table

Typical Use Case: Management VRF
• Management plane processes attached to management VRF
• User VRF does not have a route to management IP address
• End-users cannot reach management plane even though they have IP connectivity to the switch
• Simpler than protecting every switch port with ACLs
VRF-capable switch
Management plane
Fabric
Management
VLAN

Layer-3 Path Isolation (VRFs) Can Replace VLANs
Facts:
• VLANs are used in data center networks as a security zone mechanism
• Layer-3 path isolation (L3VPN) provides the same (or better) isolation
• L3VPN scales better than VLANs
Use cases:
• Retain separation while connecting L2 pods over L3 core (DC or WAN)
• Combine ACLs with firewalls
• Complex traffic forwarding policies

Example: Multi-Tier Application with VRFs
Each segment needs a different default gateway
• Layer-2-only solution or multiple routing domains
You might need routing within a segment
(example: divert backup traffic directly to backup server)
• Multiple layer-3 forwarding domains are the only option
Implementations:
• MPLS/VPN
• VRF Lite (or Cisco EVN)
• EVPN with VXLAN Outside
Web servers App servers DB servers
Default gateway
Default gateway Default gateway

Layer-3 Path Isolation in DCI Environment
Replace stretched VLANs with L3VPN
• Retain separation of security zones
• Cannot solve hot VM mobility or stretched L2 clusters
• Terminate L2 domains within data center
• Use parallel VRFs, MPLS/VPN or EVPN (over MPLS or VXLAN)
Parallel VRFs also known as: VRF-Lite, Multi VRF, EVN (Cisco), MCE (HP)
DWDM, ME, IP
Data Center A
DC core
Data Center B
DC core

Agenda

cumulusnetworks.com
Linux Implementation for VRF
Linux Implementation
 Kernel v4.3 and forward
Developed by Cumulus Networks for Linux
 Consistent API across all Linux Devices - switches
and hosts

cumulusnetworks.com 12
Distribution
Kernel
Version
VRF
status
Capabilities
Cumulus Linux 3.0 4.1 enabled All + Mgmt VRF
Debian - stretch 4.6 enabled
IPv4, IPv6 global,
“VRF All” TCP sockets
Ubuntu 16.04 4.4 enabled IPv4, IPv6 global
Fedora 24
(May beta)
4.5 disabled none
Fedora 23 4.5 disabled none
Linux OS Distributions (June 2016)

Feature by Linux Kernel Version
13
Version Feature
4.3 Basic IPv4 support, FIB tracepoint
4.4 Basic IPv6 support (global addresses only)
4.5 “VRF-all” sockets, FIB6 tracepoint
4.6 Bug fixes, IPv6 global addresses on admin down
4.7 Ingress device for IP_PKTINFO, IP6_PKTINFO
4.8 IPv6 linklocal and multicast, local VRF traffic, FIB rule
cumulusnetworks.com
v4.8 == Cumulus Linux 3.1

cumulusnetworks.com
What about Network Namespaces?
Wrong Model for VRFs
 VRF is a Layer 3 separation
 Network Namespace is a device layer separation
14
Default namespace mgmt namespace
swp4 swp5 swp6
“red” namespace
neigh
swp2 swp3swp1 eth0
FIBsockets neigh FIBsockets
neigh FIB
sockets
lldpd ipquaggaptmd
collectd

cumulusnetworks.com
Overview of VRF Design for Linux
VRF represented as a virtual (pseudo)
network device
 VRF device correlates to routing table
 FIB rules for VRF device direct lookups to VRF table
 Network interfaces are associated with VRF device
 Host and connected routes moved to VRF table
L3-only concept
 Impacts only IPv4 and IPv6 route and socket
lookups

cumulusnetworks.com
Advantages of Device Model
Network interface-to-VRF association similar
to bridges
 familiar paradigm for Linux networking
16

cumulusnetworks.com
Network interface-to-VRF association similar
to bridges
netdevice is a fundamental construct in
Linux networking stack
 Network addresses on VRF device
VRF-local loopback (127.0.0.1 and unnumbered
deployments)
 netfilter, tc, tcpdump on VRF device
17

cumulusnetworks.com
to bridges
VRFs can exist inside network namespaces
 Follows existing paradigms for network interfaces
and namespaces
18

cumulusnetworks.com
to bridges
Applications use existing socket APIs
 SO_BINDTODEVICE, IP_PKTINFO
19

cumulusnetworks.com
Network interface-to-VRF association similar to
bridges
netdevice is a fundamental construct in Linux
networking stack
Applications use existing socket APIs
Existing frameworks for monitoring,
serviceability
 iproute2 commands, netlink 20

Application Behavior With VRF
 Applications by default run in the default VRF context
 Applications that support binding to a device, can run within a
specified VRF
 Examples are ping, traceroute
 Applications that don’t support binding to a device need a
helper to set context to correct VRF
 Cumulus Linux has “vrf exec” to do this, in the process of upstreaming
 LD_PRELOAD is another option
 Applications that work in multiple VRFs concurrently need to
be modified
 Example is BGP
June 29, 2016 Cumulus Networks Confidential 21

Specifying VRF in Applications
Use existing POSIX APIs
 VRF context by binding sockets to device or cmsg
setsockopt(SO_BINDTODEVICE), IP_PKTINFO
22cumulusnetworks.com

Agenda
 Interface configuration
 Management VRF in Cumulus Linux
 Routing

cumulusnetworks.com
Cumulus Linux 3.0 Workflow: VRF Creation & Slave Port Attach
Define configuration in
/etc/network/interfaces
24
red FIB
table 1001
swp1 swp2
OR
auto all
iface red
address 127.0.0.1/8
vrf-table auto
iface swp1
vrf red
iface swp2
vrf red
auto all
iface red
address 127.0.0.1/8
vrf-table auto
%for i in range(1,4):
iface swp${i}
vrf red
%endfor

cumulusnetworks.com
Simplified Linux Workflow (Ubuntu 16.04)
25
Create VRF device
with table id
Setup FIB rules
pointing to device
Add default route for
VRF
Enslave interfaces
ip link add red type vrf table 123
ip link set red up
ip rule add iif red table 123
ip rule add oif red table 123
ip -6 rule add iif red table 123
ip -6 rule add oif red table 123
ip route add table 123 unreachable default
metric 8192
ip -6 route add table 123 unreachable default
metric 8192
ip link set eth1 master red
Ip link set eth2 master red

VRF Implementation in Cumulus Linux
VRF name is purely an operator's choice.
 Exception is Management VRF; must be “mgmt”
Table id’s limited to 1001-1255
 Hardware considerations + future growth in number of VRFs
Table-id to name mapping is saved
 allows for name-based reference in iproute2 commands

Agenda
 Routing

Management VRF
Management VRF
 Uses non-main table
 Name must be “mgmt”
 Loopback address on
mgmt device
Front panel ports
default to “Default VRF”
(main table)
auto all
iface mgmt
address 127.0.0.1/8
vrf-table auto
iface eth0 inet dhcp
vrf mgmt

Management VRF and Commands
Commands need to specify VRF to use – in general
 ping and traceroute wrappers default these commands to front
panel ports and default VRF
• Use ‘ping –I <vrf> …’ and ‘traceroute –i <vrf> …’ to specify VRF of
interest
 Helper for commands that do not support bind to device or
interface options: vrf exec <vrf> <command>
• Remember, affects IPv4 and IPv6 sockets only!

Management VRF: Default VRF “Context”
Login context defaults to Management VRF
 IPv4/IPv6 sockets opened by processes automatically bound
to Management VRF device
 Do not need to specify VRF for commands unless a different
VRF is wanted (except ping and traceroute because of the
wrappers)
 Default PS1 bash prompt provides VRF context
cumulus@cel-redxp-99:mgmt-vrf:~$

Enabling Services in VRFs
systemd based services can be run in a VRF context
 systemctl <action> <service>@<vrf>
• systemctl enable ntp@mgmt.
• systemctl start ntp@mgmt
 Possible collision with port if service is running in default VRF
• Process in default VRF with wildcard bind owns port
across all VRFs
Loopback address on Management VRF allows ntpq
to work
Cumulus Linux specific, upstream WIP

Agenda
 Routing

Routing Support for VRF
Routing protocol constructs (routes,
nexthops, router-id etc.) and processing are
VRF-aware.
 Cumulus Quagga routing protocol suite
Static routes (IPv4 and IPv6) supported in a
VRF

BGP Support for VRF
Per-VRF neighbors
 EBGP/IBGP
 Numbered & Unnumbered
Per-VRF parameters (address-families,
redistribution etc.)
BFD support (single / multihop)

VRF provisioning - Quagga
‘bgp multiple-instance’ is ON by
default
3/13/16 cumulusnetworks.com 35
router bgp 64900 vrf red
bgp router-id 6.0.3.7
network 20.7.2.0/24
network 20.7.3.0/24
neighbor 10.1.1.2 remote-as 20
neighbor 20.1.1.2 remote-as 30
!
!
router bgp 64900 vrf blue
bgp router-id 18.0.4.7
neighbor CUSTPEER peer-group
neighbor CUSTPEER remote-as external
neighbor swp1.10 peer-group CUSTPEER
!
address-family ipv4 unicast
redistribute connected
exit-address-family
!

VRF Verification and Troubleshooting – Quagga
Various existing commands are VRF-aware
 show ip route [vrf <all | name>]
 show ip bgp [vrf <name>] summary
 show interface [vrf <name>]
Couple of new commands
 show vrf
 show bgp vrfs
Existing Quagga ‘debug’ commands cover
VRF too

VRF Hardware Acceleration
Supported on all switching silicon chipsets:
 Broadcom chipsets – T+, T2, Tomahawk
 Mellanox Spectrum
Kernel table id to VRF id mapping
 VRF_id = table_id – 1000
 Except for default VRF which uses VRF 0

Scale Numbers in Cumulus Linux 3.0
Maximum VRF supported is 64
No per-VRF limit on interfaces, neighbors,
routes etc.

Routing Protocol Caveats in CL 3.0
Only BGP supported
 OSPFv2 and OSPFv3 are not VRF-aware
Each VRF table in the kernel needs a default
route
 keeps lookups from dropping to the next table
 high metric allows overrides

Demo Topology
 Show ping and traceroute
 Traffic takes the shortest
path
 Node L1 is real switch
 Rest are VMs
 VRF Loopbacks are
assigned as:
 10.253.<vlan>.<nodeid>
 Non-VRF loopbacks are
assigned as:
 10.252.<>.<>
S1 S2 S3
L1 L2
NL1 NL2
FW1 FW2
H1 H2 H3 H4
10.7.x.3 10.7.x.4 10.8.x.5 10.8.x.6
10.252.0.210.252.0.1
10.253.2.9
10.253.3.9
10.253.4.9
10.253.2.10
10.253.3.10
10.253.4.10
10.253.2.11
10.253.3.11
10.253.4.11
10.253.2.13
10.253.3.13
10.253.4.13
10.253.2.7
10.253.3.7
10.253.4.7
10.253.2.8
10.253.3.8
10.253.4.8

cumulusnetworks.com
References
ifupdown2 support for Debian/Ubuntu
 ifupdown2 with VRF support to be released soon
 https://github.com/CumulusNetworks/ifupdown2
 https://github.com/CumulusNetworks/ifupdown2/blob/master/a
ddons/vrf.py
41

Summary
VRF is a first class citizen in Linux now
 For example, distributions such as Ubuntu 16.04
support it
VRF design is such that it takes advantage
of existing socket APIs
Routing support is via BGP and static routes
 Familiar configuration model
June 29, 2016 Cumulus Networks Confidential 42

•Routing on the Host: Concepts &
Customer Case Studies
•Guest Speaker: Ivan Pepelnjak
•When: July 21
Next Month’s Webinar

© 2016 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in
the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive
licensee of Linus Torvalds, owner of the mark on a world-wide basis.
ThankYou!
cumulusnetworks.com 44June 29, 2016

Operationalizing VRF in the Data Center

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Operationalizing VRF in the Data Center (20)

More from Cumulus Networks (13)

Recently uploaded (20)

Operationalizing VRF in the Data Center