Demystifying Networking Webinar Series- Routing on the Host

v
Routing on The Host: Concepts and Case Studies
Ivan Pepelnjak, Dinesh G Dutt
July 21, 2016

Agenda
Introduction
Use Cases
Detailed Design with Case Studies
Summary
July 26, 2016 cumulusnetworks.com 2

Routing on the Host
cumulusnetworks.com 3
 Extend L3 fabric all the way down
to the host by running Cumulus’
Routing App (Cumulus Quagga)
on the host
 Advertising /32 from host
 Automatic Segmentation and L3
convergence for reachability
leaf
HostsHostsHosts Hosts
spine
L3 Routing
Layer 3
ECMP
OSPF or
BGP
1.1.1.1/32
Confidential – Do Not Distribute

•Applications and Servers are the
last bastion of bridging

How Bridging Plays A Role in Application Design
Service or node discovery relies on broadcast
Cluster heartbeat uses multicast
Assumptions about being in a single subnet
VM Mobility continued this trend

Reasons Why Bridging Is How Compute Folks Think About
Networks
In the wild west days, IP routing was a low
performance and high cost solution since L2
switching was done in hardware
Vendors still charge extra for L3 licenses on the
same box:
 BGP costs even more money than OSPF
No good routing protocol stack on the host
L3 considered complex to configure and
troubleshoot compared to (mythical) L2 which was
plug-and-playJuly 26, 2016 cumulusnetworks.com 6

So What’s Changed ?
Two Key Factors
 Modern DC Applications
 Open Networking
July 26, 2016 Cumulus Networks Confidential 7

Routing Protocol Suite on Host
Many high quality open source routing suites now
available for the host
 Cumulus Quagga
 BIRD
Also commercial offerings are coming in:
 Windows Server 2012

•Use Cases

10 © ipSpace.net 2016 Routing on Hosts
Basics: Routing on Servers
Run a routing protocol on servers
• Advertise loopback IP addresses or host-specific prefixes to the network
• Easy to migrate IP addresses between servers
BGP
Advertise loopback IP with
BGP

Routing on Servers: Use Cases
• Anycast services (examples: DNS, logging, load balancing…)
• A cluster member advertises service-specific IP address
• Large-scale virtual environments (Project Calico, Mesos, containers…)
• Workload mobility and disaster recovery scenarios

Routing on Servers: Anycast
• All servers advertise the same IP address
• ECMP load balancing of network-to-server traffic
• Direct server return to avoid NAT and centralized state
• Ideal solution for stateless services (example: UDP)
• Potential disruption of individual TCP sessions after topology changes

Routing on Servers: Clustering
Assumption: cluster services are tied to fixed IP addresses (bad idea, but still commonly used)
• Cluster members advertise IP addresses of active services
• Routing protocol announcement is revoked (or removed) after cluster member or service failure
• Another cluster member takes over and advertises service IP address

Routing on Servers: Large-Scale Virtualization
Each host acts as a PE-router
• Advertises prefix assigned to it (numerous containers per prefix)
• Alternative: advertise a host route for every VM/container
• Example: Project Calico

Routing Protocol Selection
Don’t use a link-state routing protocol across administrative boundaries
• No filtering or policy enforcement
• Updates generated by any node are flooded across the network before they’re evaluated
• OSPF areas don’t really help (exception: stub or NSSA areas)
Technology options: RIP or BGP (preferred)

Routing Protocol Selection: BGP
• Use BGP between servers and leaf switches
• Easy-to-implement filters and prefix limits on ToR switches
Ideal implementations
• Dynamic BGP neighbors on leaf switches
• BGP over unnumbered links (available in recent Quagga code)

•Configuration Deep Dive

Supported Servers
Red Hat 7
 Includes CentOS and Fedora
Ubuntu
 12.04, 14.04 and 16.04 LTS releases
Docker
 Run as a container
 Easy consumption model for other server OSes

BGP Choices
Use of ASN
 Per server OR for all servers
Use of remote-as external

Defending the ToR
BGP was designed to work across trust boundaries
 Multiple knobs to control this
Use the following knobs on ToR:
 Accept only locally originated routes from server
 Limit number of routes accepted from server
 Announce only default route to server along with loopback of
ToR
• This is more to keep the routing tables on the server intact
 Further limit what prefixes are accepted if you know the list

Case Study 1: Replacing MLAG
 Replace multiple protocols
with BGP
 MLAG, STP variations, FHRP
 Fewer Ports
 No peer link required
 Simpler failure model
10.1.20.11/32
10.254.0.1 10.254.0.2 10.254.0.1 10.254.0.2
10.1.20.11/32
 Switch maintenance
window uncoordinated
with servers
 Routing protocols support
graceful turning off of links
 Ability to connect to more
than 2 ToRs

Sample Configs; Dual Attach Servers
log file /var/log/quagga/quagga.log
router bgp 65535
bgp router-id 10.1.20.11
bgp bestpath as-path multipath-relax
neighbor TOR peer-group
neighbor TOR remote-as external
neighbor TOR capability extended-nexthop
neighbor eth1 interface peer-group TOR
neighbor eth2 interface peer-group TOR
network 10.1.20.11/32
10.254.0.1 10.254.0.2
10.1.20.11/32
log file /var/log/quagga/quagga.log
router bgp 65535
bgp router-id 10.254.0.2
neighbor SRVR peer-group
neighbor SRVR remote-as external
neighbor SRVR capability extended-nexthop
neighbor SRVR route-map INROUTE in
neighbor SRVR route-map OUTROUTE out
neighbor SRVR maximum-prefix 10
neighbor SRVR default-originate
neighbor swp1 interface peer-group TOR
network 10.254.0.2
!
ip prefix-list DEFONLY seq 5 permit 0.0.0.0/0
ip as-path access-list 1 permit ^65535$
!
route-map INROUTE permit 10
match as-path 1
route-map OUTROUTE permit 10
match ip address prefix-list DEFONLY

Some Observations on Dual-Attach
Can’t use dynamic BGP neighbor model since we
have a single IP address
 BGP runs on TCP and so needs a separate IP per interface to
peer with neighbor
BGP Unnumbered usage simplifies the
configuration

Caveats on Dual-Attach Servers
Works well with static IP addressing
With DHCP, ensure DHCP server doles out the same
IP to both interfaces
 Or else use dhclient-exit-hooks to configure loopback
PXE boot doesn’t work yet
 Solution in the works

Case Study: Dual Attach Servers Without Vendor Lockin
Dual attach Ubuntu
server to Cumulus &
Arista
Uses IPv4 link-local
address
Advertises loopback IP
Receives Default Route
from both
169.254.0.2/31 169.254.0.4/31
169.254.0.5/31169.254.0.3/31
10.1.20.11/32
10.254.0.1 10.254.0.2
server1

Case Study: Use with Containers
Replace docker’s NAT model
 Improve performance
 Improve transparency
Use case valid within DC
 Docker model is geared towards cloud deployment
Use higher performance Linux drivers
 IPVLAN and MacVlan instead of bridge or overlay

Case Study: Anycast
Common case in DC to use distributed virtual
services
Distributed services use a common anycast address
How do you announce this address ?
 Current model is to use the ToR to announce or some other
entity and use set nexthop route-map to ensure

Sysctls: The Missing Manual For OSPF on Servers
If OSPF is the routing protocol chosen, set the
following sysctls to allow ospfd to receive multicast
frames:
 net.ipv4.conf.all.rp_filter = 0
 net.ipv4.conf.default.rp_filter = 0
 net.ipv4.conf.lo.rp_filter = 0

Scaling
Summarization is difficult in a
CLOS network with BGP
~32K containers in production
today
Modern switching silicon has
upwards of 128K IPv4 routing
entries
SPINE
LEAF

Connectivity To Outside World
July 11, 2016 30cumulusnetworks.com
Pod Border Pod
Internet
Summarize here

Running Quagga As Container
 Cumulus Quagga can also be run as a container
 docker run -t -i -d --net=host --privileged=true --
name=quagga_docker cumulusnetworks/quagga:xenial-latest
 Runs as privileged container to modify routing tables,
receive netlink notifications etc.
 --net=host allows full access to host routing tables,
interfaces etc.
 https://github.com/CumulusNetworks/cldemo-docker-
quagga
 For sample Ansible playbooks
July 11, 2016 31

Resources
Try the setup out on your laptop with these
resources:
 https://cumulusnetworks.com/routing-on-the-host/
 https://cumulusnetworks.com/cumulus-vx/
 https://www.virtualbox.org/
 https://www.vagrantup.com/

Summary
A new breed of applications are ushering in a
rethink of how networking is done in the DC
Building Pure L3 Fabrics is real
 Networks, Compute and Applications are showing how to do
this
 Standards-based, robust, scalable design
Cumulus Quagga is a high quality routing suite
used in production in many places
 Community and Enterprise supported
 Available for Docker, Ubuntu and Redhat

•Tips and Tricks of Network
Automation
•Guest Speaker: To be Announced
•When: August 30
Next Month’s Webinar

© 2016 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in
the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive
licensee of Linus Torvalds, owner of the mark on a world-wide basis.
ThankYou!

Demystifying Networking Webinar Series- Routing on the Host

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Demystifying Networking Webinar Series- Routing on the Host (20)

More from Cumulus Networks (12)

Recently uploaded (20)

Demystifying Networking Webinar Series- Routing on the Host