20130325 openstack-meetup
2 likes2,893 views
This document discusses the characteristics of Massively Scalable Data Centers (MSDCs) and some of the challenges they pose for network design. MSDCs have very large numbers of servers and virtual machines, unpredictable east-west traffic flows, and objectives of maximizing resource utilization and flexibility while reducing costs. Common enterprise network architectures can struggle to meet these challenges, with problems like poor server-to-server capacity and static network assignments. The document explores some alternative approaches used by MSDCs.
![[fa:16:3e:1a:47:44]
[fa:16:3e:b9:f5:12]
ComputeNode:
VM1
VM2
Note
how
similar
these
MAC
Quantum
OVS
plugin
&
addresses
are.
Nova
KVM/LibVirt.
eth0
eth0
VLAN
mode.
vnet0
vnet1
Two
VMs
on
one
[fe:16:3e:b9:f5:12]
[fe:16:3e:1a:47:44]
Quantum
network.
qbrdf55a325-‐5f
qbr6d0c7bd9-‐5f
Linux
bridge
Linux
bridge
qvbdf55a325-‐5f
qvb6d0c7bd9-‐5f
[06:48:ce:a1:a0:ea]
[66:a0:42:85:6c:d2]
One
VLAN
per
Hence,
there
can
tenant
Quantum
be
some
~4094
tag=2
br-‐int
tag=2
network.
It
is
local
tenant
networks
to
this
host
(not
on
a
compute
qvodf55a325-‐5f
OVS
qvo6d0c7bd9-‐5f
visible
outside).
node.
[22:ba:4f:96:bd:ac]
[fe:64:41:b6:fd:64]
in_port=1
int-‐br-‐eth2
Flow
rules
on
br-‐int:
[6e:a7:fc:68:14:47]
cookie=0x0,
duraHon=51002.478s,
table=0,
n_packets=134,
n_bytes=15045,
priority=2,in_port=1
acHons=drop
Trunks
VLAN
2,
500
veth
pair
cookie=0x0,
duraHon=50998.039s,
table=0,
n_packets=0,
n_bytes=0,
priority=3,in_port=1,dl_vlan=500
phy-‐br-‐eth2
acHons=mod_vlan_vid:2,NORMAL
cookie=0x0,
duraHon=51004.225s,
table=0,
n_packets=21,
n_bytes=1530,
priority=1
acHons=NORMAL
in_port=3
[92:55:59:31:f3:94]
Flow
rules
on
br-‐eth2:
br-‐eth2
cookie=0x0,
duraHon=51187.89s,
table=0,
n_packets=55,
n_bytes=7382,
priority=2,in_port=3
acHons=drop
cookie=0x0,
duraHon=51183.923s,
table=0,
n_packets=6,
n_bytes=428,
priority=4,in_port=3,dl_vlan=2
OVS
acHons=mod_vlan_vid:500,NORMAL
cookie=0x0,
duraHon=51189.523s,
table=0,
n_packets=94,
n_bytes=8854,
priority=1
acHons=NORMAL
eth2
Should
be
connected
to
Trunks
VLAN
500
network
for
tenant
traffic.
Must
NOT
be
assigned
an
IP
address.](https://image.slidesharecdn.com/20130325-openstack-meetup-130405162851-phpapp01/85/20130325-openstack-meetup-46-320.jpg)
20130325 openstack-meetup
- 1. MSDC networks OpenFlow / OpenStack Minnesota OpenStack Meetup steve ulrich sulrich@cisco.com / @sulrich 25-‐mar, 2013 kyle mestery kmestery@cisco.com / @mestery
- 2. background – about me • networking nerd in service provider domain – core rouHng / peering / L(2|3)VPN / broadband • past few years working with some of cisco’s MSDC customers – i’m not really interested in compute – moving bits is very interesHng to me MSDC = Massively Scalable Data Center • web scale providers – Google, Yahoo!, Amazon, Facebook, MicrosoX • various hosHng providers – RackSpace, SoXLayer, etc.
- 3. MSDC characterisHcs • single administraHve domain with a focus on infrastructure – service provider operaHonal orientaHon – highly automated operaHonal environments – server guys and router router guys not that far from each other • very large number of servers and VMs – O(100K) servers, millions of VMs • very large compute, storage and b/w requirements • very large number of applicaHon specific east-‐west flows • highly unpredictable traffic matrix and mixture • very mobile workloads • warehouse scale data centers, spread across metros and around the world MSDC design objec5ves • maximize resource uHlizaHon! – run the networks hot / run the servers hot / do not crank up the AC • maximize flexibility / mobility – run apps / analyHcs / experiments on the same infrastructure • reduce costs! – every penny counts at scale e.g.: $0.01 / KwH = ~$1500 / year / rack == ~$1M / pod/cluster / year
- 4. WARNING disclaimer -‐ what follows is not comprehensive, nor a how-‐to. these discussion points are not necessarily appropriate for everyone's environment. these are not problems everyone has but some of the concepts are potenHally useful for thinking about how things may evolve in emergent DC networks driven by OpenStack and associated technologies.
- 5. common enterprise DC architecture L3 AGG L2 ACCESS
- 6. problems with ^^that picture^^ • poor uHlizaHon of network capacity • sparse address space allocaHon and uHlizaHon • big failure domains • diverse troubleshooHng methodologies – L2 domain + L3 domain • lots of protocols + agendant complexity – STP/SPB/TRILL/FP + HSRP/VRRP + IP rouHng + services MSDC network architectures commonly favor a different set of tradeoffs than most enterprises can accept
- 7. challenges -‐ server to server capacity • hierarchical network -‐ servers in different L2 domains must go through the agg (L3) layer • bandwidth between servers in different parts of a DC can be severely constrained applicaHon bandwidth management is a hard opHmizaHon problem example: load balancing use case • some LB techniques (NAT/HALF-‐NAT/DSR) require that all DIPs in a VIP’s pool be in the same L2 domain. • if an applicaHon grows and requires more front-‐end servers, it cannot use available servers in other L2 domains – drives fragmentaHon and under-‐uHlizaHon of resources
- 8. challenges – staHc network assignment • common enterprise DC pracHce is to map specific applicaHons to specific physical switches and routers – commonly depends on VLANs and L3 based VLAN spanning to cover the servers dedicated to the applicaHon • derivaHve problems – VLANs drive policy VLAN = traffic management, security, and performance isolaHon – VLAN spanning places traffic on links high in the tree unpredictable traffic loading and oversubscripHon VLAN provisioning == total hass
- 9. MSDC network requirements • east-‐west traffic dominates – scager/gather workloads è 80% E-‐W / 20% N-‐S • must provide determinisHc latency • must be robust – design to absorb failure • automated, simplified management – setup / ongoing mgmt / debug • must readily enable varying levels of oversubscripHon – 1:1 / 2:1 / 3:1 – you get the point • must be scalable – 1K – 20K ports in a cluster/pod • enable seamless addiHon of capacity and service components • increasingly pervasive dual-‐stack environment – producHon traffic & management traffic
- 10. some philosophical perspecHves • crap is beger – build out with lots of dumb, high-‐capacity links – stuff’s going to break – deal. • overlays provide mobility and reachability for available resources learn to love the tunnel – don’t get too hung up on the flavor of encap seriously – focus on the funcHon and the value and think about the long-‐term value it provides that said: be careful – here be some dragons • automate everything
- 11. modern DC architecture 3 key components • topology that is based on SPINE a CLOS (fat tree) network • distributed control plane that handles forwarding • management framework LEAF that simplifies fabric management
- 12. be robust build a system that 1 2 3 N assumes things will fail. spine width capacity loss* 4 25% 8 12.5 16 6.25% 32 3.125 * for the loss of a single spine node
- 13. modular and scalable N ports – modular (or fixed) avails full bisecHonal bandwidth M ports -‐ fixed • supports a mix of switches and sizes • enables incremental deployment and growth
- 14. simple scaling characterisHcs Total # Leaf Ports Spine Ports Servers 48 48 1,152 64 768 24,576 " # of leaf ports % maximum total ports = # of spine ports × $ ' # 2 &
- 15. flexible topology parHHon-‐able for alternate topologies 2-‐3 Hers X:1 over-‐subscripHon
- 16. leaf agachment • avoid hotspots • provides for clean separaHon of control protocols • simplifies fabric management
- 17. some MSDC wrap up… • oXen take different approaches to a problem space than enterprises have the ability to • highly mobile workloads (and their placement) drive some of the same problems within OpenStack networks – customer / rack / service alignment is rarely right where you’d like it to be – worse sHll is applicaHon placement • managing servers and network at scale drives pervasive automaHon – program your intent
- 19. the physical network (aka the underlay) start with a physical network physical devices and physical connecHons
- 20. add an overlay overlay provides a base for the logical network
- 21. add topologies logical “switch” devices overlay the physical more switching! network tunnels! physical network carries data traffic for overlay network
- 22. lather, rinse, repeat mulHple overlay networks can co-‐exist at the same Hme on the same network infrastructure overlays provides logical network constructs for different tenants, applicaHons, workloads, etc.
- 23. WHAT DOES THIS ACHIEVE?
- 24. challenges -‐ server to server capacity MSDC approach • maximize bisecHonal bandwidth through network topology and configuraHon • appropriately place the workload enabled through the use of a programmable network topology … programmable (over|under)lay program the network to get to these states
- 25. challenges -‐ staHc network assignment MSDC approach • build the topology and expand the network appropriately over the fabric to address the applicaHon requirements • wash your hands of the issues associated with L2/L3 boundaries and scale beyond VLANs program the network to get to these states
- 26. ENTER OPENFLOW (OR SDN)
- 27. programmability at mulHple layers of the network butcher what you want, where you want ApplicaHon Developer Environment Management and OrchestraHon Analysis and Monitoring, Performance and Security Network Services program for harvest opHmized network behavior intelligence Control Plane Forwarding Plane Network Elements and AbstracHon SDN / OpenFlow Transport
- 28. Openflow controller central administraHon and operaHons point for network elements
- 29. Openflow Controller -‐ Northbound API Northbound API Integral part of Controller Network enabled applicaHon can make use of Northbound API to request services from the network
- 30. Openflow Device Agent • Agent runs on the network device • Agent receives instrucHons from Controller • Agent programs device tables
- 31. Openflow protocol Openflow Protocol A mechanism for the Openflow Controller to communicate with Openflow Agents
- 32. what is this OpenFlow protocol? instruc5on set for switching • an informa5on (switch) model – Match/AcHon/Tables (MAT) abstract switch model – not an implementaHon specificaHon • an applica5on layer protocol (ALP) – binary wire protocol and messages that allow programming of the MAT • a transport protocol – SSL/TLS -‐ transports the ALP
- 33. OpenFlow switch models OpenFlow 1.0 model Redirect to Controller Packet Flow Table Forward (TCAM) with edits Drop OpenFlow Switch Specification Version 1.1.0 Implemented OpenFlow 1.1 model OpenFlow Switch Packet + Ingress Packet ingress port + Packet port Execute In Table 0 metadata Table 1 ... Table n Packet Action Action Out Action Action Set Set Set = {} Set (a) Packets are matched against multiple tables in the pipeline + Mul5cast / ECMP /Anycast / MPLS ! Find highest-priority matching flow entry Match fields: " Match fields: " Apply instructions:
- 34. OpenFlow 1.1 (notable addiHon) OPENFLOW ONLY SWITCH Data Data Openflow Processing Pipeline OPENFLOW HYBRID SWITCH Openflow Processing Pipeline O OF U T or Data Data STD P U STD Ethernet Processing T Pipeline Openflow v1.1 defines two processing pipeline opHons OPENFLOW ONLY and OPENFLOW HYBRID
- 35. OpenFlow 1.2 IPv6 supported for lookup in flow table FLOW TABLE HEADER FIELDS COUNTERS ACTIONS … … … … … … both IPv4 and IPv6 flows supported in header field lookup MPLS TCP/UDP TCP/UDP Ingress Source Dest Ether VLAN VLAN MPLS IP IP IP IP Traffic SRC DEST Port MAC MAC Type ID Priority Label SRC DEST Protocol TOS Class ICMP Type ICMP Code
- 36. Openflow 1.3 new func5onality • IPv6 extended header support (MAT) • Match on MPLS Bogom of Stack (BoS) Bit • Provider Backbone Bridging (PBB) support • DuraHon field added for StaHsHcs • Support for Tunnel encapsulaHons (i.e. GRE ) • Ability to disable packet/byte counters on a per flow basis
- 37. Openflow Protocol Summary Openflow v1.0 IniHal Standard -‐ Most prevalent in the market today Openflow v1.1 Added support for mulHple flow tables Added support for MPLS Defines two operaHng modes -‐ Hybrid | Pure Openflow Openflow v1.2 adds support for IPv6 Openflow v1.3 adds support for Rate LimiHng | IPv6 extension headers GRE – the version deemed produc'on ready Openflow v1.3.1 adds negoHaHon TLV’s
- 39. OpenStack Openstack consists of a number of components Openstack Openstack Openstack Openstack Image Compute Object Store Quantum Service (NOVA) (SWIFT) Service (GLANCE)
- 40. Quantum is used to help manage the network
- 41. QUANTUM PLUGIN
- 42. OpenFlow and Quantum Plugins • Plugins which make use of OpenFlow – Ryu OpenFlow Controller Plugin – NEC OpenFlow Plugin – Nicira NVP Plugin – Big Switch OpenFlow Controller Plugin – Midokura MidoNet Plugin – Ruijie Networks Plugin
- 43. At a high level … • Most of the OpenFlow innovaHon has been around Open vSwitch – Some hardware vendors with OpenFlow support in HW are trickling in now • This means everyone is Hed to Open vSwitch at the virtual layer … – Some wiggle room if you use OVSDB vs. the OVS CLI for example
- 44. How Do The Plugins Use OpenFlow? • Most of them program Open vSwitch – Open vSwitch exposes OpenFlow as a northbound API – A large number uHlize CLI commands from an agent on the host – Some make use of the OVSDB protocol to program hosts directly
- 45. WHAT DOES THIS ALL LOOK LIKE?
- 46. [fa:16:3e:1a:47:44] [fa:16:3e:b9:f5:12] ComputeNode: VM1 VM2 Note how similar these MAC Quantum OVS plugin & addresses are. Nova KVM/LibVirt. eth0 eth0 VLAN mode. vnet0 vnet1 Two VMs on one [fe:16:3e:b9:f5:12] [fe:16:3e:1a:47:44] Quantum network. qbrdf55a325-‐5f qbr6d0c7bd9-‐5f Linux bridge Linux bridge qvbdf55a325-‐5f qvb6d0c7bd9-‐5f [06:48:ce:a1:a0:ea] [66:a0:42:85:6c:d2] One VLAN per Hence, there can tenant Quantum be some ~4094 tag=2 br-‐int tag=2 network. It is local tenant networks to this host (not on a compute qvodf55a325-‐5f OVS qvo6d0c7bd9-‐5f visible outside). node. [22:ba:4f:96:bd:ac] [fe:64:41:b6:fd:64] in_port=1 int-‐br-‐eth2 Flow rules on br-‐int: [6e:a7:fc:68:14:47] cookie=0x0, duraHon=51002.478s, table=0, n_packets=134, n_bytes=15045, priority=2,in_port=1 acHons=drop Trunks VLAN 2, 500 veth pair cookie=0x0, duraHon=50998.039s, table=0, n_packets=0, n_bytes=0, priority=3,in_port=1,dl_vlan=500 phy-‐br-‐eth2 acHons=mod_vlan_vid:2,NORMAL cookie=0x0, duraHon=51004.225s, table=0, n_packets=21, n_bytes=1530, priority=1 acHons=NORMAL in_port=3 [92:55:59:31:f3:94] Flow rules on br-‐eth2: br-‐eth2 cookie=0x0, duraHon=51187.89s, table=0, n_packets=55, n_bytes=7382, priority=2,in_port=3 acHons=drop cookie=0x0, duraHon=51183.923s, table=0, n_packets=6, n_bytes=428, priority=4,in_port=3,dl_vlan=2 OVS acHons=mod_vlan_vid:500,NORMAL cookie=0x0, duraHon=51189.523s, table=0, n_packets=94, n_bytes=8854, priority=1 acHons=NORMAL eth2 Should be connected to Trunks VLAN 500 network for tenant traffic. Must NOT be assigned an IP address.
- 47. QUESTIONS?
- 48. acknowledgements • slide 46 – courtesy Bob Melander • nice openflow artwork and overlay graphics shamelessly ripped off from carl solder & bryan osoro