SlideShare a Scribd company logo

Network Node is Not Needed Anymore - Completed Distributed Virtual Router / Fujitsu part

Download as PPTX, PDF
T
Takanori Miyagishi

The document discusses issues with the legacy Neutron network architecture and proposes a solution called Distributed Virtual Router (DVR). With DVR: - Networking agents like the L3 agent and metadata agent are distributed from the centralized Network Node to each Compute Node. This removes bottlenecks and single points of failure. - Most east-west and north-south traffic flows no longer need to pass through the Network Node. Floating IPs and router functions are handled locally on each Compute Node. - Some traffic like accessing the DHCP server and SNAT rules still require the Network Node, but overall DVR provides improved scalability and availability compared to the legacy centralized model.

Software
1 of 52
Downloaded 71 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
Network Node is Not Needed Anymore - Completed Distributed Virtual Router Copyright 2015 FUJITSU LIMITED0 OpenStack Summit October 2015 Tokyo Tuesday, October 27 - 2:50pm – 3:30pm Takanori Miyagishi Fujitsu Limited E-mail address: miyagishi.t@jp.fujitsu.com IRC name: miygishi_t
Agenda Vision Issues Solution Efforts in Liberty Future Plan Copyright 2015 FUJITSU LIMITED1
Vision Copyright 2015 FUJITSU LIMITED2
Vision  Deployment of enterprise systems on OpenStack  To achieve this from the network perspective, we need more scalability and availability. Current Neutron has the following issues.  Performance bottle neck  Single point of failure (SPOF) Copyright 2015 FUJITSU LIMITED3
Issues Copyright 2015 FUJITSU LIMITED4
Legacy Network Architecture (Legacy Router)  Neutron centralizes networking services to Network Node  Most of traffics need to go through Network Node Copyright 2015 FUJITSU LIMITED Compute Node Network Node ML2 plugin OVS-agent DHCP-agent Controller Node Neutron Server L3-agent ML2 plugin OVS-agentOVS-agent metadata-agent : Management network : External network : Data network 5
Legacy Network Architecture (Legacy Router) Copyright 2015 FUJITSU LIMITED Compute Node Network Node ML2 plugin OVS-agent DHCP-agent Controller Node Neutron Server L3-agent ML2 plugin OVS-agentOVS-agent metadata-agent : Management network : External network : Data network  Neutron centralizes networking services to Network Node  Most of traffics need to go through Network Node What happens due to this Architecture? 6
Issue #1: Network Node is Bottle neck  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-tun br-int br-tun br-int br-ex VM VM VM : External Network : Data Network DHCP br-tun br-int vRouter 7
Issue #1: Network Node is Bottle neck  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) Copyright 2015 FUJITSU LIMITED Not need to go through Network Node in the both cases: - VMs are on the same host - VMs are on different hosts Compute node #1 Network NodeCompute node #2 br-tun br-tun br-int br-ex VM VM VM : External Network : Data Network DHCP br-tun br-int vRouter br-int 8
Issue #1: Network Node is Bottle neck Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM VM : External Network : Data Network DHCP br-tun br-int vRouter br-tun br-int br-tun br-int  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) Not need to go through Network Node in the both cases: - VMs are on the same host - VMs are on different hosts 9
Issue #1: Network Node is Bottle neck Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM VM : External Network : Data Network DHCP vRouter br-tun br-int br-tun br-int br-tun br-int  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) Need to go through Network Node in the both cases: - VMs are on the same host - VMs are on different hosts 10
Issue #1: Network Node is Bottle neck Copyright 2015 FUJITSU LIMITED Need to go through Network Node in the both cases: - VMs are on the same host - VMs are on different hosts Compute node #1 Network NodeCompute node #2 br-ex VM VM VM : External Network : Data Network DHCP vRouter br-tun br-int br-tun br-int br-tun br-int  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) 11
Issue #1: Network Node is Bottle neck Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM VM : External Network : Data Network DHCP vRouter br-tun br-int br-tun br-int br-tun br-int  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) 12
Issue #1: Network Node is Bottle neck Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-tun br-int br-tun br-int br-ex VM VM VM : External Network : Data Network DHCP br-tun br-int vRouter  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) 13
Issue #1: Network Node is Bottle neck Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-tun br-int br-tun br-int br-ex VM VM VM : External Network : Data Network DHCP br-tun br-int vRouter  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) 14
Issue #1: Network Node is Bottle neck  Five out of six traffic patterns need to go through Network Node Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-tun br-int br-tun br-int br-ex VM VM VM DHCP br-tun br-int vRouter Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes (3) VM accessing to DHCP Yes (4) VM getting metadata Yes North- South (5) Using Floating IP Yes (6) Using SNAT Yes (1) (1) (2),(3),(4),(5),(6) Bottle neck 15
Issue #2: Network Node is SPOF  If Network Node fails, that affects many traffics Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM VM DHCP br-tun br-int vRouter Affected Traffics by Network Node Failure East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes (3) VM accessing to DHCP Yes (4) VM getting metadata Yes North- South (5) Using Floating IP Yes (6) Using SNAT Yes br-tun br-int br-tun br-int 16
Solution Copyright 2015 FUJITSU LIMITED17
DVR Architecture  L3-agent and Metadata-agent can be distributed on each Compute Node  But a part of L3-agent (SNAT) and DHCP-agent still need to run on Network Node Copyright 2015 FUJITSU LIMITED Compute Node Network Node ML2 plugin OVS-agent DHCP-agent Controller Node Neutron Server L3-agent(SNAT) ML2 plugin OVS-agentOVS-agent L3-agent metadata-agent Metadata-agent 18
DVR Architecture Copyright 2015 FUJITSU LIMITED Compute Node Network Node ML2 plugin OVS-agent DHCP-agent Controller Node Neutron Server L3-agent(SNAT) ML2 plugin OVS-agentOVS-agent L3-agent metadata-agent Metadata-agent  L3-agent and Metadata-agent can be distributed on each Compute Node  But a part of L3-agent (SNAT) and DHCP-agent still need to run on Network NodeWhat are benefits of the distribution of agents? - Will explain, comparing with Legacy Router 19
Traffics with Legacy Router again  five of six types of traffics needs through Network Node Copyright 2015 FUJITSU LIMITED Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes (3) VM accessing to DHCP Yes (4) Gets metadata Yes North- South (5) Using Floating IP Yes (6) Using SNAT Yes Compute node #1 Network NodeCompute node #2 br-tun br-int br-tun br-int br-ex VM VM VM DHCP br-tun br-int vRouter (1) (1) (2),(3),(4),(5),(6) 20
Changes in the Architecture by DVR  These components are distributed to each compute node  vRouter, Floating IP, and bridge (interface) to external network Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-tun br-int br-tun br-int br-ex VM VM DHCP br-tun br-int br-exbr-ex fip fip Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes (3) VM accessing to DHCP Yes (4) VM getting metadata Yes North- South (5) Using Floating IP Yes (6) Using SNAT Yes 21
Network flow of DVR Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM DHCP br-tun br-int br-exbr-ex fip fip br-tun br-int br-tun br-int  East-West traffic - Between VMs(same subnet)  Not need to go through Network Node in the both cases:  VMs are on the same host  VMs are on different hosts Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes (3) VM accessing to DHCP Yes (4) VM getting metadata Yes North- South (5) Using Floating IP Yes (6) Using SNAT Yes 22
Network flow of DVR Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM DHCP br-tun br-int br-exbr-ex fip fip br-tun br-tun br-intbr-int  East-West traffic - Between VMs(different subnet)  No longer need to go through Network Node  VM access to vRouter that exists on own Compute Node Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes -> NO (3) VM accessing to DHCP Yes (4) VM getting metadata Yes North- South (5) Using Floating IP Yes (6) Using SNAT Yes 23
Network flow of DVR Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM DHCP br-tun br-int br-tun br-int br-tun br-int br-exbr-ex fip fip  East-West traffic – Access to DHCP  Still need to go through Network Node Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes -> NO (3) VM accessing to DHCP Yes (4) VM getting metadata Yes North- South (5) Using Floating IP Yes (6) Using SNAT Yes 24
Network flow of DVR Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM DHCP br-tun br-tun br-int br-tun br-int br-exbr-ex fip fip br-int  East-West traffic – Gets metadata  No longer need to go through Network Node  VM access to vRouter that exists on own Compute Node Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes -> NO (3) VM accessing to DHCP Yes (4) VM getting metadata Yes -> NO North- South (5) Using Floating IP Yes (6) Using SNAT Yes 25
Network flow of DVR Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM DHCP br-tun br-tun br-int br-tun br-int br-ex fip br-int br-ex fip  North-South traffic – Using Floating IP  No longer need to go through Netowork Node  VM access to vRouter and translate to floating IP that exists on own Compute node Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes -> NO (3) VM accessing to DHCP Yes (4) VM getting metadata Yes -> NO North- South (5) Using Floating IP Yes -> NO (6) Using SNAT Yes 26
Network flow of DVR Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 VM VM DHCP br-tun br-int br-exbr-ex fip fip br-exbr-tun br-int br-tun br-int  North-South traffic – Using SNAT  Still need to go through Network Node Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes -> NO (3) VM accessing to DHCP Yes (4) VM getting metadata Yes -> NO North- South (5) Using Floating IP Yes -> NO (6) Using SNAT Yes 27
Network flow of DVR  3 types of traffic are no longer need to go through Network Node due to DVR Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM DHCP br-ex fip br-tun br-int (4) (5) (2) br-tun br-int br-ex fip br-tun br-int Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes -> NO (3) VM accessing to DHCP Yes (4) VM getting metadata Yes -> NO North- South (5) Using Floating IP Yes -> NO (6) Using SNAT Yes 28
Network flow of DVR  2 types of traffics still need to go through Network Node  Meaning there are still two issues (Bottleneck and SPOF) Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 VM VM DHCP br-tun br-int br-exbr-ex fip fip (6) (3) br-exbr-tun br-int br-tun br-int Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes -> NO (3) VM accessing to DHCP Yes (4) VM getting metadata Yes -> NO North- South (5) Using Floating IP Yes -> NO (6) Using SNAT Yes 29
Completely Distributed Virtual Router  To eliminate bottle neck and SPOF completely  Distributed SNAT and DHCP are required Copyright 2015 FUJITSU LIMITED Compute Node Network Node ML2 plugin OVS-agent DHCP-agent Controller Node Neutron Server L3-agent(SNAT) ML2 plugin OVS-agentOVS-agent L3-agent metadata-agent DHCP-agent 30
Efforts in Liberty Copyright 2015 FUJITSU LIMITED31
Distributed SNAT  Distribute remaining l3-agent(SNAT) on Network Node to each Compute Node Copyright 2015 FUJITSU LIMITED Compute Node Network Node ML2 plugin OVS-agent DHCP-agent Controller Node Neutron Server L3-agent(SNAT) ML2 plugin OVS-agentOVS-agent L3-agent metadata-agent 32
Conventional behavior of SNAT  Create SNAT on Network Node for each vRouter  If a tenant has multiple vRouters, then Neutron creates multiple SNATs  The amount of public IP consumption  # of vRouter that sets a gateway - 3 public IPs are needed in the following configuration Copyright 2015 FUJITSU LIMITED Compute Node #1 Network NodeCompute Node #2 VM VM VM SNATvRouterA1 vRouterA1'vRouterA2 vRouterA2'vRouterB1 vRouterB1' SNAT SNAT VM VM VM VM 33
Possible Idea for Distributed SNAT  Five ideas that I devised w/ some hints in the past discussions #1 One SNAT for one Router #2 One SNAT for one Compute Node #3 One SNAT for one Compute Node w/o Security Concern #4 Double NAT #5 BGP(Border Gateway Protocol)  Understood why Distributed SNAT has not been implemented yet It’s not easy at all to devise the best design…  Will explain these ideas from the next page  For each idea, will show an actual amount of public IP consumption in an example system configuration  Compute Node#1 and #2 have two VMs of tenant#1 and a VM of tenant#2 respectively Copyright 2015 FUJITSU LIMITED34
#1 One SNAT for one Router  Distribute SNAT simply … Follow the conventional behavior  The number of SNAT increases because each vRouter requires one SNAT Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VM VM SNATSNAT SNAT VMVM VM SNATSNAT SNAT VM .1 .3 .5 .2 .4 .6 35
#1 One SNAT for one Router  The amount of public IP consumption  (# of Compute Nodes) * (# of vRouters per Compute Node) … Too many!!  6 public IPs are needed in the example system configuration Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VM VM SNATSNAT SNAT VMVM VM SNATSNAT SNAT VM .1 .3 .5 .2 .4 .6 36
#2 One SNAT for one Compute Node  One SNAT shared by vRouters in Compute Node  In other words, sharing SNAT between different tenants Needs an exclusion of address scope between different tenants Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VM VM SNAT VMVM VM SNAT VM .1 .2 37
#2 One SNAT for one Compute Node  The amount of public IP consumption  # of Compute nodes … Sufficiently small  2 Public IPs are needed in the example system configuration  Security concern exists  If an admin makes a wrong setting on SNAT, VMs in the different tenants can communicate. Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VM VM SNAT VMVM VM SNAT VM .1 .2 38
#3 One SNAT for one Compute Node w/o Security Concern  SNAT shared by vRouters of the same tenant in Compute Node Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VMVM VM SNAT VM .1 SNAT .3 SNAT .4 VM VM SNAT .2 39
#3 One SNAT for one Compute Node w/o Security Concern  The amount of public IP consumption  (# of Compute nodes) * (# of Tenants)  If each tenant only has one vRouter, the consumption will be the same as idea #1 because (# of Tenants) becomes (# of vRouter)  4 public IPs are needed in the example system configuration Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VMVM VM SNAT VM .1 SNAT .3 SNAT .4 VM VM SNAT .2 40
#4 Double NAT  Double NAT on SNAT and upstream physical router  SNAT: translates private IP to a dedicated local IP (like ISP shared IP)  Upstream physical router: translates the local IP to a public IP  Need some changes to Floating IP mechanism - Floating IP needs to be able to allocate the local IPs to VMs Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VM VM SNATSNAT SNAT VMVM VM SNATSNAT SNAT VM .1 .3 .5 .2 .4 .6 Upstream Router NAT table 41
#4 Double NAT  Dynamic setting to upstream physical router is required  OpenStack doesn’t have such a feature yet  The amount of public IP consumption  (# of vRouter)  3 public IPs are needed Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VM VM SNATSNAT SNAT VMVM VM SNATSNAT SNAT VM .1 .3 .5 .2 .4 .6 Upstream Router NAT table (Need dynamic setting) 42
#5 BGP(Border Gateway Protocol)  vRouters in the same group share one public IP  Upstream physical router as a gateway  BGP speaker uses BGP to advertise the router which SNAT to connect in the same group  Neutron doesn’t have a feature like BGP Speaker yet Copyright 2015 FUJITSU LIMITED Compute Node #1 BGP SpeakerCompute Node #2 Upstream Router Peering VM SNAT .6 VM VM SNAT .2 SNAT .4 VM SNAT .5 VM VM SNAT .1 SNAT .3 43
#5 BGP(Border Gateway Protocol)  High cost  Development cost: BGP support  CAPEX: Linux server for BGP speaker  Special Requirement: upstream physical router that supports BGP Copyright 2015 FUJITSU LIMITED Compute Node #1 BGP SpeakerCompute Node #2 Upstream Router Peering VM SNAT .6 VM VM SNAT .2 SNAT .4 VM SNAT .5 VM VM SNAT .1 SNAT .3 44
#5 BGP(Border Gateway Protocol)  The amount of public IP consumption  # of vRouter … same as the current DVR where SNAT is not distributed  “BGP support” is under development now  SPEC(merged): https://review.openstack.org/#/c/90833/ Copyright 2015 FUJITSU LIMITED Compute Node #1 BGP SpeakerCompute Node #2 Upstream Router Peering VM SNAT .6 VM VM SNAT .2 SNAT .4 VM SNAT .5 VM VM SNAT .1 SNAT .3 45
#5 BGP(Border Gateway Protocol)  Need more enhancements for “BGP support” to realize this idea because  BGP Speaker will be another SPOF  Possible solution is to make vRouter behave as BGP Speaker Copyright 2015 FUJITSU LIMITED Compute Node #1 BGP SpeakerCompute Node #2 Upstream Router Peering VM SNAT .6 VM VM SNAT .2 SNAT .4 VM SNAT .5 VM VM SNAT .1 SNAT .3 46
Summary of the Ideas  Ideas in the order of the amount of public IP consumption Copyright 2015 FUJITSU LIMITED Name Consumption Cost for Infrastructure #1 One SNAT for one Router (# of Compute nodes) * (# of Routers) #3 One SNAT for one Compute Node w/o Security Concern (# of Compute nodes) * (# of Tenants) #2 One SNAT for one Compute Node # of Compute Nodes #4 Double NAT # of vRouters Physical router required #5 BGP # of vRouters A server for BGP Speaker and physical router required 47
Summarized in table  Trade-off between IP consumption and cost for infrastructure Copyright 2015 FUJITSU LIMITED Name Consumption Cost for Infrastructure #1 One SNAT for one Router (# of Compute nodes) * (# of Routers) #3 One SNAT for one Compute Node w/o Security Concern (# of Compute nodes) * (# of Tenants) #2 One SNAT for one Compute Node # of Compute Nodes #4 Double NAT # of vRouters Physical router required #5 BGP # of vRouters A server for BGP Speaker and physical router required Many A few Heavy Light 48
Future Plan of Distributed SNAT Copyright 2015 FUJITSU LIMITED49
Future Plan  I’ll propose idea #3 again at the Design Summit, “One SNAT for one Compute Node w/o Security Concern”  It doesn’t require any specific hardware  Still many public IPs can be consumed in a particular environment  If you cannot get much advantages in your environment, you can use the centralized SNAT (Network Node)  Let’s discuss at the Design Summit Will discuss at Neutron’s Contributors meetup on Friday! Copyright 2015 FUJITSU LIMITED50
Network Node is Not Needed Anymore - Completed Distributed Virtual Router / Fujitsu part

More Related Content

PPTX
OpenStack: Virtual Routers On Compute Nodes
clayton_oneill
 
PPTX
Neutron DVR
Edgar Magana
 
PPTX
Overview of Distributed Virtual Router (DVR) in Openstack/Neutron
vivekkonnect
 
PPTX
OpenStack Neutron's Distributed Virtual Router
carlbaldwin
 
PDF
OpenStack in Action 4! Emilien Macchi & Sylvain Afchain - What's new in neutr...
eNovance
 
PPTX
L2 and L3 agent restructure
Rossella Sblendido
 
PDF
Accelerating SDN Applications with Open Source Network Overlays
Cumulus Networks
 
PPTX
OpenStack Neutron Dragonflow l3 SDNmeetup
Eran Gampel
 
OpenStack: Virtual Routers On Compute Nodes
clayton_oneill
 
Neutron DVR
Edgar Magana
 
Overview of Distributed Virtual Router (DVR) in Openstack/Neutron
vivekkonnect
 
OpenStack Neutron's Distributed Virtual Router
carlbaldwin
 
OpenStack in Action 4! Emilien Macchi & Sylvain Afchain - What's new in neutr...
eNovance
 
L2 and L3 agent restructure
Rossella Sblendido
 
Accelerating SDN Applications with Open Source Network Overlays
Cumulus Networks
 
OpenStack Neutron Dragonflow l3 SDNmeetup
Eran Gampel
 

What's hot (20)

PDF
OpenStack Neutron Liberty Updates
mestery
 
PDF
Technical introduction to MidoNet
MidoNet
 
PPTX
OpenStack MeetUp - OpenContrail Presentation
Stacy Véronneau
 
PPTX
Subnet Pools and Pluggable IPAM
carlbaldwin
 
PDF
Technical Deep Dive into MidoNet - Taku Fukushima, Developer at Midokura
MidoNet
 
PDF
20 - IDNOG03 - Franki Lim (ARISTA) - Overlay Networking with VXLAN
Indonesia Network Operators Group
 
PDF
VPNaaS in Neutron
Kazunori Takeuchi
 
PDF
Virtualizing the Network to enable a Software Defined Infrastructure (SDI)
Odinot Stanislas
 
PDF
ONIC Japan 2016 - Contrail アップデート
Juniper Networks (日本)
 
PPTX
2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration
James Denton
 
PPTX
Openstack Basic with Neutron
KwonSun Bae
 
PDF
OPNFV Service Function Chaining
OPNFV
 
PPTX
Orchestration tool roundup kubernetes vs. docker vs. heat vs. terra form vs...
Nati Shalom
 
PDF
Netforce: extending neutron to support routed networks at scale in ebay
Aliasgar Ginwala
 
PPTX
High Availability in Neutron
Rossella Sblendido
 
PPTX
Juniper Network Automation for KrDAG
KwonSun Bae
 
PDF
Open stack networking_101_update_2014
yfauser
 
PDF
Service Function Chaining in Openstack Neutron
Michelle Holley
 
PPTX
DevOops - Lessons Learned from an OpenStack Network Architect
James Denton
 
PDF
SDN & NFV Introduction - Open Source Data Center Networking
Thomas Graf
 
OpenStack Neutron Liberty Updates
mestery
 
Technical introduction to MidoNet
MidoNet
 
OpenStack MeetUp - OpenContrail Presentation
Stacy Véronneau
 
Subnet Pools and Pluggable IPAM
carlbaldwin
 
Technical Deep Dive into MidoNet - Taku Fukushima, Developer at Midokura
MidoNet
 
20 - IDNOG03 - Franki Lim (ARISTA) - Overlay Networking with VXLAN
Indonesia Network Operators Group
 
VPNaaS in Neutron
Kazunori Takeuchi
 
Virtualizing the Network to enable a Software Defined Infrastructure (SDI)
Odinot Stanislas
 
ONIC Japan 2016 - Contrail アップデート
Juniper Networks (日本)
 
2014 OpenStack Summit - Neutron OVS to LinuxBridge Migration
James Denton
 
Openstack Basic with Neutron
KwonSun Bae
 
OPNFV Service Function Chaining
OPNFV
 
Orchestration tool roundup kubernetes vs. docker vs. heat vs. terra form vs...
Nati Shalom
 
Netforce: extending neutron to support routed networks at scale in ebay
Aliasgar Ginwala
 
High Availability in Neutron
Rossella Sblendido
 
Juniper Network Automation for KrDAG
KwonSun Bae
 
Open stack networking_101_update_2014
yfauser
 
Service Function Chaining in Openstack Neutron
Michelle Holley
 
DevOops - Lessons Learned from an OpenStack Network Architect
James Denton
 
SDN & NFV Introduction - Open Source Data Center Networking
Thomas Graf
 
Ad

Similar to Network Node is Not Needed Anymore - Completed Distributed Virtual Router / Fujitsu part (20)

PPTX
SR-IOV benchmark
Tommy Värre
 
PDF
[OVNC 2013] Controlling Secure & Software Defined Network for Cloud Infrastru...
Ian Choi
 
PPTX
How to Configure NetFlow v5 & v9 on Cisco Routers
SolarWinds
 
PDF
Approaching hyperconvergedopenstack
Ikuo Kumagai
 
PPTX
Netsft2017 day in_life_of_nfv
Intel
 
PDF
Routed networks sydney
Miguel Lavalle
 
PDF
Ryu SDN Framework
APNIC
 
PPTX
2014/09/02 Cisco UCS HPC @ ANL
dgoodell
 
PDF
L3HA-VRRP-20141201
Manabu Ori
 
PDF
NZNOG 2020 - The Trouble With NAT
Mark Smith
 
PDF
Windows server 8 hyper v networking (aidan finn)
hypervnu
 
PDF
6th SDN Interest Group Seminar - Session6 (131210)
NAIM Networks, Inc.
 
PDF
Openstackinsideoutv10 140222065532-phpapp01
sprdd
 
PDF
OpenStack: Inside Out
Etsuji Nakai
 
PPT
Edge Device Multi-unicasting for Video Streaming
Tal Lavian Ph.D.
 
PPTX
OpenStackを利用したEnterprise Cloudを支える技術 - OpenStack最新情報セミナー 2016年5月
VirtualTech Japan Inc.
 
PDF
Network Enhancements on BitVisor for BitVisor Summit 12
cjchen22
 
PPTX
IRATI @ RINA Workshop 2014, Dublin
Eleni Trouva
 
PDF
Tutorial WiFi driver code - Opening Nuts and Bolts of Linux WiFi Subsystem
Dheryta Jaisinghani
 
PDF
Neutron-to-Neutron: interconnecting multiple OpenStack deployments
Thomas Morin
 
SR-IOV benchmark
Tommy Värre
 
[OVNC 2013] Controlling Secure & Software Defined Network for Cloud Infrastru...
Ian Choi
 
How to Configure NetFlow v5 & v9 on Cisco Routers
SolarWinds
 
Approaching hyperconvergedopenstack
Ikuo Kumagai
 
Netsft2017 day in_life_of_nfv
Intel
 
Routed networks sydney
Miguel Lavalle
 
Ryu SDN Framework
APNIC
 
2014/09/02 Cisco UCS HPC @ ANL
dgoodell
 
L3HA-VRRP-20141201
Manabu Ori
 
NZNOG 2020 - The Trouble With NAT
Mark Smith
 
Windows server 8 hyper v networking (aidan finn)
hypervnu
 
6th SDN Interest Group Seminar - Session6 (131210)
NAIM Networks, Inc.
 
Openstackinsideoutv10 140222065532-phpapp01
sprdd
 
OpenStack: Inside Out
Etsuji Nakai
 
Edge Device Multi-unicasting for Video Streaming
Tal Lavian Ph.D.
 
OpenStackを利用したEnterprise Cloudを支える技術 - OpenStack最新情報セミナー 2016年5月
VirtualTech Japan Inc.
 
Network Enhancements on BitVisor for BitVisor Summit 12
cjchen22
 
IRATI @ RINA Workshop 2014, Dublin
Eleni Trouva
 
Tutorial WiFi driver code - Opening Nuts and Bolts of Linux WiFi Subsystem
Dheryta Jaisinghani
 
Neutron-to-Neutron: interconnecting multiple OpenStack deployments
Thomas Morin
 
Ad

Recently uploaded (20)

PDF
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
lawrenceomai2
 
PDF
Digital Strategies for Manufacturing Companies
Virendra Rai, PMP
 
PPTX
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
IT IDOL Technologies
 
PDF
Raksha Bandhan Grocery Pricing Trends in India 2025.pdf
Actowiz Solustions
 
PDF
SAP S4 Hana Brochure 3 (PTS SYSTEMS AND SOLUTIONS)
pts464036
 
PDF
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
PDF
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
PDF
medical staffing services at VALiNTRY
Tiyan Grayson
 
PDF
AI in Product Development-omnex systems
omnex systems
 
PDF
wealthsignaloriginal-com-DS-text-... (1).pdf
skmiani786
 
PDF
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
PDF
PTS Company Brochure 2025 (1).pdf.......
pts464036
 
PDF
Why TechBuilder is the Future of Pickup and Delivery App Development (1).pdf
TechBuilder
 
PPTX
Introduction to Artificial Intelligence
lohedi9363
 
PDF
EN-Survey-Report-SAP-LeanIX-EA-Insights-2025.pdf
deepakkhaitan3
 
PPTX
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
PDF
How to Choose the Right IT Partner for Your Business in Malaysia
Asian Business Services & Technologies
 
PPTX
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
PDF
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
PDF
2025 Textile ERP Trends: SAP, Odoo & Oracle
Aetos Technologies
 
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
lawrenceomai2
 
Digital Strategies for Manufacturing Companies
Virendra Rai, PMP
 
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
IT IDOL Technologies
 
Raksha Bandhan Grocery Pricing Trends in India 2025.pdf
Actowiz Solustions
 
SAP S4 Hana Brochure 3 (PTS SYSTEMS AND SOLUTIONS)
pts464036
 
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
medical staffing services at VALiNTRY
Tiyan Grayson
 
AI in Product Development-omnex systems
omnex systems
 
wealthsignaloriginal-com-DS-text-... (1).pdf
skmiani786
 
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
PTS Company Brochure 2025 (1).pdf.......
pts464036
 
Why TechBuilder is the Future of Pickup and Delivery App Development (1).pdf
TechBuilder
 
Introduction to Artificial Intelligence
lohedi9363
 
EN-Survey-Report-SAP-LeanIX-EA-Insights-2025.pdf
deepakkhaitan3
 
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
How to Choose the Right IT Partner for Your Business in Malaysia
Asian Business Services & Technologies
 
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
2025 Textile ERP Trends: SAP, Odoo & Oracle
Aetos Technologies
 

Network Node is Not Needed Anymore - Completed Distributed Virtual Router / Fujitsu part

  • 1. Network Node is Not Needed Anymore - Completed Distributed Virtual Router Copyright 2015 FUJITSU LIMITED0 OpenStack Summit October 2015 Tokyo Tuesday, October 27 - 2:50pm – 3:30pm Takanori Miyagishi Fujitsu Limited E-mail address: miyagishi.t@jp.fujitsu.com IRC name: miygishi_t
  • 4. Vision  Deployment of enterprise systems on OpenStack  To achieve this from the network perspective, we need more scalability and availability. Current Neutron has the following issues.  Performance bottle neck  Single point of failure (SPOF) Copyright 2015 FUJITSU LIMITED3
  • 6. Legacy Network Architecture (Legacy Router)  Neutron centralizes networking services to Network Node  Most of traffics need to go through Network Node Copyright 2015 FUJITSU LIMITED Compute Node Network Node ML2 plugin OVS-agent DHCP-agent Controller Node Neutron Server L3-agent ML2 plugin OVS-agentOVS-agent metadata-agent : Management network : External network : Data network 5
  • 7. Legacy Network Architecture (Legacy Router) Copyright 2015 FUJITSU LIMITED Compute Node Network Node ML2 plugin OVS-agent DHCP-agent Controller Node Neutron Server L3-agent ML2 plugin OVS-agentOVS-agent metadata-agent : Management network : External network : Data network  Neutron centralizes networking services to Network Node  Most of traffics need to go through Network Node What happens due to this Architecture? 6
  • 8. Issue #1: Network Node is Bottle neck  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-tun br-int br-tun br-int br-ex VM VM VM : External Network : Data Network DHCP br-tun br-int vRouter 7
  • 9. Issue #1: Network Node is Bottle neck  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) Copyright 2015 FUJITSU LIMITED Not need to go through Network Node in the both cases: - VMs are on the same host - VMs are on different hosts Compute node #1 Network NodeCompute node #2 br-tun br-tun br-int br-ex VM VM VM : External Network : Data Network DHCP br-tun br-int vRouter br-int 8
  • 10. Issue #1: Network Node is Bottle neck Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM VM : External Network : Data Network DHCP br-tun br-int vRouter br-tun br-int br-tun br-int  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) Not need to go through Network Node in the both cases: - VMs are on the same host - VMs are on different hosts 9
  • 11. Issue #1: Network Node is Bottle neck Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM VM : External Network : Data Network DHCP vRouter br-tun br-int br-tun br-int br-tun br-int  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) Need to go through Network Node in the both cases: - VMs are on the same host - VMs are on different hosts 10
  • 12. Issue #1: Network Node is Bottle neck Copyright 2015 FUJITSU LIMITED Need to go through Network Node in the both cases: - VMs are on the same host - VMs are on different hosts Compute node #1 Network NodeCompute node #2 br-ex VM VM VM : External Network : Data Network DHCP vRouter br-tun br-int br-tun br-int br-tun br-int  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) 11
  • 13. Issue #1: Network Node is Bottle neck Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM VM : External Network : Data Network DHCP vRouter br-tun br-int br-tun br-int br-tun br-int  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) 12
  • 14. Issue #1: Network Node is Bottle neck Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-tun br-int br-tun br-int br-ex VM VM VM : External Network : Data Network DHCP br-tun br-int vRouter  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) 13
  • 15. Issue #1: Network Node is Bottle neck Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-tun br-int br-tun br-int br-ex VM VM VM : External Network : Data Network DHCP br-tun br-int vRouter  There are 6 Network Traffics  East-West Traffic • Between VMs - Same Subnet -- (1) - Different Subnets -- (2) • VM accessing to DHCP server -- (3) • VM getting metadata from vRouter -- (4)  North-South Traffic • Between VM and out of OpenStack - Using Floating IP -- (5) - Using SNAT -- (6) 14
  • 16. Issue #1: Network Node is Bottle neck  Five out of six traffic patterns need to go through Network Node Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-tun br-int br-tun br-int br-ex VM VM VM DHCP br-tun br-int vRouter Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes (3) VM accessing to DHCP Yes (4) VM getting metadata Yes North- South (5) Using Floating IP Yes (6) Using SNAT Yes (1) (1) (2),(3),(4),(5),(6) Bottle neck 15
  • 17. Issue #2: Network Node is SPOF  If Network Node fails, that affects many traffics Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM VM DHCP br-tun br-int vRouter Affected Traffics by Network Node Failure East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes (3) VM accessing to DHCP Yes (4) VM getting metadata Yes North- South (5) Using Floating IP Yes (6) Using SNAT Yes br-tun br-int br-tun br-int 16
  • 19. DVR Architecture  L3-agent and Metadata-agent can be distributed on each Compute Node  But a part of L3-agent (SNAT) and DHCP-agent still need to run on Network Node Copyright 2015 FUJITSU LIMITED Compute Node Network Node ML2 plugin OVS-agent DHCP-agent Controller Node Neutron Server L3-agent(SNAT) ML2 plugin OVS-agentOVS-agent L3-agent metadata-agent Metadata-agent 18
  • 20. DVR Architecture Copyright 2015 FUJITSU LIMITED Compute Node Network Node ML2 plugin OVS-agent DHCP-agent Controller Node Neutron Server L3-agent(SNAT) ML2 plugin OVS-agentOVS-agent L3-agent metadata-agent Metadata-agent  L3-agent and Metadata-agent can be distributed on each Compute Node  But a part of L3-agent (SNAT) and DHCP-agent still need to run on Network NodeWhat are benefits of the distribution of agents? - Will explain, comparing with Legacy Router 19
  • 21. Traffics with Legacy Router again  five of six types of traffics needs through Network Node Copyright 2015 FUJITSU LIMITED Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes (3) VM accessing to DHCP Yes (4) Gets metadata Yes North- South (5) Using Floating IP Yes (6) Using SNAT Yes Compute node #1 Network NodeCompute node #2 br-tun br-int br-tun br-int br-ex VM VM VM DHCP br-tun br-int vRouter (1) (1) (2),(3),(4),(5),(6) 20
  • 22. Changes in the Architecture by DVR  These components are distributed to each compute node  vRouter, Floating IP, and bridge (interface) to external network Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-tun br-int br-tun br-int br-ex VM VM DHCP br-tun br-int br-exbr-ex fip fip Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes (3) VM accessing to DHCP Yes (4) VM getting metadata Yes North- South (5) Using Floating IP Yes (6) Using SNAT Yes 21
  • 23. Network flow of DVR Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM DHCP br-tun br-int br-exbr-ex fip fip br-tun br-int br-tun br-int  East-West traffic - Between VMs(same subnet)  Not need to go through Network Node in the both cases:  VMs are on the same host  VMs are on different hosts Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes (3) VM accessing to DHCP Yes (4) VM getting metadata Yes North- South (5) Using Floating IP Yes (6) Using SNAT Yes 22
  • 24. Network flow of DVR Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM DHCP br-tun br-int br-exbr-ex fip fip br-tun br-tun br-intbr-int  East-West traffic - Between VMs(different subnet)  No longer need to go through Network Node  VM access to vRouter that exists on own Compute Node Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes -> NO (3) VM accessing to DHCP Yes (4) VM getting metadata Yes North- South (5) Using Floating IP Yes (6) Using SNAT Yes 23
  • 25. Network flow of DVR Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM DHCP br-tun br-int br-tun br-int br-tun br-int br-exbr-ex fip fip  East-West traffic – Access to DHCP  Still need to go through Network Node Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes -> NO (3) VM accessing to DHCP Yes (4) VM getting metadata Yes North- South (5) Using Floating IP Yes (6) Using SNAT Yes 24
  • 26. Network flow of DVR Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM DHCP br-tun br-tun br-int br-tun br-int br-exbr-ex fip fip br-int  East-West traffic – Gets metadata  No longer need to go through Network Node  VM access to vRouter that exists on own Compute Node Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes -> NO (3) VM accessing to DHCP Yes (4) VM getting metadata Yes -> NO North- South (5) Using Floating IP Yes (6) Using SNAT Yes 25
  • 27. Network flow of DVR Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM DHCP br-tun br-tun br-int br-tun br-int br-ex fip br-int br-ex fip  North-South traffic – Using Floating IP  No longer need to go through Netowork Node  VM access to vRouter and translate to floating IP that exists on own Compute node Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes -> NO (3) VM accessing to DHCP Yes (4) VM getting metadata Yes -> NO North- South (5) Using Floating IP Yes -> NO (6) Using SNAT Yes 26
  • 28. Network flow of DVR Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 VM VM DHCP br-tun br-int br-exbr-ex fip fip br-exbr-tun br-int br-tun br-int  North-South traffic – Using SNAT  Still need to go through Network Node Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes -> NO (3) VM accessing to DHCP Yes (4) VM getting metadata Yes -> NO North- South (5) Using Floating IP Yes -> NO (6) Using SNAT Yes 27
  • 29. Network flow of DVR  3 types of traffic are no longer need to go through Network Node due to DVR Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 br-ex VM VM DHCP br-ex fip br-tun br-int (4) (5) (2) br-tun br-int br-ex fip br-tun br-int Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes -> NO (3) VM accessing to DHCP Yes (4) VM getting metadata Yes -> NO North- South (5) Using Floating IP Yes -> NO (6) Using SNAT Yes 28
  • 30. Network flow of DVR  2 types of traffics still need to go through Network Node  Meaning there are still two issues (Bottleneck and SPOF) Copyright 2015 FUJITSU LIMITED Compute node #1 Network NodeCompute node #2 VM VM DHCP br-tun br-int br-exbr-ex fip fip (6) (3) br-exbr-tun br-int br-tun br-int Need through Network Node? East- West (1) Between VMs (same subnet) No (2) Between VMs (different subnet) Yes -> NO (3) VM accessing to DHCP Yes (4) VM getting metadata Yes -> NO North- South (5) Using Floating IP Yes -> NO (6) Using SNAT Yes 29
  • 31. Completely Distributed Virtual Router  To eliminate bottle neck and SPOF completely  Distributed SNAT and DHCP are required Copyright 2015 FUJITSU LIMITED Compute Node Network Node ML2 plugin OVS-agent DHCP-agent Controller Node Neutron Server L3-agent(SNAT) ML2 plugin OVS-agentOVS-agent L3-agent metadata-agent DHCP-agent 30
  • 32. Efforts in Liberty Copyright 2015 FUJITSU LIMITED31
  • 33. Distributed SNAT  Distribute remaining l3-agent(SNAT) on Network Node to each Compute Node Copyright 2015 FUJITSU LIMITED Compute Node Network Node ML2 plugin OVS-agent DHCP-agent Controller Node Neutron Server L3-agent(SNAT) ML2 plugin OVS-agentOVS-agent L3-agent metadata-agent 32
  • 34. Conventional behavior of SNAT  Create SNAT on Network Node for each vRouter  If a tenant has multiple vRouters, then Neutron creates multiple SNATs  The amount of public IP consumption  # of vRouter that sets a gateway - 3 public IPs are needed in the following configuration Copyright 2015 FUJITSU LIMITED Compute Node #1 Network NodeCompute Node #2 VM VM VM SNATvRouterA1 vRouterA1'vRouterA2 vRouterA2'vRouterB1 vRouterB1' SNAT SNAT VM VM VM VM 33
  • 35. Possible Idea for Distributed SNAT  Five ideas that I devised w/ some hints in the past discussions #1 One SNAT for one Router #2 One SNAT for one Compute Node #3 One SNAT for one Compute Node w/o Security Concern #4 Double NAT #5 BGP(Border Gateway Protocol)  Understood why Distributed SNAT has not been implemented yet It’s not easy at all to devise the best design…  Will explain these ideas from the next page  For each idea, will show an actual amount of public IP consumption in an example system configuration  Compute Node#1 and #2 have two VMs of tenant#1 and a VM of tenant#2 respectively Copyright 2015 FUJITSU LIMITED34
  • 36. #1 One SNAT for one Router  Distribute SNAT simply … Follow the conventional behavior  The number of SNAT increases because each vRouter requires one SNAT Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VM VM SNATSNAT SNAT VMVM VM SNATSNAT SNAT VM .1 .3 .5 .2 .4 .6 35
  • 37. #1 One SNAT for one Router  The amount of public IP consumption  (# of Compute Nodes) * (# of vRouters per Compute Node) … Too many!!  6 public IPs are needed in the example system configuration Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VM VM SNATSNAT SNAT VMVM VM SNATSNAT SNAT VM .1 .3 .5 .2 .4 .6 36
  • 38. #2 One SNAT for one Compute Node  One SNAT shared by vRouters in Compute Node  In other words, sharing SNAT between different tenants Needs an exclusion of address scope between different tenants Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VM VM SNAT VMVM VM SNAT VM .1 .2 37
  • 39. #2 One SNAT for one Compute Node  The amount of public IP consumption  # of Compute nodes … Sufficiently small  2 Public IPs are needed in the example system configuration  Security concern exists  If an admin makes a wrong setting on SNAT, VMs in the different tenants can communicate. Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VM VM SNAT VMVM VM SNAT VM .1 .2 38
  • 40. #3 One SNAT for one Compute Node w/o Security Concern  SNAT shared by vRouters of the same tenant in Compute Node Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VMVM VM SNAT VM .1 SNAT .3 SNAT .4 VM VM SNAT .2 39
  • 41. #3 One SNAT for one Compute Node w/o Security Concern  The amount of public IP consumption  (# of Compute nodes) * (# of Tenants)  If each tenant only has one vRouter, the consumption will be the same as idea #1 because (# of Tenants) becomes (# of vRouter)  4 public IPs are needed in the example system configuration Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VMVM VM SNAT VM .1 SNAT .3 SNAT .4 VM VM SNAT .2 40
  • 42. #4 Double NAT  Double NAT on SNAT and upstream physical router  SNAT: translates private IP to a dedicated local IP (like ISP shared IP)  Upstream physical router: translates the local IP to a public IP  Need some changes to Floating IP mechanism - Floating IP needs to be able to allocate the local IPs to VMs Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VM VM SNATSNAT SNAT VMVM VM SNATSNAT SNAT VM .1 .3 .5 .2 .4 .6 Upstream Router NAT table 41
  • 43. #4 Double NAT  Dynamic setting to upstream physical router is required  OpenStack doesn’t have such a feature yet  The amount of public IP consumption  (# of vRouter)  3 public IPs are needed Copyright 2015 FUJITSU LIMITED Compute Node #1 Compute Node #2 VM VM SNATSNAT SNAT VMVM VM SNATSNAT SNAT VM .1 .3 .5 .2 .4 .6 Upstream Router NAT table (Need dynamic setting) 42
  • 44. #5 BGP(Border Gateway Protocol)  vRouters in the same group share one public IP  Upstream physical router as a gateway  BGP speaker uses BGP to advertise the router which SNAT to connect in the same group  Neutron doesn’t have a feature like BGP Speaker yet Copyright 2015 FUJITSU LIMITED Compute Node #1 BGP SpeakerCompute Node #2 Upstream Router Peering VM SNAT .6 VM VM SNAT .2 SNAT .4 VM SNAT .5 VM VM SNAT .1 SNAT .3 43
  • 45. #5 BGP(Border Gateway Protocol)  High cost  Development cost: BGP support  CAPEX: Linux server for BGP speaker  Special Requirement: upstream physical router that supports BGP Copyright 2015 FUJITSU LIMITED Compute Node #1 BGP SpeakerCompute Node #2 Upstream Router Peering VM SNAT .6 VM VM SNAT .2 SNAT .4 VM SNAT .5 VM VM SNAT .1 SNAT .3 44
  • 46. #5 BGP(Border Gateway Protocol)  The amount of public IP consumption  # of vRouter … same as the current DVR where SNAT is not distributed  “BGP support” is under development now  SPEC(merged): https://review.openstack.org/#/c/90833/ Copyright 2015 FUJITSU LIMITED Compute Node #1 BGP SpeakerCompute Node #2 Upstream Router Peering VM SNAT .6 VM VM SNAT .2 SNAT .4 VM SNAT .5 VM VM SNAT .1 SNAT .3 45
  • 47. #5 BGP(Border Gateway Protocol)  Need more enhancements for “BGP support” to realize this idea because  BGP Speaker will be another SPOF  Possible solution is to make vRouter behave as BGP Speaker Copyright 2015 FUJITSU LIMITED Compute Node #1 BGP SpeakerCompute Node #2 Upstream Router Peering VM SNAT .6 VM VM SNAT .2 SNAT .4 VM SNAT .5 VM VM SNAT .1 SNAT .3 46
  • 48. Summary of the Ideas  Ideas in the order of the amount of public IP consumption Copyright 2015 FUJITSU LIMITED Name Consumption Cost for Infrastructure #1 One SNAT for one Router (# of Compute nodes) * (# of Routers) #3 One SNAT for one Compute Node w/o Security Concern (# of Compute nodes) * (# of Tenants) #2 One SNAT for one Compute Node # of Compute Nodes #4 Double NAT # of vRouters Physical router required #5 BGP # of vRouters A server for BGP Speaker and physical router required 47
  • 49. Summarized in table  Trade-off between IP consumption and cost for infrastructure Copyright 2015 FUJITSU LIMITED Name Consumption Cost for Infrastructure #1 One SNAT for one Router (# of Compute nodes) * (# of Routers) #3 One SNAT for one Compute Node w/o Security Concern (# of Compute nodes) * (# of Tenants) #2 One SNAT for one Compute Node # of Compute Nodes #4 Double NAT # of vRouters Physical router required #5 BGP # of vRouters A server for BGP Speaker and physical router required Many A few Heavy Light 48
  • 50. Future Plan of Distributed SNAT Copyright 2015 FUJITSU LIMITED49
  • 51. Future Plan  I’ll propose idea #3 again at the Design Summit, “One SNAT for one Compute Node w/o Security Concern”  It doesn’t require any specific hardware  Still many public IPs can be consumed in a particular environment  If you cannot get much advantages in your environment, you can use the centralized SNAT (Network Node)  Let’s discuss at the Design Summit Will discuss at Neutron’s Contributors meetup on Friday! Copyright 2015 FUJITSU LIMITED50