OpenStack DVR_What is DVR?
12 likes3,004 views
OpenStack DVR (Distributed Virtual Router) allows L3 routing functions to be distributed across compute nodes by creating router namespaces on each compute node. This avoids bottlenecks and single points of failure at network nodes. DVR supports east-west inter-subnet routing, SNAT for external access without floating IPs, and floating IPs associated with internal VMs for direct external access. Traffic flows are encapsulated in VXLAN/GRE tunnels between compute nodes and routed appropriately within each node's router namespace.
OpenStack DVR_What is DVR?
- 1. What is OpenStack DVR (Distribute Virtual Router) 내것인듯 내것아닌 내것같은 Neutron/DVR Yongyoon SHIN. SDN section, ETRI
- 2. Let’s Talk • OpenStack and Distribute Virtual Router – OpenStack – What is DVR • DVR Case – East/West Traffic – SNAT – Floating IP
- 3. OpenStack Naming • OpenStack Version Naming – Austin: Austin, Texas – Bexar: San Antonio, Texas – Cactus: Cactus, Texas – Diablo: Santa Clara, CA – Essex: Boston, Massachusetts – Folsom: San Francisco, CA – Grizzly: San Diego, CA – Havana: Portland, Oregon – Icehouse: Hong Kong – Juno: Atlanta, Georgia – Kilo: Paris, France – Liberty: Vancouver, British Columbia – M*: Tokyo, Japan (https://wiki.openstack.org/wiki/Release_Naming/M_Proposals) – N*: Austin, Texas – O*: Barcelona, Spain Quantum Neutron Nova Network * 한국은 Vancouver summit에서 2019년 summit 신청 (Naming 순서는 ‘W’) .. 2019년에도 OpenStack이 지금처럼 번성할까? ..
- 4. OpenStack Release • Main Schedule (Neutron)
- 5. OpenStack Source • github 사용 • Neutron – 현재까지 업데이트 중 • 2015.06.11. 현재 8일전 update
- 6. 3 Main Services in Neutron • East-West Communication – Data Center 내부, VM 사이의 IP Traffic 처리 • Floating IP (aka DNAT: Destination NAT) – Public Network을 통하여 직접 VM에 접근 가능할 수 있도록 VM에 Public IP를 제공 • Shard IP (aka SNAT: Source NAT) – Data Center 내부 VM들이 외부 IP를 공용으로 사용하여 public Network 에 접속할 수 있도록 함
- 7. Main Idea of DVR • Network Node의 병목현상 해소 – L3 Agent 기능을 모든 Compute Node에 분산 • 네트워크 노드의 단일 장애점 및 트래픽 집중문제 해결 Compute Node A VM VM VM VM VM VM 10.0.10.0/24 129.254.172.0/24 129.254.172.0/24 20.0.20.0/24 DVR Public Network (129.254.172.0/24) Compute Node B VM VM VM VM VM VM 10.0.10.0/24 129.254.172.0/24 129.254.172.0/24 20.0.20.0/24 DVR
- 8. DVR vs without DVR • Without DVR • With DVR Compute Node A VM Network Node Compute Node B br-int br-tun VM br-int br-tun br-int br-tun eth eth eth VR Compute Node A VM Compute Node B br-int br-tun VM br-int br-tun eth eth VR VR
- 9. OpenStack DVR • OpenStack major work – Juno version release (~ing) • 리눅스의 Name Space 사용
- 10. OpenStack DVR • Network/Compute Node NameSpace Network Node Compute Node VM01 (Network A) VM02 (Network B) br-int br-ex br-tun dnsmasq dnsmasq br-int br-ex br-tun dhcp NameSpacedhcp NameSpace Router NameSpace Router NameSpace SNAT NameSpace patch-tun patch-int patch-tun patch-int eth0 eth0 vxlan-xxxx vxlan-yyyyvxlan-xxxx vxlan-yyyy FIP NameSpace qvoVM1 qvoVM2 qrouter-xxx fip-xxx qr-aaa qr-bbb rfp-xxx fpr-xxx fg-yyy qdhcp-kkk qdhcp-qqq tap123abc tap456def int-br-ex phy-br-ex qrouter-xxx snat-xxx qr-aaa qr-bbbsg-123aaa sg-456bbb qg-789abc External
- 11. Inter subnet routing East-West • ML2 plugin • l2pop mechanism driver enabled • Openvswitch mechanism driver enavled • ovs agent installed on all of compute nodes • Tunneling (only use VXLAN / GRE) – Kilo version support VLAN
- 12. Inter subnet routing East-West
- 14. Inter subnet routing East-West • Compute Node Router – Compute Node A – Compute Node B • Controller Node - IP: 192.168.56.20 - DevStack All_in_one • Compute Node - IP: 192.168.56.25 - DevStack Compute • Network Node - IP: 192.168.56.30 - DevStack Network
- 15. Inter subnet routing East-West • Router namespace created on both nodes – Same interfaces, MAC, IP
- 16. Inter subnet routing East-West • ARP Table – Compute Node A • vm01(100.0.0.2) / vm03(100.0.20.2) • ARP Table: vm01/vm02/vm03 – Compute Node B • vm02(100.0.10.2) • ARP Table: vm01/vm02/vm03
- 17. Inter subnet routing East-West • Flows Compute Node A VM Compute Node B br-int br-tun VM br-int br-tun eth eth VR VR ① ② ③ ④ ⑤ ⑥ ⑦ ⑧ src MAC dst MAC src IP dest IP Blue VM Blue Router Interface Blue VM Red VM
- 18. Inter subnet routing East-West • Flows – ARP Tables Compute Node A VM Compute Node B br-int br-tun VM br-int br-tun eth eth VR VR ① ② ③ ④ ⑤ ⑥ ⑦ ⑧ src MAC dst MAC src IP dest IP Red Router Interface Red VM Blue VM Red VM
- 19. Inter subnet routing East-West • Flows – Matched Red VM MAC Compute Node A VM Compute Node B br-int br-tun VM br-int br-tun eth eth VR VR ① ② ③ ④ ⑤ ⑥ ⑦ ⑧ src MAC dst MAC src IP dest IP Red Router Interface Red VM Blue VM Red VM
- 20. Inter subnet routing East-West • Flows Compute Node A VM Compute Node B br-int br-tun VM br-int br-tun eth eth VR VR ① ② ③ ④ ⑤ ⑥ ⑦ ⑧ src MAC dst MAC src IP dest IP Compute Node A MAC Red VM Blue VM Red VM
- 21. Inter subnet routing East-West • Flows – Encapsulation via VXLAN/GRE tunnel Compute Node A VM Compute Node B br-int br-tun VM br-int br-tun eth eth VR VR ① ② ③ ④ ⑤ VXLAN / GRE ⑥ ⑦ ⑧ src MAC dst MAC src IP dest IP Compute Node A MAC Red VM Blue VM Red VM
- 22. Inter subnet routing East-West • Flows – Tunnel encapsulation is stripped, tagged with a local VLAN Compute Node A VM Compute Node B br-int br-tun VM br-int br-tun eth eth VR VR ① ② ③ ④ ⑤ ⑥ ⑦ ⑧ src MAC dst MAC src IP dest IP Compute Node A MAC Red VM Blue VM Red VM
- 23. Inter subnet routing East-West • Flows – Traffic matched remote host MAC Compute Node A VM Compute Node B br-int br-tun VM br-int br-tun eth eth VR VR ① ② ③ ④ ⑤ ⑥ ⑦ ⑧ src MAC dst MAC src IP dest IP Compute Node A MAC Red VM Blue VM Red VM
- 24. Inter subnet routing East-West • Flows – Traffic matched remote host MAC, Red Network VLAN tag – VLAN tag stripped Compute Node A VM Compute Node B br-int br-tun VM br-int br-tun eth eth VR VR ① ② ③ ④ ⑤ ⑥ ⑦ ⑧ src MAC dst MAC src IP dest IP Red Router Interface Red VM Blue VM Red VM
- 25. Inter subnet routing East-West • East/West Traffic Flow summary Compute Node B VM03 (Network A) br-int br-ex br-tun Router NameSpace patch-tun patch-int eth0 vxlan-xxxx vxlan-yyyy qvoVM3 qrouter-xxx qr-aaa qr-bbb Compute Node A VM01 (Network A) VM02 (Network B) br-int br-ex br-tun Router NameSpace patch-tun patch-int eth0 vxlan-xxxx vxlan-yyyy FIP NameSpace qvoVM1 qvoVM2 qrouter-xxx fip-xxx qr-aaa qr-bbb rfp-xxx fpr-xxx fg-yyy
- 26. Distribute Virtual Routing - SNAT • SNAT : Source NAT – Floating IP를 설정하지 않은 VM이 외부 통신 – 라우터는 외부 네트워크에서 단일 IP를 할당 받음 (VM은 IP 공유) • Still Centralized – 현재 완전 분산 형태로 진행 중 (neutron blueprint) • SNAT를 사용하지 않아도 되는 경우 발생 (VM이 외부망을 사용할 필요 없음) Compute Node A VM Network Node br-int br-tun br-int br-tun eth1 eth0 VR VR(SNAT) eth1 br-ex qr1 qr2 sg1 sg2 qg
- 27. Distribute Virtual Routing - SNAT
- 29. Distribute Virtual Routing - SNAT • Network node – Neutron l3_agent.ini – Network node Interface
- 30. Distribute Virtual Routing - SNAT • Network Node IP Rule
- 31. Distribute Virtual Routing - SNAT • Compute Node A – IP Rule • Compute Node B – IP Rule 모든 컴퓨트 노드의 라우팅 룰은 라우터와 연결된 모든 서브넷 에 대한 내용 dvr_snat 노드의 SNAT 네임스페이스에 대한 내용
- 32. Distribute Virtual Routing - SNAT • Flows Compute Node A VM Network Node br-int br-tun br-int br-tun eth1 eth0 VR VR(SNAT) ① ② ③ ④ ⑤ ⑥ ⑦ src MAC dst MAC src IP dest IP Blue VM Blue qr Interface Blue VM ex) Google IP eth1 br-ex qr1 qr2 sg1 sg2 qg ⑧
- 33. Distribute Virtual Routing - SNAT • Flows Compute Node A VM Network Node br-int br-tun br-int br-tun eth1 eth0 VR VR(SNAT) ① ② ③ ④ ⑤ ⑥ ⑦ src MAC dst MAC src IP dest IP Blue VM Blue qr Interface Blue VM ex) Google IP eth1 br-ex qr1 qr2 sg1 sg2 qg ⑧
- 34. Distribute Virtual Routing - SNAT • Flows Compute Node A VM Network Node br-int br-tun br-int br-tun eth1 eth0 VR VR(SNAT) ① ② ③ ④ ⑤ ⑥ ⑦ src MAC dst MAC src IP dest IP Blue qr Interface Blue sg Interface Blue VM ex) Google IP eth1 br-ex qr1 qr2 sg1 sg2 qg ⑧
- 35. Distribute Virtual Routing - SNAT • Flows Compute Node A VM Network Node br-int br-tun br-int br-tun eth1 eth0 VR VR(SNAT) ① ② ③ ④ ⑤ ⑥ ⑦ src MAC dst MAC src IP dest IP Compute Node A MAC Blue sg Interface Blue VM ex) Google IP eth1 br-ex qr1 qr2 sg1 sg2 qg ⑧
- 36. Distribute Virtual Routing - SNAT • Flows Compute Node A VM Network Node br-int br-tun br-int br-tun eth1 eth0 VR VR(SNAT) ① ② ③ ④ ⑤ ⑥ ⑦ src MAC dst MAC src IP dest IP Blue qr Interface Blue sg Interface Blue VM ex) Google IP eth1 br-ex qr1 qr2 sg1 sg2 qg ⑧
- 37. Distribute Virtual Routing - SNAT • Flows Compute Node A VM Network Node br-int br-tun br-int br-tun eth1 eth0 VR VR(SNAT) ① ② ③ ④ ⑤ ⑥ ⑦ src MAC dst MAC src IP dest IP qg Interface Default GW MAC qg Interface ex) Google IP eth1 br-ex qr1 qr2 sg1 sg2 qg ⑧
- 38. Distribute Virtual Routing - SNAT • SNAT Traffic Flow summary Network Node Compute Node VM01 (Network A) VM02 (Network B) br-int br-ex br-tun dnsmasq dnsmasq br-int br-ex br-tun dhcp NameSpacedhcp NameSpace Router NameSpace Router NameSpace SNAT NameSpace patch-tun patch-int patch-tun patch-int eth0 eth0 vxlan-xxxx vxlan-yyyyvxlan-xxxx vxlan-yyyy FIP NameSpace qvoVM1 qvoVM2 qrouter-xxx fip-xxx qr-aaa qr-bbb rfp-xxx fpr-xxx fg-yyy qdhcp-kkk qdhcp-qqq tap123abc tap456def int-br-ex phy-br-ex qrouter-xxx snat-xxx qr-aaa qr-bbbsg-123aaa sg-456bbb qg-789abc
- 39. Distribute Virtual Routing – Floating IP • Compute Node의 FIP(Floating IP) namespace – rfp: Router NameSpace에 생성 – fpr: FIP NameSpace에 생성 Compute Node VM01 (Network A) VM02 (Network B) br-int br-ex br-tun Router NameSpace patch-tun patch-int eth0 vxlan-xxxx vxlan-yyyy FIP NameSpace qvoVM1 qvoVM2 qrouter-xxx fip-xxx qr-aaa qr-bbb rfp-xxx fpr-xxx fg-yyy Veth pair Compute Node A VM br-int VR qr FIP br-ex eth0 rfp - 129.254.171.0/24 fpr - 129.254.171.0/24 fg
- 40. Distribute Virtual Routing – Floating IP
- 42. Distribute Virtual Routing – Floating IP • FIP NameSpace 생성 및 External Network 확인 – Compute Node A – Compute Node B
- 43. Distribute Virtual Routing – Floating IP • veth pare – Compute Node A • Router NameSpace – Compute Node B • Router NameSpace
- 44. Distribute Virtual Routing – Floating IP • Veth pair – Compute Node A • FIP NameSpace • External Device: fg – Compute Node B • FIP NameSpace • External Device: fg
- 45. Distribute Virtual Routing – Floating IP • Routing Rule – Compute Node A – Compute Node B
- 46. Distribute Virtual Routing – Floating IP • Floating IP Packet Tracking – Compute Node A – Compute Node B
- 47. Distribute Virtual Routing – Floating IP Compute Node A VM br-int VR ① qr FIP br-ex eth0 rfp - 129.254.171.0/24 fpr - 129.254.171.0/24 fg src MAC dst MAC src IP dest IP Blue VM Blue qr Interface Blue VM fixed IP ex) Google IP ④ ⑤ ② ③
- 48. Distribute Virtual Routing – Floating IP Compute Node A VM br-int VR ① qr FIP br-ex eth0 rfp - 129.254.171.0/24 fpr - 129.254.171.0/24 fg src MAC dst MAC src IP dest IP Blue VM Blue qr Interface Blue VM fixed IP ex) Google IP ④ ⑤ ② ③
- 49. Distribute Virtual Routing – Floating IP Compute Node A VM br-int VR ① ② qr FIP br-ex eth0 rfp - 129.254.171.0/24 fpr - 129.254.171.0/24 fg src MAC dst MAC src IP dest IP Blue VM Blue qr Interface Blue VM fixed IP ex) Google IP ③ ④ ⑤
- 50. Distribute Virtual Routing – Floating IP Compute Node A VM br-int VR ① qr FIP br-ex eth0 rfp - 129.254.171.0/24 fpr - 129.254.171.0/24 fg src MAC dst MAC src IP dest IP rfp fpr Blue VM Floating IP ex) Google IP ④ ⑤ ② ③
- 51. Distribute Virtual Routing – Floating IP Compute Node A VM br-int VR ① qr FIP br-ex eth0 rfp - 129.254.171.0/24 fpr - 129.254.171.0/24 fg src MAC dst MAC src IP dest IP fg default GW Blue VM Floating IP ex) Google IP ⑤ ④ ② ③
- 52. Distribute Virtual Routing - Floating IP • Floating-IP Traffic Flow summary Network Node Compute Node VM01 (Network A) VM02 (Network B) br-int br-ex br-tun dnsmasq dnsmasq br-int br-ex br-tun dhcp NameSpacedhcp NameSpace Router NameSpace Router NameSpace SNAT NameSpace patch-tun patch-int patch-tun patch-int eth0 eth0 vxlan-xxxx vxlan-yyyyvxlan-xxxx vxlan-yyyy FIP NameSpace qvoVM1 qvoVM2 qrouter-xxx fip-xxx qr-aaa qr-bbb rfp-xxx fpr-xxx fg-yyy qdhcp-kkk qdhcp-qqq tap123abc tap456def int-br-ex phy-br-ex qrouter-xxx snat-xxx qr-aaa qr-bbbsg-123aaa sg-456bbb qg-789abc
- 53. OpenStack DVR Summary 특성 Forwarding Mechanism Local • Source/Destination IP는 동일한 Subnet에 속함 • VM들은 동일한 노드에 배열 br-int는 관련된 MAC Address를 학습하고 로컬 트래픽을 전달 Remote • Source/Destination IP는 동일한 Subnet에 속함 • VM들은 서로 다른 노드에 배열 Segmentation type(VXLAN/GRE)에 따라 전달 East/West • Source/Destination IP는 다른 Subnet에 속함 컴퓨트 노드의 라우터 인터페이스에 의해 전달 SNAT (Source NAT) • Destination IP는 컴퓨트 노드가 알고 있는 Subnet 에 속하지 않음 • VM에 Floating IP 설정이 되어 있지 않음 dvr_snat 노드 상의 SNAT 네임스페이스에 존재하는 IP 매칭에 따라 전달 FIP (Floating IP) • Destination IP는 컴퓨트 노드가 알고 있는 Subnet 에 속하지 않음 • VM에 Floating IP 설정 컴퓨트 노드의 FIP 네임스페이스와 연결된 컴퓨트 노드 라우터에 전 달
- 54. OpenStack에서는 앞으로… • Juno 배포 이후 업데이트 예정 – L3 고가용성 지원 • HA 지원 / Cascading 관련 내용 Candidate – Security Groups 개선 • VM의 증가로 인하여 iptables의 필터링 규칙 폭증 -> 패킷 필터링 성능 저하 • Kilo 배포 이후 추가 사항 – VPNaaS (Virtual Private Network as a Service) 지원 – VLAN 지원 – IPv6 지원 – 분산 DHCP 서비스 – 분산 SNAT 서비스
- 55. Yongyoon. SHIN (SDN Research Section, ETRI) Email: uni2u@etri.re.kr http://uni2u.meximas.com Thank you