LINEʼs OpenStack Networking Challenge
4 likes1,921 views
This document discusses LINE's challenges with OpenStack networking and their new approach using an L2 isolation plugin. The key points are: 1. LINE previously used VLAN networking with Neutron which had limitations like weak east-west bandwidth and required VM scheduling. 2. Their new approach uses BGP routing between the TOR and hypervisors, removing the need for overlays, L2 agents, or network nodes. 3. This provides benefits like improved bandwidth, zero downtime TOR maintenance, portable IPs, and no network bottlenecks.
LINEʼs OpenStack Networking Challenge
- 1. LINEʼs OpenStack Networking Challenge LINE Fukuoka Developer Meetup#32
- 2. ⾃⼰紹介 ⻄脇 雄基 LINE 株式会社 インフラプラットフォーム室 Verda2 Team Yuki Nishiwaki
- 3. Whatʼs Verda ALL OpenStack based Private Cloud For Development Environment For Production Environment Released(2016) Released(2017)
- 4. OpenStack we are using Nova Neutron Glance Keystone Designate Poppy
- 5. Todayʼs Topic Nova Neutron Glance Keystone Designate Poppy • New Mechanism Driver • New Type Driver • New Custom Agent
- 7. RegionOne RegionOne RegionTwo New Region coming with Custom Network Plugin Routed Network(Custom) Shared VLAN Network
- 9. Rack X Rack X+1 Compute Node Compute Node Review: Previous OpenStack Deployment Controller Node Network NodeCompute Node nova-XXXX glance-XXXX neutron-server Outside OpenStack nova-compute linuxbridge-agent dhcp-agent metadata-agent VM1 VM2 dnsmasqdnsmasq dnsmasqns-metadata-proxy VLAN 100(net1) VLAN 101(net2) eth0 eth0.100 eth0.101 eth0.100 eth0.101 eth0 bridge bridge bridge bridge
- 10. TOR Core Aggregation Review: Previous Datacenter Network ToR Aggregation ToR Server Server Server Server Server Server Server Server Aggregation ToR Server Server Server Server Core Aggregation OSPF ECMP MC-LAG Bonding (Active-Backup) Link Redundancy Method L2 L3
- 11. TOR Core Aggregation Review: Previous Datacenter Network ToR Aggregation ToR Server Server Server Server Server Server Server Server Aggregation ToR Server Server Server Server Core Aggregation OSPF ECMP MC-LAG Bonding (Active-Backup) Link Redundancy Method L2 L3 New Region?
- 12. TOR Core Aggregation Review: Previous Datacenter Network TOR Aggregation TOR Server Server Server Server Server Server Server Server Aggregation TOR Server Server Server Server Core Aggregation OSPF ECMP MC-LAG Bonding (Active-Backup) Link Redundancy Method L2 L3 * Weak for traffic between ToR (East-West) => Only 42% bandwidth ensured between ToR * Need VM Scheduling Consideration => Schedule VM having heavy traffic between VMs to same Rack
- 13. AggregationAggregation ToR Core Aggregation Horizontally scalable DC network architecture ToR Aggregation Server Server Server Server Core AggregationAggregationAggregationAggregation ToR ToR Server Server Server Server M M > N Core Core Core N Cluster ........ EBGP ECMP L3 Link Redundancy Method New NewNewNew Add more uplinks L3 EBGP ECMP
- 14. AggregationAggregation TOR Core Aggregation Horizontally scalable DC network architecture TOR Aggregation Server Server Server Server Core AggregationAggregationAggregationAggregation TOR TOR Server Server Server Server M M > N Core Core Core N Cluster ........ EBGP ECMP L3 Link Redundancy Method New NewNewNew Add more uplinks L3 EBGP ECMP It’s common approach up to here
- 15. New Architecture bring L3 routing downstream ToR Aggregation Core OSPF ECMP(L3) MC-LAG(L2) Hypervisor Act/Backup Bonding(L2) VM ToR Aggregation Core EBGP ECMP(L3) EBGP ECMP(L3) Hypervisor VM Linuxbridge (VLAN termination) Linuxbridge? (VLAN termination) Act/Backup Bonding(L2)? L3 -> L3 L2 -> L3 L2 -> L2?
- 16. New Architecture bring L3 routing downstream ToR Aggregation Core OSPF ECMP(L3) MC-LAG(L2) Hypervisor Act/Backup Bonding(L2) VM ToR Aggregation Core EBGP ECMP(L3) EBGP ECMP(L3) Hypervisor VM Linuxbridge (VLAN termination) Linuxbridge? (VLAN termination) Act/Backup Bonding(L2)? L3 -> L3 L2 -> L3 L2 -> L2?
- 17. Use Bonding between TOR and Hypervisor? ToR Aggregation Core OSPF ECMP(L3) MC-LAG(L2) Hypervisor VM ToR Aggregation Core EBGP ECMP(L3) EBGP ECMP(L3) Hypervisor VM Linuxbridge (VLAN termination) Linuxbridge (VLAN termination) * ToR maintenance cause Down Time * Un-portable IP * Cost of complexity in managing multiple network protocols Act/Backup Bonding(L2) Act/Backup Bonding(L2)?
- 18. Use BGP even between TOR and Hypervisor TOR Aggregation Core OSPF ECMP(L3) MC-LAG(L2) Hypervisor MC-LAG(L2) VM TOR Aggregation Core EBGP ECMP(L3) EBGP ECMP(L3) Hypervisor VM Linuxbridge (VLAN termination) Linuxbridge (VLAN termination) * ToR Zero downtime maintenance * Portable IP * ALL redundancy implemented by BGP EBGP (L3) Act/Backup Bonding(L2)?
- 19. Hypervisor terminate L2 domain, then... TOR Aggregation Core OSPF ECMP(L3) MC-LAG(L2) Hypervisor MC-LAG(L2) VM ToR Aggregation Core EBGP ECMP(L3) EBGP ECMP(L3) Hypervisor VM Linuxbridge (VLAN termination) EBGP (L3) Linuxbridge? (VLAN termination) If we still use linuxbridge with vlan termination, we have to allocate vlan id for each Hypervisor.....
- 20. Use Overlay over L3? TOR Aggregation Core OSPF ECMP(L3) MC-LAG(L2) Hypervisor MC-LAG(L2) VM TOR Aggregation Core EBGP ECMP(L3) EBGP ECMP(L3) Hypervisor VM Linuxbridge (VLAN termination) Overlay over L3? (VXLAN, GRE..) EBGP (L3) * Encapsulating performance overhead * Increase complexity - Mediate between VM network to outside - Network Node? - Each Compute Node via BGP?
- 21. Use routing making VM connected to world outside TOR Aggregation Core OSPF ECMP(L3) MC-LAG(L2) Hypervisor MC-LAG(L2) VM TOR Aggregation Core EBGP ECMP(L3) EBGP ECMP(L3) Hypervisor VM Linuxbridge (VLAN termination) Overlay over L3? (VXLAN, GRE..) EBGP (L3) Static Routing & BGP Don’t need encapsulate No Overlay Network No Network Node (L3 bottleneck) Each Compute Node join BGP Network
- 22. Use routing making VM connected to world outside ToR Aggregation Core OSPF ECMP(L3) MC-LAG(L2) Hypervisor MC-LAG(L2) VM ToR Aggregation Core EBGP ECMP(L3) EBGP ECMP(L3) Hypervisor VM Linuxbridge (VLAN termination) EBGP (L3) Static Routing(L3)
- 23. Neutron doesnʼt support such a use case ToR ToR Server1 BGP VM1 VM2 VM3 192.168.0.2/32 for VM1 192.168.0.3/32 for VM2 192.168.0.4/32 for VM3 192.168.0.2/32 Next Hop = Server1
- 24. L2 Isolate Plugin for L3 only datacenter core_plugin LBaaS FWaaS ML2 Type Driver Mechanism Driver VLANFlat Linuxbridge OVS OctaviaHaproxy Server Side Agent Side OVS AgentLinuxbridge Agent L3 Agent L2 Agent Metadata AgentDHCP Agent LBaaS Agent ML2 Related Agent Define new type of network represents all end-device connected via L3 Implement logic to achieve above type of network Implement agent working with new mechanism driver Can not reuse existing agent expecting network entity to be L2 network Type Driver Mechanism Driver L2 Agent ML2 Related Agent ....
- 26. Implementation detail in Next OpenStack Summit https://www.openstack.org/summit/vancouver-2018/summit-schedule/events/21113/excitingly-simple- multi-path-openstack-networking-lag-less-l2-less-yet-fully-redundant
- 27. What this architecture/l2isolate plugin bring? * Average throughput of VM – VM traffic over ToR quite improved => Improved uplink bandwidth of ToR (Clos Network) => No encapsulating overhead for VM Network (L2 isolate Plugin) * No need to take care VM scheduling for east-west traffic * Zero downtime for ToR maintenance * Portable IP (Live migration everywhere) * No Network Node bottleneck (dhcp, metadata-proxy is distributed)