Dreamhost deploying dreamcompute at scale
2 likes826 views
The presentation discusses the collaboration between Cumulus Networks and Akanda at DreamHost to enhance cloud services through open networking and efficient scalability. It highlights the evolution of network architecture from Generation 1 to Generation 3, focusing on the implementation of VXLAN for layer 2 virtualization over layer 3 IP fabrics and the improvements in orchestration and network performance. Key benefits include simplified operations, increased scalability, and integration of open-source solutions, enabling DreamHost to manage over 1,000 customers and thousands of virtual machines effectively.
Dreamhost deploying dreamcompute at scale
- 1. Cumulus and Akanda at DreamHost Driving Scale, Efficiency, and Cost Reduction Presenters: Jonathan LaCour (DreamHost), Nolan Leake (Cumulus Networks) & Mark McClain (Akanda)
- 2. Introduction
- 3. ▪ Founded in 1997 ▪ Managed, mass-market web hosting ▪ ~400,000 customers ▪ Why Cloud? ▪ The rise of AWS ▪ The world needs a viable, open alternative • Ceph and OpenStack lead the way!
- 4. • Public cloud compute service • Built on OpenStack and Ceph • Core networking requirements • L2 tenant isolation • IPv6 • 10G+ everywhere
- 5. Network: Gen 1 ▪ Physical: White Box switches running Cumulus Linux ▪ L2 isolation: virtualized with Nicira NVP ▪ L3+ ▪ Nicira lacks L3 ▪ Software routing vendors don’t understand cloud ▪ Astara is born!
- 6. ▪ Nicira / VMWare adds L3 ▪ Time for a bake-off! ▪ Astara wins the battle, but gets some enhancements ▪ Move from OpenBSD and PF to Linux and iptables ▪ Significant optimizations to orchestration platform ▪ Gen 2 allows us to scale to 1,000+ customers, thousands of VMs Network: Gen 2
- 7. DreamCompute Network: Generation 3 ▪ VMWare NSX problems ▪ Scale: maxes out around 1,250 tenants ▪ Performance: OVS is slow and unstable ▪ Magic: difficult to debug and operate ▪ Gen 3 is built on open ▪ Physical: Cumulus Linux ▪ L2 isolation: hardware accelerated VXLAN in switch and hypervisor ▪ L3+: Astara Network: Gen 3
- 8. DreamCompute Network: Generation 3 ▪ Simple, open architecture ▪ Operational ease ▪ Proven technology: VXLAN, iptables, Linux networking stack ▪ Astara simplifies Neutron deployment ▪ Performance and scale ▪ Hardware accelerated VXLAN pervasive on switches / NICs ▪ VXLAN tunnels scale up massively ▪ Astara model of virtual network appliances scales easily Gen 3 Benefits
- 9. ● Created to fill in gaps in Neutron ● L3-L7 Service Orchestration for OpenStack ○ Dynamic Routing ○ IPv6 ● Simplified Operations ○ Using standard APIs ● Astara Project ○ Open Source ○ OpenStack Foundation top-level project
- 10. Reference Neutron Message QueueNeutron Server L2 Agent L3 Agent DHCP Agent Adv ServicesDatabase
- 11. Astara + OpenStack Neutron Message QueueNeutron Server L2 Agent Astara Database
- 12. Astara + OpenStack Neutron Astara Physical Network (L2) Nova Neutron Open: OVS/LinuxBridge Proprietary Astara OTT Platform (L2 Agnostic) Astara Network Services: Routing/LB/FW/VPN OpenStack APIs
- 13. Physical Network
- 14. Traditional L2-centric Design Falls Short ▪ Bottleneck! ▪ Core/Agg limits scale ▪ Dead Agg switch is a Big Deal ▪ Complex, Proprietary ▪ MLAG/vPC/Stacking ▪ HSRP/GLBP/NSRP ▪ Alphabet soup ▪ Complex Failure Modes ▪ Loops ▪ MAC flapping ▪ Large blast radius ▪ Scalability ▪ Limited total network size ▪ Limited number of VLANs
- 15. A Better Way L2 L3
- 16. IP Fabric: CLOS/Fat-tree ▪ No Bottleneck! ▪ Full bandwidth across racks ▪ Crucial for network virtualization ▪ Simple, Open ▪ IP ▪ BGP ▪ Fine-grained failures ▪ BGP runs the Internet ▪ Scales up to any size ▪ Just add more layers!
- 17. Open Networking: Bare-Metal Ecosystem ONIE (Open Network Install Environment)
- 18. Automation and Monitoring ▪ Only way to effectively manage large numbers of switches! ▪ Choice of Automation Tools ▪ DreamHost was already using Chef ▪ But you can use any tool that works on Linux! ▪ Choice of Monitoring Tools ▪ DreamHost was already using collectd+Graphite ▪ SNMP still there for legacy monitoring systems ▪ Other Options ▪ Elastic Search/LogStash/Kibana ▪ Sensu ▪ Even good old MRTG!
- 19. DreamCompute Gen 3 Details
- 20. VXLAN: L2 Virtualization over L3 IP Fabrics ▪ UDP tunnels between vswitches ▪ Guest L2 traffic is safely encapsulated in L3 packets on the physical network ▪ No L2 required in the physical network ▪ What about BUM Packets: Broadcast, Multicast, Unknown Unicast? ▪ “Official” RFC7348 answer: Multicast ▪ Multicast is complex and scales poorly: disabled on most networks ▪ Replicator ▪ Cumulus authored, Open Source daemon: https://github.com/CumulusNetworks/vxfld ▪ Replicates BUM packets to multiple unicast receivers ▪ Can run on Linux switches, or Linux servers/hypervisor ▪ Hardware-accelerated when run on Cumulus Linux
- 21. VXLAN: HW VTEP ▪ VTEP: “VXLAN Tunnel Encapsulation/Decapsulation Point” ▪ Thing that encapsulates virtual network L2 traffic in L3 UDP packets for physical transport ▪ Neutron-managed software VTEPs on hypervisors ▪ Encapsulations/Decapsulates packets for VMs ▪ Cumulus-managed hardware VTEP to connect to non-virtual networks ▪ Encapsulates/Decapsulates packets from VMs to routers, appliances, etc ▪ 100% in hardware, line rate.
- 22. Questions?
- 23. Extras