Building a Layer 3 network with Cumulus Linux
Download as PPTX, PDF0 likes922 views
The document discusses Outbrain's challenges with network management and scaling, highlighting issues related to manual processes, risk with changes, and setup times. It details the transition to a new data center using a Clos fabric network architecture, which enables scalability and simplified device management. Ultimately, the new network design resulted in significant improvements, with most code performing as expected except for one faulty cable among thousands.
Building a Layer 3 network with Cumulus Linux
- 1. Switches, Penguins and One Bad Cable Alex Balk
- 2. Back on AUGUST 13th 2015
- 3. ADI NAVEH ALEX BALK ALEX KARASIK CHEN SHABI DAFNA FRANK DORI SHMUEL GAI RADZI GERARDO LARACUENTE GUY MAZUZ RYAN MCQUILLAN SHAHAF SAGES YAFIT MELES We Are Core
- 4. OUR PURPOSE OUTBRAIN helps people discover content that they find interesting.
- 5. 250 Billion Content Recommendations Every Month ½ Billion People Worldwide OUTBRAIN BY THE NUMBERS
- 10. AVAILABILITYMANAGEMENT Two Main Networking Challenges
- 15. Node Stack A Stack B Backbone NodeNode
- 16. SCALE IS ABOUT DOING MORE WITH LESS
- 19. 6 Million Metrics generated every minute 150 Releases To production every day OUTBRAIN BY THE NUMBERS
- 20. SCALE IS ABOUT TURNING THE LIGHTS ON Network (gasp!) was 100% Manual! • Every change = risk • Switching stack proprietary • Debugging = fight or just a hit-n-miss • Lead time to set-up new stack measured in weeks! • No way to scale to the next 10X
- 21. June 2017
- 22. OUTBRAIN OFFICES
- 23. New Data Center = Clos Fabric — running BGP end-to-end
- 24. Node Leaf A Leaf B Spine NodeNode SpineSpine No bonding. No backbone. Everything is just a router!
- 25. All possible paths to all possible destinations constructed — hop-by-hop
- 26. Node Leaf A Spine NodeNode SpineSpine Leaf B ECMP = “Send it down any available path, they’re all the same”.
- 27. SIMPLE PREDICTABLE SCALABLE A Network That is Now
- 29. SCALE IS ABOUT BUILDING THE RIGHT CULTURE
- 30. SCALE IS ABOUT CHOOSING THE RIGHT TOOLS To bootstrap the new datacenter5 DAYS 99% Of code worked as expected 1 Bad cable...out of 3,000
- 31. END SOLUTION MODULES DRAFT 051116 Thank You
- 32. Hardcore Tech Stuff Slides shamelessly “borrowed” from Adi Naveh’s internal tech talk
- 36. Load Balancers Load Balancers ISP Traditional Network Topology in Data Center Access Aggregation Core Services North-South Traffic East-West Traffic
Editor's Notes
- #19: Publishers from all around the world
- #31: In fact, when we designed the new datacenter, We wrote Chef cookbooks to automate provisioning and config We wrote unit and integration tests using Chef’s toolchain. And setup a CI pipeline for the code. We even simulated the entire datacenter, switches, servers and all, Using Vagrant. It worked so well, that bootstrapping the new datacenter took us just 5 days. Think about it. The first time we ever saw a real Dell switch running Linux, was when we arrived onsite for the buildout. And yet, 99% of our code worked as expected. In 5 days, we were able to setup a LAN, VPN, server provisioning, DNS, LDAP, and dealt with some quirky BIOS configs. On the servers, mind you, not the switches. We even hooked Cumulus’ built-in cabling validation, To our Prometheus based monitoring system. So that right after we turned monitoring on, we got an alert. On one bad cable. Out of 3000.