Campus network refresh
0 likes333 views
The document summarizes the campus network at Imperial College London, which includes over 17,000 students and 8,000 staff across multiple campuses. The network sees over 65,000 wired and 60,000 wireless devices connected simultaneously. It utilizes routers, firewalls, switches, and over 2,900 wireless access points connected through a fiber infrastructure with MPLS and WDM technologies. The network also provides dual-stack IPv4 and IPv6 connectivity across most services and has implemented technologies like LHCONE to enhance research networking. Network configurations are increasingly automated using tools like Ansible.
Campus network refresh
- 1. Campus network refresh David Stockdale, Imperial College London
- 2. >17,000 students >8,000 staff >Main campus – South Kensington, London >Under construction – White City, London >6 other large campuses (hospitals, Silwood Park) >10+ other sites (hospitals, halls, sports grounds) >2 datacentres – Slough & South Ken >Centralised ICT Facts and figures – Imperial
- 3. >Over 65,000 unique hosts on wired network >Over 60,000 unique hosts on wireless network >Over 20,000 concurrent wireless clients at peak time >~400 active comms rooms (CWCs) >~20 dark fibre links >~15 Ethernet circuits >40G to Janet via two 2x10G trunks Facts and figures – Network
- 4. >Routers – 23x Juniper MX & Cisco 6500/6880/N7K >Smaller sites – 12x Juniper SRX 2xx/3xx >Firewalls – 2x Juniper SRX 3xxx >Switches – 1,900x Juniper & Extreme >Wireless – 2,900x Cisco lightweight APs >VoIP – 10,000x Cisco handsets Facts and figures – Equipment
- 5. Physical infrastructure Core location • Core router, dark fibre and optical equipment Distribution location • Site router, distribution switch Building ODF (BODF) • Passive fibre patching CWC • One or more stacks of edge switches
- 6. >One or more per building >Fewer, bigger CWCs >Stacks of edge switches >Until now cat 5e >In future cat 6a >Diverse fibre to BODF CWC
- 7. >One per building >No active equipment >Single-mode throughout >Fibre to dist locations BODF
- 8. >Typically 2-3 per site >Site router >Distribution switch >Fibre to core locations Distribution location
- 9. >Two in London >Core router >Lots of dark fibre >Lots of optical equipment >Also: >Firewalls >Border routers >Route reflectors >etc. Core location
- 10. >DWDM and CWDM >Physical topology != active topology >Up to 40x 10G* links over single fibre pair >Passive on shorter distances >Amplifiers on longer links >Coloured optics in our equipment >Transponders for links to other equipment WDM
- 11. Network hierarchy Core routers • 2 in London at different sites Site routers • Pairs, each attached to both core routers Distribution switches • Pairs, each attached to both site routers Edge switches • Each stack attached to both distribution switches
- 12. >2x MPLS P routers >No VRFs >Lots of 10G ports >Lots of IPv4/IPv6 P2Ps >OSPF v2/v3 as IGP – loopbacks & P2Ps >iBGP with route reflectors – other routes >PIM >LDP >ECMP >Links to border, site, datacentre and wireless routers Core routers
- 13. >VRF lite doesn’t scale well >No per-VRF P2Ps or routing protocols >VRFs don’t need to exist on intermediate routers >VRF routes in iBGP >L3VPN – IPv4 routes for VRFs >6VPE – IPv6 routes for VRFs >EoMPLS / VPLS / EVPN >MVPN – Multicast for VRFs (Draft-Rosen, not MPLS) MPLS
- 14. >2x central devices for everything >One-armed off border routers >VRFs map to zones >eBGP session per zone, landing in VRFs on routers >BGP for failover, rather than HA >Networks Group runs network side >Security Group maintains policy side Firewalls
- 15. >Production >BYOD >Guest >BMS >Device management >Registration >Banned >Many smaller VRFs VRFs / firewall zones
- 16. >MPLS PE routers >In pairs >Limited, expensive 10G ports >Dual-stack IPv4/IPv6 for production and BYOD >VRRP / HSRP >PIM, IGMP >2x10G to each core router (40G ECMP) Site routers
- 17. >Layer 2 only >MLAG pairs of stacks >Plentiful, affordable 1/10G ports >2x10G per stack to each router (40G total) Distribution switches
- 18. >1G PoE everywhere >Interested in 2.5/5G – works over cat5e >2x1/10G LACP to distribution >Standard set of per-VRF VLANs >All edge ports alike – VLANs assigned by RADIUS >No UPS >Edge SLA – higher SLA available in datacentre Edge switches
- 19. >~30% of our Internet traffic IPv6 >Dual stack on production & BYOD (including wireless) >AAAAs on most load-balanced services >Other services enabled: >Home directories (>95% IPv6!), more storage soon >Mail, DNS, Skype for Business, HEP systems >SLAAC rather than DHCPv6 (historical reasons) >Feature parity mandated in tenders IPv6
- 20. >2008 – First subnets enabled, separate firewalls >2010 – Upstream native IPv6, dual-stack firewalls >2010/11 – Most production and BYOD enabled >2010/11 – Some servers including mail & DNS >2011 – World IPv6 Day: College websites enabled >2013 – Wireless enabled >2015 – AAAAs added to most load-balanced VIPs >During a migration to new hardware >Single protocol backends IPv6 - Milestones
- 21. >IPv4 exhaustion – wireless is now using /17! >NAT will be inevitable – IPv6 minimizes it >HEP community – dependent on IPv6, run out of IPv4 >Overseas students – better IPv6 connectivity at home >People have it at home in UK! Sky, BT, Virgin (soon?) >Cost us very little, deploying opportunistically >Could cost a lot to deploy in a hurry! >We’ve seen very few problems IPv6 - Reasons
- 22. >Separate DMZ router connected to border routers >Initially for HEP, soon to be standard service >Bypasses firewall >Avoids upgrading whole path >Cheaper equipment but fewer features >Essential moving towards 100G >ECMP outbound, multiple subnets inbound = 40G! Science DMZ
- 23. >We’re gonna need a bigger pipe… :-/ Science DMZ – Success!
- 24. >Private network for participating HEP sites >Tagged VLANs on Janet links >BGP peerings into L3VPN >DMZ router has BGP peerings into Internet/LHCONE >More specific LHCONE prefixes preferred >It works… LHCONE
- 25. >Router configs built and managed by Ansible >Firewall groups fed from host database >Switch configs automatically generated >Network built from standard blocks >All edge ports alike >MPLS simplifies configuration >WDM and dark fibre surprisingly affordable >Simple is better! Automation & scalability
- 26. David Stockdale ICT Networks Group Imperial College London I have been… Exhibition Road, London, SW7 2AZ T 020 7594 6968 david@imperial.ac.uk www.imperial.ac.uk
- 27. Any questions? / Thank you