110G networking within JASMIN
Download as PPTX, PDF0 likes1,174 views
The document presents an overview of the Campus Network Engineering Workshop held on October 19, 2016, focusing on 100G networking within the Jasmin infrastructure. It highlights the unique features of Jasmin, such as high-performance storage, computing capabilities, and non-blocking networking, while addressing the challenges of providing non-blocking data access to IaaS cloud tenants. Key potential issues include performance impacts of VXLAN terminations and the need for network adjustments to accommodate the scale of 100G networking.
110G networking within JASMIN
- 1. Jonathan Churchill, Campus network engineering workshop 19/10/2016 100G networking within JASMIN
- 2. 100G networking within JASMIN Campus network engineering for data- intensive science workshop October 19th 2016 Jonathan Churchill JASMIN Infrastructure Manager ( STFC Scientific Computing Dept.)
- 3. JASMIN is a world leading, unique hybrid of: • 16PB high performance storage (~250GByte/s) • High-performance computing (~4,000 cores) • 35PB Archive and Elastic Tape • Non-blocking Networking (> 3Tbit/sec), and Optical Private Network WAN’s • Coupled with cloud hosting capabilities Cloud is here ! 17PB 40PB 5,000
- 4. JASMIN 1 JASMIN 2 JASMIN 3 DMZ Cloud Lives here JASMIN 4,5 (2016–20) …) Storage and Servers distributed over the fabric network. JASMIN 3.5 LOTUS 100G lives here
- 6. The need for speed 347Gbps 347Gbps = 34,700 Broadband connections
- 7. JC2-LSW1 JC2-LSW1 JC2-LSW1JC2-LSW1 JC2-LSW1 JC2-LSW1 JC2-LSW1 JC2-LSW1 JC2-LSW1JC2-LSW1 JC2-LSW1 JC2-LSW1 48 * 16 = 768 10GbE Non-blocking 16 x 12 x 40GbE = 192 40GbE ports S1036 = 32 x 40GbE JC2-LSW1JC2-LSW1 JC2-SP1 JC2-SP1 JC2-SP1 JC2-SP1 JC2-SP1 JC2-SP1 16 x MSX1024B-1BFS 48x10GBE + 12 40 GbE 16 x 12 40GbE = 192 Ports / 32 = 6 Total 192 40 GbE Cable 1,900 @ 10GbE Ports • Non-Blocking. Zero Contention (48x10Gb = 12x 40Gb uplinks) • Low Latency (250nS L3 / per switch/router). • Cheap(er) • But its all layer 3 routed (ECMP OSPF) 954 Routes 954 Routes
- 8. Bandwidth ?? Data via the DTZ Through the IaaS firewall DTZ Bandwidth 1:1 match to IaaS hypervisors.
- 9. Data rates inside PaaS = IaaS ? • How can we provide data rate access to Iaas Cloud tenants at similar rates to “inside” JASMIN (PaaS) ? – aka 100Gbits/sec
- 10. Non Blocking data access inside JASMIN SP1 SP2 SP3 SP4 LSW1 host001 host002 host024 iSCSI Underlay networks LSW2 host025 host026 host027 LSW3 172.26.66.64/26172.26.66.0/26 LSWn ~10x 10-12Gbps per “Bladeset” 24x 10Gbps 172.16.136.0/24 172.16.137.0/24 12x 40Gb ECMP uplinks per switch/router A non-blocking to IaaS cloud needs to duplicate or fit into this fabric. And still 1:1 using 10Gb servers
- 11. LSW21 Non Blocking data access to JASMIN IaaS via 100G ? SP1 SP2 SP3 SP4 LSW1 host001 host002 host024 iSCSI Underlay networks LSW2 host025 host026 host027 LSW3 172.26.66.64/26172.26.66.0/26 LSW20 1:10 server to client LSWn ~10x 10-12Gbps per “Bladeset” 24x 10Gbps 172.16.136.0/24 172.16.137.0/24 host-100G-1 host-100G-2 12x 40Gb ECMP uplinks per switch/router vmhost1 vmhost2 vmhost2424x 10Gbps “Blessed” private subnet
- 12. Hardware • Mellanox Connect-X4 Dual port 100Gb QSFP+ DA – Dell R730XD servers. – VXLAN/NV|GRE and Erasure Coding offload in h/w • Mellanox Dual MSN2100 16 port x 100G switch/routers
- 13. Potential Issues • Blocking “backdoor” access across the fabric – Port ingress/egress ACL’s ? • …but trunked VLAN’s or VXLAN’s at hypervisor port(s) • Performance impact of VXLAN terminations • 100G on the server at all – cf. 1->10Gb kernel tuning transition – 2x 100Gb ports > PCI3 bandwidth limited to 120Gb • Can the software keep up ?
- 14. 100G server software ? host-100G-1 tomcat-1 tomcat-4tomcat-2 tomcat-3 apache/nginx Load Balancing • OpenDAP – Parallel servers and threads ? – CPU and RAM implications – JVM memory issues ?
- 15. Summary • Target : Provide “non-blocking” data access to JASMIN IaaS. • Use of 100Gb Networking : – Reduces server count – Scaleable for growing infrastructure • Experimental. Many potential issues to resolve: – Fabric routing egress/ingress ACLs – 100G kernel tuning ? – Can the software keep up
- 16. Questions
Editor's Notes
- #4: JASMIN = Joint Analysis System Meeting Infrastructure Needs. A “Super Data Cluster” not a “Super Computer” The data analysis “missing link” between environmental data sources (satellites, aircraft, in-situ etc)and super computer simulations. Funded by NERC for all of NERC sciences. User and science management and operations by STFC RAL Space ‘CEDA’ team (Centre for Environmental Data Analysis). Physically hosted, procured and operated at STFC RAL by the Research Infrastructure group of the SCD Built in multiple phases since late 2011. Plans for expansion to 50PB on disc and matching on tape to 2020. One of NERCs 3 major compute infrastructures (along with the supercomputers ; Archer@ EPCC and MonSoon @MetOffice Exeter) Uses: * Satellite data, weather data, eg JASMIN is the ESA ‘Sentinel’ Missons UK Data hub (2-10Tbytes per day). * Climate simulation output from the biggest super computers (“Archer”, MetOffce “Monsoon”, DKRZ.) Primary use is for dataset intercomparision Eg JASMIN Holds >60% of Data used by the latest IPCC report on Climate change. Largest data set 600TB ( 10yr climate simulation from DKRZ ) Oceanography, landslip, forestry fires……plus other environmental research eg Genetic data from bugs in the environment (Environmental Genomics) Supports 5,000 world wide users (download services) and ~1,000 direct login users Stats: 16 PetaBytes useable high performance spinning disc (20PB raw) ~ 3,200,000 DVD’s = ~ 3.8 km high tower of DVD’s or > 30,000 years of MP3 Two largest Panasas ‘realms’ in the world (109 and 125 shelves). Largest single site Panasas installation in the world and Panasas’ largest customer. 900TB useable (1.44PB raw) NetApp iSCSI/NFS for virtualisation + Dell Equallogic PS6210XS for high IOPS low latency iSCSI 4,500 CPU cores split dynamically between batch cluster and cloud/virtualisation (VMware vCloud Director and vCenter/vSphere) 35PB Tape archive and Elastic tape 40 Racks >3 Tera bits per second bandwidth (~3500 DVD’s per minute). IO Capability of ~250GBytes/sec – Arguably in the top 10 in the world for IO performance. “hyper” converged network infrastructure - 10GbE + MPI low latency (~8uS) + iSCSI over same network fabric. (ie No separate SAN or Infiniband) Finalist for BCS UK industry Awards “Big Data Project of the Year” 2012 and 2014 (Won the 2012 SVC industry Public Sector Storage Project of the year award.) Designed throughout to be installed, operated and upgraded by very small (SCD) team. Currently 3FTE. Team designs, procures and integrates all components separately for best of breed solutions. You cannot buy JASMIN off the shelf (although several have tried ! Inc. Crick and MetOffice). Capital spending todate of ~£14M on a tick – tock model : ~£3-5M / ~£1M and assumed into the future.
- #8: There are no traditional “Network problems” in JASMIN ! Aka Congestion.