Next Generation Nexus 9000 Architecture
13 likes22,702 views
The document discusses the next generation of Cisco's Nexus 9000 architecture, highlighting advancements such as 25G SerDes technology and new ASIC building blocks (ASE-2, ASE-3, LSE). It covers design impacts, capabilities, and the integration of automation in network management to enhance performance and efficiency. Additionally, it emphasizes the importance of applying Moore's Law to improve functionality while maintaining cost and energy efficiency.
Next Generation Nexus 9000 Architecture
- 1. Cisco Confidential© 2015 Cisco and/or its affiliates. All rights reserved. 1 Next Generation Nexus 9000 Architecture Cisco Connect Toronto 2016 Mike Herbert – Principal Engineer May 19, 2016 In collaboration with
- 2. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2 Agenda • Nexus 9000 & 3000 • What’s New • Moore’s Law and 25G SerDes • The new building blocks (ASE-2, ASE-3, LSE) • Next Generation Capabilities • Design Impacts of 25G and 100G • Next Gen Nexus 9000 Switch Platforms • Nexus 9200/9300 (Fixed) • Nexus 9500 (Modular)
- 3. Programmable NetworkProgrammable Fabric Application Centric Infrastructure DB DB Web Web App Web App VxLAN-BGP EVPN standard-based 3rd party controller support Modern NX-OS with enhanced NX- APIs DevOps toolset used for Network Management (Puppet, Chef, Ansible etc.) Turnkey integrated solution with security, centralized management, compliance and scale Automated application centric-policy model with embedded security Broad and deep ecosystem Cisco Data Centre Networking Strategy: Providing Choice in Automation and Programmability Cisco Controller for software overlay provisioning and management across N2K-N9K Nexus 9700EX + 9300EX VTS Creation Expansion Fault MgmtReporting Connection Nexus 9400 (line cards), 9200, 3100, 3200
- 4. Nexus 9300 48p 10G & 4p 40G VXLAN routing option 36p 40G ACI 32p 40G NX-OS Nexus 9504 Nexus 9508 Nexus 9516 Nexus 9500 36p wire rate 40/50/100G 56p 40G + 8p 40/50/100G 72p 40G 48p 10/25G SFP & 4p 40/50/100G + 2p 40G 48p 10/25G SFP & 6p 40/50/100G 48p 10GT & 6p 40/50/100G Nexus 9300EX Nexus 9200 Nexus 9000 Portfolio 10/25/40/50/100G on Merchant or Cisco Silicon 48p 10G & 6p 40G 96p 10G & 6p 40G 32p 40G 36p 25/40/50/100G ACI & NX-OS 32p 25/40/50/100G NX-OS Existing Chassis Delivering on Investment Protection Promise Continued Support for All Existing Line Cards Merchant and Cisco Silicon Industry Only 25G Native VXLAN Industry Only 25G Native VXLAN
- 5. Continued Support of Broadcom Silicon Nexus 3000: 10 Million Ports Shipped Nexus 3100 64p 40G Single Chip48p 10G & 6p 100G Shipping for 3+ months Nexus 3100V Nexus 3200 VXLAN routing, 100G uplinks, No 25G VXLAN bridging, 25/100G Single NX-OS Image for Nexus 3000 & Nexus 9000 32p 40G 32p 25/50/100G 32p 40G 64p 40G 48p 10G & 6p 40G 48p 1G & 4p 10G T2+ Tomahawk
- 6. • Agenda • Nexus 9000 & 3000 • What’s New • Moore’s Law and 25G SerDes • The new building blocks (ASE-2, ASE-3, LSE) • Next Generation Capabilities • Design Impacts of 25G and 100G • Next Gen Nexus 9000 Switch Platforms • Nexus 9200/9300 (Fixed) • Nexus 9500 (Modular)
- 7. “The number of transistors incorporated into a chip will approximately double every 24 months …” “Moore’s Law” - 1975 7
- 8. Moore’s Law CMOS in out VDD VSS out p+ p+ n-well n+ n+ in “Feature size” This dimension is what Moore’s Law is all about !!
- 9. http://en.wikipedia.org/wiki/Semiconductor_device_fabrication Moore’s Law The new generation of Nexus 9000 is leveraging 16nm FF+ (FinFet) It’s all about the Economics • Increased function, efficiency • Reduced costs, power • ~ 1.6 x increase in gates between process nodes Intel 14nm - 2016 Cisco 16FF+ - 2016 BCOM 28nm - 2016 BCOM 40nm - 2013 Cisco 28nm - 2014
- 10. SerDes: Serializer + Deserializer 10 • SerDes Clocking Increases • 10.3125G (40G, 10G) • 25.78125(25G/50G/100G) - 2016
- 11. 40GE/100GE interfaces have multiple lanes (coax cables, fibers, wavelengths) MLD provides a simple (common) way to map 40G/100G to physical interfaces of different lane widths MLD (Multi Lane Distribution) Multi Lane Distribution (MLD)
- 12. Parallel Lanes 4 x10 = 40G shifts to 4 x 25 = 100G 100-GbE Backed by 10G SerDes Backed by 25G SerDes
- 13. Development Cycle Decreasing Time to Leverage Moore’s Law is Reducing 2016 2017 2018 20202019 2021 FeaturesandCapabilities Classical ASIC 2 Year Dev Cycle Tick-Tock 18 Month Dev Cycle
- 14. Metcalfe, Moore and ASIC Pin I/O Rates The Switch Architectural ChallengeTechnologyImpactsonSwitchDesigns Pin (I/O) Speed Capacity from ASIC to external components The rate of change for overall network bandwidth is growing faster than Moore’s Law which in turn is faster than the rate of change for I/O from the ASIC to off chip components Pressure from the disparity in rates of change has required a new architectural balance Moores’ Law Transistor Density Metcalfe's Law Network Bandwidth Time - t Factor Year 1990 2000 2010 2016 Switch BW 1 67 2,667 30,000 Moore’s 1 32 1,024 8,129 DRAM 1 5.6 32 90.5
- 15. Switch On Chip (SOC) Sharing Objects Among ASICs 15 SOC ASIC architecture Cross Connect Network Slice 1 Slice 3Slice 2 Slice N Central Statistic Global Component Slice Component IO Component Phys/MAC Input Forwarding Controller Parse Packet Headers L2 Lookup L3 Lookup Ingress ACL Processing Ingress Traffic Classifica on Forwarding Results Genera on Buffer Pause Accoun ng and Flow Control Sta s cs Input Data Path Controller BroadcastNetwork Packet Queuing / Shaping Egress Buffer Accoun ng Output Data Path Controller Mul cast Replica on Egress ACL Processing Mul cast Fanout Packet Rewrites Output Forwarding Path Controller Phys/MAC ASE2/ASE3 ASIC Slice Common Building Blocks
- 16. ASIC Used by Nexus 3000/9000 Merchant Merchant + Cisco 1st Gen Switches: 2013–2015 40nm 28nm Trident T2 ASE & ALE Merchant 2nd Gen Switches: 2016+ 28nm 16nm Tomahawk Trident 2+ ASE2, ASE3 & LSE 40nm Scale • Route/ Host tables • Sharding • Encap normalization • EPG/ SGT/ NSH Telemetry • Analytics • Netflow • Atomic Counters Optimization • Smart Buffers • DLB/ Flow Prioritization
- 17. ASIC Used by Nexus 3000/9000 16nm ASE-2 ASE-3 ASE2 – ACI Spine Engine 2 3.6 Tbps Forwarding (Line Rate for all packet sizes) 36x100GE, 72x40GE, 144x25GE, ... ASE3 – ACI Spine Engine 3 1.6 Tbps Forwarding (Line Rate for all packet sizes) 16x100GE, 36x40GE, 74x25GE, ... Flow Table (Netflow, …) Standalone leaf and spine, ACI spine 16K VRF, 32 SPAN, 64K MCAST fan-outs, 4K NAT MPLS: Label Edge Router (LER), Label Switch Router (LSR), Fast Re-Route (FRR), Null-label, EXP QoS classification Push /Swap maximum of 5 VPN label + 2 FRR label 8 unicast + 8 Multicast Flexible DWRR scheduler across 16 queues Active Queue Management AFD ,WRED, ECN Marking Flowlet Prioritization & Elephant-Trap for trapping 5 tuple of large flows ASE-2 ASE-3
- 18. ASIC Used by Nexus 3000/9000 16nm LSE LSE – Leaf Spine Engine Standalone leaf & spine, ACI leaf and spine Flow Table (Netflow, …) ACI feature and service and security enhancement 40MB Buffer 32G fibre channel and 8 unified port 25G and 50G RS FEC (clause 91) Energy Enhancement Ethernet, IEEE 802.3az Port TX SPAN support for multicast MPLS: Label Edge Router (LER), Label Switch Router (LSR), Fast Re-Route (FRR), Null-label, EXP QoS classification Push /Swap maximum of 5 VPN label + 2 FRR label 16K VRF, 32 SPAN, 64K MCAST fan-outs, 50K NAT 8 unicast + 8 Multicast Flexible DWRR scheduler across 16 queues Active Queue Management AFD ,WRED, ECN Marking Flowlet Prioritization, Elephant-Trap for trapping 5 tuple of large flows
- 19. ASIC Used by Nexus 3000/9000 Merchant 28nm Tomahawk Trident 2+ Broadcom Tomahawk 3.2 Tbps I/O & 2.0 Tbps Core Tomahawk supports 3200 Gbps when average packet size is greater than 250 bytes. When all ports are receiving 64 byte packets, throughput is 2000 Gbps 32 x 100GE Standalone leaf and spine VXLAN Bridging Broadcom Trident 2+ 1.28Tbps I/O & 0.96T Core (< 192B pkt) 32 x 40GE (line rate for 24 x 40G) Standalone leaf and spine VXLAN Bridging & Routing (with-out recirculation)
- 20. Responding to Fast Market Changes Sharing Platforms Among Different Architectures 20 DB DB Web Web App Web App VTS Creation Expansion Fault MgmtReporting Connection • Common hardware platforms for ACI and NX-OS fabric • Sharing platform with UCS FI
- 21. Responding to Fast Market Changes Sharing ASICs Among Platforms 21 ASE-2LSE N93180YC-EX N9272Q N9236C
- 22. Why do we discuss automation so much? Programmable NetworkProgrammable Fabric Application Centric Infrastructure DB DB Web Web App Web App VxLAN-BGP EVPN standard-based 3rd party controller support Modern NX-OS with enhanced NX- APIs DevOps toolset used for Network Management (Puppet, Chef, Ansible etc.) Turnkey integrated solution with security, centralised management, compliance and scale Automated application centric-policy model with embedded security Broad and deep ecosystem Cisco Controller for software overlay provisioning and management across N2K-N9K Automation, API’s, Controllers and Tool-chain’s VTS Creation Expansion Fault MgmtReporting Connection When you take advantage of Moore’s Law you need to shift to a server like operational models
- 23. No Changes to EOS and EOL • Will you see more rapid changes in the Networking Space from the Industry • YES • Does this mean you will be forced to upgrade faster • NO • EoS and EoL policies will still be the same • The choice is still yours
- 24. • Agenda • Nexus 9000 & 3000 • What’s New • Moore’s Law and 25G SerDes • The new building blocks (ASE-2, ASE-3, LSE) • Next Generation Capabilities • Design Impacts of 25G and 100G • Next Gen Nexus 9000 Switch Platforms • Nexus 9200/9300 (Fixed) • Nexus 9500 (Modular)
- 25. Hyper-Converged Fabrics Data Centre Implications Order(s) of Magnitude increase in density of endpoints Increased I/O traffic drives larger Bandwidth Mix of traffic types drives need for better queueing (not buffering) Security Density is Increasing as well Containers, Scale-Out Storage mixed with existing VM and Bare Metal Distributed IP storage for cloud apps and traditional storage (iSCSI/NAS,FC) for existing apps Distributed Apps via Container based Micro-services Inter-process Communication across fabric
- 26. Hypervisors vs. Linux Containers 26 Hardware Operating System Hypervisor Virtual Machine Operating System Bins / libs App App Virtual Machine Operating System Bins / libs App App Hardware Hypervisor Virtual Machine Operating System Bins / libs App App Virtual Machine Operating System Bins / libs App App Hardware Operating System Container Bins / libs App App Container Bins / libs App App Type 1 Hypervisor Type 2 Hypervisor Linux Containers (LXC) Containers share the OS kernel of the host and thus are lightweight. However, each container must have the same OS kernel. Containers are isolated, but share OS and, where appropriate, libs / bins. Source: RightScale 2015 State of the cloud report Adoption / plan to use
- 27. Bare Metal, Hypervisors, Containers & Unikernels Changes in End Point Density Unikernels, also know as “virtual library operating system” OS Bin Bin App App Multi-Application Bare Metal Growth in the Number of Endpoints Servers as End Points Micro-Servers (Processes) as End Points
- 28. IPv4/v6 Unicast FIB VPN / Prefix / Mask / Paths / Offset 1 / 10.1.2.0 / 24 / 4 / 1 1 / 10.1.3.0 / 24 / 1 / 5 3 / 10.1.2.0 / 24 / 2 / 6 3 / 10.1.3.0 / 24 / 2 / 6 Path Table Path 1 Path 2 Path 3 Path 4 Path 1 Path 1 Path 2 Why does the increase in endpoints matter? Rewrite Information ADJ 1 - Rewrite SRC A+DST A MAC ADJ 2 - Rewrite SRC A+DST B MAC ADJ 3 - Rewrite SRC A+DST C MAC ADJ 4 - Rewrite SRC A+DST D MAC ADJ 5 - Rewrite SRC A+DST E MAC ADJ 6 - Rewrite SRC A+DST F MAC ADJ 7 - Rewrite SRC A+DST G MAC ADJ 8 - Rewrite SRC A+DST H MAC ADJ 9 - Rewrite SRC A+DST I MAC ADJ 10 - Rewrite SRC A+DST J MAC Hashing Scale and Flexibility of Forwarding Tables will be stressed
- 29. ASE2, ASE3 & LSE Tile Based Forwarding Tables 29 8K x 104 bits 8K x 104 bits 8K x 104 bits 8K x 104 bits 8K x 104 bits 8K x 104 bits 8K x 104 bits 8K x 104 bits 8K x 104 bits 8K x 104 bits 8K x 104 bits 8K x 104 bits 8K x 104 bits 8K x 104 bits 8K x 104 bits 8K x 104 bits • Improve flexibility by breaking the lookup table into small re-usable portions, “tiles” • Chain lookups through the “tiles” allocated to the specific forwarding entry type • IP LPM, IP Host, ECMP, Adjacency, MAC, Multicast, Policy Entry • e.g. Network Prefix chained to ECMP lookup chained to Adjacency chained to MAC • Re-allocation of forwarding table allows maximized utilization for each node in the network • Templates will be supported initially Initial Lookup for LPM Chained Lookup for ECMP Entries for the Route Chained Lookups for Adjacency & MAC
- 30. Forwarding Table Compression 30 Destination IP Next_hop 100.1.1.1/32 2.2.2.2 100.1.1.2/32 2.2.2.2 100.1.1.3/32 2.2.2.2 100.1.1.4/32 2.2.2.2 100.1.1.5/32 2.2.2.2 Common Information that can be eliminated Pivot Entry 100.1.1.0/29 TRIE Entry Next_Hop .1 2.2.2.2 .2 2.2.2.2 .3 2.2.2.2 .4 2.2.2.2 .5 2.2.2.2 • Eliminating repetitive information from forwarding table. Increased table scale with same amount of SRA. Effectively compress forwarding table entries. • Applicable for IPv4 host, IPv4 LPM routes and IPv6 /64 LPM routes 100.1.1.0/29 .0 .1 .2 .3 .4 .5 .6 .7 3 bits required per entry. Able to pack more entries with same amount of memory
- 31. Enabling Group-Based Policies Across the Enterprise Web App DB ACI Fabric Data Center APIC Policy Domain APIC ACI Policy Domain • Goal: Consistent Security Policy Groups and Identity shared between TrustSec and ACI domains • Allow TrustSec security groups to be used in ACI policies • Allow ACI EndPoint Groups to be used in policies across the Enterprise • Simplify management of security functions using both TrustSec and ACI classifications TrustSec Policy Domain TrustSec domain Voice Employee Supplier BYOD Campus / Branch / Non-ACI DC TrustSec Policy Domain Voice VLAN Data VLAN
- 32. ISE Policy Domain CMD/SGT APIC Policy Domain GBP VXLAN (GBP ID == SGT) TrustSec Border Router (ASR1K, ASR9K) Consistent Campus Security – DC Policy Enforcement ACI Spine SXP IPSec EPG SGT <-> EPG Mapping Policy Federation
- 33. Fabric Wide Troubleshooting Real Time Monitoring, Debugging and Analysis Debug Understand ‘what’ and ‘where’ for drops and determine application impact Monitor Track Latency (avg/min/max), buffer utilization, network events Analyze Specific events and suggest potential solution (e.g. trigger automatic rollback) Flow Monitoring Flow Monitoring Application Traffic operating within SLA Granular Fabric Wide Flow Monitoring Delivering Diagnostic Correlation Impacted Application Traffic
- 34. Real-time Flow Sensors ASE-3 & LSE • Granular flow information • Per flow statistics • Per packet visibility Length 66 Length 9000 Accumulated Flow Information (Volume…) Per Packet Variations
- 35. VXLAN & Fabric Design Requirements Host-based Forwarding VXLAN, MPLS, dot1q VTEPVTEPVTEPVTEP VTEP VTEP Spine – No VTEP Required Collapsed Border Spine – VTEP Required Border Leaf VXLAN Overlay EVPN MP-BGP or ACI VTEP VXLAN VLAN VTEPVTEPVTEPVTEP VTEP VTEP VXLAN to VXLAN AnyCast Gateway VTEP VXLAN, MPLS Multi-Protocol Border Leaf
- 36. VXLAN Support Gateway, Bridging, Routing* VXLAN to VLAN Bridging (L2 Gateway) VXLANORANGE Ingress VXLAN packet on Orange segment Egress interface chosen (bridge may .1Q tag the packet) VXLAN L2 Gateway Egress is a tagged interface. Packet is routed to the new VLAN VXLAN to VLAN Routing (L3 Gateway) VXLANORANGE Ingress VXLAN packet on Orange segment VXLAN Router Destination is in another segment. Packet is routed to the new segment VXLANORANGE VXLANBLUE Ingress VXLAN packet on Orange segment VXLAN Router VXLAN to VXLAN Routing (L3 Gateway) VLANBLUE VLANORANGE
- 37. Trident BCM VLAN Subnet 10.20.20.0/24 VxLAN Subnet 10.10.10.0/24 VxLAN to VLAN Routing – Trident 2 Decap Recirculate Route Packet Outer IP Outer UDP VXLAN Outer Ethernet Inner Ethernet Payload New FCS Inner IP Inner Ethernet Payload FCS Inner IP Perform a FIB lookup when DMAC = This Router Match against this TEP address VxLAN routed mode via loopback is possible, packet is de-encapsulated, forwarded out through a loopback (either Tx/Rx loopback or via external component), on second pass the match for ‘my router’ MAC results in L3 lookup and subsequent forward via L2 VLAN
- 38. ASE2, ASE3, LSE Decap + Route + Encap VLAN/VxLAN to VxLAN Routing Nexus 9300EX, 9200 Standalone Mode VxLAN, VLAN, Label Subnet 10.20.20.0/24 VxLAN, VLAN, Label Subnet 10.10.10.0/24 Outer IP Outer UDP VXLAN Outer Ethernet Inner Ethernet Payload New FCS Inner IP Parse the Outer IP & UDP Header Inner TCP/UDP Parse the VXLAN Header • ASE2, ASE3 & LSE ASIC parse the full outer MAC, IP/UDP header, VXLAN header in one pipeline pass • VLAN to VXLAN ‘and’ VXLAN to VXLAN routing is performed in a single pass • Line rate performance for all encapsulations with all packet sizes
- 39. ASE2, ASE3, LSE Decap + Route + Encap VLAN/VxLAN to VxLAN Routing Nexus 9300EX ACI Mode VxLAN/VLAN Subnet 10.20.20.0/24 VxLAN Subnet 10.10.10.0/24 Outer IP Outer UDP VXLAN Outer Ethernet Inner Ethernet Payload New FCS Inner IP Parse the Outer IP & UDP Header Inner TCP/UDP Parse the VXLAN Header Parse the Inner MAC, IP & TCP/UDP Header • LSE (Leaf and Spine) ASIC parse the full outer MAC, IP/UDP header, VXLAN and inner MAC, IP & UDP/TCP header in one pipeline pass • VLAN to VXLAN ‘and’ VXLAN to VXLAN routing is performed in a single pass • Line rate performance for all encapsulations with all packet sizes
- 40. Hyper-Converged Fabric Distributed IPC & IP Storage Distributed IP Storage Smart Buffer Spines Smart Buffer TOR/ Leaf IP Storage Traffic 300m Smart Buffers in Leaf / Spine Dedicated buffer for guaranteeing lossless traffic Data Traffic Requirements Mix of application workloads Dynamic change of traffic profile (Distributed IP storage, voice/video, big data, virtual network services, distributed micro-services – IPC) Maximise application performance Cisco Solution Flow-let switching across spine/ leaf fabric Dynamic load balancing based on congestion/ latency Adaptive scheduling Improvement of Application Flow Completion Time
- 41. Buffering Data Centre Two Requirements for Buffers • Long Lived TCP flows • Maximize the utilization of the available network capacity (ensure links are able to run at line rate) • Window Size Increases to probe the capacity of the network • Delay x Bandwidth Product (C x RTT)* • e.g if your network had 100 msec of latency with 10G interface, 125KBytes is required to keep the interface running at maximum capacity (line rate) • Incast Scenarios • Headroom, how much space is available to absorb the burst of traffic (excess beyond the buffer required by long lived TCP flows) Buffer Available for Incast Burst Buffer Required for Maximizing Network Utilization
- 42. Buffer Occupancy Why Buffer ? Sender 1 Sender 2 Receiver 10Gbps 100% Throughput Buffer Occ > 0 Throughput = 100%
- 43. Buffer Occupancy More Buffer = Additional Latency Sender 1 Sender 2 Receiver 10Gbps 100% Throughput Application does not go faster
- 44. Elephants Waste Buffer Sender 1 Sender 2 Receiver Elephants buildup large queues No buffer left for mice Because TCP eats up all available buffer (until packet drop) Applications performance suffers
- 45. Multiple TCP flows in reality full WindowSizeBufferOccupancy empty
- 46. Long Lived TCP Flows TCP Congestion Control and Buffer Requirements • Rule of thumb is for one TCP flow, B = • But, typical link carries 10’s - 1000s of flows and it turns out that the actual buffer requirement is less than this Required buffer is instead of • Proven by theory and experiments in real operational networks • For example, see Beheshti et al. 2008: “Experimental Study of Router Buffer Sizing” C ´ RTT n C´RTT C´RTT
- 47. Micro-bursts Need Headroom Where does it come from? Burst Link average full empty Headroom Queuing Latency ?
- 48. Micro-bursts Need Headroom Where does it come from? Burst Link average full empty More Headroom More Queuing Latency The Bigger Buffer Approach • Larger Buffer can increase the burst headroom but • Increases queuing latency which decreases application performance • You can still have large flows fill up the entire buffer resulting in no increase in burst headroom • Impacts application performance
- 49. We want the best of both worlds Burst Link average full empty More Headroom Less Queuing Latency • Maximize the amount of buffer always available for bursts • Minimize the latency for high throughput flows • Better application performance for both types of traffic
- 50. Innovation gives us the best of both worlds AFD & DPP • How to minimize the buffer used by long lived flows while ensuring maximal use of network capacity • Approximate Fair Drop (AFD) for active queue management • Computes a “fair” rate for each flow at the output queue and dropping flows in proportion to the amount they exceed the approximated fair rate • How to ensure the incast flows are serviced as fast as possible to keep the buffer available • Dynamic Packet (Flow) Prioritization (DPP) AFD Discard Threshold for Large TCP Flows Buffer Headroom for Mice Flows
- 51. AFD Increases Headroom, Reduces Latency full BufferOccupancy empty average Buffer Headroom Available for Burst Buffer Required for Max Throughput
- 52. DPP (Dynamic Packet Prioritization) Prioritized Packets Express Lane Normal Lane Egress Queue • Separate flows into short and long • Put short flows in high priority queue • Put long flows in low priority queue • The 10% of bytes that are in short flows means high priority queue will be empty • Prioritization guarantees packet order • We want to prevent the drops of the mice, the incast and burst traffic
- 53. All flows are short until they become long Flow hilo Link Prio lohi Sortshort long 10% 90% long medium short packet
- 54. Elephant Trap packet • Mechanism to identify large volume flows • Identified based on 5-tuple • Elephant trap threshold is byte-count-based. • When received packets in a flow exceeds the number of bytes specified by the threshold, the flow is considered an elephant flow • Only elephant flows are submitted to AFD dropping algorithm. Mice flows are protected and not subject to AFD dropping • Arriving data rate is measured on the ingress, and compared against a calculated fair rate on the egress port to decide dropping capability 10 msec E* E* E* == Elephant Flow
- 55. DPP looks for Any Burst TCP, UDP, Multicast, .. A Long-lived TCP Session An Elephant Flow Source Dest One Long-lived Session Multiple Flowlets • The elephant trap and DPP algorithm are not tracking only TCP sessions • The algorithm is 5-tuple based which means it can find TCP, UDP, Unicast and Multicast bursts • A very long lived session that is quiet and then bursts will be prioritized for that burst • Traffic arriving due to a link failure will be prioritized, etc …
- 56. Better Application Performance in an Incast Environment 56 0.00 20.00 40.00 60.00 80.00 100.00 120.00 140.00 160.00 180.00 200.00 20 30 40 50 60 70 80 90 95 FlowCompletiontime(msec) Traffic Load (% line rate) Data Mining Workload Average Flow Completion Time Nexus 92160 Nexus 9272Q Merchant (BCOM Dune 9GB) http://miercom.com/cisco-systems-speeding-applications-in-data-Centre-networks/
- 57. Better Application Performance in an Incast Environment 57 Nexus 92160 Nexus 9272Q Merchant (BCOM Dune 9GB) 0.00 5.00 10.00 15.00 20.00 25.00 20 30 40 50 60 70 80 90 95 FlowCompletiontime(msec) Traffic Load (% line rate) Data Mining Workload Under 100KB Flow Completion Time http://miercom.com/cisco-systems-speeding-applications-in-data-Centre-networks/
- 58. Better Application Performance in an Incast Environment 58 Nexus 92160 Nexus 9272Q Merchant (BCOM Dune 9GB) 0.00 500.00 1000.00 1500.00 2000.00 2500.00 20 30 40 50 60 70 80 90 95 FlowCompletiontime(msec) Traffic Load (% line rate) Data Mining Workload > 10MB Flow Completion Time http://miercom.com/cisco-systems-speeding-applications-in-data-Centre-networks/
- 59. ACI Fabric Load Balancing Flowlet Switching H1 H2 TCP flow • State-of-the-art ECMP hashes flows (5- tuples) to path to prevent reordering TCP packets. • Flowlet switching* routes bursts of packets from the same flow independently. • No packet re-ordering Gap ≥ |d1 – d2| d1 d2 *Flowlet Switching (Kandula et al ’04)
- 60. ACI Fabric Load Balancing Congestion Monitoring P1 5 At every ASIC hop along the path DRE field is updated with the congestion metric (if my DRE > current DRE then update DRE field else keep current DRE) 2 P10 0 Every packet is sourced from an iLeaf with an identifier for the ingress port to the Fabric and a DRE = 0 metric 0P1 1 P10 5 Next packet sent back to the originating iLeaf will carry with it feedback information on the max load recorded on the ingress path 4 P4 On receipt of the metric feedback the original leaf will update the Flowlet load balancing weights (It will also receive indication of the load from the other direction) 6 P13 5P4 P1 5 On receipt of each packet update the DRE weight in feedback table tracking LBTag (source identifier) 3 P13 5P4 5
- 61. • Agenda • Nexus 9000 & 3000 • What’s New • Moore’s Law and 25G SerDes • The new building blocks (ASE-2, ASE-3, LSE) • Next Generation Capabilities • Design Impacts of 25G and 100G • Next Gen Nexus 9000 Switch Platforms • Nexus 9200/9300 (Fixed) • Nexus 9500 (Modular)
- 62. Design for Optimal Capacity What does 25/50/100G mean 2015 2016 2017 Transaction ResponseTime 2015 2016 2017 Storage-Petabytes 2015 2016 2017 Transaction Volume 2015 2016 2017 StorageAccess Time
- 63. Design for Optimal Capacity What does 100G mean 6 x 100G Uplinks 0.8 : 1 Undersubscribed • You do not need to ‘and’ should not be designing a network that is capacity constrained • Capacity has a ‘very’ different cost point than it did even as recently as last year • Design for the Optimal Capacity Requirements • Bandwidth solves problems, buffering at best masks them • Consider “undersubscription” • It is now possible N9K-C93180YC-EX
- 64. Design for Optimal Capacity What does 100G mean 64 20×10Gbps Uplinks 2×100Gbps Uplinks 11×10Gbps flows (55% load) 1 2 1 2 20 Prob of 100% throughput = 3.27% Prob of 100% throughput = 99.95%
- 65. What about 25G? 25 Gbps Ethernet & Standards Consortium Passive: 1,2,3 meter Within Rack N9200, N9300-EX N3200 Mellanox Qlogic, BRCM IEEE Passive: 1,2,3,5 meter Optics: SR Across Rack Roadmap N9300-EXU None available yet Distance Deployment Supporting Platform 25G NIC ( Verified ) 25G NIC (Ongoing Testing)
- 67. QSFP-BIDI vs. QSFP-40G-SR4 TX RX TX/RX TX/RX 4 x 10G 4 x 10G 2 x 20G 2 x 20G 12-Fiber ribbon cable with MPO connectors at both ends Duplex multimode fiber with Duplex LC connectors at both ends 12-Fiber infrastructure Duplex Multimode Fiber Higher cost to upgrade from 10G to 40G due to 12-Fiber infrastructure Use of duplex multimode fiber lowers cost of upgrading from 10G to 40G by leveraging existing 10G multimode infrastructure QSFP-BIDI QSFP SR 12-Fiber infrastructure Duplex Multimode Fiber QSFP-BIDIQSFP SR 67
- 68. Cisco 100G QSFP28 Optics Portfolio Optics Type Description Connector Availability QSFP-100G-SR-BD 40/100G, 100m MMF LC 2HCY16 QSFP-100G-SR4-S 100GBASE-SR4, 100m MMF MPO Q4CY15 QSFP-100G-LR4-S 100GBASE-LR4, 10km SMF LC Q4CY15 QSFP-100G-CWDM4-S 100GE CWDM4, 2km SMF LC Q4CY15 QSFP-100G-PSM4-S 100GBASE-PSM4, 2km SMF MPO Q4CY15 QSFP-100G-CU 100GBASE QSFP to QSFP copper direct-attach cables Twinax Q4CY15 QSFP-4SFP25G-CU 100GBASE QSFP to 4x25G SFP+ copper break-out cables Twinax Q4CY15 QSFP-100G-AOC 100GBASE QSFP to QSFP active optical cables AOC (Active Optic Cable) Q4CY15 MPO: 8 strands LC: 2 strands SMF: Single Mode Fiber MMF: Multi Mode Fiber
- 69. • Agenda • Nexus 9000 & 3000 • What’s New • Moore’s Law and 25G SerDes • The new building blocks (ASE-2, ASE-3, LSE) • Next Generation Capabilities • Design Impacts of 25G and 100G • Next Gen Nexus 9000 Switch Platforms • Nexus 9200/9300 (Fixed) • Nexus 9500 (Modular)
- 70. Next Gen – 9200 & 9300EX Nexus 9300-EX 48p 1/10GT + 6p 40/100G QSFP Nexus 93108TC-EX 48p 10/25G SFP + 6p 40/100G QSFP Nexus 93180YC-EX Dual personality – ACI and NX-OS mode Industry’s first native 25G VXLAN capable switch Flexible port configurations – 1/10/25/40/50/100G Up to 40 MB shared buffer Native Netflow 72p 40G QSFP Nexus 9272Q Nexus 9200 36p 40/100G QSFP Nexus 9236C 48p 10/25G SFP + 4p 100G/ 6p 40G QSFP Nexus 92160YC-X 56p 40G + 8p 40/100G QSFP Nexus 92304QC NX-OS switches Industry’s first 36p 100G 1RU switch Industry’s first native 25G VXLAN capable switch Up to 30 MB shared buffer High density compact 40/100G aggregation
- 71. Cisco Nexus® Nexus 92160YC-X Nexus 9200 Series Switch Architecture ASE3 Based 71 • 1 RU height • No GEM module • 48 10/25 Gb SFP+ ports • 6 x 40 Gbps or 4 x 100Gbps QSFP28 ports • 1 100/1000baseT management port • 1 RS232 console port • 2 USB 2.0 ports • Front-to-back and back-to-front airflow options • 1+1 redundant power supply options • 2+1 redundant fans • Full line rate performance for all packet sizes • VXLAN bridging and routing • Full Netflow • Capable of supporting both NX-OS modes 48x 1/10/25 Gbps ports 6 x 40 or 4 x 100 Gbps QSFP28 ports N9K-C92160YC-X Console Port Management Port USB Ports Power supply and fan Power supply and fan 4 System Fan Trays
- 72. Cisco Nexus® Nexus N9K-C9236C Nexus 9200 Series Switch Architecture ASE2 Based 72 ASIC: ASE2 4-core CPU (Intel Ivy Bridge Gladden 4 core at 1.8 GHz) 8G DIMM memory 2MB NVRAM Two Power supply (1200W) 1 + 1 redundant Power consumption 450 W Two Fans 3 + 1 redundant 36 x 40/100G ports 144 10/25G ports (when all ports in breakout mode Each 100G Port Break-out Options: • 1p 100G SR, AOC • 1p 40G • 4p 10G • 4p 25G 1m, 3m • QSA (roadmap) 36 x 100 Gbps QSFP28 ports N9K-C9236C Power supply and fan Power supply and fan 4 System Fan Trays ASE-2 Network Interfaces 36 x 100G QSFP28
- 73. Nexus 9236C Port Configuration 1 RU 36 Port 100G Fiber Ports 1 - 36 are 100G QSFP28 (Breakout Capable) QSFP28 Each 100G Port Supports Multiple Speed and Break-out Options * (QSA in a future SW release) 100G 40G 4 x 25G 4 x 10G QSA 10G*
- 74. Ports 37 - 72 are 40G QSFP+ (Breakout Capable 144 x 10G) QSFP+ Ports 1 - 36 are 40G QSFP+ Nexus 9272Q Port Configuration 2RU 72 Port 40G Fiber 9272Q 40 40 40G Agg 40 40 9272Q 40 40 10G Access 10 10
- 75. Nexus 92304QC Port Configuration 2RU 56p 40G Fiber + 8p 40G/00G Ports 1-16 are 40G QSFP+ (Breakout Capable 4 x10G) QSFP28 Ports 33-56 are 40G QSFP+ Ports 57-64 100G QSFP28 QSFP+ Ports 17-32 are 40G QSFP+
- 76. Cisco Nexus® Nexus N9K- C93180YC-EX Nexus 9300EX Series LSE Based 76 ASIC: LSE 1RU 2-core CPU (Intel Ivy Bridge Gladden) 2MB NVRAM 64MB Two Power supply (650W) 1 + 1 redundant Power consumption 248 W Four Fans 3 + 1 redundant 48 x 10G/25G SFP28 and 6 x 40G/100G QSFP28 Support both NX-OS mode and ACI mode (ACI leaf) Netflow and data analytics 48x 1/10/25 Gbps ports 6 x 100 Gbps QSFP28 ports N9K-C93180YC-EX Power supply and fan Power supply and fan 4 System Fan Trays LSE Network Interfaces 6 x100G QSFP28
- 77. Nexus 9500 Platform Architecture 77 Nexus® 9508 Front View Nexus 9508 Rear View 8 line card slots Max 3.84 Tbps per slot duplex Redundant supervisor engines 3000 W AC power supplies 2+0, 2+1, 2+2 redundancy Supports up to 8 power supplies Redundant system controller cards 3 or 6 fabric modules (behind fan trays) 3 fan trays, front-to-back airflow No mid-plane for LC-to-FM connectivity Designed for Power and Cooling Efficiency Designed for Reliability Designed for Future Scale Chassis Dimensions: 13 RU x 30 in. x 17.5 in (HxWxD)
- 78. Modular Nexus 9500 A CLOS Based SOC Architecture (Leaf and Spine) SOC SOC SOC SOC SOC SOC SOC SOC Non Blocking CLOS Leverage Switch on Chip (SOC) Based components
- 79. Fabric Module •Back-ward compatible w/ existing Broadcom T2 based line cards Nexus 9500 – Moving Forward Existing 4-, 8-, 16- slot chassis No mid-plane to update Power and cooling within existing shipping system profile Existing shipping Power Supply, Supervisor and System Controllers 9504 9508 9516 32p 100G QSFP Line card •10/25/40/50/100G Merchant ASIC 9500 Series + Fabric Module •Back-ward compatible w/ existing Nexus 9300 ACI Leafs (40G uplinks) in ACI mode 32p 100G QSFP Line card •10/25/40/50/100G • Analytics Readiness Cisco ASIC + X9700-EX (NX-OS and ACI) X9400-S (NX-OS) Migrate From NX-OS to ACI Spine with Just a Software Upgrade Upgrade to 100G Infrastructure While Reusing Existing Chassis 16nm Technology 28nm Technology
- 80. 40/100G - Merchant N9K-X9432C-S Investment Protection with Supervisors, System Controller, PS and Chassis Flexible Speed 10,25,40,50,100G Supported in NX- OS mode QSFP28 Connector, Pin compatible with 40G QSFP+ Supports Mix and Match Current Linecards* 4, 8 and 16* Chassis * future
- 82. Nexus 9500 – LC vs Fabric Compatibility Merchant ASIC Note: 9636PQ works in 4 and 8 slot only X9400 Series X9600 Series X9500 Series NFE Fabric Module (3- 6) Supervisor, System Controller, Power Supply X9400-S Series NFE2 Fabric Module (4) Q1CY16 X9400-S Series NFE2 Fabric Module (4) X9400 Series 2HCY16Shipping 4, 8 and 16 Slot 4 and 8 Slot 4, 8 and 16 Slot Supervisor, System Controller, Power Supply Supervisor, System Controller, Power Supply NX-OS NX-OS NX-OS
- 83. Investment Protection with Supervisors, System Controller, PS and Chassis Flexible Speed 10,25,40,50,100G QSFP28 Connector, Pin compatible with 40G QSFP+ Supported in ACI and NX- OS mode 4, 8 and 16* Chassis * future 40/100G - LSE N9K-X9736C-EX (Q2CY16)
- 84. Modular Nexus 9500 Generation 2 Line Cards and Fabric Modules LSE LSE LSE LSE ASE-2 N9K-X9732C-EX (LSE) Summarization and Balance of Host and Network Routes Shift Network (LPM) Routes Host Routes 1. IPv4: 1M LPM+ host 2. IPv4: 750K LPM + host AND IPv6 /64: 256K N9K-C9504-FM-E*, N9K-C9508-FM-E* (ASE-2) ASE-2 ASE-2 ASE-2
- 85. ASE2-Based New Fabric Module for Nexus 9500 85 • The new fabric module is built with ASE2 ASICs • Continue to use an internal CLOS architecture with fabric modules at the spine and line cards at the leaf • Each Nexus 9500 switch needs up to 4 ASE2-based fabric modules • Each ASE2 ASIC on a fabric module provides 32 x 100GE internal ports to interconnect line cards. (32 out of 36 100GE ports on an ASE2 are used to build the CLOS architecture with evenly distributed bandwidth between each line card and fabric module.) • The number of ASE2 on a fabric module depends on the chassis types it supports: Chassis Type Nexus 9504 Nexus 9508 Nexus 9516 # of ASE2 on FM 1 2 4
- 86. LSE-Based New Line Card for Nexus 9500 9700EX 86 Fabric Module LSE 1 LSE 2 LSE 3 LSE 4 8 x 100G 8 x 100G 8 x 100G 8 x 100G 8 x 100G 8 x 100G 8 x 100G 8 x 100G 32 x 100GE Built with 4 LSE ASICs, 32 x 100GE front panel ports + 32x 100GE internal links to fabric modules Each LSE provides 8 x 100GE front panel ports Each LSE has 8 x 100GE internal links to fabric modules, evenly distributed among the 4 fabric modules
- 87. Nexus 9500 – LC and Fabric Compatibility Cisco ASIC X9700 Series NFE Fabric Module (6) X9700-EX Series ASE2 Fabric Module (4) Q2CY16Shipping 4, 8 and 16 Slot 4, 8 and 16* Slot ACI Only ACI, NX-OS Supervisor, System Controller, Power Supply Supervisor, System Controller, Power Supply
- 88. Programmable NetworkProgrammable Fabric Application Centric Infrastructure DB DB Web Web App Web App VxLAN-BGP EVPN standard-based 3rd party controller support Modern NX-OS with enhanced NX- APIs DevOps toolset used for Network Management (Puppet, Chef, Ansible etc.) Turnkey integrated solution with security, centralized management, compliance and scale Automated application centric-policy model with embedded security Broad and deep ecosystem Cisco Data Centre Networking Strategy: Providing Choice in Automation and Programmability Cisco Controller for software overlay provisioning and management across N2K-N9K Nexus 9700EX + 9300EX VTS Creation Expansion Fault MgmtReporting Connection Nexus 9400 (line cards), 9200, 3100, 3200
- 89. Thank you. In collaboration with