Cisco ASR 9000 Architecture - BRKARC-2003 3rd session.pdf
- 1. #CLUS
- 2. #CLUS Yongzhong Peng Manager, Technical Marketing SP Routing Infrastructure BRKARC-2003 Cisco ASR 9000 System Architecture
- 3. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Agenda • ASR9000 Products Introduction • ASR9000 System Hardware Architecture • ASR9000 Distributed Control Plane • ASR9000 Data Packet Processing • ASR9000 QoS Architecture • ASR9000 TCAM Architecture and Usage • Conclusion BRKARC-2003 3
- 4. Questions? Use Cisco Webex Teams to chat with the speaker after the session Find this session in the Cisco Live Mobile App Click “Join the Discussion” Install Webex Teams or go directly to the team space Enter messages/questions in the team space How Webex Teams will be moderated by the speaker until June 16, 2019. 1 2 3 4 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Cisco Webex Teams cs.co/ciscolivebot#BRKARC-2003 4
- 6. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Compact & Powerful Access/Aggregation Flexible Service Edge High Density Service Edge and Core • Small footprint with full IOS- XR for distributed environments • Optimized for MSE with high M-D scale for medium to large sites • Scalable, ultra high density service routers for large, high- growth sites One Platform, One OS, One Family Fixed 2RU 240 Gbps 2 LC/6RU 16 Tbps 8 LC/21RU 14 Tbps 10 LC/30RU 80 Tbps 20 LC/44RU 160 Tbps 4 LC/10RU 7 Tbps ASR 9904 ASR 9001 ASR 9006 ASR 9010 ASR 9912 ASR 9922 MSE DCI Peering P/PE Mobility Broadband nV Satellites ASR 9000v, NCS5000 CE 8 LC/21RU 64 Tbps ASR 9910 4 LC/14RU 32 Tbps ASR 9906 ASR 9901 Cisco ASR 9000 System Portfolio 456Gbps BRKARC-2003 6
- 7. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS ASR 9000 Hardware Components – RP/RSP ASR 9904 ASR 9006 ASR 9010 ASR 9910 ASR 9906 ASR 9912 ASR 9922 BRKARC-2003 7
- 8. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS RSP5/RP3 Front Panel Configuration • 2x BITS ports on RJ-45 • 100Mbps, 1588 port – RJ-45 • TOD – RJ-45 • 10Mhz on SMA • 1x CMP • 1PPS on SMA • Alarm output serial port • USB • 2x Management ports on RJ-45 • AUX & Console on RJ-45 connectors • LED’s for major/critical and normal oper alarms or states RSP5 RP3 BRKARC-2003 8
- 9. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS ASR 9000 Hardware Components – Linecards ASR 9904 ASR 9006 ASR 9010 ASR 9910 ASR 9906 ASR 9912 ASR 9922 BRKARC-2003 9
- 10. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS ASR 9900 - Switch Fabric Cards Common for ASR 9912 and ASR 9922 SFC-S for ASR 9910 7 Fabric Card Slots 5 Fabric Card Slots for ASR 9910 (2 on RSPs => total 7) Decoupled, multi-stage switch fabric hardware True HW separation between control and data plane Add bandwidth per slot easily & independently SFC2/SFC-S SFC3 Fabric Capacity per SFC 230G 600G Fabric Capacity Per Line Card Slot 1.38T N+1 1.61T N+0 3.6T N+1 4.2T N+0 Fabric Redundancy N+1 N+1 LC Support Typhoon Tomahawk Tomahawk LightSpeed ASR 9912 ASR 9922 ASR 9910 ASR 9906 BRKARC-2003 10
- 11. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS • ASR-9922 • A99-RP3-SE/TR • A99-SFC3 • ASR-9922-FAN-V3 • ASR-9912 • A99-RP3-SE/TR • A99-SFC3 • ASR-9910 • A9K-RSP5-SE/TR • A99-SFC3-S • ASR-9906 • A9K-RSP5-SE/TR • A99-SFC3-T • ASR-9904 / ASR-9010 / ASR-9006 • A9K-RSP5-SE/TR 4th Generation Hardware Commons PIDs A99-SFC3-T A99-SFC3-S A99-SFC3 BRKARC-2003 11
- 12. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS 5 Fabric Cards 2 Fan Trays 2 RSPs 4 Line Card Slots 1 Power Tray Cisco ASR 9906 Overview Feature Description Total Capacity >32T Capacity per Slot >4T Slots 6 slots - 4 Line Cards and 2 RSPs Rack size 14RU, front-to-back airflow. Power 1 Power Tray 4.4 KW DC supplies (max 4) 6.0 KW AC supplies (max 3) Fan Front-to-Back Airflow 2 Fan Trays, FRU RSPs Integrated Fabric, 1+1 Redundancy Fabric Cards 5 Fabric Cards + 2 RSP Integrated Fabric Total 7 Fabric for 6+1 Redundancy 230G per FC2 or 600G per FC3 Line cards Tomahawk Lightspeed Service Card: VSM with 32-bit XR BRKARC-2003 12
- 13. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS 3rd Gen Tomahawk 1.2T Tomahawk 28nm 240 Gbps Tigershark 28nm 200 Gbps SM15 28nm 1.20 Tbps X86 6 Core 2 Ghz 2nd Gen Typhoon 360G Typhoon 55nm 60 Gbps Skytrain 65nm 60 Gbps Sacramento 65nm 220 Gbps PowerPC Quad Core 1.5 Ghz Trident 90nm 15 Gbps Octopus 130nm 60 Gbps Santa Cruz 130nm 90 Gbps PowerPC Dual Core 1.2 Ghz 1st Gen Trident 120G 4th Gen Lightspeed 3.2T Lightspeed 16nm 400 Gbps SKB 16nm 3.59 Tbps X86 8 Core 2.2 Ghz ASR 9000 Silicon Evolution BRKARC-2003 13
- 14. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS 14 BRKARC-2003 ASR 9000 3rd Ethernet Line Card Overview MPAs 20x1GE 2x10GE 4x10GE 1x40GE 2x40GE -TR, -SE MPAs 1x100GE 2x100GE 20x10GE MOD400/MOD200 3rd gen LC Tomahawk NPU: 240Gbps, ~150Mpps A9K-400G-DWDM A9K-48X10GE-1G MPA 32x1GE 4x100GE & 8x100G CPAK OTN Line Card 4x100GE & 12x100G QSFP28 LAN Line Card
- 15. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS ASR 9000 4th Generation Line cards 1.6T-3.2T per slot 9922 9912 9904 9910 9906 32 x 100GE TR line card 16 x 100GE TR line card 16 x 100GE SE line card 9010 9006 16 x 100GE TR line card BRKARC-2003 15
- 16. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Pop Quiz ???? How much fabric bandwidth can one SFC3 provide per slot? • 200G • 4.2T • 3.0T • 600G BRKARC-2003 16 With 3rd Generation LC With 4th Generation LC
- 18. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS ASR 9000 System Architecture “At-a-Glance” Fully Distributed Architecture for High Performance and High Multi- dimensional Control Plane Scale RSP RSP/RP FIC CP U BITS/DTI FIA CPU Switch Fabric FIA FIA FIA FIA NP NP NP NP PHY PHY PHY PHY CPAK CPAK CPAK CPAK CPAK CPAK CPAK CPAK CPU CPU Switch Fabric FIA FIA NP NP SerDes XBAR Bay Bay CPU Switch Fabric Line Card Data forwarding is fully distributed across NPs Control plane split among RSP/RP and LC CPU Network Processor Network Processor BRKARC-2003 18
- 19. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS ASR90xx – RSP5 and Mixed LC FIA FIA FIA RSP5 Arbiter RSP5 FIA FIA FIA Stage 1 Stage 2 Stage 3 Tomahawk Line Card LightSpeed Linecard Ingress Linecard Egress Linecard Tomahawk LC Fabric bandwidth: Dual RSP: 860Gbps/slot Single RSP: 430Gbps/slot LightSpeed LC Fabric bandwidth: dual RSP: 1.8T/slot single RSP: 900G/slot Fabric Fabric Fabric SM15 Arbiter Fabri c BRKARC-2003 19 Fabric frame format: Super-frame Fabric load balancing: Unicast is per-packet Multicast is per-flow Virtual Output Queue Arbitration Active-Active Fabric
- 20. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS SFC3 SM15 Tomahawk LCs (7-fab) 3.2T LC at line-rate with fabric redundancy LightSpeed LC (7-fab) SKB 600G ASR9922/12 System - LightSpeed & Tomahawk Tomahawk LCs interoperate at full throughput w/ fabric redundancy 1.5T with 7 Fabrics 1.29T w/ 6 Fabrics 4.2T w/ 7 Fabrics 3.6T w/ 6 Fabrics A99-12X100GE A99-8X100GE-SE/TR A99-32X100GE-TR RP3 ARBITER RP2 and RP3 have enough capacity to arbiter 16x100G linecards. Only 32x100G requires RP3. BRKARC-2003 20
- 21. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS SFC3 SM15 Tomahawk LCs (7-fab) 3.2T LC at line-rate with fabric redundancy LightSpeed LC (7-fab) SKB 600G ASR9910/06 System - LightSpeed & Tomahawk Tomahawk LCs interoperate at full throughput w/ fabric redundancy 1.5T with 7 Fabrics 1.29T w/ 6 Fabrics 4.2T w/ 7 Fabrics 3.6T w/ 6 Fabrics A99-12X100GE A99-8X100GE-SE/TR A99-32X100GE-TR RSP5 RSP5 BRKARC-2003 21
- 22. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS 215G SFC3 SM15 Tomahawk LC (5-fabric) 3T with 5-fabrics Prevents “high priority” packet drop due to fabric congestion 3.2T LC (7-fab) SKB 215G 600G ASR99xx Mix Mode SKB LS LC (5-fab) 1.6T LC at line-rate w/ fabric redundancy 800G LC at line-rate w/ fabric redundancy All Tomahawk LCs interoperate at full throughput w/ fabric redundancy; except for 12x100GE LC, which will be at 1.075T 1.075T with 5 fabrics 860G w/ 4 fabrics 3T w/ 5 fabrics 2.4T w/ 4 fabrics System automatically rate-limits each slice to 375G (93.75% linerate per slice; single 100G port can reach 100% linerate) For 100G linerate on all ports: • Shut down one slice BRKARC-2003 22
- 23. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS 32x100GE (7-fabric) Linecard Architecture LightSpeed NP0 FIA LightSpeed NP1 FIA LightSpeed NP2 FIA LightSpeed NP3 FIA LightSpeed NP4 FIA LightSpeed NP5 FIA LightSpeed NP6 FIA LightSpeed NP7 FIA QSFP 3 QSFP 0 CPU Retimer 8x Slices Switch Fabric 0 (SKB) Switch Fabric 1 (SKB) QSFP 7 QSFP 4 Retimer QSFP 11 QSFP 8 Retimer QSFP 15 QSFP 12 Retimer QSFP 19 QSFP 16 Retimer QSFP 23 QSFP 20 Retimer QSFP 27 QSFP 24 Retimer QSFP 31 QSFP 28 Retimer x8 x8 x8 x8 x8 x8 x8 x8 x8 x8 x8 x8 x8 x8 Slice Based Architecture BRKARC-2003 23
- 24. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS LightSpeed NP2 FIA LightSpeed NP3 FIA LightSpeed NP0 FIA QSFP 3 QSFP 0 CPU Retimer 4x Slices Switch Fabric 0 (SKB) QSFP 11 QSFP 8 Retimer QSFP 19 QSFP 16 Retimer QSFP 27 QSFP 24 Retimer x8 x8 x8 x8 x8 x8 x8 x8 x8 x8 LightSpeed NP1 FIA 16x100GE (5-fabric) Linecard Architecture RSP 0 RSP 1 Slice Based Architecture BRKARC-2003 24
- 25. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Line Card Components NPU FIA Switch Fabric ASIC TM PHY CPU P1 P2 P3 BE P1 P2 P3 BE Runs distributed control plane protocols for increased scale BFD, CFM, ARP Receive FIB table from RP and program hardware forwarding table Provides data connection to switch fabric Manage VoQ, Superframe and loadbalancing data traffic across switch fabric Mcast replication table for replication toward NPs Main forwarding engine L2 and L3 lookups Multicast replication toward Optics User level QoS and Security features Dedicated queue ASIC – TM (traffic manager) per NPU for QoS functions User Configurable Queue on TM. Default Port Queue Always Created. BRKARC-2003 25
- 26. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Network Processor Architecture Details • TCAM: VLAN tag, QoS and ACL classification • Stats memory: interface statistics, forwarding statistics etc • Frame memory: buffer, Queues • Lookup Memory: forwarding tables, FIB, MAC, ADJ • TR/SE • Different TCAM/frame/stats memory size for different per-LC QoS, ACL, logical interface scale • Same lookup memory for same system wide scale mixing different variation of LCs doesn’t impact system wide scale - STATS MEMORY FRAME MEMORY LOOKUP MEMORY TCAM FIB MAC NPU Complex Forwarding chip (multi core) TR and SE has different memory size TR and SE has same memory size -TR: transport optimized, -SE: Service edge optimized BRKARC-2003 26
- 27. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Pop Quiz ???? Can 16x100GE(5-Fabric) be used in ASR-9910 Chassis? • Yes • No BRKARC-2003 27
- 29. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Switch Fabric FIA FIA FIA FIA NP NP NP NP PHY PHY PHY PHY CPAK CPAK CPAK CPAK CPAK CPAK CPAK CPAK CPU ASR9000 Fully Distributed Control Plane Switch Fabric Punt FPGA FIA Switch Fabric RP LPTS LC CPU: ARP, ICMP, BFD, NetFlow, OAM/CFM, L2 Protocols etc RP CPU: Routing, MPLS, IGMP, PIM, HSRP/VRRP, etc LPTS (local packet transport service): control plane policing Control packet Punt Switch CPU CPU NP Offloading: BFD LC BRKARC-2003 29 CPU
- 30. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS LC NP L3 Control Plane Architecture LC CPU RSP/RP CPU LDP RSVP-TE BGP ISIS OSPF EIGRP Static HW FIB Adjacency ARP/NDP LSD RIB AIB SW FIB AIB: Adjacency Information Base RIB: Routing Information Base FIB: Forwarding Information Base LSD: Label Switch Database RSP/RP LC BRKARC-2003 30 ISIS/OSPF BGP Static
- 31. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Distributed ARP Processing LC CPU PHY NP FIA LPTS Punt Switch ARP spio netio Tsec Driver SPP Line card FIA Switch Fabric RP Switch Fabric ARP RP CPU Incomplete ADJ Packets Incoming ARP Packets Outgoing ARP Packets BRKARC-2003 31
- 32. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS NP Switch Fabric FIA FIA FIA FIA Switch Fabric ASIC CPU Punt FPGA FIA CPU Switch Fabric RP LC1 NP NP NP NP NP NP NP NP FIA FIA FIA FIA Switch Fabric ASIC CPU LC2 1 2 2 NP learn MAC address in hardware (around 4M pps) NP flood MAC notification (data plane) message to all other NPs in the system to sync up the MAC address system-wide. MAC notification and MAC sync are all done in hardware 1 2 NP NP NP NP NP NP NP Hardware based MAC learning: ~4Mpps/NP MAC Learning and Sync Data packet BRKARC-2003 32
- 33. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Distributed Netflow Architecture NPU LC-CPU 1M Record Cache NPU NPU NPU RP Netflow show/clear Command Netflow Configure NPU: • Traffic Filtering • Traffic Sampling • Extract flow header information • LPTS Policer • Aggregate 200Kpps netflow policer rate • Evenly Divided among netflow-enabled NPs netIO EXPORT • Manage flow table • Export flow information LC BRKARC-2003 33
- 34. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Pop Quiz ???? In which hardware component does MAC learning happen? • RP/RSP the Control Module • LC-CPU • LC-NPU • LC-PHY BRKARC-2003 34
- 35. ASR 9000 Data Packet Processing
- 36. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS CPAK 0 NP FIA FIA FIA FIA CPU CPAK 1 PHY CPAK 2 NP CPAK 3 PHY CPAK 4 NP CPAK 5 PHY CPAK 6 NP CPAK 7 PHY … Up to 14x120 G Switch Fabric (SM15) CPAK 0 NP FIA FIA FIA FIA CPU CPAK 1 PHY CPAK 2 NP CPAK 3 PHY CPAK 4 NP CPAK 5 PHY CPAK 6 NP CPAK 7 PHY … Up to 14x120G Switch Fabric (SM15) Switch Fabric Switch Fabric Distributed Two-Stage Packet Processing Uniform packet flow for simplicity and predictable performance 1 2 1 2 • Ingress lookup yields packet egress port and applies ingress features • Egress lookup performs packet-rewrite and applies egress features BRKARC-2003 36
- 37. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS ASR9000 Life of a Packet • MACSEC Decryption • G.709/OTN/WAN- PHY/LAN-PHY • Line Clocking PHY FIA CPU CPU Switch Fabric CPAK 0 NP FIA CPAK 1 PHY TM 1 2 3 4 5 NP TM 6 7 8 1 • Ingress L2/L3 FIB lookup, ACL/QoS lookup • Ingress PBR/ABR, ACL, uRPF • Ingress QoS: classification, marking, policing • Packet Punting • Ingress ECMP/LAG hashing 2 Ingress side of LC Egress side of LC • Ingress Queuing Processing • Bypassed in case no ingress queuing support 3 • Buffering packet from NP • Requesting fabric credit • Manage superframe and load-balancing packet across fabric • Manage system VoQ 4 • Re-assembling packets from superframe • Send packet to corresponding NP • Release buffer and fabric credit 5 • Egress L2/L3 FIB lookup, ACL/QoS lookup • Egress PBR/ABR, ACL, uRPF • Egress QoS: classification, marking, policing, shaping • Incomplete Adj Packet Punting • Egress ECMP/LAG hashing 6 • MACSEC Emcryption • G.709/OTN/WAN- PHY/LAN-PHY • Line Clocking 8 • Egress Queuing Processing 7 BRKARC-2003 37
- 38. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Switch Fabric Arbitration Fabric ASIC and VOQ SFC or RSP1 Arbitration Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Arbitration Fabric ASIC and VOQ 1: Fabric Request 3: Fabric Grant 2: Arbitration 4: load-balanced transmission across fabric links 5: credit return SFC or RSP0 BRKARC-2003 38
- 39. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Fabric Load Balancing – Unicast Fabric ASIC and VOQ SFC or RSP1 Arbitration Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Arbitration Fabric ASIC and VOQ SFC or RSP0 4 3 2 1 • Unicast traffic sent across first available fabric link to destination (maximizes efficiency) • Each frame (or super frame) contains sequencing information • All destination fabric ASIC have re-sequencing logic • Additional re-sequencing latency is measured in nanoseconds BRKARC-2003 39
- 40. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS RSP5 RSP5 Tomahawk Line Card FIA Fabric SM15 FIA FIA FIA NP SFP+ SFP+ SFP+ SFP+ SFP+ NP NP NP SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ Distributed Multi-Stage Multicast Replication CPAK 0 NP FIA FIA FIA FIA CPAK 1 PHY CPAK 2 NP CPAK 3 PHY CPAK 4 NP CPAK 5 PHY CPAK 6 NP CPAK 7 PHY Switch Fabric (SM15) Tomahawk Line Card FIA Fabric SM15 FIA FIA FIA NP SFP+ SFP+ SFP+ SFP+ SFP+ NP NP NP SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ 1 1. 512k MGID/FGID lookup Flow based (RBH key calc) multi link load balance 2 3 2 & 3. Flow based (using RBH) multi link load balance 4 4. 22-bit FGID for slot bitmask based unique fabric LC replications and FPOE lookup 5 5 5. MGID lookup into FPOE table (256k entries) for unique FIA replications 6 6 6 6 6. MGID in FIA DI table lookup for unique NPU replication Flow based load balance 7 7 7. MGID lookup for NPU to egress OIF interface replication 7 7 BRKARC-2003 40
- 41. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Fabric Load Balancing – Multicast Fabric ASIC and VOQ SFC or RSP1 Arbitration Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Arbitration Fabric ASIC and VOQ SFC or RSP0 A1 A2 B1 A3 B2 C1 Flows exit in-order 41 BRKARC-2003 • Multicast traffic hashed based on (S,G) info to maintain flow integrity • Very large set of multicast destinations preclude re-sequencing • Multicast traffic is non arbitrated – sent across a different fabric plane BRKARC-2003 41
- 42. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS ASR9000 Traffic Load-Balancing 42 BRKARC-2003 • Hashing based Load-Balancing A1 A2 B1 A3 B2 C1 C2 Extract Header Info Hashing based on header Info Hashing Value Shift A1 A2 A3 B1 B2 C1 C2 A1 A2 A3 B1 B2 C1 C2 Bundle ECMP A1 A2 A3 B1 B2 C1 C2 UCMP Weight=20 Weight=40 • Same Hashing applied to Bundle, ECMP, UCMP • UCMP need factor in path weight • Load-Balancing is Flow-based Hashing Value Modulo
- 43. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Default Fields Used for Hashing 43 BRKARC-2003 MAC DST DOT1Q EtherType PayLoad FCS MAC SRC Layer 2 Ethernet: Router ID + IP SRC Port SRC Port DST IP DST Hash Function Hash Value Native IPv4/IPv6: IP SRC Port SRC Port DST PayLoad IP DST L4 L3 L2 L1 Labeled IPv4/IPv6:*1 PayLoad IP SRC Port SRC Port DST PayLoad IP DST L8 L7 L6 L5 Labeled IPv4/IPv6:*2 L4 L3 L2 L1 *1:XR 6.2.2 & Before. *2:XR 6.2.3 & After.
- 44. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Default Fields Used for Hashing-Labeled non-IP 44 BRKARC-2003 MAC DST DOT1Q EtherType PayLoad FCS MAC SRC Labeled non-IP (n<=4): Router ID + Hash Function Hash Value *2:XR 6.2.3 & After. Ln …… L1 MAC DST DOT1Q EtherType PayLoad FCS MAC SRC Labeled non-IP (5<=n<=8):*2 Ln L1 L5 L2 … L3 L4 Innermost Label + Input Interface Handle
- 45. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Default Fields Used for Hashing-Deep Label Stack 45 BRKARC-2003 Router ID + Hash Function Hash Value *1:XR 6.2.2 & Before. *2:XR 6.2.3 & After. Any PayLoad Labeled any-Payload (n>=5):*1 Ln L1 L5 … … Labeled any-Payload (n>=9):*2 Ln L1 L5 L2 … L3 L4 Fifth Label L6 L7 L8 Any PayLoad + Input Interface Handle
- 47. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS ASR9000 Priority-Based QoS Architecture • Ingress (sub-)interface QoS Queues • User Configurable Ingress QoS Policy • 4xVOQ per VQI • Up to 8K VOQs per TSK FIA (vs 4k per SKT FIA) 4x Egress Destination Qs per VQI, aggregated at egress port rate • Dedicated Traffic Manager(TM) for Traffic Queuing • User Configurable QoS Policy on Ingress/Egress NP • End-to-End priority propagation Guarantee bandwidth, low latency for high priority traffic • Unicast VOQ and back pressure User-configuration with Ingress MQC Egress (sub-)interface QoS Queues User-configuration with Egress MQC Implicit Configuration Not User-controllable Ingress side of LC Egress side of LC NP0 PHY NP2 PHY NP3 PHY NP1 PHY FIA CPU CPU Switch Fabric 3 4 1 2 3 4 CPAK 0 NP FIA CPAK 1 PHY 1 2 P1 P2 P3 BE P1 P2 P3 BE P1 P2 P3 BE TM TM P1 P2 P3 BE BRKARC-2003 47
- 48. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Default Implicit Trust Model internal cos = 1 L2 Bridging 802.1p = 1 * 802.1p = 1 IPP=5 DSCP=44 IPP=5 DSCP=44 IPP=5 DSCP=44 internal cos = 0 L2 Bridging Untagged IPP=5 DSCP=44 802.1p = 0 * IPP=5 DSCP=44 IPP=5 DSCP=44 Ingress line card Egress line card Carried in internal buffer header, by default, internal cos is used for impositioned fields only, For example, added vlan tag, impositioned MPLS label, It doesn’t include VLAN tag translation or MPLS label swap ASR 9000 would never modify packet DSCP/IP without a policy-map configured L2 IF: trust outer Cos L3 IF: trust DSCP L3 MPLS: trust outer EXP BRKARC-2003 48
- 49. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS •Priority Level 1 •Priority Level 2 •Shape •bandwidth or bandwidth remaining •(W)RED •Police •Set (marking) •Priority Level 1 •Priority Level 2 •Shape, •bandwidth or bandwidth remaining •(W)RED •Police •Set (marking) •Shape •Bandwidth remaining •1R2C policer L4 Child policy (child of Parent) with user- defined classes & class-default L3 Parent Policy on L2 (EFP) or L3 subint. class-default or physical port /w user-defined classes L2 Grand-Parent Policy L1 Port scheduler not configurable •N/A 3-Layer Hierarchical QoS (H-QoS) L0 Port Group scheduler not configurable •N/A policy-map child class Pr1 police rate 64 kbps priority level 1 class Pr2 police rate 10 mbps priority level 2 class Cl3 bandwidth 3 mbps class Cl4 set precedence routine bandwidth 1 mbps ! policy-map parent class parent1 shape average 100 mbps service-policy child class parent2 shape average 25 mbps service-policy child class class-default ! policy-map grand-parent class class-default shape average 500 mbps service-policy parent BRKARC-2003 49
- 50. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Default Interface Queues P1 P2 P3 L Level 4 Level 3 Level 2 Level 1 Queues Schedulers Schedulers Schedulers BRKARC-2003 50
- 51. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS MQC Hierarchy in Queuing ASIC policy-map child class c1 priority level 1 police rate 640 kbps class c2 bandwidth 20 mbps class class-default bandwidth 1 mbps ! policy-map parent class class-default shape average 35 mbps service-policy child ! interface GigabitEthernet0/0/0/0 service-policy output parent P1 P2 P3 L Port default queues c1 c2 cd-c MQC queues Inactive entity Active entity cd-p L1 L2 L3 L4 cd-c cd-p 51 BRKARC-2003 51
- 52. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS MQC Hierarchy in Queuing ASIC policy-map child class c1 priority level 1 police rate 640 kbps class c2 bandwidth 20 mbps class class-default bandwidth 1 mbps ! policy-map parent class class-default shape average 35 mbps service-policy child ! interface GigabitEthernet0/0/0/0.1 service-policy output parent ! interface GigabitEthernet0/0/0/0.2 service-policy output parent Port default queues c1 c2 cd-c G0/0/0/0.1 Inactive entity Active entity P1 P2 P3 L c1 c2 cd-c G0/0/0/0.2 cd-p L1 L2 L3 L4 cd-p cd-c cd-p BRKARC-2003 52
- 53. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Pop Quiz ???? For an incoming untagged layer 2 frame, what is the priority value used when the frame is processed inside ASR9000? • 2 • COS bit • 802.1p Value • 0 BRKARC-2003 53
- 55. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS 55 BRKARC-2003 55 TCAM Used for Traffic Classification I/F classification ACL classification Fwd lookup QOS classification IFIB lookup IFIB action QoS action ACL action L2 rewrite QoS action ACL action QOS classification L2 rewrite ACL classification Fwd lookup From wire To wire Ingress linecard egress linecard To fabric From fabric TCAM TCAM
- 56. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS 56 BRKARC-2003 4th Generation LC TCAM Bank Sets • TCAM organized into bank sets, with 4 arrays in each bank set. • Each array composed of 512x160b entries or 128x640b entries • Entries can be arranged as 20B/80B wide and can span multiple lines/blocks • LPTS on 4th Generation LC uses hash table instead of iTCAM which saves iTCAM usage Bank3 Bank2 Bank1 Bank0 512 lines 20B 20B 20B 20B
- 57. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS 57 BRKARC-2003 TCAM Partitions for Different Linecards TCAM Regions 3rd Generation LC 4th Generation LC L2 Partition Physical Ports, Bundles Reserved Partition for each type N/A, L2 classification does not require TCAM resource Encap Default Encap Untagged, Encap Any Single VLAN, PWHE Double VLAN, BVI ODS2 (160 bit entries) ODS2 iFIB Reserved Partition for iFIB (IPv4 LPTS) N/A, LPTS does not require TCAM resource Common Reserved Partition for all ODS2 features Reserved Partition for all ODS2 features ODS8(640 bit entries) ODS8 iFIB Reserved Partition for iFIB (IPv6 LPTS) N/A, LPTS does not require TCAM resource Common Reserved Partition for all ODS8 features Reserved Partition for all ODS8 features
- 58. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS 58 BRKARC-2003 4th Generation LC TCAM Feature Lookup Region TCAM Region Features sharing resources Search mode 160-ING L2-ACL 160 bits IPV4-ACL IPv4-QOS PBR-IPV4 PBR-MPLS PBR-L2 IPV4-LI 160-EGR L2-ACL 160 bits IPV4-ACL IPv4-QOS 640-ING IPV6-ACL 640 bits IPv6-QOS PBR-IPV6 IPV6-LI EDPL BGP Flowspec 640-EGR IPV6-ACL 640 bits IPv6-QOS
- 59. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS 59 BRKARC-2003 TCAM Partition Example –3rd Generation LC Note: “show controllers rm tcam summary all all np X location X/X/X” for 4th Generation LC
- 60. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS TCAM Partition Example – 4th Generation LC 60 BRKARC-2003 160-ING Region
- 61. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS TCAM Partition Example – 4th Generation LC 61 BRKARC-2003 160-ING Region
- 62. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS 62 BRKARC-2003 TCAM Usage Summary • IPv4/IPv6-ACL/QoS support on both ingress/egress • IPv4-ACL/QoS search mode use 160bits Partition • IPv6-ACL/QoS search mode use 640bits • PBR/Lawful Interception(LI) are both ingress features • BGP FlowSpec (both v4 and v6) uses 640 bit TCAM format, and only in the ingress direction
- 63. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS 63 BRKARC-2003 Pop Quiz ???? As we discussed, most of IPv4 related features use ODS2 partition and most of IPv6 related features use ODS8 partition. When BGP flowspec is configured for IPv4 traffic and rules has been pushed, in which TCAM region are the rules programed? • L2 Partition • ODS8 Common • ODS2 Common • ODS2 iFIB
- 64. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Conclusion • ASR9000 - Truly Carrier-Class Edge Router Provides: • Rich Features, Flexible Service Capability • Variety of Hardware to Meet Different Capacity Requirements • Fully Distributed Architecture for High Performance and System Scalability • Uniform, Open and Modularized Software Architecture BRKARC-2003 64
- 65. Complete your online session evaluation • Please complete your session survey after each session. Your feedback is very important. • Complete a minimum of 4 session surveys and the Overall Conference survey (starting on Thursday) to receive your Cisco Live water bottle. • All surveys can be taken in the Cisco Live Mobile App or by logging in to the Session Catalog on ciscolive.cisco.com/us. Cisco Live sessions will be available for viewing on demand after the event at ciscolive.cisco.com. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS BRKARC-2003 65
- 66. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public #CLUS Continue your education Related sessions Walk-in labs Demos in the Cisco campus Meet the engineer 1:1 meetings BRKARC-2003 66
- 67. Thank you #CLUS
- 68. #CLUS