Trill and Datacenter Alternatives
Download as PPTX, PDF14 likes3,665 views
The document discusses the significance of TRILL and various data center technologies in enhancing data center operations through improved virtualization, convergence, and cloud services. It outlines key imperatives such as increasing price/performance ratios, energy efficiency, and the need for standardized solutions in network management. Additionally, it presents alternatives to TRILL, such as Shortest Path Bridging (SPB), OpenFlow, and SDN, while addressing challenges and scalability issues within data center networks.
Trill and Datacenter Alternatives
- 1. TRILL & Datacenter technologies – their importance, and alternatives to datacenter network convergence Rajesh Kumar Sundararajan Assistant VP Product Management, Aricent May 10, Interop Las Vegas, 2012
- 2. About Me ARICENT Group • Global innovation, technology, and services company • Focused exclusively on communications • Co-creating most innovative communications products and applications with customers • Complete lifecycle solutions for networks, management, applications Me • Rajesh Kumar Sundararajan • Assistant Vice President – Product Line Management • Ethernet, IP and Datacenter offerings 2
- 3. Agenda Datacenter imperatives Solutions proposed to datacenter imperatives Categorization of the solutions Technological overview of each of the solutions TRILL and alternatives Summary comparison Conclusion 3
- 4. 3 Famous C’s of Datacenter Operations Operational trends in the Datacenter driven by Virtualization, Convergence, & Cloud Services • Increasing amount of physical space to accommodate new hardware • Additional CAPEX for hardware and software • Increased OPEX for staffing and power COST • Physical separation of users from applications • Latency-related concerns due to geographical distribution of applications • High performance demands of Cloud Services applications COMPLEXITY • Ever-increasing amounts of bandwidth demanded by consumers and enterprise applications • Increasing proliferation of video to deliver both consumer as well as business-related content CAPACITY 4
- 5. Virtualization, Convergence, & Cloud Services Improved efficiencies in the Datacenter from • Increased utilization of individual servers • Consolidation of servers and network ports • Simplified management and operations • Network virtualization (beyond storage and server virtualization) VIRTUALIZATION • Convergence of equipment and network architectures • Simplified design and increased flexibility of network architecture • LAN/SAN convergence enables the ubiquity and extends the reach of Ethernet in the Datacenter CONVERGENCE • Ability to push hardware (storage, server) and software (SaaS) to a 3rd party provider • Eliminates need to procure, install, update and upgrade hardware & software, resources can be obtained on as-needed basis • Drive performance / load across datacenters CLOUD SERVICES • Remote datacenter backup 5
- 6. Imperatives and Solutions Increase Price/ Performance Ratio • 10Gb Ethernet support required for all new Datacenter Ethernet equipment • Migration to 40GbE/100GbE needs to be planned • Next gen products need to balance high performance with low CAPEX & OPEX Improve Energy Efficiency • Networking equipment needs to lower energy consumption costs • New server chipsets, architectures and software needed to improve overall energy efficiency Support Multiple Migration Options • Multiple migration options available for evolution to a converged network • Companies may start by migrating to converged network adaptors to top-of- rack switches to a converged core or vice versa • Equipment vendors need to support different migration options in products Evolve Standards • DCBX, ETS, PFC, QCN • FCoE, FIP, FIP Snooping • Openflow, SDN, TRILL, SPB, MC-LAG • VxLAN, NVGRE 6
- 7. Supporting Technologies Fabric scalability & performance Network Virtualization Interconnecting datacenters Uncomplicate Switching SPB VEPA TRILL OpenFlow MCLAG SDN Endpoint Virtualization QCN DCBX NVGRE VxLAN PFC ETS NPIV NPV Lossless Ethernet Convergence FCoE FIP FIP Snooping 7
- 8. Lossless Ethernet Why existing QoS and Flow Control are not enough QoS techniques are primarily – Flow control – pause frames between switches to control sender’s rate – 802.1p and DSCP based queuing – flat or hierarchical queuing – Congestion avoidance methods – WRED, etc. Issues – Flow control – no distinction between different applications or frame priorities – 802.1p and DSCP based QoS methods • Need to differentiate classes of applications (LAN, SAN, IPC, Management) • Need to allocate deterministic bandwidth to classes of applications – Congestion avoidance methods • Rely on dropping frames at the switches • Source may continue to transmit at same rate • Fine for IP based applications which assume channel loss • Don’t work for storage applications which are loss intolerant 8
- 9. Lossless Ethernet Flow control – All traffic on port is Priority Flow Control (PFC) affected DCBX Internal back pressure Q1 – CoS 1 Q2 – CoS 2 PAUSE frames Q3 – CoS 3 Q4 – CoS 4 Priority Flow Control – Traffic for specific CoS on port is affected Internal back pressure TLVs in LLDP messages Q1 – CoS 1 Q2 – CoS 2 Advertise own capabilities Q3 – CoS 3 PAUSE (CoS = 3) frames Q4 – CoS 4 Priority groups = x; PFC = Yes, which priorities; Quantized Congestion Notification (QCN) Congestion notification = Yes; Reaction point Congestion point (switch facing (source/ ETH congestion on egress port) TLVs in LLDP messages end-point) Accept or No Advertise own capabilities Throttle Tx rate ETH Congestion Switches advertise and notification know capabilities to use message on links 9
- 10. NPIV & NPV •NPIV = N_PortID_Virtualization FiberChannel FiberChannel Storage node switch switch - host based technology •NPV = N_Port_Virtualization E_Port F_Port E_Port N_Port – switch based technology (N_PortID) •Technology for the storage side FiberChannel •Relevant to Datacenter Ethernet because of switch Storage node the virtualization capabilities and bearing on NPIV FCoE Physical port F_Port •NPIV – virtualization of storage device port Logical port N_Port1 (N_PortID1) to support VMs, multiple zones on same link N_Port2 (N_PortID2) •Requires support on storage endpoint and N_Port2 (N_PortID3) connected switch as well NPV •NPV – endpoint is unaware; switch proxies for multiple endpoints (N_Ports) using a F_Port NP_Port single NP_Port •Reduces number of switches required Multiple N_PortIDs 10
- 11. FCoE (FiberChannel Over Ethernet) FCoE switch •Means to carry FiberChannel frames within FC endpoint FC link Ethernet frames Ethernet •Interconnects FiberChannel endpoints or Ethernet switch Ethernet switches across an Ethernet (DataCenter interconnect Bridged Ethernet) network Ethernet switch Ethernet switch •FC frames encapsulated in Ethernet Ethernet switch header •New EtherType to transport FC frames Ethernet switch •FCoE can be enabled on – FC endpoint devices / FC switches / Ethernet switches Ethernet FCoE switch FCoE switch FC Endpoint Ethernet switch Ethernet switch FCoE switch FC switch FC link Ethernet Ethernet Ethernet FC link FC endpoints FC frame FCoE frame FCoE frame Eth header FC frame FC frame FCoE frame 11
- 12. FIP (FCoE Initialization Protocol) •Protocol between FC devices built on assumption of direct connection •Traversing an Ethernet cloud requires additional procedures •Addressed by FIP Device discovery Initializing communication Maintaining communication •FCoE Control frames – FIP – uses different EtherType than FCoE Data frames – FC frames encapsulated with FCoE EtherType FCoE device discovery Initializing communication Maintaining communication FCoE device FCoE device Ethernet 12
- 13. FIP Snooping •Protocol between FC devices built on assumption of direct connection •FC switch enforces many configurations, performs validations and access control on attached endpoints •Security concerns when this is exposed over non-secure Ethernet •Addressed by FIPSnooping Done on transit switches carrying FCoE, on VLANs dedicated to FCoE Switches install firewall filters to protect FCoE ports Filters are based on inspection of the FLOGI procedure Example (a) deny Enodes using FC-Switch MAC address as source MAC (b) ensure address assigned to Enode is used only for FCoE traffic FIP FIP Snooping FIP Snooping Ethernet switch Ethernet switch FCoE device FCoE device Ethernet 13
- 14. Fabric Scalability and Performance Why Spanning Tree (RSTP/MSTP/PVRST) is not enough Necessary fundamentals for FCoE to work Multipath through the network Lossless fabric Rapid convergence in fabric Spanning tree (with variants like RSTP, MSTP, PVRST) is the universal way to provide redundancy and stability (loop avoidance) in Ethernet networks Spanning tree is a distance vector based protocol • Routing equivalent of spanning tree (distance vector based) = RIP • Limits the size of the network that it can handle • Much smaller network size than link state based protocols (OSPF / ISIS) Datacenter networks have got much bigger (and getting bigger still !!) Spanning tree blocks links / paths to create redundancy inefficient capacity utilization Does not support multipath which is important for SAN/LAN convergence The TRILL solution • Apply link state routing to bridging / Layer 2 Ethernet • Use technique like ECMP for alternate paths without blocking any links or paths 14
- 15. Fabric Scalability and Performance TRILL – Transparent Interconnection of Lot of Links Focus on problem of dense collection of RBridge RBridge interconnected clients and switches Attempt to: • Eliminate limitations of spanning tree TRILL control protocol centric solutions (IS-IS extension) • Bring the benefits of routing technologies to the L2 network (without the need for RBridge RBridge IP/subnets, etc.) Objectives: Learn • Zero configuration and zero assumptions MAC • Forwarding loop mitigation MAC TRILL control frame frame (advertise learnt MAC) • No changes to spanning tree protocols Normal bridge Key components: RBridge RBridge RBridge or destination • R-bridges (Routing bridges) • Extensions to IS-IS • Apply link state routing to VLAN aware bridging problem MAC TRILL MAC TRILL MAC MAC frame header frame header frame frame 15
- 16. Fabric Scalability and Performance TRILL – Handling Multicast and Broadcast • Create distribution tree with selected root Root for • Distribute from root to rest of tree Distribution Tree 1 • Multiple distribution trees for Root for – Multipath and load distribution Distribution Tree 2 – Alternate paths and resilience • All Rbridges pre-calculate and maintain the distribution trees • Algorithm specified for ensuring identical calculations at all RBRidges • By default - distribution tree is shared across all VLANs and multicast groups Distribution Tree 1 • How an ingress node selects a specific tree (from multiple Distribution Tree 2 existing trees) is not specified • Ingress Rbridge receiving multicast encapsulates in TRILL header and sends to root of tree and to downstream branches • Frame with TRILL header is distributed down branches of tree • Rbridges at edges remove TRILL header and send to receivers • Rbridges listen to IGMP messages • Rbridges prune trees based in presence of multicast receivers • Information from IGMP messages propagated through the TRILL core to prune the distribution trees 16
- 17. Fabric Scalability and Performance TRILL – Issues and Problems •Does not (yet) address different types of virtual networks (VxLAN, NVGRE…) •Provides for L2 multipathing (traffic within a VLAN) but L3 (routed traffic across VLANs) is unipath only •Initial scope of TRILL defined to address spanning tree limitations •IP maybe an afterthought; only 1 default router with VRRP = unipath for L3 •Result of above – forces datacenter operators to provision larger VLANs (more members per VLAN) – so, restricts segmentation using VLANs •Requires hardware replacement in switching infrastructure •Existing security processes have to be enhanced – Existing security processes rely on packet scanning and analysis – Encapsulation changes packet headers, existing tools must be modified / enhanced •Does not inherently address fault isolation •Does not inherently address QoS mapping between edge & core (example – congestion management requires congestion in network to be signaled to source ) •Does not clearly address source specific multicast; so multicast based on groups only 17
- 18. Alternatives? Network centric approaches SPB EVB – VEPA / VN-Tag MC-LAG Openflow / SDN •Endpoint /server centric approaches VxLAN NVGRE 18
- 19. Fabric Scalability and Performance SPB – Shortest Path Bridging • Key Components SP-Bridge SP-Bridge – Extend IS-IS to compute paths between shortest path bridges – Encapsulate MAC frames in an additional header for transport Control protocol (IS-IS extension) between shortest path bridges • Variations in Encapsulation – SPB – VID Normal bridge or – SPB – MAC (reuse 802.1ah SP-Bridge SP-Bridge SP-Bridge destination encapsulation) • Allows reuse of reliable Ethernet Learn Learn OAM technology (802.1ag, MAC MAC Y.1731) MAC SPB MAC SPB MAC MAC frame header frame header frame frame • Source MAC learning from SPB encapsulated frames at the edge SP-bridges 19
- 20. EVB (Edge Virtual Bridging) Addresses interaction between virtual switching environments in a hypervisor and VM-11 VM-12 ……… VM-1n 1st layer of physical switching infrastructure Virtualizer 2 different methods – VEPA (Virtual Ethernet Port Aggregator) & VN-Tag VEB / vSwitch Without VEPA – in virtualized environment, Ethernet switch traffic between VMs is switched within the virtualizer VM-11 VM-12 ……… VM-1n Key issues – monitoring of traffic, security policies, etc, between VMs is broken Virtualizer With VEPA – all traffic is pushed out to the Negotiation switch and then to the appropriate VM VEPA Key issue – additional external link bandwidth Ethernet switch requirement, additional latency Switch must be prepared to do “hairpin turn” Accomplished by software negotiation between switch and virtualizer 20
- 21. MC-LAG (Multi Chassis LAG) Relies on fact that datacenter network is large but with predictable topology CORE Downstream node has multiple links to different upstream nodes AGGREGATION & ACCESS Links are link-aggregated (trunked) into single logical interface Can be used in redundant or load-shared Typical datacenter network mode Load-shared mode offers multipath Resilience and multipath inherent Coordination protocol across switches No hardware changes required LAG Accomplished with software upgrade Switches must have protocol extension to LAG LAG LAG coordinate LAG termination across switches Does nothing about address reuse problem from endpoint virtualization 21
- 22. OpenFlow and Software Defined Networking (SDN) Paradigm shift to networking SDN Controller [ Service Creation, Flow management, first packet processing, route creation) Flowvisor ( Responsible for Network Partitioning based on Rules ( e.g. Bridge IDs, Flow ids, User credentials) Secure connection Open Flow Open Flow Open Flow Open Flow enabled enabled enabled enabled Switches Switches Switches Switches Open Flow Open Flow Open Flow Source -ONF enabled enabled enabled Switches Switches Switches SDN Controller – Focus on Service OpenFlow – Enabling Network Virtualization • Simplify the network (make it dumb? • Move the intelligence outside 22
- 23. Fabric Scalability and Performance OpenFlow and Software Defined Networking (SDN) paradigm SDN Controller – Focus on OpenFlow – Enabling Network Service Virtualization • Open platform for managing the traffic on • Light weight software (strong resemblance to “open flow” complying switches client software) • Functions – network discovery, network • Standard interface for access and service creation, provisioning, QoS, “flow” provisioning management, first packet handling • Secure access to controller • Interoperability with existing networking • Push/pull support for statistics infrastructure – hybrid networks • Unknown flow packet trap and disposition • Overlay networks, application aware through controller routing, performance routing, extensions to existing network behavior • Comply to OpenFlow specifications (current version 1.2) • Accessed and managed by multiple controllers 23
- 24. Fabric Scalability and Performance OpenFlow/SDN based switches • Open flow enabled switches – 2 Rule Action Stats types – Hybrid (OpenFlow HAL and current Packet + byte counters network control plane) 1. Forward packet to port(s) – Pure OpenFlow Switches 2. Encapsulate and forward to controller 3. Drop packet • Pure OpenFlow 4. Send to normal processing pipeline – Simpler, low in software content, lower Switch MAC MAC Eth VLAN IP IP IP TCP TCP cost Port src dst type ID Src Dst Prot sport dport + mask • Primarily contains – SSL (for secure management) Management (CLI, SNMP, WEB) Secure Connection (SSL) – Encap/decap – Hardware programming layer/driver Routing Block ( Protocols –RIP, DCBX –for Data centers OSFP, ISIS, BGP, Encap / Event – Event handler RTM, Multicast) Decap Handler PFC , ETS LLDP • Open Flow switches receive IP forwarding Chassis Infrastructure software instructions from service controllers Management and Congestion Notification Master Policy Engine System Monitoring • Architectural aspects Layer -2 Block – QOS (Hierarchical, multiple Vlan, STP, LACP, Scheduling Scheme) & ACL HAL Layer for open flow – Resource partitioning IGMP Management – Packet flow aspects 24
- 25. Openflow – key issues • Enormous amount of provisioning for rules in each switch • In today’s switches – must rely on setting up ACLs • Switches typically have low limits on number of ACLs that can be set up • Will need hardware upgrade to use switches with large amounts of ACLs • Ensuring consistency of ACLs across all switches in the network? – troubleshooting challenges 25
- 26. NVGRE and VXLAN – Background Challenges with Virtualization: • More VMs = more MAC addresses and more IP addresses • Multi-user datacenter + VMs = need to reuse MAC and IP addresses across users • Moving applications to cloud = avoid having to renumber all client applications = need to reuse MAC addresses, IP addresses and VLAN-IDs across users • More MAC addresses and IP addresses = larger table sizes in switches = larger network, more links and paths Necessity = Create a virtual network for each user Possible Solutions: • VLANs per user – limitations of VLAN-Id range • Provider bridging (Q-in-Q) – Limitations in number of users (limited by VLAN-ID range) – Proliferation of VM MAC addresses in switches in the network (requiring larger table sizes in switches) – Switches must support use of same MAC address in multiple VLANs (independent VLAN learning) • VXLAN, NVGRE – new methods 26
- 27. VxLAN (Virtual eXtensible LAN) – How it Works VM-11 VM-12 ……… VM-1n VM-21 VM-22 ……… VM-2n ip-11 ip-12 mac-11 mac-12 ip-1n mac-1n ip-21 ip-22 mac-21 mac-22 ip-2n mac-2n Without HYPERVISOR HYPERVISOR VxLAN SERVER – A (ip-A, mac-A) SERVER – B (ip-B, mac-B) Ethernet frame from VM-11 to VM-12 D-MAC = mac-21 S-MAC = mac-11 D-IP = ip-21 S-IP = ip-11 payload Ethernet header IP header VM-11 VM-12 ……… VM-1n VM-21 VM-22 ……… VM-2n To ip-21 ip-11 ip-12 ip-1n ip-21 ip-22 ip-2n Using To mac-21 mac-11 mac-12 mac-1n mac-21 mac-22 mac-2n HYPERVISOR HYPERVISOR VxLAN To ip-B SERVER – A (ip-A, mac-A) SERVER – B (ip-B, mac-B) (tunneling in To mac-B UDP/IP) Ethernet frame from VM-11 to VM-12 D-MAC = mac-B S-MAC = mac-A D-IP = ip-B S-IP = ip-A VNI D-MAC = mac-21 S-MAC = mac-11 D-IP = ip-21 S-IP = ip-11 payload Outer Ethernet header Outer IP header Inner Ethernet header Inner IP header 27 27
- 28. VXLAN - Internals VN-X (User X) VM-11 VM-12 ……… VM-1n VM-21 VM-22 ……… VM-2n VN-Y (User Y) ip-11 ip-12 ip-1n ip-21 ip-22 ip-2n mac-11 mac-12 mac-1n mac-21 mac-22 mac-2n HYPERVISOR HYPERVISOR SERVER – A (ip-A, mac-A) SERVER – B (ip-B, mac-B) Self table VM-11 VNI-X VM-12 VNI-Y VM-13 VNI-Z Remote table ip-21 VNI-X ip-B ip-22 VNI-X ip-B ip-2n VNI-Y ip-B Remote table – learnt and aged continuously based on actual traffic 28
- 29. NVGRE • NVGRE = Network Virtualization using Generic Routing Encapsulation • Transport Ethernet frames from VMs by tunneling in GRE (Generic Routing Encapsulation) • Tunneling involves GRE header + outer IP header + outer Ethernet header • Relies on existing standardized GRE protocol – avoids new protocol, new Assigned Number, etc • Use of GRE (as opposed to UDP/IP) loss of multipath capability VXLAN NVGRE VNI – VXLAN Network Identifier (or VXLAN TNI – Tenant Network Identifier Segment ID) VxLAN header + UDP header + IP header + GRE header + IP header + Ethernet header Ethernet header = 8+8+40+16 = 72 bytes = 8+40+16 = 64 bytes addition per Ethernet addition per Ethernet frame frame VTEP - VXLAN Tunnel End Point - originates NVGRE endpoint or terminates VXLAN tunnels VXLAN Gateway - forwards traffic between NVGRE gateway VXLAN and non-VXLAN environments New protocol Extends existing protocol for new usage 5/29/201 29 Multipath using different UDP ports No multipath since GRE header is same 2 29
- 30. Problems with NVGRE and VXLAN Configuration and Management: Security: • Controlled multicast (with the use of say, IGMPv3) • Existing security processes are broken – Existing security processes rely on packet within tenant network now gets broadcast to all scanning and analysis endpoints in the tenant’s virtual network – since – Encapsulating MAC in IP changes packet broadcast and multicast will get mapped to one headers, existing tools must be modified / enhanced multicast address for the entire VXLAN/VNI • Starts to put more emphasis on firewalls • Requires configuration of the virtual network mapping and IPS in virtualizers – redoing Linux consistently on all the virtual machines – network stack !! management nightmare without tools to debug and isolate misconfiguration • These maybe just the tip of the iceberg – will we need virtualized DHCP, virtualized DNS, etc. Partial Support: New Ecosystem: • Does not address QoS • Existing network analysis tools won’t – Encapsulation / tunneling techniques like Provider Bridging or work – partner ecosystem for technology PBB clearly addressed QoS by mapping the internal has to be developed “marking” to external “marking” • Existing ACLs installed in network • What is tunneled may be already tunneled – questions infrastructure are broken of backward compatibility with existing apps • Needs additional gateway to communicate outside the virtual network 30
- 31. TRILL vs. VXLAN (or) TRILL & VXLAN? TRILL VxLAN Addresses network; tries to optimize Addresses end points (like servers) network Technology to be implemented in network Technology to be implemented in virtualizers infrastructure (switches/routers) Needs hardware replacement of switch Needs software upgrade in virtualizers infrastructure (assuming virtualizer supplier supports this) Restricted to handling VLANs Agnostic about switches/routers between end- No optimizations for VXLAN/NVGRE points (leaves them as good/bad as they are) No changes required in end-points More computing power requirement from virtualizers (additional packet header handling, additional table maintenance, associated timers) Need is for large datacenters (lots of links Need is primarily for multi tenant datacenters and switches) Need for both – depending on what requirement a datacenter addresses 31
- 32. TRILL & the other networking alternatives TRILL SPB VEPA VN-Tag MC-LAG SDN / Openflow Multipath Multipath No / NA No / NA Multipath Multipath Requires Requires No h/w H/w change No h/w No h/w change (for h/w change h/w change change at analyzer change small number of point flows) Requires h/w change for large numbers More More Negligible More complex Negligible Simpler switch complex complex additional endpoint additional Complex controller switch switch complexity complexity Does not Tools Existing Requires Existing Existing tools address available tools enhancement tools continue to work network from Mac- continue to to existing continue to trouble in-Mac work tools at work shooting technology analyzer point Lesser provisioning Least Lesser Least High amount of provisioning provisioning provisioning provisioning 32
- 33. Aricent’s Datacenter Ethernet Software Frameworks Aricent IPRs Intelligent Switching Solution (ISS) with Datacenter Switching Extensions Services 33
- 34. Feel free to meet me at booth #2626 or get in touch with me for any questions at: Rajesh Kumar Sundararajan: rajeshkumar.sundararajan@aricent.com
- 35. Thank you.
Editor's Notes
- #15: The TRILL solution:Apply link state routing to bridging/Layer 2 EthernetUse techniques like ECMP for alternate paths without blocking any links or paths