CXL at OCP
- 1. Siamak Tavallaei CXL Advisor to the Board, CXL Consortium Steering Committee, OCP Nov 14, 2023 TheStateofCXL-relatedActivitieswithinOCP SupercomputingConference(SC’23)
- 2. An Update: CXL™ 3.1 Specification Released! (Nov 14, 2023) Presented by Siamak Tavallaei CXL Advisor to the Board, CXL Consortium EMBARGOED UNTIL NOVEMBER 14 AT 8:00 am EST
- 3. 11 CXL 3.1 Featureenhancements ComputeExpressLink™ andCXL™aretrademarksof the ComputeExpressLinkConsortium.
- 4. • The CXL specification continues to evolve to meet new usage models • New features introduced in the CXL 3.1 specification: • CXL Fabric Improvements/Extensions • Scale-outof CXL fabrics using PBR (Port Based Routing) • Trusted-Execution-Environment Security Protocol (TSP) • Allows for Virtualization-basedTrustedExecutionEnvironments (TEEs) to host ConfidentialComputing Workloads • Memory Expander Improvements • Up to 34-bit of metadata and RAS capability enhancements CXL 3.1 Feature Enhancements 12 ComputeExpressLink™ andCXL™aretrademarksof the ComputeExpressLinkConsortium.
- 5. 13 CXL Fabric Improvements/ Extensions ComputeExpressLink™ andCXL™aretrademarksof the ComputeExpressLinkConsortium.
- 6. CXL 3.1: Fabric Enhancement Features Port-Based Routing (PBR) compared to Hierarchy-Based Routing (HBR) Support fabric topologiesother thantreetopologiesthatHBR switches offer • Address-based,non-prescriptiveroutingfor large memory fabrics • Supports tree,mesh, ring,star,butterfly,and multi-dimensionaltopologies ComputeExpressLink™ andCXL™aretrademarksof the ComputeExpressLinkConsortium. Host Host CXL CXLHBRSwitch CXL HBR Switch CXL HBR Switch CXL HBR Switch Type-3 CXLMem Type-3 CXLMem Type-3 CXLMem Type-3 CXLMem Type-3 CXLMem Type-3 CXLMem CXL PBR Switc h Host Host Host Host Host Host Host Host CXL PBR Switch CXL PBR Switch CXL PBR Switch CXL PBR Switch Type-1/2/3 CXLDevice Type-1/2/3 CXLDevice Type-1/2/3 CXLDevice Type-1/2/3 CXLDevice Type-1/2/3 CXLDevice Type-1/2/3 CXLDevice Type-1/2/3 CXLDevice Type-1/2/3 CXLDevice
- 7. ComputeExpressLink™ andCXL™aretrademarksof the ComputeExpressLinkConsortium.
- 8. 19 Trusted Execution Environment (TEE) & TEE Security Protocol (TSP) ComputeExpressLink™ andCXL™aretrademarksof the ComputeExpressLinkConsortium.
- 9. RECAP: CXL 2.0 Security Benefits CXL 2.0 provides Integrity and Data Encryption (IDE) of traffic across all entities (Root Complex, Switch, Device) at the Link Layer CXL 2.0 Switch CPU/SoC Root Complex CXL Device Home Agent MC Host Memory Coherent Bridge IO Bridge IOMMU CXL.memory CXL.io Area of Protection Coherent Cache (Optional) DTLB MC Device Memory CXL.cache ComputeExpressLink™ andCXL™aretrademarksof the ComputeExpressLinkConsortium.
- 10. CXL 3.1 Trusted Security Protocol (TSP) Allows for Virtualization-based, Trusted Execution Environments (TEEs) to host Confidential Computing Workloads (CC WL) Benefits: • Freedom tomigratesensitiveWLs to TSP-enabledClouds • Collaborationwithmultiplepartiesfor sharingdata • Conform to Compliance & Datasovereigntyprograms • StrengthenApplicationsecurity& SoftwareIP protection KeyCapabilities: • Separationbetween TVM*& CSP’s infrastructure(VMM) • Configurationof CXL device • Encryptionof sensitivedatain both Host/Devicememory • Cryptographicallyverify correctconfigurationof trusted computingenvironment *TVM = Trusted VM ComputeExpressLink™ andCXL™aretrademarksof the ComputeExpressLinkConsortium. CXL Device CXL Link MC TVM* VM VM VM VM CXL.io CXL.mem CXL IDE (Defined in CXL v2.0) TEE Capable Host MC Device Memory Device Memory Host Mem Host Mem CXL.io CXL.mem
- 11. Elements of TSP / TSP Overview 22 ComputeExpressLink™ andCXL™aretrademarksof the ComputeExpressLinkConsortium. TSP Components for Confidential Computing • TrustedExecutionState& Access Control • How access to memoryis controlled • Configuration • Ability to determinethe supported security featureson the device, enable required features,and lock the configuration • Attestation& Authentication • Trusting who you are talking to • Memory Encryption • Encrypting data-at-rest • Transport Security • Encrypting the link to protect data-in-flight and detect/preventphysical attacks Confidential Computing (initiator) CXL TrustedExecutionEnvironment SecurityProtocol (TSP) Attestation Authentication Trusted Execution State & Access Control Memory Encryption (Data-at-rest) Transport Security (Data-in-flight) Configuration CXL memoryexpander (target) HDM-H (CXL 3.1), HDM-DB (CXL 3.1 ECN)
- 12. 23 Memory Expander Improvements ComputeExpressLink™ andCXL™aretrademarksof the ComputeExpressLinkConsortium.
- 14. CXL Specification Feature Summary Features CXL 1.0 / 1.1 CXL 2.0 CXL 3.0 CXL 3.1 Releasedate 2019 2020 August 2022 November2023 Max link rate 32GT/s 32GT/s 64GT/s 64GT/s Flit 68 byte (up to 32 GT/s) Type 1, Type 2 and Type 3 Devices MemoryPooling w/ MLDs Global PersistentFlush CXL IDE Switching (Single-level) Switching (Multi-level) Multiple Type 1/Type 2 devicesper rootport Direct memoryaccess for peer-to-peer 256-byte Flit (up to 64 GT/s PAM4) 256-byte Flit (Enhanced coherency) 256-byte Flit (Memorysharing) 256-byte Flit (Fabric capabilities) Fabric Manager API definition for PBR Switch Host-to-Hostcommunication with Global Integrated Memory(GIM) concept Trusted-Execution-Environment(TEE) SecurityProtocol Memoryexpander enhancements (up to 34-bit ofmeta data, RAS capability enhancements) ComputeExpressLink™ andCXL™aretrademarksof the ComputeExpressLinkConsortium. Supported Not Supported
- 15. Siamak Tavallaei Chief Systems Architect CXL Advisor to the Board, CXL Consortium Excerpts from: The State of CXL-related Activities within OCP CXL™ Forum at OCP Summit
- 16. CXL as the standard protocol for data-movement through coherent memory and for the associated management and system composability OCP as a community to realize integrated systems around CXL Proposal for an OCP reference system for CXL- enabled compute disaggregation
- 17. CXL Technical Work Groups CXL Consortium https://www.computeexpresslink.org CXL Board of Directors (BoD) https://www.computeexpresslink.org/meettheboard Marketing Work Group (MWG) Technical Task Force (TTF) Active CXL WGs run under Technical Task Force (TTF) PWG (Protocol WG) SSWG (System and Software WG) PHY (Physical and Link Work Group) MSWG (Memory System WG) Compliance WG
- 18. OCP Server Project Active OCP Server Project subprojects and workstreams Drive contributions (Base Spec, Design Spec, Products, …) https://www.opencompute.org/wiki/Server CMS (Composable Memory System) https://www.opencompute.org/projects/composable-memory-system DC-MHS (Datacenter-ready Modular Hardware System) https://www.opencompute.org/projects/dc-mhs Extended Connectivity Workstream (for PCIe and CXL) https://www.opencompute.org/wiki/Server/PCIe_Extended_Connectivity_Requirements_Workstream ODSA (Open Domain-Specific Architecture) https://www.opencompute.org/wiki/Server/ODSA OAI (Open Accelerator Infrastructure) https://www.opencompute.org/wiki/Server/OAI HPC (High-performance Computing) https://www.opencompute.org/wiki/HPC OCP NIC https://www.opencompute.org/wiki/Server/NIC
- 19. CXL Resources 36 Webinars: • Webinar: A look into the CXLdevice ecosystemand the evolutionof CXL use cases • Webinar: Introducing CXL 3.0: Enabling composable systems with expanded fabric capabilities • Webinar: CXL 1.1 vs. CXL 2.0 – What’s the difference? • Webinar Archive: https://www.computeexpresslink.org/webinars Blogs: • Upcoming Webinar: A Look into the CXL™ DeviceEcosystemand the Evolution of CXL Use Cases • CXL 3.0 Webinar Q&A Recap • CXL™ Consortium MemberSpotlight: UnifabriX • Blog Archive: https://www.computeexpresslink.org/blog White Papers: • CXL 3.0 specification • An Overviewof Reliability, Availability, and Serviceability(RAS) in ComputeExpress Link™ 2.0 ComputeExpressLink™ andCXL™aretrademarksof the ComputeExpressLinkConsortium.
- 23. A converged memory environment is the main key for ease of software programing! and efficient data-movement!
- 24. 11/21/2023 ComputeExpressLink™ andCXL™aretrademarksof the ComputeExpressLinkConsortium. • Use cases keep driving the need for higher IO and memory bandwidth: e.g., high performance accelerators, system memory, SmartNIC etc. • CPU capability requiring more memory capacity and bandwidth per core • Efficient peer-to-peer resource-sharing & messaging across multiple domains • Need to overcome memory bottlenecks due to CPU pin and thermal constraints Industry trends
- 32. The CXL 3.0 Spec allows for a large interconnected Fabric
- 33. ComputeExpressLink™ andCXL™Consortiumaretrademarksof the ComputeExpressLink Consortium. 59 CXL3.0Fabrics Composable Systems with Spine/Leaf Architecture CXL 3.0Fabric Architecture • InterconnectedSpine Switch System • Leaf Switch NIC Enclosure • Leaf Switch CPU Enclosure • Leaf Switch AcceleratorEnclosure • Leaf Switch Memory Enclosure CXL Switch Accelerator Memory CPUs CPUs CXL Switch GFAM GFAM GFAM CXL Switch CXL Switch NIC NIC CXL Switch CXL Switch Spine Switches LeafSwitches EndDevices Fabric Manager Example traffic flow
- 34. ComputeExpressLink™ andCXL™Consortiumaretrademarksof the ComputeExpressLink Consortium. 60 CXL3.1Fabrics Enables more efficient “East/West” interconnect CXL 3.0/3.1 Composable Fabric growth for disaggregation/pooling/accelerator
- 35. Composable Architecture n e w p a ra d i g m fo r d i s a g g re ga te d c o m p u t i n g ! H P C , A I / M L , I n - m e m o r y D ata b a s e s
- 43. Challenges
- 45. Successful Disaggregation Approach First do no harm The OS running on a Server: The Platform and the CXL Fabric Manager provide the same experience as a static server system Remedy every new fault mode Ride on PCIe, UEFI, and traditional RAS and Security Reduce the problem to that which has been solved before! Put things where they belong • Partition the system efficiently: ease of use, serviceability, maintenance While pushing the envelop, if it hurts, don’t do it! • Retreat from the extremes and avoid too many variables for the first generation • Fail Fast, learn, and grow the solution through PoCs
- 46. Successful Disaggregation Approach First do no harm The OS running on a Server: The Platform and the CXL Fabric Manager provide the same experience as a static server system Remedy every new fault mode Ride on PCIe, UEFI, and traditional RAS and Security Reduce the problem to that which has been solved before! Put things where they belong • Partition the system efficiently: ease of use, serviceability, maintenance While pushing the envelop, if it hurts, don’t do it! • Retreat from the extremes and avoid too many variables for the first generation • Fail Fast, learn, and grow the solution through PoCs
- 47. Successful Disaggregation Approach First do no harm The OS running on a Server: The Platform and the CXL Fabric Manager provide the same experience as a static server system Remedy every new fault mode Ride on PCIe, UEFI, and traditional RAS and Security Reduce the problem to that which has been solved before! Put things where they belong • Partition the system efficiently: ease of use, serviceability, maintenance While pushing the envelop, if it hurts, don’t do it! • Retreat from the extremes and avoid too many variables for the first generation • Fail Fast, learn, and grow the solution through PoCs
- 48. Based on the Modular Building Block Architecture (MBA), Build a reference hardware system to allow the software architecture to be ready for the full set of features!
- 56. Enablers (Software and Firmware Ingredients) CXL Fabric Manager • Secure composability, allocation, on-lining/off-lining Pre-boot Environment • Discovery, enumeration, setup, … CXL Bus/Class Driver • Configuration, Resource Allocation CXL Memory Device Driver • Interactions with Bus/Class Driver, Fabric Manager, VMM, … • RAS, Security, Fault-isolation, On-lining, Off-lining, … • Error Isolation, Telemetry, Performance Monitoring OS-specific Software • VMM, Hypervisor • VM Allocation, Orchestration, Fault-isolation & Recovery
- 57. O C P i s t h e p l a c e w h e r e w e R e a l i z e Te c h n o l o g i e s i n t o I n t e g r a t e d S y s t e m s
- 58. Activities within OCP Server Project in support of CXL-enabled Systems CMS (Composable Memory System) Software Infrastructure for managing tiered, composable, disaggregated systems https://www.opencompute.org/projects/composable-memory-system DC-MHS (Datacenter-ready Modular Hardware System) M-SIF (modular shared infrastructure) Partition the system efficiently: ease of use, serviceability, maintenance https://www.opencompute.org/projects/dc-mhs https://www.opencompute.org/wiki/Server/DC-MHS Extended Connectivity Workstream (for PCIe and CXL) https://www.opencompute.org/wiki/Server/PCIe_Extended_Connectivity_Requirements_Workstream Interconnect for the Disaggregated Computing Local Disaggregation within a Chassis With the option to Extend connectivity to Expansion Chassis Considerations for Copper and Photonic Interconnect
- 61. B a s ed on O C P O p en Ac c el er a tor I nf r as tructure (OA I )
- 64. Summary: Successful Disaggregation Approach (at OCP) First do no harm (software compatibility, security, and management: CMS) The OS running on a Server: The Platform and the CXL Fabric Manager provide the same experience as a static server system Remedy every new fault mode Ride on PCIe, UEFI, and traditional RAS and Security Reduce the problem to that which has been solved before! Put things where they belong (modular hardware system: DC-MHS/M-SIF) Partition the system efficiently: ease of use, serviceability, maintenance While pushing the envelop, if it hurts, don’t do it! (robust Extendeded Connectivity) • Retreat from the extremes; avoid too many variables for the first generation • Fail Fast, learn, and grow the solution through PoCs
- 66. Siamak Tavallaei has recently served as the CXL Consortium President, Chief Systems Architect at Google Cloud, and the Incubation Committee (IC) Representative for the Server Project. He is currently the CXL Advisor to the Board at CXL Consortium and actively participates in OCP Steering Committee. His current focus is the system optimization for large-scale, mega-datacenters for general-purpose and tightly-connected, accelerated machines built on co-designed hardware, software, security, and management. He continues to drive the architecture and productization of CXL-enabled solutions for AI/ML, HPC, and large memory-footprint Databases. In 2016, he joined OCP as a co-lead of Server Project where he drove open-sourced modular design concepts for integrated hardware/software solutions (OAI, DC-SCM, CMS, DC-MHS, and DC-Stack). His experiences as Chief Systems Architect at Google, Principal Architect at Microsoft Azure, Distinguished Technologist at HP, and Principal Member of Technical Staff at Compaq along with his contributions to industry collaborations such as EISA, PCI, InfiniBand, and CXL give Siamak a broad understanding of requirements and solutions for the Enterprise, Hyperscale, and Edge datacenters and industry-wide initiatives. Bio