AI Networking - Push to Standardization

Scale-Up vs. Scale-Out

Artificial Intelligence (AI) workloads, especially driven by the explosive growth of Large Language Models (LLMs), are dramatically reshaping networking requirements, primarily through two crucial dimensions: Scale-Up and Scale-Out. Effective AI networking, grounded in optimized protocols and scalable architectures, is critical to enabling high throughput, low latency, and efficient handling of vast data flows.

Scale-Up refers to enhancing performance and capacity within a single node or system. This involves ultra-low latency and high-bandwidth communication between CPUs, accelerators (GPUs, FPGAs, ASICs), and memory within a tightly integrated environment. High-speed interconnects like NVIDIA’s NVLink or AMD’s Infinity Fabric exemplify such scale-up solutions.

Scale-Out involves increasing capacity and performance across distributed systems or clusters, emphasizing seamless handling of parallelism, network congestion, and synchronization across multiple interconnected nodes. This typically involves inter-GPU communications and requires robust network infrastructures capable of managing data parallelism and collective communications effectively.

Challenges Due to Lack of a Standard Interface

The current AI networking ecosystem faces fragmentation due to multiple proprietary solutions, resulting in:

·       Interoperability Issues - Integration complexities across diverse systems.

·       Scalability Constraints - Limited flexibility in rapidly scaling infrastructure.

·       Vendor Lock-In: Dependence on specific vendors' technologies, stifling innovation.

·       Complex System Optimization: Difficulty in optimizing communication pathways efficiently, leading to potential performance degradation.

Addressing Technical Challenges: Latency, Performance, and Congestion Management

AI workloads have distinct networking requirements during their lifecycle:

• Training Phase - Demands sustained, high bandwidth with minimal packet loss tolerance, characterized by large, persistent "elephant flows" requiring uninterrupted communication channels.
• Inference Phase - Prioritizes ultra-low latency to ensure quick response and enhanced user experience for interactive applications.
• Congestion Management - Effective congestion control strategies are essential to maintain optimal network performance, minimize packet loss, and reduce latency variations, especially at scale.

Two emerging standards addressing these networking challenges are the Universal Accelerator Link (UAL) and the Ultra Ethernet Consortium (UEC).

Universal Accelerator Link (UAL)

UAL is an optimized interconnect protocol addressing scale-up challenges by enabling efficient, ultra-low latency data transfers between intra-node components. Key specifications include sub-1 microsecond (µs) round-trip latency for request-response transactions, less than 100 nanoseconds (ns) pin-to-pin latency, and bandwidth support up to 200 Giga transfers per second (GT/s) per lane.

Ultra Ethernet Consortium (UEC)

UEC targets scale-out solutions by enhancing Ethernet standards specifically for AI and HPC workloads. It emphasizes congestion control, advanced telemetry, precise synchronization, multi-path packet spraying, and tail latency reduction. Ultra Ethernet combines traditional Ethernet’s scalability with enhanced performance features, supporting large-scale deployments with up to 1,000,000 endpoints.

Core Technologies in AI Networking

Several networking technologies underpin these standards and enhance AI system performance:

• Remote Direct Memory Access (RDMA): Enables direct memory-to-memory data transfers, bypassing CPU involvement, thereby significantly minimizing latency and maximizing bandwidth.
• Collective Communication Libraries (CCLs): Facilitate data parallelism and synchronization across multiple GPUs or nodes using optimized methods like ring based All-Reduce algorithms to ensure efficient bandwidth utilization.
• In-Network Computing (INC): Supported by Ultra Ethernet, INC offloads specific computational tasks to network devices, significantly reducing data movement and latency.

Congestion Management: RDMA vs. UEC

• RDMA Congestion Management: Uses Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) mechanisms to ensure lossless data transfers, ideal for environments requiring ultra-low latency. 
• UEC Congestion Management: Incorporates sophisticated endpoint-driven congestion control strategies and adaptive multi-path packet spraying, dynamically mitigating congestion at scale and proactively maintaining network performance.

Standardization through UAL and UEC offers promising solutions to address current fragmentation and technical limitations, enhancing interoperability, reducing latency, and significantly boosting performance to create scalable, efficient AI infrastructures.