Breaking the Memory Wall

Editor's Notes

• #5: Peak compute has increased by 60,000x over the past 20 years As opposed to 100x for DRAM or 30x for interconnect bandwidth.  Adopting emerging memory technologies that try to address this problem, takes a long time, and be able to scale, manufacture reliably With ours, we’re using standard DRAM…
• #6: eCommerce & Data Warehousing Boost OLTP throughput by 2.5x Cuts OLAP query times in half Caching LLM prompts is a great way to reduce expenses. It works by using vector search to identify similar prompts and then returning its response. If there are no similar responses, the request is passed onto the LLM provider to generate the completion. Vector DBs don’t need to be bound by inference server. It may be decoupled from the process life cycle, enabling multiple instances to leverage the same cache. After receiving a query, inference server can Computes a hash of the input query, including the tensor and some metadata. This becomes the inference key. Checks Redis for a previous run inference. Returns that inference, if it exists. Runs the tensor through the model if the inference does not exist. Stores the inference in the vector DB. Returns the inference.
• #8: 12 DIMMs per socket vs 28 DIMMs per socket 6 channels per socket vs 16 channels per socket RAM per Core is roughly the same. 48 DIMMs * 256GB = 12.2TB (~76GB per core), 56 * 256GB = 14.3 (~75GB per core) Ice Lake supports 40 cores per socket. 4 socket = 160 cores. Genoa supports 96 cores. 2 socket = 192 cores. PCIe 4.0 system: Lenovo ThinkSystem SR860 V2 PCIe 5.0 system: ThinkSystem SR675 V3
• #9: M-SIF: Shared Infrastructure: Improve interoperability related to share infrastructure enclosures with multiple serviceable modules. Modules containing elements are blind-matable and hot-pluggable into shared infrastructure enclosure. PMM: 167 mm long x 125 mm wide x 18.30 mm thick form factor 6.5” long x 4.9” wide x .72” thick
• #10: Backplane Specification:  OIF CEI-28G-LR: Delivers low crosstalk noise and low insertion loss while minimizing channel performance variation for every differential pair. Enables a scalable migration path beyond 25Gb/s.
• #12: At Astera, we’ve launched a Cloud-Scale Interop Lab for our retimer and CXL Type 3 device. Not only are we testing with all major CPU and memory vendors, but OS vendors as well. We believe seamless interoperability is critical for ecosystem enablement, and there are a myriad of host systems & DDR5-4800 & -5600 modules that we support with our Leo Memory Connectivity Platform. Jonathan Zhang, Sr. Director of Software Engineering at Astera Labs, presented some of the challenges of DDR5 interop testing yesterday in the CMS track, but I’ll just mention here that it takes a tremendous engineering effort to continuously run regression tests on as many DIMMs as possible, but of course, we “don’t want to confuse the work behind the work with delivering actual customer value.” So, it’s important to continue collaborating across the ecosystem to optimize benchmarks with different HW & SW configurations to understand what works best per use case and customer scenario. RAS & Telemetry are also vital for the usability of the technology. These features are essential to support pre- and post- OS, which we’re working diligently on with customers and partners to support software development and integration to configure, control CXL memory, as well as log events and alert the host as needed. This is really a continuum to support not just orchestration for cloud-scale fleet management, but for managing memory tiers as well as hypervisors. Thanks to community efforts through the CXL Linux Sync, we’re seeing significant progress for core infrastructure services, such as media management, error handling, and CXL performance monitoring Through this foundational work, we’re seeing fruitful collaborations with all OS vendors to support applications on bare-metal, VMs, containers and back-end applications, such as databases, without major software changes or provisioning scripts.
• #17: eCommerce & Data Warehousing Boost OLTP throughput by 2.5x Cuts OLAP query times in half Caching LLM prompts is a great way to reduce expenses. It works by using vector search to identify similar prompts and then returning its response. If there are no similar responses, the request is passed onto the LLM provider to generate the completion.
• #18: eCommerce & Data Warehousing Boost OLTP throughput by 2.5x Cuts OLAP query times in half AI Inferencing Boosts ResNet-50 IPS by 1.5x Accelerate Vector Search by 2x
• #19: eCommerce & Data Warehousing Boost OLTP throughput by 2.5x Cuts OLAP query times in half AI Inferencing Boosts ResNet-50 IPS by 1.5x Accelerate Vector Search by 2x