![Growing Memory Bandwidth and
Capacity Gap
• Increase in core counts requires continued increase in memory bandwidth &
capacity
• The gap between such requirements and platform provisioning capability is
growing
• CXL creates new opportunities beyond physical limitations, and efficient memory
disaggregation is possible
[Memory Bandwidth Requirement] [Memory Capacity Requirement]](https://image.slidesharecdn.com/ocp23gcxlforumskhynixjungminchoi-231025005113-e9b72a8f/85/SK-hynix-CXL-Disaggregated-Memory-Solution-4-320.jpg)
![Challenges in Today’s Datacenter
• Challenge 1 : Memory stranding & data spill
• The memory utilization of each node in a compute cluster varies time to time
• Unused memory in each node can never be utilized by other nodes, which causes memory
stranding and data spill
VM 1
VM 2
VM 3
Cor
e
Cor
e
Cor
e
Cor
e
Cor
e
Cor
e
Cor
e
Cor
e
Memory
Stranded
Cor
e
Cor
e
Cor
e
Cor
e
Cor
e
Cor
e
Cor
e
Cor
e
Memory
Spilled
VM 1
VM 2
VM 3
[Memory Stranding] [Data Spill]
Memory underutilization & Waste of memory
costs
Storage swap & Performance
degradation
Two sides of a
coin](https://image.slidesharecdn.com/ocp23gcxlforumskhynixjungminchoi-231025005113-e9b72a8f/85/SK-hynix-CXL-Disaggregated-Memory-Solution-5-320.jpg)
![Solution: CXL Disaggregated Memory
System
• CXL disaggregated memory system can support memory pooling & sharing
• Memory pooling : Mitigate memory stranding and data spill by sharing memory resources
between nodes
• Memory sharing : Remove data transfer overhead and data duplication by sharing data
between nodes
Node #1
[Memory Pooling]
Allocate CXL memory based on memory usage for
each node
Share data objects based on zero-copy between
nodes
Node #2 Node #3 Node #4
CXL Disaggregated Memory
[Memory Sharing]
Node #1 Node #2 Node #3 Node #4
CXL Disaggregated Memory
Shared
Object
#1 Shared
Object
#2
Shared
Object
#N](https://image.slidesharecdn.com/ocp23gcxlforumskhynixjungminchoi-231025005113-e9b72a8f/85/SK-hynix-CXL-Disaggregated-Memory-Solution-7-320.jpg)
![Niagara: CXL Disaggregated Memory
Prototype
• Niagara is a 4U FPGA based multi-port CXL disaggregated memory prototype
• Up to 4 CXL host servers can be connected, Support up to 4 channels of DDR4 DIMM (1TB)
• Support memory pooling, sharing and other HW-assisted features
[Rack-Scale System with Niagara]
[Niagara Block Diagram]
[Niagara Specification]
CXL
Interface
CXL 2.0, Gen4x8
Up to 4-port
Memory
4CH DDR4 DIMM
Up to 1 TB
Performan
ce
Latency : 600ns
Bandwidth : 11 GB/s
CXL EP CXL EP CXL EP CXL EP
Memor
y
FPGA
CXL RP
CXL
HW-
assisted
Features
Pooled
Memory
Manage
r
CXL RP CXL RP CXL RP
Memory
Memor
y
Memor
y
Node #1 Node #2 Node #3 Node #4
Niagara
Server #1
Server #2
Server #3
Server #4](https://image.slidesharecdn.com/ocp23gcxlforumskhynixjungminchoi-231025005113-e9b72a8f/85/SK-hynix-CXL-Disaggregated-Memory-Solution-8-320.jpg)
![Use Cases of CXL Disaggregated
Memory
• Evaluation results of memory pooling
• Reduce execution time by mitigating data spill and improve memory utilization of a
system
[Execution Time of CloudSuite
Benchmark]
Spill to Niagara outperforms NVMe by up to 2.5x
(even though Niagara is an FPGA-based
prototype)
Utilization improved by 35% by applying DCS to
Kubernetes
[Memory Utilization on Kubernetes]
0 50 100 150 200 250 300 350
20
10
0
Normalized Execution Time vs Baseline [%]
Workload
Spilled
to
Medium
[%]
No Spill (Baseline) Spill to Niagara Spill to SSD
2.5x improvement
[ Collaboration with
MemVerge ]
Baseline Kubernetes
DCS-enabled
Kubernetes
Worker
#1
Worker
#2
Worker
#1
Worker
#2
# CPU Cores 112 176 112 176
Local Memory
Capacity
62 GB 62 GB 62 GB 62 GB
CXL Memory Capacity 32 GB 32 GB 64 GB (Pooled)
Total Memory
Capacity
94 GB 94 GB
62-126
GB
62-126
GB
Avg. Memory
Utilization
55.7% 75.1%
35% improvement](https://image.slidesharecdn.com/ocp23gcxlforumskhynixjungminchoi-231025005113-e9b72a8f/85/SK-hynix-CXL-Disaggregated-Memory-Solution-10-320.jpg)
![Use Cases of CXL Disaggregated
Memory
• Evaluation results of memory sharing
• Reduce execution time by eliminating network based data transfer overhead
[Execution Time of *Ray Shuffle
Benchmark]
Ray with Niagara outperforms native Ray by up to
5.9x
[Execution Time of Spark Benchmark]
5.9x improvement
*Ray is an open source based distributed computing framework for AI/ML
[ Collaboration with
MemVerge ]
Merge Hash Join outperforms Sort Merge join by
up to 1.8x
0 20 40 60 80 100 120 140
Query 1
Query 2
Query 3
End-to-End Execution Time [sec]
Mege Hash Join
1.8x improvement
Merge Hash Join
0 20 40 60 80 100
25
50
75
100
Latency [sec]
Partitions
Ray with Shared Memory
End-to-End Execution Time
[sec]](https://image.slidesharecdn.com/ocp23gcxlforumskhynixjungminchoi-231025005113-e9b72a8f/85/SK-hynix-CXL-Disaggregated-Memory-Solution-12-320.jpg)
![Other HW-assisted Features of Niagara
• Niagara supports HW-assisted features to CXL disaggregated memory efficiency
• Block Data Management : Copy or move data within CXL pooled memory
• Snapshot : Save and restore data in CXL Pooled Memory to/from storage device
• Memory Failure Prediction : Predict memory Uncorrectable Errors(UEs) by analyzing memory
failure patterns
[Block Data Management] [Snapshot] [Memory Failure Prediction]
CXL RP
Pooled
Memory
Manage
r
CXL RP
Node #1 Node #N
CXL EP CXL EP
CXL Disaggregated Memory
Data
Copy/Mov
e
CXL RP CXL RP
Node #1 Node #N
CXL EP CXL EP
CXL Disaggregated Memory
Data
Save/Restore
Storage
CXL RP CXL RP
Node #1 Node #N
CXL EP CXL EP
CXL Disaggregated Memory
UE prediction
Notify
predicted
UE
Pooled
Memory
Manage
r
Pooled
Memory
Manage
r](https://image.slidesharecdn.com/ocp23gcxlforumskhynixjungminchoi-231025005113-e9b72a8f/85/SK-hynix-CXL-Disaggregated-Memory-Solution-13-320.jpg)
![• Niagara 2.0 will be available by the end of 2023
• Niagara 2.0 is a 2U CXL disaggregated memory prototype which can connect up to 8 CXL
host servers
• Support DCD (Dynamic Capacity Device) feature defined in CXL specification 3.1
• We look forward to an open collaboration with industry partners to enable HW/SW
ecosystem
Next Step
[Niagara 2.0 Prototype] [Server Cluster with Niagara 2.0]
Server Server
Pooled
Memory
Manager
Cloud
Orchestrato
r
DCD Engine
CXL Disaggregated
Memory
Niagara
2.0](https://image.slidesharecdn.com/ocp23gcxlforumskhynixjungminchoi-231025005113-e9b72a8f/85/SK-hynix-CXL-Disaggregated-Memory-Solution-14-320.jpg)
![Niagara Live Demo
• Demonstrate the allocation/deallocation of
memory space in CXL disaggregated memory
based on dynamic changes in memory
requirements of VM servers
• Check out SK hynix booth (#A8) for a live demo
[Niagara Live Demo System]
Niagara
Server #1
Server #2
Server #3
Server #4
[Demo GUI]](https://image.slidesharecdn.com/ocp23gcxlforumskhynixjungminchoi-231025005113-e9b72a8f/85/SK-hynix-CXL-Disaggregated-Memory-Solution-15-320.jpg)
SK hynix CXL Disaggregated Memory Solution
- 1. SK hynix CXL Disaggregated Memory Solution
- 2. SK hynix CXL Disaggregated Memory Solution Jungmin Choi, Memory System Architect, SK hynix
- 3. Agenda Motivation Growing Memory Bandwidth and Capacity Gap Challenges in Today’s Datacenter Solution Niagara: CXL Disaggregated Memory Prototype Use Cases of CXL Disaggregated Memory Other HW-assisted Features of Niagara Next Step
- 4. Growing Memory Bandwidth and Capacity Gap • Increase in core counts requires continued increase in memory bandwidth & capacity • The gap between such requirements and platform provisioning capability is growing • CXL creates new opportunities beyond physical limitations, and efficient memory disaggregation is possible [Memory Bandwidth Requirement] [Memory Capacity Requirement]
- 5. Challenges in Today’s Datacenter • Challenge 1 : Memory stranding & data spill • The memory utilization of each node in a compute cluster varies time to time • Unused memory in each node can never be utilized by other nodes, which causes memory stranding and data spill VM 1 VM 2 VM 3 Cor e Cor e Cor e Cor e Cor e Cor e Cor e Cor e Memory Stranded Cor e Cor e Cor e Cor e Cor e Cor e Cor e Cor e Memory Spilled VM 1 VM 2 VM 3 [Memory Stranding] [Data Spill] Memory underutilization & Waste of memory costs Storage swap & Performance degradation Two sides of a coin
- 6. Challenges in Today’s Datacenter • Challenge 2 : Data transfer overhead & data duplication • In a distributed computing system, there is a network-based data transfer overhead between remote nodes • Duplication of shared data between nodes increases local memory pressure Node App DRAM Node App DRAM Node App DRAM Node App DRAM Network Serialization & Deserialization Data Duplication
- 7. Solution: CXL Disaggregated Memory System • CXL disaggregated memory system can support memory pooling & sharing • Memory pooling : Mitigate memory stranding and data spill by sharing memory resources between nodes • Memory sharing : Remove data transfer overhead and data duplication by sharing data between nodes Node #1 [Memory Pooling] Allocate CXL memory based on memory usage for each node Share data objects based on zero-copy between nodes Node #2 Node #3 Node #4 CXL Disaggregated Memory [Memory Sharing] Node #1 Node #2 Node #3 Node #4 CXL Disaggregated Memory Shared Object #1 Shared Object #2 Shared Object #N
- 8. Niagara: CXL Disaggregated Memory Prototype • Niagara is a 4U FPGA based multi-port CXL disaggregated memory prototype • Up to 4 CXL host servers can be connected, Support up to 4 channels of DDR4 DIMM (1TB) • Support memory pooling, sharing and other HW-assisted features [Rack-Scale System with Niagara] [Niagara Block Diagram] [Niagara Specification] CXL Interface CXL 2.0, Gen4x8 Up to 4-port Memory 4CH DDR4 DIMM Up to 1 TB Performan ce Latency : 600ns Bandwidth : 11 GB/s CXL EP CXL EP CXL EP CXL EP Memor y FPGA CXL RP CXL HW- assisted Features Pooled Memory Manage r CXL RP CXL RP CXL RP Memory Memor y Memor y Node #1 Node #2 Node #3 Node #4 Niagara Server #1 Server #2 Server #3 Server #4
- 9. Use Cases of CXL Disaggregated Memory • Use case 1 : Memory pooling • Niagara supports Dynamic Capacity Service (DCS), a HW/SW integrated solution, for memory pooling • DCS can dynamically allocate/deallocate disaggregated memory resources for each node without RESET • Improve memory utilization and performance of a system equipped with CXL disaggregated memory Niagara Prototype Node 1 Node 2 Node 3 Not Used Node 4 Resizing Allocate d Memory Mgmt. Subsystem Local DRAM Zone (Static Size) Cluster Management Framework Host VM Manager OS/Hypervis or Virtual Machine CXL Node #2 Node #3 Node #4 Node #1 VM Local DRAM CXL Memory Memory Pooling Daemon VM Local DRAM CXL Memory Memory Pooling Daemon CXL Memory Zone (Resizable) VM Scheduler Memory *PMM Mailbox Secure Eraser **MPU #1 Memory Section Table to MPU #1 to MPU #2 to MPU #3 to MPU #4 from Table Mailbox MPU #2 from Table Mailbox MPU #3 from Table Mailbox MPU #4 from Table Memory Memory Memory Dynamic Allocation Engine CXL.io CXL.mem CXL.io CXL.mem CXL.io CXL.mem CXL.io CXL.mem CXL IP CXL IP CXL IP CXL IP * PMM : Pooled Memory Manager ** MPU : Memory Protection Unit
- 10. Use Cases of CXL Disaggregated Memory • Evaluation results of memory pooling • Reduce execution time by mitigating data spill and improve memory utilization of a system [Execution Time of CloudSuite Benchmark] Spill to Niagara outperforms NVMe by up to 2.5x (even though Niagara is an FPGA-based prototype) Utilization improved by 35% by applying DCS to Kubernetes [Memory Utilization on Kubernetes] 0 50 100 150 200 250 300 350 20 10 0 Normalized Execution Time vs Baseline [%] Workload Spilled to Medium [%] No Spill (Baseline) Spill to Niagara Spill to SSD 2.5x improvement [ Collaboration with MemVerge ] Baseline Kubernetes DCS-enabled Kubernetes Worker #1 Worker #2 Worker #1 Worker #2 # CPU Cores 112 176 112 176 Local Memory Capacity 62 GB 62 GB 62 GB 62 GB CXL Memory Capacity 32 GB 32 GB 64 GB (Pooled) Total Memory Capacity 94 GB 94 GB 62-126 GB 62-126 GB Avg. Memory Utilization 55.7% 75.1% 35% improvement
- 11. Use Cases of CXL Disaggregated Memory • Use case 2 : Memory sharing • Niagara supports firmware that allows all hosts to access shared data objects in Niagara memory • No more object serialization and transfer over network for remote object access • No more duplicate object copies on different nodes Zero copy CXL EP CXL EP CXL EP CXL EP CXL RP Pooled Memory Manager CXL RP CXL RP CXL RP Node #1 Node #2 Node #3 Node #4 Shared Object Data Write Data Read Data Read Data Read
- 12. Use Cases of CXL Disaggregated Memory • Evaluation results of memory sharing • Reduce execution time by eliminating network based data transfer overhead [Execution Time of *Ray Shuffle Benchmark] Ray with Niagara outperforms native Ray by up to 5.9x [Execution Time of Spark Benchmark] 5.9x improvement *Ray is an open source based distributed computing framework for AI/ML [ Collaboration with MemVerge ] Merge Hash Join outperforms Sort Merge join by up to 1.8x 0 20 40 60 80 100 120 140 Query 1 Query 2 Query 3 End-to-End Execution Time [sec] Mege Hash Join 1.8x improvement Merge Hash Join 0 20 40 60 80 100 25 50 75 100 Latency [sec] Partitions Ray with Shared Memory End-to-End Execution Time [sec]
- 13. Other HW-assisted Features of Niagara • Niagara supports HW-assisted features to CXL disaggregated memory efficiency • Block Data Management : Copy or move data within CXL pooled memory • Snapshot : Save and restore data in CXL Pooled Memory to/from storage device • Memory Failure Prediction : Predict memory Uncorrectable Errors(UEs) by analyzing memory failure patterns [Block Data Management] [Snapshot] [Memory Failure Prediction] CXL RP Pooled Memory Manage r CXL RP Node #1 Node #N CXL EP CXL EP CXL Disaggregated Memory Data Copy/Mov e CXL RP CXL RP Node #1 Node #N CXL EP CXL EP CXL Disaggregated Memory Data Save/Restore Storage CXL RP CXL RP Node #1 Node #N CXL EP CXL EP CXL Disaggregated Memory UE prediction Notify predicted UE Pooled Memory Manage r Pooled Memory Manage r
- 14. • Niagara 2.0 will be available by the end of 2023 • Niagara 2.0 is a 2U CXL disaggregated memory prototype which can connect up to 8 CXL host servers • Support DCD (Dynamic Capacity Device) feature defined in CXL specification 3.1 • We look forward to an open collaboration with industry partners to enable HW/SW ecosystem Next Step [Niagara 2.0 Prototype] [Server Cluster with Niagara 2.0] Server Server Pooled Memory Manager Cloud Orchestrato r DCD Engine CXL Disaggregated Memory Niagara 2.0
- 15. Niagara Live Demo • Demonstrate the allocation/deallocation of memory space in CXL disaggregated memory based on dynamic changes in memory requirements of VM servers • Check out SK hynix booth (#A8) for a live demo [Niagara Live Demo System] Niagara Server #1 Server #2 Server #3 Server #4 [Demo GUI]
- 16. Thank you!