![© Copyright 2020 Xilinx
Coloring Configuration
Xen command line
Dom0-less DomUs
xl domU config files
way_size=65536 xen_colors=0 dom0_colors=1-6
fdt set /chosen/domU0 colors <0x0 0x80>
colors=["10-11"]](https://image.slidesharecdn.com/cachecoloringelcna-200630220407/85/Xen-Cache-Coloring-Interference-Free-Real-Time-System-13-320.jpg)
![© Copyright 2020 Xilinx
Benchmarking
Hardware: Xilinx MPSoC
Workload: Bare Metal application
Stress: Bare Metal application
Test configuration:
Measured load: 1500 sums
Internal load: 8000 sums every 250us
Interrupt triggered every 8ms
Samples:1000
Colored configuration:
Xen: ["0,1"]
Motor Control DomU: ["4,7"]
Stress DomU: ["8,11"]
Stress2 DomU:["12,15"]
Linux stress:["2,3"]
Linux stress command: stress -m 1 --vm-bytes 8M --vm-stride 64 --vm-keep -t 3600 &](https://image.slidesharecdn.com/cachecoloringelcna-200630220407/85/Xen-Cache-Coloring-Interference-Free-Real-Time-System-14-320.jpg)
![© Copyright 2020 Xilinx
Xen Cache Coloring: Demo
Xen
[0]
Baremetal
Latency
Measurements
[7]
Baremetal Memory
Stress
[10 11]
Dom0
[1 2 3 4 5 6]](https://image.slidesharecdn.com/cachecoloringelcna-200630220407/85/Xen-Cache-Coloring-Interference-Free-Real-Time-System-30-320.jpg)
Xen Cache Coloring: Interference-Free Real-Time System
- 1. © Copyright 2020 Xilinx Xen Cache Coloring & Real-Time Stefano Stabellini Marco Solieri Luca Miccio Giulio Corradi
- 2. © Copyright 2020 Xilinx Xen in Embedded Consolidation and Componentization Isolation Security Reliability Safety Interference& Real-Time
- 3. © Copyright 2020 Xilinx Xen Schedulers CPU CPU CPU CPU
- 4. © Copyright 2020 Xilinx Xen Schedulers CPU CPU CPU CPU sched=null sched=credit vCPU vCPU vCPU vCPU vCPU vCPU vCPU vCPU vCPU vCPU
- 5. © Copyright 2020 Xilinx Xen Schedulers CPU CPU CPU CPU sched=null sched=credit vCPU vCPU vCPU vCPU vCPU vCPU vCPU vCPU vCPU vCPU
- 6. © Copyright 2020 Xilinx Xen Schedulers CPU CPU CPU CPU sched=null sched=rtds vCPU vCPU vCPU vCPU vCPU vCPU vCPU vCPU vCPU vCPU
- 7. © Copyright 2020 Xilinx Shared L2 cache L2 Core 1 Core 2 Core 3 Core 4 DDR L1 L1 L1 L1
- 8. © Copyright 2020 Xilinx Shared L2 cache L2 Core 1 Core 2 Core 3 Core 4 DDR L1 L1 L1 L1
- 9. © Copyright 2020 Xilinx Shared L2 cache L2 Core 1 Core 2 Core 3 Core 4 DDR L1 L1 L1 L1
- 10. © Copyright 2020 Xilinx Xen Cache Coloring CPUs clusters often share L2 cache Interference via L2 cache affects performance App0 running on CPU0 can cause cache entries evictions, which affect App1 running on CPU1 App1 running on CPU1 could miss a deadline due to App0’s behavior It can happen betweenApps running on the same OS & between VMs on the same hypervisor Hypervisor Solution: Cache Partitioning, AKA Cache Coloring Each VM gets its own allocation of cache entries No shared cache entries between VMs Allows real-time apps to run with deterministic IRQ latency Core 1 Core 2 Core 3 Core 4 L2 D D R
- 11. © Copyright 2020 Xilinx Colors Calculation Detect Way Size via registers via xen command line Calculate colors masks and total number of available colors Xilinx ZynqMP: way size is 65536 Order 16 -> color bitmask 0xF000 -> 16 colors D D R
- 12. © Copyright 2020 Xilinx Colored Allocation New colored memory allocator In alternative to Buddy Pages are stored by color in separate lists Can be used for Xen memory as well as Domain memory
- 13. © Copyright 2020 Xilinx Coloring Configuration Xen command line Dom0-less DomUs xl domU config files way_size=65536 xen_colors=0 dom0_colors=1-6 fdt set /chosen/domU0 colors <0x0 0x80> colors=["10-11"]
- 14. © Copyright 2020 Xilinx Benchmarking Hardware: Xilinx MPSoC Workload: Bare Metal application Stress: Bare Metal application Test configuration: Measured load: 1500 sums Internal load: 8000 sums every 250us Interrupt triggered every 8ms Samples:1000 Colored configuration: Xen: ["0,1"] Motor Control DomU: ["4,7"] Stress DomU: ["8,11"] Stress2 DomU:["12,15"] Linux stress:["2,3"] Linux stress command: stress -m 1 --vm-bytes 8M --vm-stride 64 --vm-keep -t 3600 &
- 15. © Copyright 2020 Xilinx L1 vs L2 cache L1 cache can hide L2 cache interference effects in simple test scenarios Make sure not use the L1 cache by accident during the measurements
- 16. © Copyright 2020 Xilinx Benchmark #1: Motor Control Execution Time Measure the time it takes to run a typical motor control execution routine, with and without interference
- 17. © Copyright 2020 Xilinx Results No interference: results for reference Lower is better
- 18. © Copyright 2020 Xilinx Results Adding interference: Xen Coloring has the best execution time
- 19. © Copyright 2020 Xilinx Results Increasing interference: • Xen Coloring is stable • Other results worsen
- 20. © Copyright 2020 Xilinx Results Increasing interference again: • Xen Coloring is stable • Other results worsen still
- 21. © Copyright 2020 Xilinx Results Comparing Xen vs. Xen Colored Lower is better
- 22. © Copyright 2020 Xilinx Benchmark #2: interrupt response time Measure the difference between the time when the interrupt service routine runs and the time the interrupt is set to fire, with and without interference Timer in Programmable Logic
- 23. © Copyright 2020 Xilinx Results No interference: results for reference Lower is better
- 24. © Copyright 2020 Xilinx Results Adding interference: • Xen without Coloring worsen • Xen Coloring is stable
- 25. © Copyright 2020 Xilinx Results Adding more interference: • Xen without Coloring worsen again • Xen Coloring is stable
- 26. © Copyright 2020 Xilinx Results Adding even more interference: • Xen without Coloring worsen still • Xen Coloring is stable
- 27. © Copyright 2020 Xilinx Results All results together Xen Coloring provides stable IRQ latency even under severe levels of interference
- 28. © Copyright 2020 Xilinx Conclusions Xen Cache Coloring offers Much lower execution times under stress Much lower IRQ latency under stress Greatly improves Determinism -- measurements have far lower variance Xen Cache Coloring effectively reduces the effects of cache interference Enable deployments of real-time and non-real-time workloads on a single SoC
- 29. © Copyright 2020 Xilinx Status All patches online at: https://github.com/Xilinx/xen.git xilinx/release-2020.1 TODO Upstreaming Linux Dom0 not 1:1 mapped, i.e. PV drivers support
- 30. © Copyright 2020 Xilinx Xen Cache Coloring: Demo Xen [0] Baremetal Latency Measurements [7] Baremetal Memory Stress [10 11] Dom0 [1 2 3 4 5 6]
- 31. © Copyright 2020 Xilinx Baremetal Latency Measurements
- 32. © Copyright 2020 Xilinx Baremetal Memory Stress
- 33. © Copyright 2020 Xilinx Thank You