Optimizing the Design and Implementation of KVM/ARM - SFO17-403
0 likes803 views
The document discusses the optimization of the KVM/ARM virtualization architecture, focusing on the Virtualization Host Extensions (VHE) and their benefits. It highlights various performance improvements achieved, including a significant reduction in hypercall overhead and cycle counts compared to non-VHE and x86 systems. The conclusion emphasizes the advantages of the redesigned KVM/ARM, including faster operations and reduced workload overheads.
![ENGINEERS AND DEVICES
WORKING TOGETHER
–Popek and Golberg
[Formal requirements for virtualizable third generation architectures ’74]
““Efficient, isolated duplicate
of the real machine””](https://image.slidesharecdn.com/sfo17-403optimizingthedesignandimplementationofkvmarm-171020175829/85/Optimizing-the-Design-and-Implementation-of-KVM-ARM-SFO17-403-2-320.jpg)
Optimizing the Design and Implementation of KVM/ARM - SFO17-403
- 1. connect.linaro.org SFO17-403: Optimizing the Design and Implementation of KVM/ARM Christoffer Dall
- 2. ENGINEERS AND DEVICES WORKING TOGETHER –Popek and Golberg [Formal requirements for virtualizable third generation architectures ’74] ““Efficient, isolated duplicate of the real machine””
- 3. ENGINEERS AND DEVICES WORKING TOGETHER Hardware OS Kernel App AppApp Hardware Hypervisor VM Kernel App App VM Kernel App App Native Virtual Machines Virtualization
- 4. ENGINEERS AND DEVICES WORKING TOGETHER Hypervisor Design Hardware Hypervisor VM Kernel App App VM Kernel App App Type 1 (Standalone)
- 5. ENGINEERS AND DEVICES WORKING TOGETHER Hypervisor Design Hardware Hypervisor VM Kernel App App VM Kernel App App Type 1 (Standalone) Hardware OS Kernel VM Kernel App App VM Kernel App App Type 2 (Hosted) Hypervisor App
- 6. ENGINEERS AND DEVICES WORKING TOGETHER Hypervisor Design Hardware Xen Dom0 Linux App App DomU Linux App App Hardware Linux VM Linux App App VM Linux App App KVM App
- 7. ENGINEERS AND DEVICES WORKING TOGETHER ARM Virtualization Extensions Kernel UserEL0 EL1 HypervisorEL2
- 8. ENGINEERS AND DEVICES WORKING TOGETHER ARM VE and Hypervisors Xen Dom0 Linux App App DomU Linux App AppEL0 EL1 EL2 ?
- 9. ENGINEERS AND DEVICES WORKING TOGETHER KVM/ARM Host Linux AppApp VM Kernel AppApp KVM KVM lowvisor EL0 EL1 EL2 1. Hypercall 2. Return3. Hypercall 4. Return switch state
- 10. ENGINEERS AND DEVICES WORKING TOGETHER KVM/ARM Host Linux AppApp VM Kernel AppApp KVM EL0 EL1 EL2 1. Hypercall 2. Return
- 11. ENGINEERS AND DEVICES WORKING TOGETHER ARMv8.1 VHE • Virtualization Host Extensions • Supports running unmodified OSes in EL2 without using EL1 Linux EL0 EL1 EL2 AppApp
- 12. ENGINEERS AND DEVICES WORKING TOGETHER VHE: Backwards Compatible • HCR_EL2.E2H complete enables and disables VHE • When disabled, completely backwards compatible with ARMv8.0 • Example: Xen disables VHE
- 13. ENGINEERS AND DEVICES WORKING TOGETHER VHE: Expands Functionality of EL2 • Expanded EL2 functionality • New registers: TTBR1_EL2, CONTEXTIDR_EL2 • New virtual EL2 timer
- 14. ENGINEERS AND DEVICES WORKING TOGETHER VHE: Support Userspace in EL0 • TGE: Trap General Exceptions • Routes all exceptions to EL2 • VHE no longer disables EL0 stage 1 MMU Linux EL0 EL1 EL2 AppApp Exceptions
- 15. ENGINEERS AND DEVICES WORKING TOGETHER VHE: EL2&0 Translation Regime • Same page table format as EL1 • Used in EL0 with TGE bit set
- 16. ENGINEERS AND DEVICES WORKING TOGETHER VHE: System Register Redirection TCR_EL1 mrs x0, TCR_EL1 HCR_EL2.E2H == 0 TCR_EL2
- 17. ENGINEERS AND DEVICES WORKING TOGETHER VHE: System Register Redirection TCR_EL1 mrs x0, TCR_EL1 TCR_EL2 HCR_EL2.E2H == 1
- 18. ENGINEERS AND DEVICES WORKING TOGETHER VHE Register Redirection TCR_EL1mrs x0, TCR_EL12
- 19. ENGINEERS AND DEVICES WORKING TOGETHER More VHE Register Redirection • Some registers change bit position to be similar between EL1 and EL2 • Example: CNTHTCL_EL2 changes layout to match CNTKCTL_EL1 with extra bits
- 20. ENGINEERS AND DEVICES WORKING TOGETHER Legacy KVM/ARM without VHE HypervisorLinux EL2 EL1 KVM Lowvisor Trap Run VM
- 21. ENGINEERS AND DEVICES WORKING TOGETHER KVM/ARM with VHE HypervisorLinux EL2 KVM Lowvisor Function Call Run VM
- 22. ENGINEERS AND DEVICES WORKING TOGETHER Experimental Setup • AMD Seattle B0 • 64-bit ARMv8-A • 2.0 GHz AMD A1100 CPU • 8-way SMP • 16 GB RAM • 10 GB Ethernet (passthrough) *Measurements obtained using Linux in EL2. See BKK16 talk.
- 23. ENGINEERS AND DEVICES WORKING TOGETHER VHE Performance at First Glance CPU Clock Cycles non-VHE VHE* Hypercall 3.181 3.045 *Measurements obtained using Linux in EL2. See BKK16 talk.
- 24. ENGINEERS AND DEVICES WORKING TOGETHER KVM/ARM Optimization #1 VM Kernel AppAppEL0 EL1 EL2 Host AppApp Linux KVM • Avoid saving/restoring EL1 register state
- 25. ENGINEERS AND DEVICES WORKING TOGETHER KVM/ARM Optimization #2 VM Kernel AppAppEL0 EL1 EL2 Host AppApp Linux KVM • Legacy KVM/ARM design enabled/disabled virtualization features on every transition • Virtual/Physical interrupts • Stage 2 memory translation KVM Lowvisor Disable traps Enable traps
- 26. ENGINEERS AND DEVICES WORKING TOGETHER KVM/ARM Optimization #2 VM Kernel AppAppEL0 EL1 EL2 Host AppApp Linux KVM • Leave virtualization features enabled • Host EL2 never uses stage 2 translations and always has full hardware access.
- 27. ENGINEERS AND DEVICES WORKING TOGETHER KVM/ARM Optimization #3 • Don’t context switch the timer on every exit from the VM • Completely reworks the timer code • 20 patches on list
- 28. ENGINEERS AND DEVICES WORKING TOGETHER KVM/ARM Optimization #4 • Reduce run loop work • Do work in vcpu_load and vcpu_put instead • Called when entering/exiting run-loop • Called when preempted/scheduled • Requires VHE vcpu_load vcpu_put vcpu run loop
- 29. ENGINEERS AND DEVICES WORKING TOGETHER KVM/ARM Optimization #5 • Rewrite the world switch code kvm_arch_vcpu_ioctl_run { ... while (1) { ... if (has_vhe() /* static key */ ret = kvm_vcpu_vhe_run(vcpu); else ret = kvm_call_hyp(__kvm_vcpu_run, vcpu); ... } ... }
- 30. ENGINEERS AND DEVICES WORKING TOGETHER Microbenchmark Results CPU Clock Cycles non-VHE VHE OPT * x86 Hypercall 3.181 752 1.437 I/O Kernel 3.992 1.604 2.565 I/O User 6.665 7.630 6.732 Virtual IPI 14.155 2.526 3.102 *Measurements obtained using Linux in EL2. See BKK16 talk.
- 31. Application Workloads Application Description Kernbench Kernel compile Hackbench Scheduler stress Netperf Network performance Apache Web server stress Memcached Key-Value store
- 32. ENGINEERS AND DEVICES WORKING TOGETHER Application Workloads 0.00 0.50 1.00 1.50 2.00 Kernbench Hackbench TCP_STREAM TCP_MAERTS TCP_RR Apache Memcached non-VHE VHE OPT* *Measurements obtained using Linux in EL2. See BKK16 talk. Normalized overhead (lower is better)
- 33. ENGINEERS AND DEVICES WORKING TOGETHER Conclusions • Optimize and redesign KVM/ARM for VHE • Reduce hypercall overhead by more than 75% • Better cycle counts than x86 for key hypervisor operations • Network benchmark overhead reduced by 50% • Key-value store workload overhead reduced by more than 80%
- 34. ENGINEERS AND DEVICES WORKING TOGETHER Upstream Status • Timer patches on list • Core optimization patches coming soon