Multiple Device Emulators for HVM Guests, Paul Durrant, Citrix
1 like7,265 views
The document discusses advancements in device emulation for HVM guests in XenServer, specifically focusing on the implementation and future of NVIDIA Grid virtual GPU (vGPU) integration. It highlights challenges in current emulation methods and introduces a new ioreq server abstraction to accommodate multiple device emulators. Future plans include improved vGPU support via a 'pass-through' approach and better management of device emulation using multiple instances of QEMU.
![IOREQ Server Abstraction
• Based on patch created by Julien Grall from XenClient
HVM_PARAM_DM_DOMAIN
HVM_PARAM_IOREQ_PFN
vcpu[X].arch.hvm_vcpu.xen_port
HVM_PARAM_BUFIOREQ_PFN
HVM_PARAM_BUFIOREQ_EVTCHN
IOREQ
Server](https://image.slidesharecdn.com/multipleemulators-131114051942-phpapp02/85/Multiple-Device-Emulators-for-HVM-Guests-Paul-Durrant-Citrix-6-320.jpg)
Multiple Device Emulators for HVM Guests, Paul Durrant, Citrix
- 1. Multiple Device Emulators for HVM Guests Paul Durrant Principal Software Engineer, Citrix
- 2. Agenda “What we’ve done with device emulation in XenServer and what we’re planning to do” • Background • How device emulation is done for an HVM guest • nVidia GRID vGPU • Current implementation of vGPU • Future implementation of vGPU • Potential spin-off work
- 3. How are devices emulated for an HVM guest? Guest DOM0 QEMU PIO MMIO Instruction Trap Page Fault VMEXIT Xen IOREQ RTC, HPET, etc. Problem: • All device emulation has to be done directly in Xen or in a single emulator i.e. QEMU
- 4. nVidia GRID Virtual GPU Windows Guest OpenGL Device Model DirectX Standard Hardware Driver Kernel Driver Xen Need an emulator for this bit GRID GPU
- 5. nVidia GRID Virtual GPU • Build into Xen? • Closed source codebase • GPL problems • Needs hooks into VGA • Build into QEMU? • Large patch • Maintenance issue • Binary plugin interface • Unlikely to gain traction upstream “Neither of those sounds good. Is there another way?”
- 6. IOREQ Server Abstraction • Based on patch created by Julien Grall from XenClient HVM_PARAM_DM_DOMAIN HVM_PARAM_IOREQ_PFN vcpu[X].arch.hvm_vcpu.xen_port HVM_PARAM_BUFIOREQ_PFN HVM_PARAM_BUFIOREQ_EVTCHN IOREQ Server
- 7. Multiple Emulators Guest EMU0 PIO EMU1 MMIO Instruction Trap Page Fault VMEXIT Xen IOREQ IOREQ IOREQ Server Table IOREQ Server Table: • Demux on port address, memory address or PCI BDF
- 8. Where XenServer vGPU is now • Emulator process for each vGPU • Wrapper for nVidia plugin • Runs in dom0 • Patched QEMU • Stripped out graphics device model • Uses IOREQ Server API “That doesn’t sound ideal”
- 9. Where XenServer vGPU is going • ‘Appliance’ VM for vGPU • Pass GPU through • vGPU emulators • Use IOREQ Server API to register • libVNC based console • Unpatched QEMU • No graphics emulation required • Uses old HVM_PARAM interface • Instantiates ‘catch all’ IOREQ server “Multiple emulators sound cool. What else could we do with them?”
- 10. Service Domains Storage PV Guest EMU Network PV EMU Xen Co-locate emulators and PV backends: • No need to build complex emulated-to-PV data paths “Does this mean we’re not going to use QEMU any more?”
- 11. QEMU Disaggregation • We still want to use QEMU for most devices, but we want to use several instances • May be issues separating device emulation from machine emulation • Work already done by Julien • ‘Unplug’ becomes a problem
- 12. Idea • New HVMOPs to request unplug • Usual IOREQs issued to ‘catch-all’ emulator for compatibility • New emulators receive new UNPLUG IOREQs • If no ‘catch-all’ is present then we may be able to unplug individual devices • Xen Platform ‘fixed IO port’ emulation handled in Xen • Maps onto new HVMOP code • Needs prototyping
- 13. Q&A