PCI Pass-through - FreeBSD VM on Hyper-V (MeetBSD California 2016)
1 like2,478 views
The document discusses PCI pass-through for FreeBSD virtual machines on Hyper-V, covering architecture, integration services, and system requirements. It highlights the enhancements in performance when specific physical PCIe devices are directly controlled by guest VMs. Additionally, it mentions upcoming steps including debugging and integration for improving functionality and performance in Azure environments.
PCI Pass-through - FreeBSD VM on Hyper-V (MeetBSD California 2016)
- 1. PCI Pass-through - FreeBSD VM on Hyper-V Dexuan Cui (dexuan@freebsd.org) Yanmin Qiao (sephe@freebsd.org ) Kylie Liang (kyliel@Microsoft.com) Microsoft Enterprise Open Source Group
- 2. Agenda • FreeBSD VM on Hyper-V Architecture • PCI Pass-through • Demo with FreeBSD VM on Hyper-V • Next Step • PCI Pass-through in Azure
- 3. Running a Guest OS on Hyper-V Have Integration Services? Yes Yes No No Emulated Devices • No Hyper-V integration • Limited manageability • Limited functionality • OK CPU, weak disk/net
- 4. Integration Services for FreeBSD - High Level Architecture Windows Server Virtual Machine Management Services VMMS)( Virtual Infrastructure Driver VID)( Virtualization Service Provider (VSP) VMBusDrivers Hyper-v Hardware Windows Kernel VMBus FreeBSD Kernel ifnetCAM StorVSC NetVSC FreeBSD VM Virtual Machine Worker Process (VMWP) Utilities Daemon User mode Kernel mode
- 5. Changes to FreeBSD • Driver source code under sys/dev/hyperv/ • include/hyperv.h • netvsc • storvsc • utilities • vmbus • Daemon • http://svnweb.freebsd.org/base/head/contrib/hyperv/tools/
- 6. Discrete Device Assignment • Discrete Device Assignment (also known as PCI Passthrough) available as part of the Hyper-V role in Microsoft Windows Server 2016. • Performance enhancement that allows a specific physical PCIe device to be directly controlled by a guest VM running on the Hyper- V instance. • GPU • Network adapter • NVMe device
- 7. System Requirements - Host • Processor • Intel's Extended Page Table (EPT) • or AMD's Nested Page Table (NPT). • Chipset: • Interrupt remapping - Intel's VT-d with the Interrupt Remapping capability (VT-d2) or any version of AMD I/O Memory Management Unit (I/O MMU). • DMA remapping - Intel's VT-d with Queued Invalidations or any AMD I/O MMU. • Access control services (ACS) on PCI Express root ports. • The firmware tables must expose the I/O MMU to Hyper-V. • Note: This feature might be turned off in the UEFI or BIOS. For instructions, see the hardware documentation or contact your hardware manufacturer.
- 8. System Requirements - Device • Some devices in your computer don’t mark, or tag, their traffic in way that individually identifies the device, making it impossible for the I/O MMU to redirect this traffic to the memory owned by a specific VM. These devices, mostly older PCI-style logic, can’t be assigned to a guest VM. • A survey script available to find potential DDA devices on your hosts • Check machine/firmware (iommu: for intel it’s VT-d), host os (WS 2016), devices (PCIe + no Port IO BAR + MSI/MSI-X + ACS), etc • https://github.com/Microsoft/Virtualization- Documentation/tree/master/hyperv-samples/benarm-powershell/DDA
- 9. Guest on Hyper-V vs. XEN & KVM • Xen (HVM)/KVM • Full emulation of PCI/PCIe bus • XEN(PV guest)/Hyper-V • Partial emulation of PCI/PCIe bus + PV hotplug message
- 10. Generation 1 vs. 2 VM on Hyper-V Generation 1 Device Generation 2 Replacement Generation 2 Enhancements IDE controller Virtual SCSI controller Boot from .vhdx (64 TB maximum size, and online resize capability) IDE CD-ROM Virtual SCSI CD-ROM Support for up to 64 SCSI DVD devices per SCSI controller. Legacy BIOS UEFI firmware Secure Boot Legacy network adapter Synthetic network adapter Network boot with IPv4 and IPv6 Floppy controller and DMA controller No floppy controller support Universal asynchronous receiver/transmitter (UART) for COM ports Optional UART for debugging Faster and more reliable i8042 keyboard controller Software-based input Uses fewer resources because there is no emulation. Also reduces the attack surface from the guest operating system. PS/2 keyboard Software-based keyboard Uses fewer resources because there is no emulation. Also reduces the attack surface from the guest operating system. PS/2 mouse Software-based mouse Uses fewer resources because there is no emulation. Also reduces the attack surface from the guest operating system. S3 video Software-based video Uses fewer resources because there is no emulation. Also reduces the attack surface from the guest operating system. PCI bus No longer required Programmable interrupt controller (PIC) No longer required Programmable interval timer (PIT) No longer required Super I/O device No longer required
- 11. Guest and FreeBSD sys/dev/hyperv/pcib/pcib.c - In code review phase The patch implements a PCI bridge driver to support PCI pass-through. • Talks to the host to discover device(s) and presents the device(s) to FreeBSD's pci driver via PCI configuration space • Allocates resources for the device(s) and initialize the related BARs, when the device driver's attach method is invoked. • Talks to the host to create MSI/MSI-X interrupt remapping between the guest and the host. • Supports device hot add/remove.
- 12. Demo Dismount the device from the host Hot Add Configure and test the performance Hot Remove Mount the device to the host
- 13. Next Step • Debug: hypervisor’s scheduling can affect the performance greatly • SR-IOV • Allocates a portion of a NIC to the virtual machine for improved latency and throughput. • VF driver in FreeBSD • Integrate it to Virtual Appliances with ISV vendors
- 14. Hybrid Hyper-scale Open & flexible Applications Clients Infrastructure Management Databases & Middleware App Frameworks & Tools DevOps PaaS & DevOps Enterprise proven AzureOpen & flexible >80%of Fortune 500 use the Microsoft Cloud
- 15. PCI Pass-through in Azure – GPU Access • Used on N-Series Azure VMs to give the guests access to GPUs. • VM Size: N-Series instances are enabled with NVIDIA’s cutting-edge GPUs so you can run GPU-accelerated workloads and visualize them. • Windows and Linux VMs • Start from South Central US region • In public preview now. • Want to learn more – http://gpu.azure.com/ .
- 16. PCI Pass-through in Azure – Accelerated Networking • Accelerated Networking enables Single Root I/O Virtualization (SR-IOV) to a virtual machine (VM), greatly improving its networking performance. • VM Size: Standard_D15_v2 and Standard_DS15_v2 • Windows Server 2012 R2 and Windows Server 2016 Technical Preview 5. Linux and Windows Server 2012 support will be added soon. • West Central US, West Europe Azure regions • Start from South Central US region • In preview now.
- 17. BSD Based Virtual Appliance in Azure Marketplace
- 18. FreeBSD 10.3 is available in Azure Marketplace June: Global Azure Sep: Azure China
- 19. BSD Meetup in China – First Time Meet other local geeks and discuss the BSD including FreeBSD, NetBSD, OpenBSD, MacOS X, DragonFlyBSD and any other BSD derived systems. If you want to know about BSD running in the cloud or how BSD is used as networking and security product, you definitely should join us! BSD的开发者，爱好者们，无论您是在做FreeBSD，NetBSD，OpenBSD ，MacOS X, DragonFlyBSD或任何BSD衍生产品的开发，或者您听过并 想了解BSD和UNIX，让我们聚集一堂吧！又或者您关心云计算，网络 和安全，这样的聚会也必定让您受益匪！ http://www.huodongxing.com/go/bsd_sh 11/21 @ Shanghai http://www.huodongxing.com/go/bsd_bj 12/5 @ Beijing
- 20. Thank You