Implementing SR-IOv failover for Windows guests during live migration
0 likes336 views
The document discusses the implementation of SR-IOV failover for Windows guests during migration, detailing virtio-win drivers, network configurations, and various proposed solutions to live migration challenges. It compares Windows NIC teaming and Hyper-V solutions, highlighting the complexities of the mux driver model versus the simplified Hyper-V model. Additionally, it outlines technical specifications, installation prerequisites, certification processes, and performance metrics for virtio network devices.
Implementing SR-IOv failover for Windows guests during live migration
- 1. Implementing SR-IOV Failover for Windows Guests During Migration Annie Li - Principal Software Engineer, Oracle Yan Vugenfirer - CEO, Daynix http://github.com/virtio-win/
- 2. Agenda • Virtio-win drivers • Windows guest terminology • The problem • Different solutions • Failover solution with virtio-net on Windows guest
- 3. • Upstream: https://github.com/virtio-win/kvm-guest- drivers-windows/ • Drivers for the major virtio devices: • virtio-net • virtio-blk, virtio-scsi • virtio-balloon, virtio-serial, virtio-vsock, virtio-input, virtio-rng • Panic, fw-cfg • INF files (pci-serial, sm-bus on Q35) Drivers for Windows
- 4. • WDF drivers for the “simple” devices • Miniport architecture for network and storage • NDIS • Storport • Scsiport VirtIO Drivers for Windows
- 5. • Supported OS • Windows XP, Vista, 7, 8, 8.1, 10 (up to recent builds) • Widows Server 2003, 2008, 2008R2, 2012, 2012R2, 2016, 2019 VirtIO Drivers for Windows
- 6. • Send PRs - https://github.com/virtio- win/kvm-guest-drivers-windows/ pulls • Code changes should pass WHQL • We are running WHQL CI on upstream (HCK-CI) How to Contribute
- 8. NDIS (Network Driver Interface Specification) Architecture NotifyObject Network configuration subsystem HW device HW device Protocol drivers Protocol Driver Miniport Drivers Network Device Miniport driver Miniport edge Protocol edge Intermediate drivers Protocol driver Intermediate driver Protocol driver Miniport driver NDIS User space
- 9. • NDIS miniport driver • Basic driver package: • INF file – installation description • SYS file – driver binary • PDB file – symbols for debugging • CAT file - package digital signature Virtio-net (NetKVM) Driver for Windows
- 10. The Problem – Live Migration and SR-IOV
- 11. Overview of SR-IOV Migration Solutions • Previous efforts and vendor specific HW solutions • Hyper-V and Windows • Linux and VirtIO
- 12. Previous Efforts • KVM Forum 2015 Live Migration with SR-IOV Pass-through - Weidong Han, Huawei • KVM Forum 2018 - Live Migration Support for GPU with SR-IOV - Zheng Xiao, Alibaba Cloud; Jerry Jiang & Ken Xue, AMD • KVM Forum 2020 (parallel session) - Device Keepalive State for Local Live Migration and VMM Fast Restart - Jason Zeng, Intel
- 13. Overview of Software Solutions • Windows NIC Teaming • Windows MUX Intermediate driver • Hyper-V Solution
- 14. Windows NIC Teaming • Similar to bond in Linux • Provides failover capability • Configured through GUI or Powershell Cmdlets in user space
- 15. Windows MUX Intermediate Driver • Kernel space solution with various models • One-to-two model for SR-IOV live migration
- 16. Windows MUX Intermediate Driver TCPIP VF miniport driver NotifyObject Open Adapter1 Open Adapter2 Network configuration subsystem VirtIO network device Virtual Function network device Protocol driver Protocol Driver Miniport Driver Network Device Virtual Adapter NetKVM Miniport edge Protocol edge One-to-Two MUX Intermediate Driver Internal Binding Other Protocols ndisndis ndis ndis ndis • Netkvm is Open Source driver code for VirtIO network • VF miniport driver is provided by vendor NDIS(Kernel Space) User space
- 17. Network Binding of NIC Teaming or MUX
- 18. Hyper-V VM Network • Network virtual service client(NetVSC ) • Synthetic data path • SR-IOV data path • Two Installation files(INF) • Share same driver binary
- 19. Hyper-V SR-IOV VF Failover TCPIP Other Protocols VF Miniport Driver Hyper-V NetVSC Driver Hyper-V NetVSC Miniport Driver Hyper-V NetVSC Protocol Driver Miniport Driver 1:1 binding Hyper-V Virtual Network Device Virtual Function Device Virtual Network Device Protocol Driver Protocol Driver ndisvf ndisvf • No Bond/Teaming • No NotifyObject • No new Virtual Adapter NDIS(Kernel space)
- 20. Network Binding in Hyper-V
- 21. Comparison Summary • MUX Driver model: • Complicated, New virtual adapter, Restore offload, NotifyObject. • Hyper-V model: • Simplified, Appropriate for Hyper-V • 2-netdev model in KVM
- 22. Windows Network of VirtIO and VF TCPIP Other Protocols VF Miniport Driver NetKVM Miniport Driver VirtIO Network Device Virtual Function Device Protocol Driver Miniport Driver Virtual Network Device ndis ndis NDIS(Kernel sapce)
- 23. TCPIP Other Protocols VF Miniport Driver VirtIO NetKVM Driver NotifyObject Network configuration subsystem Virtual Function Device Protocol Driver Miniport Driver Virtual Network Device 1:1 binding ndis ndis ndis ndis VirtIO SR-IOV VF Failover VirtIO Network Device VirtIO Protocol Driver NetKVM Miniport Driver User Space NDIS in Kernel Space • Netkvm is Open Source driver code for VirtIO network • VF miniport driver is provided by vendor
- 24. Protocol Driver in 2-netdev Model • Behaves like a bridge • VF adapter is coupled to VirtIO adapter with the same MAC address • Handling TX/RX network data • Object identifiers(OIDs) are wrapped and forwarded, offloads are propagated
- 25. Network Binding of VirtIO SR-IOV
- 26. Known Issues • DHCP issue • Only happens in specific scenario • Statistics is missing • NetKVM driver needs to keep the statistics of packets sent to or received from VF driver • Old Windows system • Windows Server 2003 and Windows XP • Need to add VF PNP to Notification Object code or to the registry • Possible race during boot?
- 27. VirtIO and SR-IOV Failover • VirtIO specification specifics • Feature bit called VIRTIO_NET_F_STANDBY. It is appropriate for 3-netdev model in Linux, but not for Windows 2-netdev model.
- 28. Installation • Before • INF for Miniport • After • INF for Miniport • INF for Protocol driver definition and Notify Object • Not part of the guest tools installer: https://github.com/virtio-win/virtio-win-guest-tools-installer
- 29. WHQL Certification • Before • Certification of miniport • Automatic review • After • Two steps certification • Automatic review for miniport • Manual review of protocol driver and the final package
- 30. Performance From VM to Remote Host(NetPerf, 32 vcpus, 32 TCP Streams, MTU=9000) 0 25 50 75 100 Default Default 16k 32k 16k 128k 32k 64k 32k 256k 64k 128k 128k 128k 256k 256k VF 2-netdev NetKVM • x axis: “TX/RX Buffer size” and “TX/RX Socket Buffer size” • y axis: Network throughput from 0 to 100G
- 31. Performance From Remote Host to VM(NetPerf, 32 vcpus, 32 TCP Streams, MTU=9000) 0 25 50 75 100 Default Default 16k 32k 16k 128k 32k 64k 32k 256k 64k 128k 128k 128k 256k 256k VF 2-netdev NetKVM • x axis: “TX/RX Buffer size” and “TX/RX Socket Buffer size” • y axis: Network throughput from 0 to 100G
- 32. Performance From VM to VM on Remote Host (NetPerf, 32 vcpus, 32 TCP Streams, MTU=9000) 0 25 50 75 100 Default Default 16k 32k 16k 128k 32k 64k 32k 256k 64k 128k 128k 128k 256k 256k VF 2-netdev NetKVM • x axis: “TX/RX Buffer size” and “TX/RX Socket Buffer size” • y axis: Network throughput from 0 to 100G
- 34. Links – source code • virtio-win drivers source code: • https://github.com/virtio-win/kvm-guest-drivers- windows • MiniPort and Protocol driver: • https://github.com/virtio-win/kvm-guest-drivers- windows/tree/master/NetKVM • Notification object: • https://github.com/virtio-win/kvm-guest-drivers- windows/tree/master/NetKVM/NotifyObject
- 35. Links – download binary drivers • https://docs.fedoraproject.org/en-US/quick-docs/ creating-windows-virtual-machines-using-virtio- drivers/index.html
- 36. Links – related presentations • KVM Forum 2015 Live Migration with SR-IOV Pass-through - Weidong Han, Huawei • https://www.youtube.com/watch?v=vnwEnzVp9Zo • https://www.linux-kvm.org/images/9/9a/03x07-Juniper-Weidong_Han- LiveMigrationWithSR-IOVPass-through.pdf • KVM Forum 2018 - Live Migration Support for GPU with SR-IOV - Zheng Xiao, Alibaba Cloud; Jerry Jiang & Ken Xue, AMD • https://events19.linuxfoundation.org/wp-content/uploads/2017/12/Live- Migration-Support-for-GPU-with-SRIOV-Challenges-and-Solution-Zheng-Xiao- Alibaba-Cloud-Jerry-Jiang-Ken-Xue-AMD.pdf • KVM Forum 2020 (parallel session) - Device Keepalive State for Local Live Migration and VMM Fast Restart - Jason Zeng, Intel • https://sched.co/eE3W