XPDDS17: Reworking the ARM GIC Emulation & Xen Challenges in the ARM ITS Emulation
2 likes1,128 views
The document presents an overview of ARM interrupt virtualization, specifically focusing on the GICv2 and GICv3 architectures and their implementation challenges. It discusses the Xen VGIC implementation, limitations faced during virtualization support, and outlines necessary rework for improved interrupt handling and support for large interrupt configurations. Additionally, it addresses security concerns and the complexities associated with dynamically allocated interrupts and communication between guest and host systems.
XPDDS17: Reworking the ARM GIC Emulation & Xen Challenges in the ARM ITS Emulation
- 1. ARM® Interrupt Virtualization Andre Przywara <andre.przywara@arm.com> ARM Ltd., Cambridge, UK Xen summit 12/07/2017 © ARM 2017
- 2. ARM interrrupt virtualization agenda GICv2 and virtualization overview Xen vGIC implementation Rework plans GICv3/ITS architecture changes ITS emulation challenges 2 © ARM 2017
- 3. GICv2 architecture Device CPU interface CPU interface CPU interface Device Device Core 0 Core 1 Core n.... AXI bus IRQ lines (SPIs) GICv2 .... SGIs Distributor PPIs IRQ/FIQ 3 © ARM 2017
- 4. ARM GICv2 at a glance ”distributor” is the central component supports up to 8 CPUs and 1020 interrupts (988 shared + 32 private) has an input pin for each wired interrupt (SPIs) connects to separate CPU interfaces (one per core) has per-core input pins for private interrupts (PPIs) handles inter-processor interrupts internally (SGIs) Programmed via MMIO accesses Configuration affects always a group of interrupts (32-bit registers) Some registers are banked per CPU (at the same memory address) 4 © ARM 2017
- 5. Virtualization support in GICv2 Virtual CPU interface allows IRQ ACKs and EOIs without exiting the guest Hypervisor sets up virtual IRQs in List Registers (LRs) In the guest the (virtual) CPU interface relates to these Allows connecting a physical interrupt to a virtual one Physical IRQs gets EOIed at the same time No need to trap or monitor EOI in this case anymore Distributor IRQ lines (SPIs) GICCGICV LRs GICH CPU interface 5 © ARM 2017
- 6. Xen vGIC implementation Lives in xen/arch/arm (shared between arm(32) and arm64) Distributor is emulated (mostly in vgic-v2.c) Pending interrupts are written into the list registers (LRs) Interrupt acknowledgment and EOI is handled without exiting the guest (by the GIC hardware) After guest run distributor emulation syncs back from the LRs Virtual IRQ configuration is stored in ranks (groups of IRQs) GICV LRs CPU interface Distributor (emulated) GICCGICH virtual IRQs traps 6 © ARM 2017
- 7. vGIC implementation design Code design is centered around MMIO IRQ grouping (ranks) One lock per VCPU and one lock per rank Sophisticated lockless tricks to avoid excessive rank locking IRQ 32 IRQ 33 ISENABLER: . . . . IRQ 63 IRQ 62 IRQ 61 031 IRQ 34 ITARGETSR: IRQ 32IRQ 33IRQ 34IRQ 35 0781516232431 BUT: 7 © ARM 2017
- 8. Implementation issues and limitations Distributor accesses are rare (during IRQ configuration only) VCPU entry/exit path is much more frequent and sensitive State is held in bitmaps and bytemaps covering contigious IRQ numbers proves problematic with sparse LPI allocation (later) Handling of level vs. edge triggered interrupts 8 © ARM 2017
- 9. vGIC rework To bring ITS/LPI support closer to the existing code base: Store all virtual IRQ properties in struct pending_irq Protect with one lock per IRQ Get rid of the ranks and their locks at all Simplify IRQ list management (remove lr_queue list) Introduce reference counting for dynamic pending_irq’s Model level-triggered interrupts correctly 9 © ARM 2017
- 10. GICv3 architecture Device Device Device Device Device .... AXI bus IRQ lines (SPIs) .... SGIs Distributor PPIs GICv3 Redistributor Redistributor Redistributor Core 0 Core 1 Core n CPU interface CPU interface CPU interface ITS Memory IRQ/FIQ MSI MSI 10 © ARM 2017
- 11. GICv3 changes System register access to CPU interface (drops banked MMIO) IRQ routing allows millions of cores Lifts the 8-CPU limit of GICv2 Uses MPIDR based values to specify one target core per IRQ Splits distributor to separate private and shared IRQs New class of interrupts (LPI) via an Interrupt Translation Service (ITS) Allows MSI/MSI-X support Supports indirections for target cores (via collections) Introduces device ID sampled from the bus New IRQ class with possibly thousands of LPIs and probably sparse allocation Tables are held in physical memory 11 © ARM 2017
- 12. GICv3/ITS Xen implications Potentially large, sparsely allocated LPIs spoil vGICv2 ranks Requires dynamic allocation (and de-allocation → ref-counting) Motivates storing all IRQ property in struct pending_irq Brings up locking issues (solution: per-IRQ lock) ITS data structures are held in guest physical memory Need to reach into domain memory (fortunately cheap on ARM64) Creates security question (we can’t trust this memory) Provides natural resource limitation (more LPIs → more memory) Caching is architected and common in hardware too 12 © ARM 2017
- 13. ITS tables (simplified) device command prop pend coll 13 © ARM 2017
- 14. ITS challenges Security and DOS challenges: Untrusted guest memory holds internal information Need to sanitise information on every access (sometimes not feasible) Accessing guest memory is not for free Looking at getting rid of storing information there Guest could fill (potentially large) emulated command queue Architecture allows defered execution But we need to schedule this! Some guest actions may require host action Disabling or enabling interrupts is one case To avoid IRQ storms we want to disable IRQs on the hardware Requires queueing an ITS command to the host command queue May open DOS vector when a domain often disables/enables LPIs 14 © ARM 2017
- 15. Preliminary fixes ITS emulation only for Dom0 → trust the guest Untrusted guest memory holds internal information Sanitise as much as possible Trust the guest! Guest could fill (potentially large) emulated command queue Real World Dom0s only schedule two or three commands Trust the guest! Some guest actions may require host action Keep interrupts enabled on the host Detect guest disabled IRQs early Ignore IRQ storms for now (trust the guest!) 15 © ARM 2017
- 16. Xen questions Accessing shared host resources on behalf of guests Could introduce deferred host command queueing How to schedule commands? Suspend vCPUs until commands have been executed? More costly emulation on behalf of a guest Return early when timeslice expires! But when to return to continue the work? Deschedule guest while in an MMIO emulation? Schedule tasklet on account of a guest? 16 © ARM 2017
- 17. The trademarks featured in this presentation are registered and/or unregistered trademarks of ARM limited (or its subsidiaries) in the EU and/or elsewhere. All rights reserved. All other marks featured may be trademarks of their respective owners. Copyright © 2016 ARM Limited © ARM 2017