Making GPU resets less painful on Linux
1 like34 views
The document discusses the complexities of handling GPU resets on Linux, particularly focusing on the challenges faced when a GPU hangs during operation. It emphasizes the need for better reporting and handling mechanisms in the DRM and Mesa layers to improve user experience and application resilience against GPU failures. The roadmap for improving GPU resets includes standardizing reporting procedures and enhancing the ability to identify and recover from hangs, including developing tools for capturing relevant state and error information.
Making GPU resets less painful on Linux
- 1. Making GPU resets less painful on Linux André Almeida @ Igalia OSS NA 2023 1
- 2. Hi! Kernel developer Working in the Steam Deck 2
- 3. Oh no, my GPU hanged! You are playing your game on Linux Something wrong is sent to the device ??? Game over, reboot your machine 3
- 4. Modern GPUs are complex Really complex AMD Radeon RX 7900 XTX 96 Compute units, 384 texture units, 6 shader engines, 58 B transistors... Shaders are Turing Complete 4
- 5. Modern GPUs are complex DCHUB HUBP (n) DPP (n) MPC OPTC DIO DCCG DMU AZ MMHUBBUB DWB (n) Global sync Pixel data Sideband signal Config. Bus SDP Monitor OPP dc_plane dc_stream dc_state Code struct dc_link Floating point calculation bit-depth reduction/dither } Notes 5
- 6. Modern GPUs are complex If you have an infinity loop in the CPU, it's not that bad CPU programs has virtual memory and virtual processor Things might be more barebone in the GPU But in a GPU, the display won't be able to update 6
- 7. Detecting GPU hangs From the hardware to the application 7
- 8. Detecting GPU hangs Device to Kernel Submit a job and wait until is done Check fences Or timeouts The driver does a GPU reset This can be "soft" resets, one hw engine reset or full device reset More complete resets are more destructive Now, report to userspace 8
- 9. Reporting GPU hangs Kernel to Mesa DRM has no API for that I915_GET_RESET_STATS AMDGPU_CTX_OP_QUERY_STATE2 MSM_PARAM_FAULTS Return -ERROR for ioctls It's not really hw specific 9
- 10. Reporting GPU hangs Mesa to application APIs provide a way to tell apps that a reset happened: VK_ERROR_DEVICE_LOST GL_ARB_robustness Non-robust GL apps are just killed Applications then can recreate the context 10
- 11. What happens in practice? DRM <-> Mesa it's not really hw specific How about we have a DRM_GET_RESET_STATE? WIP: A DRM documentation explaining what DRM drivers and usermode drivers (Mesa) should do when a reset happens, with a DRM IOCTL to query the reset 11
- 12. What happens in practice? Each vendor reacts differently to resets My focus is on amdgpu The state was that it would be unrecoverable for any kind of reset Just a black screen and not responsive. Access via ssh/tty sometimes worked Pierre-Eric (AMD) and I fixed this for KDE compositor radeonsi wasn't following the spec More testing is need for robustness 12
- 13. What happens in practice? Other OSs have more control in the stack, so they can be more reliable In particular in the compositor side, so it's easier to get in a standard behavior 13
- 14. Good reporting of GPU hangs Apart from reporting to userspace that the GPU was reset, would be nice to tell what happened Currently Mesa developers have a hard time figuring out what in the game caused the hang 14
- 15. Good reporting of GPU hangs GPU hang have two main sources: Hardware settings (voltage, frequency) Application errors (infinite loops) There's no way to distinguish this right now 15
- 16. Good reporting of GPU hangs Ideally without overhead so can be enabled by default WIP: AMDGPU_INFO_GUILTY_APP to capture data about the hanged app (e.g. buffer in use) This callbacks need to be platform specific Reads some registers 16
- 17. Good reporting of GPU hangs Challenge: when the GPU hangs the hardware state can be a bit unreliable. How to get the right info correctly? Using the GPU in "debug" mode or inserting fences, barrier and extra information causes overhead No easy way to deploy to all users 17
- 18. Roadmap for better GPU resets Standardization of how DRM reports GPU hangs to userspace of how usermode driver deals with a hang and with the guilty application what compositors should do after a hang Better hang log Show which buffer caused the hang Dump hardware state reliably devcoredump 18
- 21. 21