Implementing a Vulkan Video Encoder From Mesa to GStreamer
0 likes119 views
The document discusses the development of a Vulkan video encoder from Mesa to GStreamer, highlighting the utilization of stateless codecs for hardware-accelerated video processing. It covers the timeline of Vulkan video development, the status of encoding support for H.264 and H.265 on Intel and AMD GPUs, and the challenges faced in the process. Additionally, it outlines plans for future support of AV1 encoding and improvements in synchronization and memory management for better performance.
Implementing a Vulkan Video Encoder From Mesa to GStreamer
- 1. A Vulkan Video Encoder from Mesa to GStreamer Hyunjun Ko / Stéphane Cerveau 2024-01-06 1
- 2. 2
- 3. Agenda 1. Vulkan Video 2. Mesa - driver 3. GStreamer - application 4. Demos 3
- 4. Vulkan Video Stateless codecs using GPU hardware acceleration Supported codecs: H.264, H.265, AV1 Closer integration with Graphics and Displays. Cross-platform and vendor-neutral low-level HW state(- less) video codecs API Each driver can operate differently depending on its capabilities with a common API. 4
- 5. Vulkan Video Timeline March 2018: TSG was created and driven by IHVs such as AMD/Intel/Nvidia and open source operators April 2021: including the Video Decode and Encode extensions January 2023: Video Extensions for Accelerated H.264 and H.265 Decode December 2023: Khronos Vulkan Video Extensions for Accelerated H.264 and H.265 Encode Provisional extensions released released finalized 5
- 8. The status of Vulkan Video Encoder Support in Mesa project 8
- 9. Contents What is Mesa? History of Vulkan Video development in Mesa Drivers supporting Vulkan Video in Mesa Development of Vulkan Video Encoding on Intel GPUs Challenges Plan 9
- 10. What is Mesa? Began as an open source implementation of the OpenGL. Now actively implementing Vulkan specification on various GPUs. Intel, AMD, Qualcomm Adreno(R/E), Raspberry PI, etc.. Contributors: Igalia, Intel, Google, Collabora, Mesa community. https://mesa3d.org/ https://gitlab.freedesktop.org/mesa/mesa 10
- 11. History of Vulkan Video development in Mesa Dave Airlie started in 2022 on AMD and Intel GPU(RADV and ANV) With on FFmpeg. Igalia joined on GStreamer later. Hyunjun joined in 2023, started working on Intel GPU(ANV) So Dave fully dedicated to AMD GPU(RADV). Dave's blog post Lynne 11
- 12. The status of Vulkan Video development in Mesa Implemented and landed decoder first for h264 and h265. Now MR for h264/5 encoding are almost ready. ANV H264 MR #20782 ANV H265 MR #22202 RADV H264/265 MR #20388 RADV MR ANV Branch 12
- 13. Drivers supporting Vulkan Video in Mesa GPU H264 dec H265 dec H264 enc H265 enc Intel(ANV) O O WIP WIP AMD (RADV) O O Ready Ready 13
- 14. Working on Intel GPUs (1) Dived into Intel Vulkan driver(ANV) in 2023. Started working on H265 decoding first. Tons of documents and source code of Intel VAAPI drivers. Exhausted to learn lots of Video commands. But better than nothing :) 14
- 15. Working on Intel GPUs (2) H265 Encoding Sequnce 15
- 16. Working on Intel GPUs (3) Complete each command very carefully. Otherwise you got a GPU hang or even whole system down. When you get a GPU hang and don't see any clue. Dumping whole video commands encoding a frame(by VAAPI Driver) into a file. Compare to commands that you created 16
- 17. Working on Intel GPUs (4) Thanks to the existing infrastrutures of ANV Easy to handle memories and images. Thanks to ANV maintainers. They actively reviewed relevant merge requests. 17
- 18. Co-working with Applications GStreamer, VK CTS, FFMpeg... Each uses different parameters and makes it find bugs easily. Different resolution, profile, SPS, PPS parameters. 18
- 19. Challenges GPU hang. Not enough useful tools to investigate. Lots of generations of Intel GPUs. Different commands, parameters, memory size, alignment, etc... 19
- 20. Plan 2024 Land h264/h265 encoding support. AV1 support. Support other GPUs? 20
- 21. a 20-year-old framework for streaming media applications. Black boxes interconnection system Native, multiplatform, highly-optimized framework 21
- 23. Vulkan Video support Follow Vulkan H.264 decoder in December 2023 Vulkan H.264/H.265 encoder . Vulkan Video Status merged under review 23
- 24. State machine 24
- 25. Challenges 25
- 26. Cross platform API Hardware crashes, Thanks Validation Layers! Exact behavior varies by hardware vendor Rate control and quality issues 26
- 27. Synchronization Major issues with both decoder and encoder Old memories from Vulkanised 2023, green screen... Rework of GStreamer state machine with memory barriers, fences. GstVulkanOperation to handle commands synchronization. 27
- 28. DPB management Understand the correct use of Begin and Encode reference slots. Need to declare the reference within Begin command and use it during the Encode command. Various crashes in drivers not detected by the Validation Layers when the standard H.26x parameters (SPS, slice header) were not filled properly. Vulkan reference slots management. 28
- 29. Vulkan tooling Validation Layers: Help validate we understand the specifications correctly Do not prevent a misconfiguration of the std parameters Mesa drivers helped to understand driver's pitfalls when VL was clear. GFXReconstruct, VK_LAYER_LUNARG_api_dump layer. CTS: Help with a reference design reviewed by IHVs. 29
- 30. Demos 30
- 31. Questions ? 31
- 33. 33