Embedded Recipes 2017 - Proper APIs to HW video codec accelerators - Olivier Crete
3 likes175 views
The document discusses the development of better APIs for video codec accelerators, outlining the different types of codecs, users, and use cases such as broadcast production and video calls. It emphasizes the importance of various exchange formats, memory allocation, metadata, and performance requirements for optimal processing and low latency. Additionally, the document critiques existing solutions like OpenMAX IL, libv4l, va-api, and GStreamer, proposing a new design inspired by the LV2 audio plugin API for improved functionality.
![Exchange formats: Encoded
●
Packetization
– Frame
– Subframe
●
Variants
– H.264 AVC: AVC[1-4], byte-stream
Embedded Recipes 2017 9](https://image.slidesharecdn.com/codec-apisembeddedrecipes2017-171007124133/85/Embedded-Recipes-2017-Proper-APIs-to-HW-video-codec-accelerators-Olivier-Crete-9-320.jpg)
Embedded Recipes 2017 - Proper APIs to HW video codec accelerators - Olivier Crete
- 1. Open First Better APIs for Video Codec accelerators Olivier Crête
- 2. Who am I ? At Collabora since 2007 Close to 5 years on VVoIP for Nokia Maemo / Meego GStreamer core maintainer Multimedia Lead Embedded Recipes 2017 2
- 3. Type of codecs ● Software ● Hardware – Small CPU/GPU to parse bitstream ● Hardware Accelerators – Need CPU software ● USERSPACE LIBRARY! Embedded Recipes 2017 3
- 5. Use cases ● Broadcast production – High quality, high bit-rate, high expectations ● User generated content – Medium quality, low expectations ● Physical security – Low quality, high latency, low cost Embedded Recipes 2017 5
- 6. Use cases ● Broadcast production – High quality, high bitrate ● End user playback – Mobile – TV / STB Embedded Recipes 2017 6
- 7. Use cases ● Video calls – Latency, latency, low bitrate ● Transcoding – Live: Latency! – Stored: Quality per bit Embedded Recipes 2017 7
- 8. ● Exchange formats – Encoded – Raw ● Memory allocation & registration ● Metadata ● Long term maintainability Embedded Recipes 2017 8 Requirements
- 9. Exchange formats: Encoded ● Packetization – Frame – Subframe ● Variants – H.264 AVC: AVC[1-4], byte-stream Embedded Recipes 2017 9
- 10. Exchange formats: Raw content ● Subsampling: 4:4:4, 4:2:2, 4:2:0, etc ● Color space: BT.601, BT.709, BT.2020, BT.2100 – Critical for HDR Embedded Recipes 2017 10
- 11. Exchange formats: Raw memory layout ● Planar vs packed ● Multiple planes – Multiple DMAbuf fds ● Alignment requirements ● Tiled formats ● Opaque, possible compressed formats ● Padding, etc Embedded Recipes 2017 11
- 12. Memory allocation & registration ● External vs Internal allocation – Import vs Export ● Types of memory – DMAbuf – malloc() Olivier Crête 12
- 13. Memory pre-allocation & dynamic allocation ● Pre-registration – No run-time allocation – Hard real time ● Dynamic allocation/registration – Dynamic pipeline modifications Embedded Recipes 2017 13
- 14. Metadata ● Per frame metadata – Timestamps and timecodes – Vertical Ancillary data (VANC): AfD, etc ● Inter-frame data – SCTE-35/104 ● Ad insertion points Embedded Recipes 2017 14
- 19. OpenMAX IL ● Android's subset ● Minimal largely incomplete vendor implementations ● Dead upstream , Khronos WG has not met in years“ ” ● Spec lacks important features leading to fragmentation ● Forces threading and allocation model ● Whole framework Embedded Recipes 2017 19
- 20. libv4l ● “Transparent wrapper over kernel API” ● Similar limitations to v4l API – Tied to kernel API rules – Still breaks dmabuf/userptr ● Only limited maintenance Embedded Recipes 2017 20
- 22. GStreamer ● Whole framework ● Forces specific way of working – Threading – Pipeline formation – Allocations ● Not designed for low latency Embedded Recipes 2017 22
- 24. Simple Plugin API (SPA) Pipewire Embedded Recipes 2017 24
- 25. The solution I wish I had designed ● Designed – for Pipewire – by Wim Taymans, GStreamer architect Embedded Recipes 2017 25
- 27. Thank you!