Video complexity analyzer (VCA) for streaming applications
0 likes1,067 views
The document presents the Video Complexity Analyzer (VCA), designed to develop online prediction systems optimized for live streaming applications, focusing on the spatial and temporal complexity of video encoding. The VCA aims to address the high computational time of existing features, making it suitable for real-time applications, and highlights experimental results showing its effectiveness in shot detection and faster feature computation. Future plans include enhancing the system with multi-threading and CUDA/OpenCL support for improved performance.
![5
Motivation
● We aim to develop online prediction systems tailor-made for live streaming applications.
● The state-of-the-art spatial and temporal complexity feature is SI-TI.
Fig.1: Correlation of SI feature with number of bits (in kb) per frame in IDR encoding with QP27 of x265 for 24 test sequences from MCML[1] and
SJTU[2] dataset.
[1] M. Cheon and J.-S. Lee, “Subjective and Objective Quality Assessment of Compressed 4K UHD Videos for Immersive Experience,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 28,
no. 7, pp. 1467–1480, 2018.
[2] L. Song, X. Tang, W. Zhang, X. Yang, and P. Xia, “The SJTU 4K Video Sequence Dataset,”Fifth International Workshop on Quality of Multimedia Experience (QoMEX2013), Jul. 2013.
Pearson correlation coefficient (PCC) of SI with bits per frame is ~0.79.](https://image.slidesharecdn.com/vqegnorm2021209-211220215147/85/Video-complexity-analyzer-VCA-for-streaming-applications-5-320.jpg)
![6
Motivation
● Time taken to compute SI-TI features is very high!
➢ ~0.05 seconds per frame for 1080p, ~0.2 seconds per frame for 2160p
➢ Not suitable for live applications
➢ Higher computational cost in VoD applications
Fig.2: Time taken to compute SI-TI features [1] in Intel Xeon Gold 5218R
[1] SITI source code: https://github.com/Telecommunication-Telemedia-Assessment/SITI](https://image.slidesharecdn.com/vqegnorm2021209-211220215147/85/Video-complexity-analyzer-VCA-for-streaming-applications-6-320.jpg)
Video complexity analyzer (VCA) for streaming applications
- 1. All rights reserved. ©2020 All rights reserved. ©2021 Video complexity analyzer (VCA) for streaming applications December 14, 2021 Presenters: Vignesh V Menon & Hadi Amirpour 1
- 2. All rights reserved. ©2020 Vignesh V Menon Hadi Amirpour Researchers @ Christian Doppler Laboratory ATHENA, University of Klagenfurt, Austria All rights reserved. ©2021 2 About Us
- 3. All rights reserved. ©2020 ● Motivation for VCA ● Features ● Experimental Results ● Applications ● Future Roadmap All rights reserved. ©2021 3
- 4. All rights reserved. ©2020 Motivation All rights reserved. ©2021 4
- 5. 5 Motivation ● We aim to develop online prediction systems tailor-made for live streaming applications. ● The state-of-the-art spatial and temporal complexity feature is SI-TI. Fig.1: Correlation of SI feature with number of bits (in kb) per frame in IDR encoding with QP27 of x265 for 24 test sequences from MCML[1] and SJTU[2] dataset. [1] M. Cheon and J.-S. Lee, “Subjective and Objective Quality Assessment of Compressed 4K UHD Videos for Immersive Experience,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, no. 7, pp. 1467–1480, 2018. [2] L. Song, X. Tang, W. Zhang, X. Yang, and P. Xia, “The SJTU 4K Video Sequence Dataset,”Fifth International Workshop on Quality of Multimedia Experience (QoMEX2013), Jul. 2013. Pearson correlation coefficient (PCC) of SI with bits per frame is ~0.79.
- 6. 6 Motivation ● Time taken to compute SI-TI features is very high! ➢ ~0.05 seconds per frame for 1080p, ~0.2 seconds per frame for 2160p ➢ Not suitable for live applications ➢ Higher computational cost in VoD applications Fig.2: Time taken to compute SI-TI features [1] in Intel Xeon Gold 5218R [1] SITI source code: https://github.com/Telecommunication-Telemedia-Assessment/SITI
- 7. 7 Motivation Video Complexity Analyzer (VCA) can be realized as a fast preprocessor which determines the spatial and temporal complexity of videos (segments) to aid the encoding process. Fig.3: The proposed framework for streaming applications.
- 8. All rights reserved. ©2020 Features All rights reserved. ©2021 8
- 9. 9 Spatial complexity feature k is the block address in the pth frame,w×w pixels is the size of the block, and DCT(i, j) is the (i, j)th DCT component when i+j >1, and 0 otherwise.
- 10. 10 Spatial complexity feature C represents the number of blocks per frame. Fig. 4: Number of bits (in kb) per frame and E feature of Wood SJTU sequence.
- 11. 11 Temporal complexity feature The block-wise SAD of the texture energy of each frame (p) compared to its previous frame (p-1) is computed.
- 12. All rights reserved. ©2020 Experimental Results All rights reserved. ©2021 12
- 13. 13 Results PCC(SI, Bits per frame) = 0.787 PCC(E, Bits per frame) = 0.856 Fig. 5: Correlation of SI and E features with number of bits (in kb) per frame.
- 14. 14 Results Fig. 6: Average time to compute E-h for various resolutions (with x86 SIMD). Note: Presently, E-h computation is 5 times faster than SI-TI computation.
- 15. All rights reserved. ©2020 Applications All rights reserved. ©2021 15
- 16. 16 Shot Detection We define the gradient of ‘h’ per frame ‘p’ as: Fig. 7: Epsilon values for ToS sequence. Please note that shot transitions happen at frames: 107, 110, 238,338,465,531,596,778,850,917,1018,1361,1437.
- 17. 17 Shot Detection The algorithm is classified into two steps: ● Feature extraction ● Successive Elimination Algorithm Source: V. V. Menon, H. Amirpour, M. Ghanbari, and C. Timmerer,“Efficient Content-Adaptive Feature-Based Shot Detection for HTTP Adaptive Streaming,” in 2021 IEEE International Conference on Image Processing (ICIP), 2021, pp. 2174–2178. Benchmark is the default shot detection algorithm of x265.
- 18. All rights reserved. ©2020 Future Roadmap All rights reserved. ©2021 18
- 19. 19 Future Roadmap ● The initial version will be released before March 1, 2022. ● Adding Multi-threading support ○ H computation for blocks in each frame can be realized concurrently. ○ ~6x speedup expected with 8 threads. ● Adding CUDA/ OpenCL support
- 20. All rights reserved. ©2020 Thanks for your attention! All rights reserved. ©2021 20 Vignesh V Menon (vignesh.menon@aau.at) Hadi Amirpour (hadi.amirpour@aau.at)