CODA_presentation.pdf

CODA: Content-aware Frame Dropping Algorithm for High
Frame-rate Video Streaming
Vignesh V Menon, Hadi Amirpour, Mohammad Ghanbari, Christian Timmerer
Christian Doppler Laboratory ATHENA, Institute of Information Technology (ITEC), University of Klagenfurt, Austria
Data Compression Conference (DCC)
24 March 2022
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 1

Outline
1 Introduction
2 CODA
3 Evaluation
4 Conclusions and Future Directions

Introduction
Introduction

Introduction
Introduction
High Framerate (HFR) videos
High Framerate (HFR) video streaming enhances the viewing experience and improves
visual clarity.1
However, it may lead to an increase of both encoding time complexity and compression
artifacts, particularly at lower bitrates.
1
ITU-R BT.2020-2. “Parameter values for ultra-high definition television systems for production and international programme exchange”. In: 2015.

Introduction
Introduction
0.2 0.5 1.2 4.5 16.8
Bitrate (in Mbps)
40
50
60
70
80
90
VMAF
120fps
60fps
30fps
24fps
(a) HoneyBee
0.2 0.5 1.2 4.5 16.8
Bit-rate (in Mbps)
30
40
50
60
70
80
VMAF
120fps
60fps
30fps
24fps
(b) Lips
Figure: Rate-Distortion (RD) curves of UHD encodings of (a) HoneyBee and (b) Lips sequences for
multiple framerates.

Introduction
Introduction
Variable Framerate (VFR) coding
Input
Video
. Temporal
Downsampling
Framerate Selection
Encoder Decoder
Temporal Up-
sampling
Display
f
d ∈ D
ˆ
f = f ∗ (1 − d) ˆ
f
f =
ˆ
f
1−d
Figure: Block diagram of a variable framerate (VFR) coding scheme2
in the context of video encoding.
f and ˆ
f denote the original framerate of the video and the framerate at which the video is encoded. d
represents the frame dropping factor.
2
G. Herrou et al. “Quality-driven Variable Frame-Rate for Green Video Coding in Broadcast Applications”. In: IEEE Transactions on Circuits and Systems for
Video Technology (2020), pp. 1–1. doi: 10.1109/TCSVT.2020.3046881.

CODA
CODA

CODA Phase 1: Feature Extraction
CODA
Phase 1: Feature Extraction
Compute texture energy per block
A DCT-based energy function is used to determine the block-wise feature of each frame
defined as:
Hk =
w
X
i=1
h
X
j=1
e|( ij
wh
)2−1|
|DCT(i − 1, j − 1)| (1)
where w and h are the width and height of the block, and DCT(i, j) is the (i, j)th DCT
component when i + j > 2, and 0 otherwise.
The energy values of blocks in a frame is averaged to determine the energy per frame.3
3
Michael King, Zinovi Tauber, and Ze-Nian Li. “A New Energy Function for Segmentation and Compression”. In: 2007 IEEE International Conference on
Multimedia and Expo. 2007, pp. 1647–1650. doi: 10.1109/ICME.2007.4284983.

CODA Phase 1: Feature Extraction
Proposed Algorithm
Phase 1: Feature Extraction
hk: SAD of the block level energy values of frame k to that of the previous frame k − 1.
hk =
SAD(Hk(i) − Hk−1(i))
M
(2)
where M denotes the number of CTUs in frame k.

CODA Phase 2: Framerate prediction
CODA
Phase 2: Framerate prediction
Inputs:
E, h : spatial and temporal complexities
r : video resolution
fmax : original framerate
D : set of all frame drop factors ˜
d
B : set of all target bitrates b (in kbps)
Output: ˆ
f (b) ∀ b ∈ B
Step 1: Determine ˆ
d(b).
ˆ
d(b) = d0e−
βMA(r,fmax )·h·b
E
Step 2: The predicted optimized framerate for the video is given by:
ˆ
f (b) = fmax · (1 − ˆ
d(b))

Evaluation
Evaluation

Evaluation
Evaluation
E and h values are extracted using VCA open-source software.4
0.2 0.5 1.2 4.5 16.8
Bit-rate (in Mbps)
20
40
60
80
100
120
Frame-rate
Ground truth (fG)
Predicted (f)
(a)
0.2 0.5 1.2 4.5 16.8
Bit-rate (in Mbps)
20
40
60
80
100
120
Frame-rate
Ground truth (fG)
Predicted (f)
(b)
Figure: Optimized framerate prediction results of Beauty (a) and ShakeNDry (b) sequences. Please
note that, depending on the content, the optimized framerate in various bitrates are different.
4
V. V Menon, C. Feldmann, and H. Amirpour. VCA. Version 1.0.0. Available: https://github.com/cd-athena/VCA. Feb. 2022.

Evaluation
Evaluation
0.2 0.5 1.2 4.5 16.8
Bit-rate (in Mbps)
30
40
50
60
70
VMAF
Original (120fps)
CODA (VFR)
(a)
0.2 0.5 1.2 4.5 16.8
Bit-rate (in Mbps)
40
50
60
70
VMAF
Original (120fps)
CODA (VFR)
(b)
Figure: Rate-Distortion (RD) results of Beauty (a) and ShakeNDry (b) sequences for the default
encoding and CODA-based VFR encoding.

Conclusions and Future Directions

Conclusions
Presented a content-aware frame dropping algorithm for video streaming applications, es-
pecially for HFR videos.
Predicts the optimized framerate for a set of bitrates defined in a bitrate ladder and res-
olution for every video, which helps in improving the overall performance of HFR video
streaming in terms of encoding time and compression efficiency.
UHD encoding using the proposed algorithm requires 15.87% fewer bits to maintain the
same PSNR and 18.20% fewer bits to maintain the same VMAF as compared to the original
framerate encoding. An overall encoding time reduction of 21.82% is also observed.

Q & A
Thank you for your attention!
Vignesh V Menon (vignesh.menon@aau.at)
Hadi Amirpour (hadi.amirpourazarian@aau.at)
Mohammad Ghanbari (ghan@essex.ac.uk)
Christian Timmerer (christian.timmerer@aau.at)

CODA_presentation.pdf

More Related Content

Similar to CODA_presentation.pdf (20)

More from JunZhao68 (20)

Recently uploaded (20)

CODA_presentation.pdf