Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementations

Gain of Grain: A Film Grain Handling Toolchain
for VVC based Open Implementations
—
Vignesh V Menon, Postdoctoral Researcher, Fraunhofer HHI
Feb. 12, 2024, Mile High Video (MHV‘24)

MHV’24
Introduction
12.02.2024 © Fraunhofer
Slide 2
Film grain in video coding
• Film grain, an inherent characteristic of analog film, contributes to the unique visual aesthetics and cinematic experience in movies [1].
• The emergence of the Versatile Video Coding (VVC) [2, 3] standard brings new opportunities and challenges in efficiently representing film
grain, aiming to preserve its artistic value while ensuring compatibility with modern video compression techniques.
• Compression artifacts like blockiness are introduced when grainy video content is encoded at low bitrates. Film grain presents a challenge in
video coding due to its random and non-uniform nature, which can amplify compression artifacts if not adequately handled during
encoding.
[1] Inseong Hwang et al. “Enhanced Film Grain Noise Removal for High Fidelity Video Coding”. In: 2013 International Conference on Information Science and Cloud Computing Companion. 2013, pp. 668–674. doi: 10.1109/ISCC-
C.2013.69.
[2] Benjamin Bross et al. “Overview of the Versatile Video Coding (VVC) Standard and its Applications”. In: IEEE Transactions on Circuits and Systems for Video Technology. Vol. 31. 10. 2021, pp. 3736–3764. doi:
10.1109/TCSVT.2021.3101953
[3] Adam Wieckowski et al. “A Complete End to End Open Source Toolchain for the Versatile Video Coding (VVC) Standard”. In: Proceedings of the 29th ACM International Conference on Multimedia. New York, NY, USA:
Association for Computing Machinery, 2021, 3795–3798. isbn: 9781450386517. doi: 10.1145/3474085.3478320.

MHV’24
Introduction
Slide 3
Film grain in video coding
Figure: Illustration of compression artifacts introduced due to film grain when encoded using VVenC encoder
at low bitrates.
▪ Film grain is observed to amplify compression artifacts if
not adequately handled during encoding.
▪ Integrating film grain handling into VVC requires
adherence to standardized encoding practices while
accommodating unique film grain characteristics [4].
[4] Zoubida Ameur, Wassim Hamidouche, Edouard François, Miloš Radosavljević, Daniel Ménard, and Claire-Hélène Demarty, “Deep-Based Film Grain Removal and Synthesis,” IEEE Transactions on Image Processing, vol. 32, pp. 5046–
5059, 2022.

MHV’24
State-of-the-art toolchain
Slide 4
Components:
▪ Denoising: reducing or removing noise from the video, including film grain.
▪ Encoding: compressing the video data for efficient storage and transmission.
▪ Decoding: reconstructing the video data from the compressed format.
▪ Grain synthesis: generating or enhancing film grain in the video.

MHV’24
Proposed toolchain
Slide 5

MHV’24
Proposed toolchain
Slide 6
MCTF in VVenC:
▪ VVenC already employs a denoising stage [5], based on a framework proposed
initially in [6]– motion compensated temporal (pre-)filtering.
▪ The filter performs a blockwise motion search for each filtered frame in
neighboring frames to remove the noise.
▪ Using up to eight predictors, a weighted average of the current frame block and
its predictors is generated and used for further encoding.
▪ Improved search strategies, reference number reduction, and flexible block size
were introduced, improving the filter runtime and operation [5].
Denoising
[5] Adam Wieckowski, Tobias Hinz, Christian R. Helmrich, Benjamin Bross, and Detlev Marpe, “An optimized temporal filter implementation for practical applications,” in 2022 Picture Coding Symposium (PCS), 2022, pp. 247–251.
[6] Jack Enhorn, Rickard Sjöberg, and Per Wennersten, “A Temporal Pre-Filter For Video Coding Based On Bilateral Filtering,” in 2020 IEEE International Conference on Image Processing (ICIP), 2020, pp. 1161–1165.

MHV’24
Proposed toolchain
Slide 7
▪ The framework for film grain handling based on frequency filtering and
parameterization of grain is implemented in VVenC (to be released soon).
▪ The film grain pattern is characterized using a horizontal high cut-off
frequency and a vertical high cut-off frequency obtained in the discrete cosine
transform domain.
▪ The film grain pattern is scaled to the appropriate level using a stepwise
scaling function that considers the underlying image’s characteristics.
▪ Afterward, the film grain pattern is blended into the image using additive
blending [7].
Film grain estimation
[7] M. Sean, Y. Peng, H. Walt, P. Fangjun, L. Taoran, C. Tao, M. Radosavljević, E. François, G. Vijayakumar, P. Kaustubh et al., “Fixed-point grain blending process for film grain characteristics sei message,” in Joint Video Experts Team
(JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, document JVET-W0095, 2021.

MHV’24
Proposed toolchain
Slide 8
▪ To convey the grain parameters to the decoder, the encoder embeds it as
supplemental enhancement information (SEI) in the bitstream [8], as the "Film
Grain SEI.“
▪ Film Grain SEI inherits the same syntax and semantics of the AVC film grain
SEI message [9].
▪ Since we implement a frequency filtering model for film grain estimation,
film_grain_model_id is set to 0.
▪ Additive blending is used when blending_mode_id is set to 0.
▪ Since our implementation analyses film grain for only the luma channel,
comp_model_present_flag[0] is set to 1.
▪ FGC SEI message is inserted at each frame, which is indicated by setting the
film_grain_characteristics_persistence_flag to 0. It also means the FGC SEI
message only applies to the current decoded frame.
Film Grain SEI
[8] “RDD 5:2006 - SMPTE Registered Disclosure Doc - Film Grain Technology —Specifications for H.264 | MPEG-4 AVC Bitstreams,” RDD 5:2006, pp. 1–18, 2006.
[9] Vijayakumar Gayathri Ramakrishna, Kaustubh Shripad Patankar, and Mukund Srinivasan, “Cloud-Based Workflow for AVC Film Grain Synthesis,” ser. MHV’23. New York, NY, USA: Association for Computing Machinery, 2023, p.
66–71.

MHV’24
Experimental setup
Slide 9
Comparisons:
▪ We run experiments on an AMD EYPC 7502P processor (32 cores),
where we run each VVenC v1.10 instance using four CPU threads,
with two-pass rate control, enabling adaptive quantization.
▪ All sequences are down-scaled to 1920x1080 8bit.
▪ We consider the Default toolchain as the benchmark, where :
▪ Encode the input video using VVenC with MCTF disabled.
▪ Decode the resulting bitstream using the VTM decoder.
▪ FGA and FGS are disabled in this toolchain.
Table: Experimental parameters used to evaluate the proposed film grain handling
toolchain.

MHV’24
Quality assessment metrics
Slide 10
Observations:
▪ Traditional metrics like PSNR and SSIM are not suitable for evaluating
the perceptual quality of film grain coding owing to their lack of
texture sensitivity.
▪ PSNR and SSIM are sensitive to noise, such that they penalize the
addition of synthesized film grain.
▪ VMAF [10], while more advanced, is not trained to evaluate the
perceptual quality of VVC-coded videos [11]. Figure: RD curves using the Default toolchain, MCTF (without FGS), and the
proposed toolchain for IntoTree test sequence encoded at faster preset.
[10] Zhi Li, Christos Bampis, Julie Novak, Anne Aaron, Kyle Swanson, Anush Moorthy, and Jan De Cock, “VMAF: The journey continues,” in Netflix Technology Blog, vol. 25, 2018.
[11] Christian R. Helmrich, Benjamin Bross, Jonathan Pfaff, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand, “Information on and analysis of the VVC encoders in the SDR UHD verification test,” in WG 05 MPEG Joint Video
Coding Team(s) with ITU-T SG 16, document JVET-T0103, Oct. 2020.
Take aways:
▪ Given these limitations, specialized metrics focusing on texture enhancement, perception of controlled noise, and overall film-like appearance
would be more appropriate for evaluating film grain coding, subject to future work.
▪ Metrics that include human perception aspects and consider texture fidelity alongside noise would offer a better assessment of the quality
enhancements film grain brings to video content.

MHV’24
Subjective quality assessment
Slide 11
▪ Original ▪ No MCTF (250 kbps)
▪ With the proposed toolchain (250 kbps)
▪ MCTF (250 kbps)

MHV’24
Subjective quality assessment
Slide 12
▪ Original ▪ MCTF (3 Mbps) ▪ With the proposed toolchain (3 Mbps)

MHV’24
Runtime complexity
Slide 13
Encoding and decoding
Encoding time:
▪ FGA contributes to the increased relative duration required for encoding
as the preset progresses towards faster configuration.
▪ The overall encoding time using the proposed toolchain reduces up to
11.59 % using slower preset.
Table: Runtime complexity of the proposed toolchain compared to
the default toolchain with the same preset.
[13] SVT. “The SVT High Definition Multi Format Test Set”. In: Feb. 2006. url: https://tech.ebu.ch/docs/hdtv/svt-multiformat-conditions-v10.pdf
[14] Li Song et al. “The SJTU 4K video sequence dataset”. In: 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX). 2013, pp. 34–35. doi: 10.1109/QoMEX.2013.6603201. url:
https://doi.org/10.1109/QoMEX.2013.6603201.
[15] Jill Boyce et al. JVET-J1010: JVET common test conditions and software reference configurations. July 2018. url: https://www.researchgate.net/publication/326506581
Decoding time:
▪ The decoding time has increased due to the supplementary
computational load of FGS.
▪ The effect on decoding time is intertwined with the encoding presets;
as encoding presets progress towards slower configurations, the
proportional increase in decoding time tends to diminish.

MHV’24
Conclusions and future directions
Slide 14
Conclusions
▪ We presented an overview of the film grain handling toolchain for VVC-based open implementation using the VVenC encoder and VTM
decoder.
▪ The experimental results show that the proposed toolchain improves the subjective quality of the grainy video content encoded at multiple
bitrates, compared to default toolchain, considering VVenC encoding.
▪ Using slower preset, the proposed toolchain reduces the encoding duration by up to 11.59% while the decoding time increases by up to
15.32%.
▪ These outcomes underscore the inherent trade-offs in optimizing the film grain handling toolchain, emphasizing the need for a balanced
approach that prioritizes encoding efficiency and video quality.
Future directions
▪ Denoising and film grain estimation shall be tuned for various encoding presets in VVenC.
▪ More sophisticated models shall be investigated to represent film grain accurately in the digital domain. This could involve advanced statistical
models and/or machine learning approaches to capture the intricate characteristics of film grain.
▪ Develop quality assessment metrics that better capture the human perception of film grain.

Thank you for your attention
— ▪ Vignesh V Menon (vignesh.menon@hhi.fraunhofer.de)
▪ Adam Wieckowski (adam.wieckowski@hhi.fraunhofer.de)
▪ Jens Brandenburg (jens.Brandenburg@hhi.Fraunhofer.de)
▪ Benjamin Bross (benjamin.bross@hhi.fraunhofer.de)
▪ Thomas Schierl (thomas.schierl@hhi.fraunhofer.de)
▪ Detlev Marpe (detlev.marpe@hhi.fraunhofer.de)

Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementations

More Related Content

Similar to Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementations (20)

More from Vignesh V Menon (20)

Recently uploaded (20)

Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementations