Resolution enhancement of low-quality videos using a high-resolution frame
Download as PPT, PDF2 likes1,052 views
The document presents a method for enhancing the resolution of low-quality videos using super-resolution (SR) techniques in the discrete cosine transform (DCT) domain. It discusses challenges posed by compression artifacts, outlines an example-based SR synthesis algorithm, and compares DCT-domain SR with spatial domain SR, highlighting the advantages of the former. The findings suggest that DCT-domain SR improves video quality and efficiency, making it suitable for devices with limited resources.
Resolution enhancement of low-quality videos using a high-resolution frame
- 1. Visual Communications and Image Processing 2006 SPIE paper 6077-8 Resolution enhancement of low-quality videos using a high-resolution frame Tuan Pham 1 Lucas van Vliet 1 Klamer Schutte 2 1. Quantitative Imaging Group, Delft University of Technology, The Netherlands 2. TNO Physics and Electronics Laboratory, The Netherlands
- 2. Contents 1. Influence of compression on Super-Resolution 2. Example-based SR in the DCT domain 3. Application in video compression © Tuan Pham (pham@tnw.tudelft.nl) 2
- 3. Super-Resolution reconstruction •Problems with compressed inputs: – Registration: less accurate motion vectors due to compression error – Fusion: outlier intensities due to ringing and blocking artifacts – Deblur: distorted and truncated frequency spectrum due to quantization © Tuan Pham (pham@tnw.tudelft.nl) 3
- 4. Discrete Cosine Transform •Each 8x8 image block is projected onto 64 orthogonal bases → 64 DCT coefficients per block Input image 64 DCT basis functions DCT coefficients •The coefficients are then quantized (lossy) & compressed (lossless) © Tuan Pham (pham@tnw.tudelft.nl) 4
- 5. Quantization noise and blur JPEG quality 80 Quantization noise 8 MSE of all 8x8 blocks 7 6 5 4 8 6 8 4 6 y 4 2 2 x Average DCT attenuation factor Average error variance of 8x8 blocks © Tuan Pham (pham@tnw.tudelft.nl) 5
- 6. SR by texture synthesis •Example-based SR1 aims to fill-in the missing frequencies: Hi-Res texture source Interpolation or Multi-frame Texture fusion transfer Lo-Res compressed Hi-Res blurred Super-Resolved input video intermediate video output video •We propose a DCT-domain SR synthesis algorithm which can be applied directly on the compressed video stream. © Tuan Pham al., “Example-based Super-Resolution”, IEEE Comp.Graph. & App., 2001 1. Freeman et (pham@tnw.tudelft.nl) 6
- 7. Resize versus SR in DCT domain •Image resize1 in DCT domain: – zero filling of high-frequency content followed by transcoding 8x8 8x8 8x8 DCT 0 DCT DCT enlarge transcode 8x8 DCT 0 fill-in 8x8 8x8 0 0 DCT DCT LR DCT 16x16 DCT 2x HR DCT •Super-resolution in DCT domain: – fill-in plausible high-frequency coefficients instead of zero-ing © Tuan Phamand Mitra, “Image resizing in the compressed domain using subband DCT”, CSVT, 2002 7 1. Mukherjee (pham@tnw.tudelft.nl)
- 8. Example-based SR in DCT domain •Block-wise SR synthesis in a raster-scan order: – Correspondence constraint: low-freq. coefficients of LR - HR blocks should match – Spatial continuity constraint: boundary pixels of adjacent HR blocks should match 8x8 quantized DCT SR decoded image AC DC MeanAbs + ε ×α 15 LR DCTs 33 overlapping pixels + Training data × ÷ Best Match © Tuan Pham (pham@tnw.tudelft.nl) 8
- 9. DCT-domain SR vs spatial domain SR + Texture source: CIF frame 0 Input QCIF frame 20 (JPEG quality 80) Example-based SR (Freeman) © Tuan Pham (pham@tnw.tudelft.nl) DCT-domain SR synthesis 9
- 10. DCT-domain SR vs spatial domain SR + Texture source: CIF frame 0 Input QCIF frame 50 (JPEG quality 80) © Tuan Linear Embedding SR (CVPR’04) Local Pham (pham@tnw.tudelft.nl) DCT-domain SR 10
- 11. Importance of a good texture source + input (JPEG quality 50) source from 30-frame away wrong source QCIF input © Tuan Pham (pham@tnw.tudelft.nl) good but not striking poor SR result 11
- 12. SR using updated HR frames •LR compressed video + a regularly updated HR source QCIF input at JPEG quality 80 CIF output using 1 HR frame / sec •Application: video coding for devices with limited resources © Tuan Pham (pham@tnw.tudelft.nl) 12
- 13. Conclusions •SR reconstruction of compressed image is difficult because of: – truncated frequency spectrum due to heavy quantization – space-variant quantization noise •DCT-domain SR is better than spatial domain SR because it: – produces better results – is more efficient •Application of SR synthesis using a HR frame: – video compression for systems with limited resources (phones, cameras) © Tuan Pham (pham@tnw.tudelft.nl) 13
- 14. Thank you ! Tuan Pham pham@tnw.tudelft.nl © Tuan Pham (pham@tnw.tudelft.nl) 14