Rethinking Data Augmentation for Image Super-resolution: A Comprehensive Analysis and a New Strategy
0 likes647 views
The document presents a comprehensive analysis of data augmentation methods for image super-resolution, highlighting their impact on performance and identifying limitations such as sharp transitions and pixel relationship loss. It introduces a new augmentation strategy called 'cutmixup' that preserves image structure while improving results. Additionally, simple methods like blending and RGB channel permutation show significant enhancement without altering image integrity.
Rethinking Data Augmentation for Image Super-resolution: A Comprehensive Analysis and a New Strategy
- 1. Rethinking Data Augmentation for Image Super-resolution : A Comprehensive Analysis and a New Strategy; CutBlur & MoA Jaejun Yoo*, Namhyuk Ahn*, and Kyung-Ah Sohn
- 2. Motivation * Various levels of vision tasks High-level: Semantic recognition (e.g., classification, object detection) Mid-level: Super-pixel (e.g., segmentation) Low-level: Pixels, Edges, Colors (e.g., super-resolution, denoising) ????? MixUp, Cutout, CutMix …
- 3. Analysis on existing DA methods “Sharp transitions, mixed image contents or losing the relationships of pixels can degrade SR performance.” e.g., Cutout fails (discarding pixels) and every feature method fails (manipulation). Training curves when applied feature DA’s
- 4. Analysis on existing DA methods • DA methods in pixel space bring some improvements when applied very carefully.
- 5. Analysis on existing DA methods • DA methods in pixel space bring some improvements when applied very carefully. • Cutout: Original setting (drop 25% of of pixels in a rectangular shape) significantly degrades the performance because it erases spatial information too much. However, erasing tiny amount of pixels (0.1% random pixels) boosts the performance (2~3 pixels of 48x48 input patch) Cutout
- 6. Analysis on existing DA methods • DA methods in pixel space bring some improvements when applied very carefully. • Mixup & CutMix: Improvements of using CutMix are marginal. We suspect this happens because CutMix generates a drastic sharp transition between two different images. Improvements of using Mixup is better than CutMix but it still generates unrealistic image and affects to the image structure. Mixup CutMix
- 7. Analysis on existing DA methods • DA methods in pixel space bring some improvements when applied very carefully. • CutMixup: To verify our hypothesis, we combine benefits of Mixup and CutMix; CutMixup. CutMixup provides various boundary cases while minimizes the sharp transition by retaining partial cues as Mixup does. CutMixup
- 8. Analysis on existing DA methods • DA methods in pixel space bring some improvements when applied very carefully. • Blend & RGB permutation: To push further, we tried a constant blending and RGB channel permutation, which turn out to be very simple but effective strategies showing big performance enhancement (dB). Note that both methods do not incur any structure modification to an image. BlendRGB perm.
- 9. CutBlur
- 10. CutBlur • What does the model learn from CutBlur? • CutBlur prevents the SR model from over-sharpening an image and helps it to super-resolve only the necessary region. Super-resolution results of a model (EDSR) trained without CutBlur and its error residual (Δ) Error residual (Δ)Output
- 11. CutBlur • What does the model learn from CutBlur? • CutBlur prevents the SR model from over-sharpening an image and helps it to super-resolve only the necessary region. Super-resolution results of a model (EDSR) trained CutBlur and its error residual (Δ) Error residual (Δ)Output with
- 12. Improved generalization: over-sharpening • Super-resolution (SR) • Trained on ×4 scale factor dataset and tested on different scale factor (×2)
- 13. Improved generalization: over-smoothing • Denoising • Trained on severe noise (! = 70) & tested on mild noise (! = 30).
- 14. Improved generalization: over-removal • JPEG artifact removal • Trained on a mild compression rate & tested on a severe compression rate
- 15. Mixture of Augmentation (MoA) • During the training phase … • Randomly select single augmentation at every step. (among the curated DA list) • Apply it!
- 16. Comparison on diverse benchmark models and datasets • SRCNN (0.07M) – ECCV’14, CARN (1.14M) – ECCV’18, RCAN (15.6M) – ECCV’18, EDSR (43.1M) – CVPRW’17 • DIV2K (synthetic), RealSR (real-world) • Our method shows consistent improvement for different models (parameters) and datasets (different environments and size):
- 17. Code: https://github.com/clovaai/cutblur Paper: https://arxiv.org/abs/2004.00448 QR code for code & paper For more details, please visit our website: