[DLHacks 実装]Perceptual Adversarial Networks for Image-to-Image Transformation
2 likes2,174 views
The document discusses two papers on image-to-image translation: Pix2Pix and PAN. Pix2Pix uses a conditional GAN with adversarial and per-pixel losses. PAN builds on Pix2Pix by adding a perceptual adversarial loss using the discriminator's features. This additional loss helps the discriminator better detect differences between real and fake images. The document provides examples of tasks like deraining and inpainting that both frameworks can address and compares their results. It concludes by discussing the author's implementation of Pix2Pix and PAN and results showing Pix2Pix generally outperforming PAN, potentially due to issues like the need for per-pixel loss or patch discriminator design.
![Info
Pix2Pix [CVPR2017]
• Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros
- iGAN [ECCV 2016]
- interactive-deep-colorization [SIGGRAPH 2017]
- Context-Encoder [CVPR 2016]
- Image Quilting [SIGGRAPH 2001]
- Texture Synthesis by Non-parametric Sampling [ICCV 1999]
• University of California
• 178 citations
PAN [arXiv2017]
• Chaoyue Wang, Chang Xu, Chaohui Wang, Dacheng Tao
• University of Technology Sydney, The University of Sydney,
Universite Paris-Est
2](https://image.slidesharecdn.com/20171017dlhacks-171019082642/85/DLHacks-Perceptual-Adversarial-Networks-for-Image-to-Image-Transformation-3-320.jpg)
![Example1 : Image De-Raining
• Removing rain from single images via a deep
detail network [Fu, CVPR2017]
• ID-GAN (cGAN) [Zhang, arXiv2017]
- per-pixel loss
- adversarial loss
- pre-trained VGG’s
perceptual loss
10
Input Output
(Ground Truth)](https://image.slidesharecdn.com/20171017dlhacks-171019082642/85/DLHacks-Perceptual-Adversarial-Networks-for-Image-to-Image-Transformation-11-320.jpg)
![Example1 : Image De-Raining
• Removing rain from single images via a deep
detail network [Fu, CVPR2017]
• ID-GAN (cGAN) [Zhang, arXiv2017]
- per-pixel loss
- adversarial loss
- pre-trained VGG’s
perceptual loss
11
Input Output
(Ground Truth)
(cf. PAN uses discriminator’s
perceptual loss)](https://image.slidesharecdn.com/20171017dlhacks-171019082642/85/DLHacks-Perceptual-Adversarial-Networks-for-Image-to-Image-Transformation-12-320.jpg)
![Example2 : Image Inpainting
• Globally and Locally Consistent Image
Completion [Iizuka, SIGGRAPH2017]
• Context Encoders (cGAN) [Pathak, CVPR2016]
- per-pixel loss
- adversarial loss
12
Input Output
(Ground Truth)](https://image.slidesharecdn.com/20171017dlhacks-171019082642/85/DLHacks-Perceptual-Adversarial-Networks-for-Image-to-Image-Transformation-13-320.jpg)
![Example3 : Semantic Segmentation
Cityscape / Pascal VOC
• DeepLabv3 [Chen, arXiv2017]
• PSPNet [Zhao, CVPR2017]
http://host.robots.ox.ac.uk:8080/leaderboard/displaylb.php?
cls=mean&challengeid=11&compid=6
Cell Tracking / CREMI
• Learned Watershed
[Wolf, ICCV2017]
• U-Net
[Ronneberger, MICCAI2015]
http://www.codesolorzano.com/Challenges/CTC/Welcome.html
13
Input Output
(Ground Truth)](https://image.slidesharecdn.com/20171017dlhacks-171019082642/85/DLHacks-Perceptual-Adversarial-Networks-for-Image-to-Image-Transformation-14-320.jpg)
![Discussion
vs. No perceptual loss (Pix2Pix)
- Perceptual loss enables D to detect more
discrepancy between True/False images
vs. Pre-trained VGG perceptual loss (ID-GAN)
- VGG features tend to focus on content
- PAN features tend to focus on discrepancy
- PAN’s loss leads to avoid adversarial
examples [Goodfellow, ICLR2015] (?)
17
Why is perceptual adversarial loss so efficient?](https://image.slidesharecdn.com/20171017dlhacks-171019082642/85/DLHacks-Perceptual-Adversarial-Networks-for-Image-to-Image-Transformation-18-320.jpg)
![Result (PSNR[dB])
30](https://image.slidesharecdn.com/20171017dlhacks-171019082642/85/DLHacks-Perceptual-Adversarial-Networks-for-Image-to-Image-Transformation-31-320.jpg)
[DLHacks 実装]Perceptual Adversarial Networks for Image-to-Image Transformation
- 1. Image-to-Image Translation with Conditional Adversarial Nets (Pix2Pix) & Perceptual Adversarial Networks for Image-to-Image Transformation (PAN) 2017/10/2 DLHacks Otsubo
- 2. Topic : image-to-image “translation” 1
- 3. Info Pix2Pix [CVPR2017] • Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros - iGAN [ECCV 2016] - interactive-deep-colorization [SIGGRAPH 2017] - Context-Encoder [CVPR 2016] - Image Quilting [SIGGRAPH 2001] - Texture Synthesis by Non-parametric Sampling [ICCV 1999] • University of California • 178 citations PAN [arXiv2017] • Chaoyue Wang, Chang Xu, Chaohui Wang, Dacheng Tao • University of Technology Sydney, The University of Sydney, Universite Paris-Est 2
- 4. Background • Many tasks are regarded as “translation” from input image to output image - Diverse methods exist for them 3 Is there single framework to achieve them?
- 5. Overview Pix2Pix • General-purpose solution to image-to-image translation using single framework - Single framework: conditional GAN (cGAN) PAN • Pix2Pix - (per-pixel loss) + (perceptual adversarial loss) 4
- 6. Naive Implementation : U-Net (①) 5 ①per-pixel loss (L1/L2)
- 10. PAN’s loss (②+③) 9 L1 norm ② ② ③ ③ m : constant
- 11. Example1 : Image De-Raining • Removing rain from single images via a deep detail network [Fu, CVPR2017] • ID-GAN (cGAN) [Zhang, arXiv2017] - per-pixel loss - adversarial loss - pre-trained VGG’s perceptual loss 10 Input Output (Ground Truth)
- 12. Example1 : Image De-Raining • Removing rain from single images via a deep detail network [Fu, CVPR2017] • ID-GAN (cGAN) [Zhang, arXiv2017] - per-pixel loss - adversarial loss - pre-trained VGG’s perceptual loss 11 Input Output (Ground Truth) (cf. PAN uses discriminator’s perceptual loss)
- 13. Example2 : Image Inpainting • Globally and Locally Consistent Image Completion [Iizuka, SIGGRAPH2017] • Context Encoders (cGAN) [Pathak, CVPR2016] - per-pixel loss - adversarial loss 12 Input Output (Ground Truth)
- 14. Example3 : Semantic Segmentation Cityscape / Pascal VOC • DeepLabv3 [Chen, arXiv2017] • PSPNet [Zhao, CVPR2017] http://host.robots.ox.ac.uk:8080/leaderboard/displaylb.php? cls=mean&challengeid=11&compid=6 Cell Tracking / CREMI • Learned Watershed [Wolf, ICCV2017] • U-Net [Ronneberger, MICCAI2015] http://www.codesolorzano.com/Challenges/CTC/Welcome.html 13 Input Output (Ground Truth)
- 15. Result1 : Image De-Raining 14 (≒pix2pix)→ (≒pix2pix)
- 16. Result2 : Image Inpainting 15
- 17. Result3 : Semantic Segmentation 16
- 18. Discussion vs. No perceptual loss (Pix2Pix) - Perceptual loss enables D to detect more discrepancy between True/False images vs. Pre-trained VGG perceptual loss (ID-GAN) - VGG features tend to focus on content - PAN features tend to focus on discrepancy - PAN’s loss leads to avoid adversarial examples [Goodfellow, ICLR2015] (?) 17 Why is perceptual adversarial loss so efficient?
- 19. Minor Difference • Pix2Pix uses Patch-GAN - Small size(70×70) patch-discriminator - Final output of D is average of patch-discriminator’s responses (convolutionally applied) 18
- 20. To Do • Implement 1. Pix2Pix (Patch Discriminator) 2. PAN (Patch Discriminator) 3. PAN (Normal Discriminator) Wang et al. might compare 1 with 3. 19
- 21. 20
- 23. My Implementation • https://github.com/DLHacks/pix2pix_PAN • pix2pix - https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix • PAN - per-pixel loss à perceptual adversarial loss - not same as paper’s original architecture - num of parameters is same as pix2pix 22
- 24. My Experiments • Facade (label à picture) • Map (picture à Google map) • Cityscape (picture à label) 23
- 32. Discussion – Why pix2pix > PAN? • per-pixel loss is needed? • patch discriminator is not suited for PAN? • positive margin m • (bad pix2pix implementation in PAN’s paper…?) 31