Conditional CycleGANによる食事画像変換
7 likes1,327 views
The document discusses conditional generative adversarial networks (GANs) for image-to-image translation tasks. It presents the conditional CycleGAN model which uses cycle consistency loss to learn mappings between domains without paired training examples. The model consists of generators and discriminators trained in an adversarial manner to translate images from one domain to another and back again.
![ⓒ 2018 UEC Tokyo.
• StarGAN: Unified Generative Adversarial Networks for
Multi-Domain Image-to-Image Translation [Choi+ CVPR-18]
– :
–
– +1
15](https://image.slidesharecdn.com/180306tanno1-180315191525/85/Conditional-CycleGAN-15-320.jpg)
![ⓒ 2018 UEC Tokyo.
AC-GAN[Odena+ ICML-17]
• Conditional Image Synthesis With Auxiliary Classifier GANs
– ) (
)
1616
fake or real
one-hot vector
!
C[0, 1, 0…]
one-hot vector
"
#
$%
#](https://image.slidesharecdn.com/180306tanno1-180315191525/85/Conditional-CycleGAN-16-320.jpg)
![ⓒ 2018 UEC Tokyo.
Real Image
In domain
Real Image
In domain B
Fake Image
In domain
Reconstructed
Image
Domain
SelectC[0, 1]
fake or real class[1, 0]
Domain
Select
C[1, 0]
!
(G) (G)
17
•
– 3](https://image.slidesharecdn.com/180306tanno1-180315191525/85/Conditional-CycleGAN-17-320.jpg)
![ⓒ 2018 UEC Tokyo. 18
•
– 3
Real Image
In domain
Real Image
In domain B
Fake Image
In domain
Reconstructed
Image
Domain
SelectC[0, 1]
fake or real class[1, 0]
Domain
Select
C[1, 0]
!
(G) (G)](https://image.slidesharecdn.com/180306tanno1-180315191525/85/Conditional-CycleGAN-18-320.jpg)
![ⓒ 2018 UEC Tokyo.
• 1 6
+ B + 6
- [
S RTCP ]
– 1 L
1
– aL EV J
19
Conv(st=1)
7x7x64
Conv(st=2)
4x4x128
Conv(st=2)
4x4x256
ResBlock
ResBlock
ResBlock
ResBlock
ResBlock
ResBlock
Conv(st=1/2)
4x4x128
Conv(st=1/2)
4x4x64
Conv(st=1)
7x7x3
3x3Conv
InsNorm
LeakyReLU
3x3Conv
InsNorm
D](https://image.slidesharecdn.com/180306tanno1-180315191525/85/Conditional-CycleGAN-19-320.jpg)
![ⓒ 2018 UEC Tokyo.
•
– 1
– M
1,0 0 ,) ( )
[Tanno+ MMM2017]
S
3](https://image.slidesharecdn.com/180306tanno1-180315191525/85/Conditional-CycleGAN-25-320.jpg)
![ⓒ 2018 UEC Tokyo.
• :
–
27
G )(
A N
Real Image
In domain
Real Image
In domain B
Fake Image
In domain
Reconstructed
Image
Domain
SelectC[0, 1]
fake or real class[1, 0]
C
Domain
Select
C[1, 0]
!
(G) (G)](https://image.slidesharecdn.com/180306tanno1-180315191525/85/Conditional-CycleGAN-27-320.jpg)
Conditional CycleGANによる食事画像変換
- 1. ⓒ 2018 UEC Tokyo. Conditional CycleGAN † †† ††† ††† ††† † 1 †† 3 †††
- 2. ⓒ 2018 UEC Tokyo. • A – ( ( ( ( ( – ( ( )( 2 Input noise vector fake or real D G 100 dim Training set fake image real image
- 3. ⓒ 2018 UEC Tokyo. • A GI GI A – 4 - 1 - 1- ( ( + ( ) 3 Input noise vector fake or realD G 100 dim Training set fake image real image G D G: N A D D: G N (real) G (fake)
- 4. ⓒ 2018 UEC Tokyo. • 5 1 – 2 I 0 ( A ( 0 5 – A 1 ) 2 2 1 ( 1 – 1 5 2 0 81 4
- 5. ⓒ 2018 UEC Tokyo. • 5 …
- 6. ⓒ 2018 UEC Tokyo. 6
- 7. ⓒ 2018 UEC Tokyo. • : – 2 • : – 2 2 1 7 : 2 0 1 2 2 +++ +++
- 8. ⓒ 2018 UEC Tokyo. • : – 2 • : – 2 2 1 8 : 2 0 1 2 2 +++ +++
- 9. ⓒ 2018 UEC Tokyo. • . . • : . . : 9 Conditional CycleGAN 11 2 Image-to-Image CycleGAN :
- 10. ⓒ 2018 UEC Tokyo. • 1 1 -( 2 ) – C I F I 10 D + 7 2 G
- 11. ⓒ 2018 UEC Tokyo. 11 • 1 ) + ( - – A 7
- 12. ⓒ 2018 UEC Tokyo. 12 Cycle Consistency Loss • 1 ) + ( - – A 7 G CI
- 13. ⓒ 2018 UEC Tokyo. 13 Cycle Consistency Loss • 1 ) + ( - – A 7 G CI
- 14. ⓒ 2018 UEC Tokyo. . 2 2 : . 14 to 1 to to 1:n
- 15. ⓒ 2018 UEC Tokyo. • StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation [Choi+ CVPR-18] – : – – +1 15
- 16. ⓒ 2018 UEC Tokyo. AC-GAN[Odena+ ICML-17] • Conditional Image Synthesis With Auxiliary Classifier GANs – ) ( ) 1616 fake or real one-hot vector ! C[0, 1, 0…] one-hot vector " # $% #
- 17. ⓒ 2018 UEC Tokyo. Real Image In domain Real Image In domain B Fake Image In domain Reconstructed Image Domain SelectC[0, 1] fake or real class[1, 0] Domain Select C[1, 0] ! (G) (G) 17 • – 3
- 18. ⓒ 2018 UEC Tokyo. 18 • – 3 Real Image In domain Real Image In domain B Fake Image In domain Reconstructed Image Domain SelectC[0, 1] fake or real class[1, 0] Domain Select C[1, 0] ! (G) (G)
- 19. ⓒ 2018 UEC Tokyo. • 1 6 + B + 6 - [ S RTCP ] – 1 L 1 – aL EV J 19 Conv(st=1) 7x7x64 Conv(st=2) 4x4x128 Conv(st=2) 4x4x256 ResBlock ResBlock ResBlock ResBlock ResBlock ResBlock Conv(st=1/2) 4x4x128 Conv(st=1/2) 4x4x64 Conv(st=1) 7x7x3 3x3Conv InsNorm LeakyReLU 3x3Conv InsNorm D
- 20. ⓒ 2018 UEC Tokyo. • 1 0 1 1 0, – 1 !"#"$% 20 !"$&''#(#%)
- 21. ⓒ 2018 UEC Tokyo. • : – 2 • : – 2 2 1 21 : 2 0 1 2 2 +++ +++
- 22. ⓒ 2018 UEC Tokyo. • 5 1 – 2 I 0 ( A ( 0 5 – A 1 ) 2 2 1 ( 1 – 1 5 2 0 81 22 ( ) (
- 23. ⓒ 2018 UEC Tokyo. - • 0 2 20 610 P 6 + – VT N – 0 20 G C S – R • – GP – V 23 argmin '()(*+ = -'.)/(0/( + 2'3(4+0 '()(*+ '.)/(0/( '3(4+0 0 0 0 0 20 input conv3_1 conv1_1 conv2_1 conv4_1 conv5_1 conv4_2 ) )( 5+ = 6789+ 6789+ : U ⃗< ⃗= ⃗>
- 24. ⓒ 2018 UEC Tokyo. • – - – • 0 0 0 • ) ( 1 G1 • – 1 1 J • a0 • 1 G ( 1 G 24 !"
- 25. ⓒ 2018 UEC Tokyo. • – 1 – M 1,0 0 ,) ( ) [Tanno+ MMM2017] S 3
- 26. ⓒ 2018 UEC Tokyo. • T MN – MN ) + 6 : + : – MN ) - : 2 : 3 8 • P – A 6 1 6 0 • P C – + 6 : + : ) G O Q – - : 2 : 3 8 ) -2 + + O(Q S G : S 26
- 27. ⓒ 2018 UEC Tokyo. • : – 27 G )( A N Real Image In domain Real Image In domain B Fake Image In domain Reconstructed Image Domain SelectC[0, 1] fake or real class[1, 0] C Domain Select C[1, 0] ! (G) (G)
- 28. ⓒ 2018 UEC Tokyo. • N : 28 G ( ( ) A C
- 29. ⓒ 2018 UEC Tokyo. • – A N 29 )( G : C
- 30. ⓒ 2018 UEC Tokyo. 30 )( • 合計約２３万枚を利用 – 学習用：９割 – テスト用：１割 : 74007 34216 27854 24760 21324 18396 13499 7138 5329 3530 230053
- 31. ⓒ 2018 UEC Tokyo. • 1 31 2 ) ( 1 …
- 32. ⓒ 2018 UEC Tokyo. • 32 0 032 1 1 – G T • 1 0 1 2 – a G A 1 f c S 32 dC il : eN M
- 33. ⓒ 2018 UEC Tokyo. • 33 : )( 33
- 34. ⓒ 2018 UEC Tokyo. • 34 : )( 34 A
- 35. ⓒ 2018 UEC Tokyo. • 1 35 ) ( Content Style
- 36. ⓒ 2018 UEC Tokyo. cCycleGAN Multi Style Transfer 36
- 37. ⓒ 2018 UEC Tokyo. 37
- 38. ⓒ 2018 UEC Tokyo. • – lS e n i • AC if • ( ) 1 a • c – A a M Na – T – G S G 38
- 39. ⓒ 2018 UEC Tokyo. • – • – • – 39
- 40. ⓒ 2018 UEC Tokyo. • a m A – e A N A a e m – gm C A tC r u – a • xf a a – 1 A - l n A xf A 0 I pi 40
- 41. ⓒ 2018 UEC Tokyo. • A ( ( – 8 1 2) 8 – 8 41 G 30% 8 G 32% N
- 42. ⓒ 2018 UEC Tokyo. • – 0M e 2 – l 0M N T 2 • l A – 2 0M b I i 0S 0M N – C 2 A 42
- 43. ⓒ 2018 UEC Tokyo. • 43 : )( 43 A
- 44. ⓒ 2018 UEC Tokyo. • 1 44 ) ( Content Style
- 45. ⓒ 2018 UEC Tokyo. • 45 : )(
- 46. ⓒ 2018 UEC Tokyo. 46 2 ) ( • –
- 47. ⓒ 2018 UEC Tokyo. • 47 ) ( Content Style
- 48. ⓒ 2018 UEC Tokyo. • 48 ) ( Content Style