3D Volumetric Data Generation with Generative Adversarial Networks
2 likes3,915 views
This document discusses using generative adversarial networks (GANs) to generate 3D volumetric data. Specifically, it aims to extend GANs to 3D voxel data by applying 3D convolutions and deconvolutions. To do so, it trains a GAN on chair models from a dataset, representing the 3D models as binary voxel grids. Techniques like minibatch discrimination and mutual information reconstruction are used to improve training stability and add semantic meaning. The results generated 3D chair-like models but training convergence was an issue due to the small dataset size.
![Background
Generative Adversarial Networks (GAN) [1] have achieved state-of-the-art performance in unsupervised
learning, generating synthetic images by training on the MNIST dataset or ImageNet for multi-channel
images.
However, these networks have not yet been extended to higher dimensions such as volumetric 3D data.
Generated 3D model have various applications in entertainment and could be used as an alternative to
existing procedural methods for creating graphics.
This study demonstrates the capabilities of GAN-based architectures for generating practical 3D models
by applying 3 dimensional convolutions and deconvolutions* on voxel data.
Goal
• Extension of GANs to 3D volumetric data, training on a single class
• Control the shapes of the generated models by e.g. interpolation
1. Introduction
*Transposed Convolutions](https://image.slidesharecdn.com/yamazakifinal-161027093919/85/3D-Volumetric-Data-Generation-with-Generative-Adversarial-Networks-2-320.jpg)
![2. Training Data
3D CAD models from ShapeNet [2]
• Class: Chair
• Instances: 4846
Preprocessing
• Voxelization
• 3D CAD models are converted into binary 0, 1 voxels with dimensions (32, 32, 32). [3]
• Normalization
• No normalization is applied. Data is in range [0, 1]
• Other
• Remove bad samples and centre the models in the
space
Training Data Volume DistributionMean 3D Model](https://image.slidesharecdn.com/yamazakifinal-161027093919/85/3D-Volumetric-Data-Generation-with-Generative-Adversarial-Networks-3-320.jpg)
![A GAN consists of a generator G and a discriminator D, in this
case, both of them are represented as a feed forward neural
network that are trained simultaneously.
• Random noise z vectors sampled from a uniform or
Gaussian distribution
Loss
• Softmax cross-entropies based on the predictions of D
• Separate losses for G and D defined by the minimax game
Optimal Discriminator Strategy
Optimization
• Adam for both G and D
• Learning rate of G is larger than D
3. Generative Adversarial Network
Random Noise
Random Index
Generator
(Linear, Deconvolution,
Batch Normalization,
ReLU, Sigmoid)
Discriminator
(Convolution, Linear, Leaky ReLU)
Training
Data
Generated
3D Model
Real
3D Model
Generated/Real
Prediction
See Appendix for the network architecture and Adam parameters
min
G
max
D
V (G, D) = Ex⇠Pdata(x)[log D(x)] + Ez⇠Pz(z)[log(1 D(G(z)))]
D(x) =
pdata(x)
pdata(x) + pG(x)](https://image.slidesharecdn.com/yamazakifinal-161027093919/85/3D-Volumetric-Data-Generation-with-Generative-Adversarial-Networks-4-320.jpg)
![Issues with GAN
• Collapsing Generator
• G outputs similar 3D models for different inputs
• Non-semantic input z
• Interpolation of z indicate on sharp edges in the latent
space. Hence no way to control the shape of the output
Improving the GAN
• Avoid Generator from collapsing
• Minibatch Discrimination [4] layer in D
• Embed semantic meaning into the input [5]
• With z, concatenate additional latent codes before feeding
it to G
• Additional loss based on mutual information reconstruction
by D
Random Noise
+ Latent Codes
Random Index
Generator
(Linear, Deconvolution,
Batch Normalization,
ReLU, Sigmoid)
Discriminator
(Convolution, Linear, Leaky ReLU, Minibatch Discrimination)
Training
Data
Generated
3D Model
Real
3D Model
Generated/Real
Prediction
Mutual Information
Reconstruction](https://image.slidesharecdn.com/yamazakifinal-161027093919/85/3D-Volumetric-Data-Generation-with-Generative-Adversarial-Networks-5-320.jpg)
![Minibatch Discrimination
Motivation
Avoid generator from collapsing to a single point
Idea
Reproduce the diversity in the training data
Minibatch Discrimination layer to D, before the
generated/real prediction
For each minibatch fed to this layer, compute the
L1 distance between all input vectors
Add this information to the given minibatch
Mutual Information Reconstruction
Motivation
Embed semantic meanings in z
Idea
Maximize the mutual information being preserved
for latent codes C that are passed through the
networks
Latent Codes, input to G
• C = [C1, C2, C3] (Concatenations)
• Categorical one-hot vector C1~Cat(K=2, p=0.5)
• Continuous C2~Unif(-1, 1)
• Continuous C3~Unif(-1, 1)
Reconstruction, output from D
• Categorical
• Softmax Cross Entropy
• Continuous
• Assume a fixed variance and compute the Gaussian
negative log-likelihood based on the mean.
z c1, e.g. [0, 1] c2 c3
Softmax1
𝞵2 𝞵3
Minibatch Discrimination Layer
Kernel
… …](https://image.slidesharecdn.com/yamazakifinal-161027093919/85/3D-Volumetric-Data-Generation-with-Generative-Adversarial-Networks-6-320.jpg)
![Reference
[1] Goodfellow et al. (2014). Generative Adversarial Networks. abs/1406.2661, .
[2] Angel X. Chang and (2015). ShapeNet: An Information-Rich 3D Model Repository. CoRR, abs/1512.03012, .
[3] Patrick Min, Binvox, 3D Mesh Voxelizer, http://www.patrickmin.com/binvox/
[4] Tim Salimans et al. (2016). Improved Techniques for Training GANs. CoRR, abs/1606.03498, .
[5] Xi Chen et al. (2016). InfoGAN: Interpretable Representation Learning by Information Maximizing. CoRR, abs/1606.03657, .
Appendix
Generator Discriminator
Input ∈ R128+2+2 Input 32x32x32 3D voxel data
FC 1024, BN, ReLU Conv 1 → 64, Kernel 4, Stride 2, lReLU (leaky ReLU)
FC 16384, BN, ReLU Conv 64 → 128, Kernel 4, Stride 2, BN, lReLU
DC 256 → 128, Kernel 4, Stride 2, BN, ReLU Conv 128 → 256, Kernel 4, Stride 2, BN, lReLU
DC 128 → 64, Kernel 4, Stride 2, BN, ReLU FC 1024, BN, lReLU
Output DC 64 → 1, Kernel 4, Stride 2, BN, ReLU Minibatch Discrimination, Kernels 64, Kernel Dimension 16
Output FC 2 (Generated/Real prediction)
FC 256, BN, lReLU
Output FC 2+2 (Mutual Information Reconstruction)
Adam Optimizer Parameters
Generator Discriminator
ɑ 0.001 0.00005
β1 0.5 0.5
β1 0.999 0.999
GAN Architecture](https://image.slidesharecdn.com/yamazakifinal-161027093919/85/3D-Volumetric-Data-Generation-with-Generative-Adversarial-Networks-9-320.jpg)
3D Volumetric Data Generation with Generative Adversarial Networks
- 1. 3D Volumetric Data Generation with Generative Adversarial Networks Hiroyuki Vincent Yamazaki Keio University hvy@keio.jp Preferred Networks Summer Internship, 2016
- 2. Background Generative Adversarial Networks (GAN) [1] have achieved state-of-the-art performance in unsupervised learning, generating synthetic images by training on the MNIST dataset or ImageNet for multi-channel images. However, these networks have not yet been extended to higher dimensions such as volumetric 3D data. Generated 3D model have various applications in entertainment and could be used as an alternative to existing procedural methods for creating graphics. This study demonstrates the capabilities of GAN-based architectures for generating practical 3D models by applying 3 dimensional convolutions and deconvolutions* on voxel data. Goal • Extension of GANs to 3D volumetric data, training on a single class • Control the shapes of the generated models by e.g. interpolation 1. Introduction *Transposed Convolutions
- 3. 2. Training Data 3D CAD models from ShapeNet [2] • Class: Chair • Instances: 4846 Preprocessing • Voxelization • 3D CAD models are converted into binary 0, 1 voxels with dimensions (32, 32, 32). [3] • Normalization • No normalization is applied. Data is in range [0, 1] • Other • Remove bad samples and centre the models in the space Training Data Volume DistributionMean 3D Model
- 4. A GAN consists of a generator G and a discriminator D, in this case, both of them are represented as a feed forward neural network that are trained simultaneously. • Random noise z vectors sampled from a uniform or Gaussian distribution Loss • Softmax cross-entropies based on the predictions of D • Separate losses for G and D defined by the minimax game Optimal Discriminator Strategy Optimization • Adam for both G and D • Learning rate of G is larger than D 3. Generative Adversarial Network Random Noise Random Index Generator (Linear, Deconvolution, Batch Normalization, ReLU, Sigmoid) Discriminator (Convolution, Linear, Leaky ReLU) Training Data Generated 3D Model Real 3D Model Generated/Real Prediction See Appendix for the network architecture and Adam parameters min G max D V (G, D) = Ex⇠Pdata(x)[log D(x)] + Ez⇠Pz(z)[log(1 D(G(z)))] D(x) = pdata(x) pdata(x) + pG(x)
- 5. Issues with GAN • Collapsing Generator • G outputs similar 3D models for different inputs • Non-semantic input z • Interpolation of z indicate on sharp edges in the latent space. Hence no way to control the shape of the output Improving the GAN • Avoid Generator from collapsing • Minibatch Discrimination [4] layer in D • Embed semantic meaning into the input [5] • With z, concatenate additional latent codes before feeding it to G • Additional loss based on mutual information reconstruction by D Random Noise + Latent Codes Random Index Generator (Linear, Deconvolution, Batch Normalization, ReLU, Sigmoid) Discriminator (Convolution, Linear, Leaky ReLU, Minibatch Discrimination) Training Data Generated 3D Model Real 3D Model Generated/Real Prediction Mutual Information Reconstruction
- 6. Minibatch Discrimination Motivation Avoid generator from collapsing to a single point Idea Reproduce the diversity in the training data Minibatch Discrimination layer to D, before the generated/real prediction For each minibatch fed to this layer, compute the L1 distance between all input vectors Add this information to the given minibatch Mutual Information Reconstruction Motivation Embed semantic meanings in z Idea Maximize the mutual information being preserved for latent codes C that are passed through the networks Latent Codes, input to G • C = [C1, C2, C3] (Concatenations) • Categorical one-hot vector C1~Cat(K=2, p=0.5) • Continuous C2~Unif(-1, 1) • Continuous C3~Unif(-1, 1) Reconstruction, output from D • Categorical • Softmax Cross Entropy • Continuous • Assume a fixed variance and compute the Gaussian negative log-likelihood based on the mean. z c1, e.g. [0, 1] c2 c3 Softmax1 𝞵2 𝞵3 Minibatch Discrimination Layer Kernel … …
- 7. • Minibatch size: 128 • Epochs: 100 4. Results Generated 3D Models *The blue models are their nearest models in the training dataset 3D Volume Distributions Chair-likeness Learned Distribution True DistributionLosses
- 8. 5. Conclusions • GANs can be extended to 3D volumetric data using 3 dimensional convolutions and deconvolutions • Smaller datasets (sparse data) leads to worse looking models with noise • Partially mitigated by reconstructing mutual information reconstruction and minibatch discrimination • In many cases, D improves faster than G • Gradients back propagated through G saturates and training stops • Training not converging Future Work • Larger dataset with potentially multiple classes • Balance training between G and D • Heuristic • Stop updating D while it is too strong • Larger G, i.e. more parameters
- 9. Reference [1] Goodfellow et al. (2014). Generative Adversarial Networks. abs/1406.2661, . [2] Angel X. Chang and (2015). ShapeNet: An Information-Rich 3D Model Repository. CoRR, abs/1512.03012, . [3] Patrick Min, Binvox, 3D Mesh Voxelizer, http://www.patrickmin.com/binvox/ [4] Tim Salimans et al. (2016). Improved Techniques for Training GANs. CoRR, abs/1606.03498, . [5] Xi Chen et al. (2016). InfoGAN: Interpretable Representation Learning by Information Maximizing. CoRR, abs/1606.03657, . Appendix Generator Discriminator Input ∈ R128+2+2 Input 32x32x32 3D voxel data FC 1024, BN, ReLU Conv 1 → 64, Kernel 4, Stride 2, lReLU (leaky ReLU) FC 16384, BN, ReLU Conv 64 → 128, Kernel 4, Stride 2, BN, lReLU DC 256 → 128, Kernel 4, Stride 2, BN, ReLU Conv 128 → 256, Kernel 4, Stride 2, BN, lReLU DC 128 → 64, Kernel 4, Stride 2, BN, ReLU FC 1024, BN, lReLU Output DC 64 → 1, Kernel 4, Stride 2, BN, ReLU Minibatch Discrimination, Kernels 64, Kernel Dimension 16 Output FC 2 (Generated/Real prediction) FC 256, BN, lReLU Output FC 2+2 (Mutual Information Reconstruction) Adam Optimizer Parameters Generator Discriminator ɑ 0.001 0.00005 β1 0.5 0.5 β1 0.999 0.999 GAN Architecture