010_20160216_Variational Gaussian Process
3 likes894 views
The document discusses the development of the Variational Gaussian Process (VGP) as a powerful variational model capable of capturing any continuous posterior distribution, proven through a universal approximation theorem. It highlights the use of deep generative models and variational inference methods to effectively represent complex data. Additionally, an efficient black box algorithm for VGP is introduced within the context of hierarchical variational models.
![4
Variational Models
• We want to compute posterior p(z|x) (z: latent variables, x: data)
• Variational inference seeks to minimize
for a family q(z; )
KL(q(z; )||p(z|x))
• Maximizing evidence lower bound (ELBO)
log p(x) Eq(z; )[log p(x|z)] KL(q(z; )||p(z))
• (Common) Mean-field distribution q(z; ) =
Y
i
q(zi; i)
• Hierarchical variational models
• (Newer) Interpret the family as a variational model for posterior
latent variables z (introducing new latent variables)[1]
Lawrence, N. (2000). Variational Inference in Probabilistic Models. PhD thesis.](https://image.slidesharecdn.com/01020160216variationalgaussianprocess-160216094213/85/010_20160216_Variational-Gaussian-Process-4-320.jpg)
010_20160216_Variational Gaussian Process
- 1. Variational Gaussian Process Tran Quoc Hoan @k09hthaduonght.wordpress.com/ 10 February 2016, Paper Alert, Hasegawa lab., Tokyo The University of Tokyo Dustin Tran, Rajesh Ranganath, David M.Blei ICLR 2016
- 2. 2 Background p(x) = p(x|z)p(z) ……… ……… Parameters ……… ……… Parameters ✓ Inference model Generative model Observations x Hidden variables z q (z|x) p✓(x|z) z ⇠ p✓(z) In variational auto encoder (VAE), parameters are displayed as neural networks
- 3. 3 Summary • Deep generative models provide complex representation of data • Variational inference methods require a rich family of approximating distribution • They develop a powerful variational model - the variational Gaussian process (VGP) • They prove a universal approximation theorem: the VGP can capture any continuous posterior distribution. • They derive an efficient black box algorithm.
- 4. 4 Variational Models • We want to compute posterior p(z|x) (z: latent variables, x: data) • Variational inference seeks to minimize for a family q(z; ) KL(q(z; )||p(z|x)) • Maximizing evidence lower bound (ELBO) log p(x) Eq(z; )[log p(x|z)] KL(q(z; )||p(z)) • (Common) Mean-field distribution q(z; ) = Y i q(zi; i) • Hierarchical variational models • (Newer) Interpret the family as a variational model for posterior latent variables z (introducing new latent variables)[1] Lawrence, N. (2000). Variational Inference in Probabilistic Models. PhD thesis.
- 9. 9 Variational Gaussian Processes Mean-fields parameters Induces correlation btw latent variables of the variational model
- 11. 11 Variational Lower Bound auxiliary model Variational latent variable space Posterior latent variable space Data space
- 12. 12 Auto-Encoding Variational Models Take both xn, zn as input
- 13. 13 Black Box Stochastic Optimization
- 14. 14 Black Box Stochastic Optimization ???
- 16. 16 Experiments
- 17. 17 Experiments