Automatic variational inference with latent categorical variables
0 likes7,061 views
This document discusses using normalizing flows and mixture models for automatic variational inference with latent categorical variables. It introduces discrete normalizing flows which allow generating samples and evaluating probabilities of categorical distributions. Mixture models of discrete flows are proposed to better approximate complex categorical distributions. The document outlines ideas for specifying multivariate categorical distributions and evaluating probabilities in the entropy term. Finally, it lists several experiments including on Gaussian mixtures, Bayesian networks, hidden Markov models, and variational autoencoders.
Automatic variational inference with latent categorical variables
- 1. Automatic variational inference with latent categorical variables Tomasz KuĹmierczyk 2020-04-15
- 2. Resources Mixture of discrete normalizing ďŹows for variational inference: â ArXiv text: https://arxiv.org/abs/2006.15568 â GitHub code: https://github.com/tkusmierczyk/mixture_of_discrete_normalizing_ďŹows Discrete normalizing ďŹows: â ArXiv text: https://arxiv.org/abs/1905.10347 â GitHub implementation: https://github.com/google/edward2/blob/master/edward2/tensorďŹow/layers/discrete_ďŹo ws.py
- 3. Agile AI with Probabilistic Programming â easy speciďŹcation & model development â scalability thanks to variational inference â handling latent discrete variables?
- 4. Latent categorical variables â allocation models with hand-crafted algorithms â mixture models â hidden markov models â topic models â expertâs knowledge encoding â ⌠?
- 5. 1D categorical distribution speciďŹcation (unordered) set of categories xâ {a,b,c,d} â probability p p(x=a) = ½ p(x=b) = 0 p(x=c) = ½ p(x=d) = 0
- 6. 2D categorical distribution speciďŹcation p(x=a) = ½ p(x=b) = 0 p(x=c) = ½ p(x=d) = 0 0.25 0.20 0.05 0 0 0 0 0 0.20 0.05 0.05 0.20 0 0 0 0 e f g h z = joint distribution p(x, z) 0.50 0.40 0.10 0 0 0 0 0 0.40 0.10 0.10 0.40 0 0 0 0 e f g h z = conditional distribution p(z | x) or
- 8. Automatic Variational Inference Why: eďŹcient inference of approximate posteriors q without tedious math qx(x) â p(x|D) Reparametrized ELBO: Requirements for q to perform automatic VI with reparametrization gradients: â generating sampling diďŹerentiable w.r.t distribution parameters â log-probability of samples
- 9. Reparametrization: For example: normal distribution with parameters Îť = (Îź, Ď): fÎť(u) = Îź + Ď u , u ~ N(0, 1) Normalizing ďŹows: â parameters Îť do not have this interpretation Reparametrization vs normalizing ďŹows , u ~ pu
- 10. Discrete ďŹows , u ~ pu fÎť sampling: probability evaluation:
- 11. Individual ďŹow vs. mixture of ďŹows b-th ďŹow mixing weightsum over ďŹows
- 12. Accuracy of B-ďŹows Base distributions with probability mass concentrated at exactly one category: p(ub=c)=1 Equal mixing weights: Ďb = 1/B â each ďŹow allocates probability 1/B â | true probability for category k - approximation | â 1/B â works well for concentrated distributions â fails for uniform distribution
- 13. Multivariate categorical distributions p(x) = p(x1) p(x2 | x1) p(x3 | x2, x1) ⌠p(xd | xd-1, xd-2, âŚ, x1) ... fd(u) = (Îźd + Ďd u) mod Kd where Îźd := Îźd(xd-1, xd-2, âŚ, x1, * ) Ďd := Ďd(xd-1, xd-2, âŚ, x1, * ) neural networks trained with straight-through estimator of argmax
- 14. Practical probability evaluation in entropy term assuming independence: full covariance:
- 15. Toy example p( ⌠| grass wet = yes ) sprinkler = no sprinkler = yes rain = no 0.000 0.642 rain = yes 0.353 0.004 p( ⌠| grass wet = yes ) sprinkler = no sprinkler = yes rain = no 0.000 0.600 rain = yes 0.400 0.000 true posterior approximate posterior (10 ďŹows, found in 15 iterations)
- 16. Experiments â gaussian mixture model â (large) bayesian networks â (higher order) hidden markov model â variational autoencoder