Simplifying Gaussian Mixture Models Via Entropic Quantization (EUSIPCO 2009)
0 likes450 views
The document discusses the simplification of Gaussian Mixture Models (GMMs) using entropic quantization, primarily through Bregman divergence and Kullback-Leibler divergence. It presents algorithms for reducing the number of components in GMMs while retaining statistical measures, and compares the performance of the BKM and UTAC algorithms in terms of speed and accuracy. Experimental results indicate that the BKM method offers superior performance over UTAC in simplifying GMMs.
![Mixture model simplification
Sided BKMC
Right-sided BKMC algorithm
1: Initialize the GMM g
2: repeat
3:
Compute the cluster C : the Gaussian fi belongs to cluster Cj if and only if
˜ ˜
˜ ˜
DF (Θi Θj ) < DF (Θi Θl ), ∀l ∈ [1, m] {j}
4:
Compute the centroids: the weight and the natural parameters of the j-th
centroid (i.e. Gaussian gj ) are given by:
αj =
αi ,
i
The sum
i
θj =
i
αi θi
,
i αi
Θj =
i
αi Θi
i αi
is performed on i ∈ [1, m] such as fi ∈ Cj
5: until the cluster does not change between two iterations
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
12 / 23](https://image.slidesharecdn.com/2009eusipco-131216043317-phpapp02/85/Simplifying-Gaussian-Mixture-Models-Via-Entropic-Quantization-EUSIPCO-2009-12-320.jpg)
![Mixture model simplification
Sided BKMC
Left-sided BKMC algorithm
1: Initialize the GMM g
2: repeat
3:
Compute the cluster C : the Gaussian fi belongs to cluster Cj if and only if
˜ ˜
˜ ˜
DF (Θj Θi ) < DF (Θl Θi ), ∀l ∈ [1, m] {j}
4:
Compute the centroids: the weight and the natural parameters of the j-th
centroid (i.e. Gaussian gj ) are given by:
αj =
αi ,
˜
Θj =
F −1
i
i
where
˜
F −1 (Θ) =
− Θ + θθT
−1
θ, −
αi
αj
˜
F Θi
1
Θ + θθT
2
−1
5: until the cluster does not change between two iterations
V. Garcia (X, Paris, France)
Simplifying GMMs
28th august 2009
13 / 23](https://image.slidesharecdn.com/2009eusipco-131216043317-phpapp02/85/Simplifying-Gaussian-Mixture-Models-Via-Entropic-Quantization-EUSIPCO-2009-13-320.jpg)
Simplifying Gaussian Mixture Models Via Entropic Quantization (EUSIPCO 2009)
- 1. Simplifying Gaussian Mixture Models Via Entropic Quantization Frank Nielsen1 2 , Vincent Garcia1 , and Richard Nock3 1 Ecole Polytechnique (Paris, France) Sony Computer Science Laboratories (Tokyo, Japan) Universit´ des Antilles et de la Guyane (Guadeloupe, France) e 2 3 28th august 2009 V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 1 / 23
- 2. Introduction Plan 1 Introduction Mixture models Problem Mixture model simplification 2 Mixture model simplification KLD and Bregman divergence Sided BKMC Symmetric BKMC jMEF 3 Experiments Quality measure and initialization Sided BKMC BKMC vs UTAC 4 Conclusion V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 2 / 23
- 3. Introduction Mixture models Mixture models Mixture model is a powerful framework to estimate PDF Mixture model f n f (x) = αi fi (x) i=1 where αi ≥ 0 denotes a weight with n i=1 αi =1 If f is a Gaussian mixture model (GMM), (x − µi )T Σ−1 (x − µi ) 1 i fi (x) = exp − 2 (2π)d/2 |Σi |1/2 with µi mean and Σi covariance matrix V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 3 / 23
- 4. Introduction Problem Problem 2.5 2 1.5 1 0.5 0 −0.5 0 0.5 1 1.5 Density estimation using kernel-based Parzen estimator Mixture models usually contain a lot of components Estimation of statistical measures is computationally expensive Need to reduce the number of components Re-lear a simpler mixture model from dataset Simplify the mixture model f V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 4 / 23
- 5. Introduction Mixture model simplification Mixture model simplification Given a mixture model f of n components n f (x) = αi fi (x) i=1 Compute a mixture model g of m components m αj gj (x) g (x) = j=1 such as g is the best approximation of f V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 5 / 23
- 6. Mixture model simplification Plan 1 Introduction Mixture models Problem Mixture model simplification 2 Mixture model simplification KLD and Bregman divergence Sided BKMC Symmetric BKMC jMEF 3 Experiments Quality measure and initialization Sided BKMC BKMC vs UTAC 4 Conclusion V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 6 / 23
- 7. Mixture model simplification KLD and Bregman divergence Relative entropy and Bregman divergence The fundamental measure between statistical distributions is the relative entropy, also called the Kullback-Leibler divergence Given fi and fj two distributions, the KLD is given by KLD(fi ||fj ) = fi (x) log fi (x) dx fj (x) In the case of normal distriubtions det Σj 1 1 KLD(fi ||fj ) = log + tr Σ−1 Σi j 2 det Σi 2 1 d + (µj − µi )T Σ−1 (µj − µi ) − j 2 2 V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 7 / 23
- 8. Mixture model simplification KLD and Bregman divergence Relative entropy and Bregman divergence Nomral distributions belong to the class of exponential families Canonical form of exponential families f (x) = exp ˜ ˜ Θ, t(x) − F (Θ) + C (x) Estimation of the KLD by computing the Bregman divergence defined for the log normalizer F ˜ ˜ KLD(fi ||fj ) = DF (Θj ||Θi ) where ˜ ˜ ˜ ˜ ˜ ˜ ˜ DF (Θj ||Θi ) = F (Θj ) − F (Θi ) − Θj − Θi , F (Θi ) V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 8 / 23
- 9. Mixture model simplification KLD and Bregman divergence Relative entropy and Bregman divergence For multivariate normal distributions Sufficient statistics 1 t(x) = (x, − xx T ) 2 Natural parameters 1 ˜ Θ = (θ, Θ) = (Σ−1 µ, Σ−1 ) 2 Log normalizer 1 d 1 ˜ F (Θ) = tr(Θ−1 θθT ) − log det Θ + log π 4 2 2 ˜ F (Θ) = V. Garcia (X, Paris, France) 1 −1 1 1 Θ θ , − Θ−1 − (Θ−1 θ)(Θ−1 θ)T 2 2 4 Simplifying GMMs 28th august 2009 9 / 23
- 10. Mixture model simplification Sided BKMC Bregman k-means clustering K-means clustering Set of points Initialize k centroids = k classes Repetition until convergence Repartition step (distance) Computation of centroids (centers of mass) Bregman K-means clustering Set of distributions Initialize k centroids (αi , gi ) = GMM with k components Repetition until convergence Repartition step (sided Bregman divergence) Computation of centroids (sided centroids) V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 10 / 23
- 11. Mixture model simplification Sided BKMC Sided centroids 5 multivariate Gaussians Right-centroid Left-centroid http://www.sonycsl.co.jp/person/nielsen/BNCj/ V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 11 / 23
- 12. Mixture model simplification Sided BKMC Right-sided BKMC algorithm 1: Initialize the GMM g 2: repeat 3: Compute the cluster C : the Gaussian fi belongs to cluster Cj if and only if ˜ ˜ ˜ ˜ DF (Θi Θj ) < DF (Θi Θl ), ∀l ∈ [1, m] {j} 4: Compute the centroids: the weight and the natural parameters of the j-th centroid (i.e. Gaussian gj ) are given by: αj = αi , i The sum i θj = i αi θi , i αi Θj = i αi Θi i αi is performed on i ∈ [1, m] such as fi ∈ Cj 5: until the cluster does not change between two iterations V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 12 / 23
- 13. Mixture model simplification Sided BKMC Left-sided BKMC algorithm 1: Initialize the GMM g 2: repeat 3: Compute the cluster C : the Gaussian fi belongs to cluster Cj if and only if ˜ ˜ ˜ ˜ DF (Θj Θi ) < DF (Θl Θi ), ∀l ∈ [1, m] {j} 4: Compute the centroids: the weight and the natural parameters of the j-th centroid (i.e. Gaussian gj ) are given by: αj = αi , ˜ Θj = F −1 i i where ˜ F −1 (Θ) = − Θ + θθT −1 θ, − αi αj ˜ F Θi 1 Θ + θθT 2 −1 5: until the cluster does not change between two iterations V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 13 / 23
- 14. Mixture model simplification Symmetric BKMC Symmetric BKMC algorithm Symmetric similarity measure can be required (e.g. CBIR) Repartition step: Symmetric Bregman divergence ˜ ˜ SDF (Θp , Θq ) = ˜ ˜ ˜ ˜ DF (Θq ||Θp ) + DF (Θp ||Θq ) 2 Computation of symmetric centroid: Compute right and left centroids (cr and cl ) The symmetric centroid cs belongs to the geodesic link joining cr and cl cλ = F −1 (λ F (cr ) + (1 − λ) F (cl )) The symmetric centroid cs = cλ verifies SDF (cλ , cr ) = SDF (cλ , cl ). V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 14 / 23
- 15. Mixture model simplification jMEF jMEF jMEF : Java library for Mixture of Exponential Families Create and manage MEF Simplify MEF using BKMC Available on line at www.lix.polytechnique.fr/∼nielsen/MEF V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 15 / 23
- 16. Experiments Plan 1 Introduction Mixture models Problem Mixture model simplification 2 Mixture model simplification KLD and Bregman divergence Sided BKMC Symmetric BKMC jMEF 3 Experiments Quality measure and initialization Sided BKMC BKMC vs UTAC 4 Conclusion V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 16 / 23
- 17. Experiments Quality measure and initialization Quality measure and initialization Simplification quality measure KLD(f g ) (right-sided) No closed-form expression Draw 10,000 points to estimate this KLD (Monte-Carlo) Initial GMM f Learnt from an image K-means on RGB pixels ⇒ 32 classes EM algorithm ⇒ fi Weights αi : proportion of pixels in each class V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 17 / 23
- 18. Experiments Sided BKMC Sided BKMC Evolution of KLD(f g ) as a function of m The simplification quality increases with m Left-sided BKMC provides the best results Right-sided BKMC provides the worst results V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 18 / 23
- 19. Experiments BKMC vs UTAC BKMC vs UTAC UTAC algorithm based on sigma points + EM algorithm BKMC provides better results than UTAC BKMC is faster than UTAC: 20ms vs 100ms V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 19 / 23
- 20. Experiments BKMC vs UTAC Clustering-based image segmentation Image UTAC BKMC KLD=0.23 KLD=0.11 KLD=0.16 V. Garcia (X, Paris, France) f KLD=0.13 Simplifying GMMs 28th august 2009 20 / 23
- 21. Experiments BKMC vs UTAC Clustering-based image segmentation Image UTAC BKMC KLD=0.69 KLD=0.53 KLD=0.36 V. Garcia (X, Paris, France) f KLD=0.18 Simplifying GMMs 28th august 2009 21 / 23
- 22. Conclusion Plan 1 Introduction Mixture models Problem Mixture model simplification 2 Mixture model simplification KLD and Bregman divergence Sided BKMC Symmetric BKMC jMEF 3 Experiments Quality measure and initialization Sided BKMC BKMC vs UTAC 4 Conclusion V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 22 / 23
- 23. Conclusion Conclusion GMM simplification algorithm based on k-means and Bregman divergence BKMC is faster and provides better results than UTAC algorithm BKMC extends to mixtures of exponential families jMEF available on line at www.lix.polytechnique.fr/∼nielsen/MEF Included features: Create/manage mixtures of exponential families BKMC algorithm Hierarchical GMM (ACCV 2009) V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 23 / 23