A note on variational inference for the univariate Gaussian
0 likes485 views
1. The document summarizes variational inference for the univariate Gaussian model. 2. It derives an approximate posterior distribution q(μ, τ) that factorizes into q(μ) and q(τ). 3. q(μ) is Gaussian and q(τ) is Gamma, where their parameters are updated through an iterative procedure that maximizes a lower bound on the log evidence.
![This is a negative KL-divergence and is thus maximized when
ln q(µ) = q(τ) ln p(X|µ, τ)p(µ|τ; µ0, λ0)dτ + const. (12)
= q(τ) ln
τ
2π
N/2
exp −
τ
2
N
n=1
(xn − µ)2
dτ (13)
+ q(τ) ln
λ0τ
2π
1/2
exp −
λ0τ
2
(µ − µ0)2
dτ (14)
= −
Eq(τ)[τ]
2
N
n=1
(xn − µ)2
+ λ0(µ − µ0)2
+ const. (15)
where Eq(τ)[·] denotes the expectation with respect to the approximate posterior q(τ). This result means
that q(µ) is a Gaussian distribution, whose mean and precision is given by the following math:
N
n=1
(xn − µ)2
+ λ0(µ − µ0)2
= Nµ2
− 2µ
N
n=1
xn +
N
n=1
x2
n + λ0µ2
− 2λ0µ0µ + λ0µ2
0 (16)
= (λ0 + N) µ2
− 2
λ0µ0 + N ¯x
λ0 + N
µ + const. (17)
= (λ0 + N) µ −
λ0µ0 + N ¯x
λ0 + N
2
+ const. (18)
∴ µN =
λ0µ0 + N ¯x
λ0 + N
(19)
λN = (λ0 + N)Eq(τ)[τ] (20)
Next, we keep q(µ) and maximize L.
L = q(µ)q(τ) ln
p(X|µ, τ)p(µ|τ; µ0, λ0)p(τ; a0, b0)
q(µ)q(τ)
dµdτ (21)
= q(τ) q(µ) ln p(X|µ, τ)p(µ|τ; µ0, λ0)dµ + const. dτ (22)
+ q(τ) ln p(τ; a0, b0)dτ − q(τ) ln q(τ)dτ + const. (23)
This is maximized when
ln q(τ) = q(µ) ln p(X|µ, τ)p(µ|τ; µ0, λ0)dµ + ln p(τ; a0, b0) + const. (24)
= q(µ) ln
τ
2π
N/2
exp −
τ
2
N
n=1
(xn − µ)2
×
λ0τ
2π
1/2
exp −
λ0τ
2
(µ − µ0)2
dµ (25)
+ ln
1
Γ(a0)
ba0
0 τa0−1
exp(−b0τ) + const. (26)
=
N + 1
2
ln(τ) −
τ
2
q(µ)
N
n=1
(xn − µ)2
+ λ0(µ − µ0)2
dµ + (a0 − 1) ln(τ) − b0τ + const. (27)
= a0 +
N − 1
2
ln(τ) − b0 +
1
2
Eq(µ)
N
n=1
(xn − µ)2
+ λ0(µ − µ0)2
τ + const. (28)
(29)
This result means that q(τ) is a gamma distribution, whose parameters are
aN = a0 +
N − 1
2
(30)
bN = b0 +
1
2
Eq(µ)
N
n=1
(xn − µ)2
+ λ0(µ − µ0)2
(31)
2](https://image.slidesharecdn.com/vb-univariate-gaussian-180410085735/85/A-note-on-variational-inference-for-the-univariate-Gaussian-2-320.jpg)
![Consequently, µN and λN can be computed as
µN =
λ0µ0 + N ¯x
λ0 + N
(32)
λN = (λ0 + N)
aN
bN
(33)
when aN and bN are given. Further, aN and bN can be computed as
aN = a0 +
N − 1
2
(34)
bN = b0 +
1
2
Eq(µ)
N
n=1
(xn − µ)2
+ λ0(µ − µ0)2
(35)
= b0 +
1
2
N
n=1
x2
n + λ0µ2
0 − (λ0µ0 + N ¯x)Eq(µ)[µ] + (λ0 + N)Eq(µ)[µ2
] (36)
= b0 +
1
2
N
n=1
x2
n + λ0µ2
0 − (λ0µ0 + N ¯x)µN + (λ0 + N)
1
λN
+ µ2
N (37)
= b0 +
1
2
N
n=1
x2
n + λ0µ2
0 +
λ0 + N
λN
(38)
2 True posterior
The full joint distribution is obtained as follows:
p(X, µ, τ; µ0, λ0, a0, b0) = p(X|µ, τ)p(µ|τ; µ0, λ0)p(τ; a0, b0)
=
τ
2π
N/2
exp −
τ
2
N
n=1
(xn − µ)2
×
λ0τ
2π
1/2
exp −
λ0τ
2
(µ − µ0)2
×
1
Γ(a0)
ba0
0 τa0−1
exp(−b0τ)
(39)
exp −
τ
2
N
n=1
(xn − µ)2
× exp −
λ0τ
2
(µ − µ0)2
(40)
= exp −
τ
2
N
n=1
(xn − µ)2
−
λ0τ
2
(µ − µ0)2
(41)
= exp −
τ
2
Nµ2
− 2N ¯xµ + Nx2 + λ0µ2
− 2λ0µ0 + λ0µ2
0 (42)
= exp −
τ
2
(λ0 + N)µ2
− 2(λ0µ0 + N ¯x)µ + (λ0µ2
0 + Nx2) (43)
= exp −
τ(λ0 + N)
2
µ2
− 2
λ0µ0 + N ¯x
λ0 + N
µ −
τ
2
(λ0µ2
0 + Nx2) (44)
= exp −
τ(λ0 + N)
2
µ −
λ0µ0 + N ¯x
λ0 + N
2
−
λ0µ0 + N ¯x
λ0 + N
2
−
τ
2
(λ0µ2
0 + Nx2) (45)
= exp −
τ(λ0 + N)
2
µ −
λ0µ0 + N ¯x
λ0 + N
2
+
τ
2
(λ0µ0 + N ¯x)2
λ0 + N
− (λ0µ2
0 + Nx2) (46)
= exp −
τ(λ0 + N)
2
µ −
λ0µ0 + N ¯x
λ0 + N
2
× exp
τ
2
(λ0µ0 + N ¯x)2
λ0 + N
− (λ0µ2
0 + Nx2) (47)
(48)
3](https://image.slidesharecdn.com/vb-univariate-gaussian-180410085735/85/A-note-on-variational-inference-for-the-univariate-Gaussian-3-320.jpg)
A note on variational inference for the univariate Gaussian
- 1. Variational Inference for the univariate Gaussian Tomonari MASADA @ Nagasaki University April 10, 2018 1 Approximate posterior We consider the univariate Gaussian model for modeling a set of real-valued observations. (Our discussion is based on Section 10.1.3 of PRML by C. M. Bishop.) When we have a set of N observations X = {x1, . . . , xN }, the likelihood function is given by p(X|µ, τ) = N n=1 1 √ 2πσ2 exp − (xn − µ)2 2σ2 = τ 2π N/2 exp − τ 2 N n=1 (xn − µ)2 (1) where τ ≡ 1/σ2 is called precision. We introduce conjugate prior distributions for µ and τ as follows: p(µ|τ; µ0, λ0) = N(µ|µ0, (λ0τ)−1 ) (2) p(τ; a0, b0) = Gam(τ|a0, b0) (3) where Gam(τ|a0, b0) is the gamma distribution. While we can find the posterior exactly for this problem, we approximate the exact posterior by a factorized approximate posterior given by q(µ, τ) = q(µ)q(τ) (4) The full joint distribution can be obtained as follows: p(X, µ, τ; µ0, λ0, a0, b0) = p(X|µ, τ)p(µ|τ; µ0, λ0)p(τ; a0, b0) = τ 2π N/2 exp − τ 2 N n=1 (xn − µ)2 × λ0τ 2π 1/2 exp − λ0τ 2 (µ − µ0)2 × 1 Γ(a0) ba0 0 τa0−1 exp(−b0τ) (5) A lower bound of the log evidence can be obtained as follows: ln p(X; µ0, λ0, a0, b0) = ln p(X, µ, τ; µ0, λ0, a0, b0)dµdτ (6) = ln p(X|µ, τ)p(µ|τ; µ0, λ0)p(τ; a0, b0)dµdτ (7) = ln q(µ)q(τ) p(X|µ, τ)p(µ|τ; µ0, λ0)p(τ; a0, b0) q(µ)q(τ) dµdτ (8) ≥ q(µ)q(τ) ln p(X|µ, τ)p(µ|τ; µ0, λ0)p(τ; a0, b0) q(µ)q(τ) dµdτ (9) We refer to the above lower bound by L. First, we keep q(τ) and maximize L. L = q(µ)q(τ) ln p(X|µ, τ)p(µ|τ; µ0, λ0)p(τ; a0, b0) q(µ)q(τ) dµdτ (10) = q(µ) q(τ) ln p(X|µ, τ)p(µ|τ; µ0, λ0)dτ + const. dµ − q(µ) ln q(µ)dµ + const. (11) 1
- 2. This is a negative KL-divergence and is thus maximized when ln q(µ) = q(τ) ln p(X|µ, τ)p(µ|τ; µ0, λ0)dτ + const. (12) = q(τ) ln τ 2π N/2 exp − τ 2 N n=1 (xn − µ)2 dτ (13) + q(τ) ln λ0τ 2π 1/2 exp − λ0τ 2 (µ − µ0)2 dτ (14) = − Eq(τ)[τ] 2 N n=1 (xn − µ)2 + λ0(µ − µ0)2 + const. (15) where Eq(τ)[·] denotes the expectation with respect to the approximate posterior q(τ). This result means that q(µ) is a Gaussian distribution, whose mean and precision is given by the following math: N n=1 (xn − µ)2 + λ0(µ − µ0)2 = Nµ2 − 2µ N n=1 xn + N n=1 x2 n + λ0µ2 − 2λ0µ0µ + λ0µ2 0 (16) = (λ0 + N) µ2 − 2 λ0µ0 + N ¯x λ0 + N µ + const. (17) = (λ0 + N) µ − λ0µ0 + N ¯x λ0 + N 2 + const. (18) ∴ µN = λ0µ0 + N ¯x λ0 + N (19) λN = (λ0 + N)Eq(τ)[τ] (20) Next, we keep q(µ) and maximize L. L = q(µ)q(τ) ln p(X|µ, τ)p(µ|τ; µ0, λ0)p(τ; a0, b0) q(µ)q(τ) dµdτ (21) = q(τ) q(µ) ln p(X|µ, τ)p(µ|τ; µ0, λ0)dµ + const. dτ (22) + q(τ) ln p(τ; a0, b0)dτ − q(τ) ln q(τ)dτ + const. (23) This is maximized when ln q(τ) = q(µ) ln p(X|µ, τ)p(µ|τ; µ0, λ0)dµ + ln p(τ; a0, b0) + const. (24) = q(µ) ln τ 2π N/2 exp − τ 2 N n=1 (xn − µ)2 × λ0τ 2π 1/2 exp − λ0τ 2 (µ − µ0)2 dµ (25) + ln 1 Γ(a0) ba0 0 τa0−1 exp(−b0τ) + const. (26) = N + 1 2 ln(τ) − τ 2 q(µ) N n=1 (xn − µ)2 + λ0(µ − µ0)2 dµ + (a0 − 1) ln(τ) − b0τ + const. (27) = a0 + N − 1 2 ln(τ) − b0 + 1 2 Eq(µ) N n=1 (xn − µ)2 + λ0(µ − µ0)2 τ + const. (28) (29) This result means that q(τ) is a gamma distribution, whose parameters are aN = a0 + N − 1 2 (30) bN = b0 + 1 2 Eq(µ) N n=1 (xn − µ)2 + λ0(µ − µ0)2 (31) 2
- 3. Consequently, µN and λN can be computed as µN = λ0µ0 + N ¯x λ0 + N (32) λN = (λ0 + N) aN bN (33) when aN and bN are given. Further, aN and bN can be computed as aN = a0 + N − 1 2 (34) bN = b0 + 1 2 Eq(µ) N n=1 (xn − µ)2 + λ0(µ − µ0)2 (35) = b0 + 1 2 N n=1 x2 n + λ0µ2 0 − (λ0µ0 + N ¯x)Eq(µ)[µ] + (λ0 + N)Eq(µ)[µ2 ] (36) = b0 + 1 2 N n=1 x2 n + λ0µ2 0 − (λ0µ0 + N ¯x)µN + (λ0 + N) 1 λN + µ2 N (37) = b0 + 1 2 N n=1 x2 n + λ0µ2 0 + λ0 + N λN (38) 2 True posterior The full joint distribution is obtained as follows: p(X, µ, τ; µ0, λ0, a0, b0) = p(X|µ, τ)p(µ|τ; µ0, λ0)p(τ; a0, b0) = τ 2π N/2 exp − τ 2 N n=1 (xn − µ)2 × λ0τ 2π 1/2 exp − λ0τ 2 (µ − µ0)2 × 1 Γ(a0) ba0 0 τa0−1 exp(−b0τ) (39) exp − τ 2 N n=1 (xn − µ)2 × exp − λ0τ 2 (µ − µ0)2 (40) = exp − τ 2 N n=1 (xn − µ)2 − λ0τ 2 (µ − µ0)2 (41) = exp − τ 2 Nµ2 − 2N ¯xµ + Nx2 + λ0µ2 − 2λ0µ0 + λ0µ2 0 (42) = exp − τ 2 (λ0 + N)µ2 − 2(λ0µ0 + N ¯x)µ + (λ0µ2 0 + Nx2) (43) = exp − τ(λ0 + N) 2 µ2 − 2 λ0µ0 + N ¯x λ0 + N µ − τ 2 (λ0µ2 0 + Nx2) (44) = exp − τ(λ0 + N) 2 µ − λ0µ0 + N ¯x λ0 + N 2 − λ0µ0 + N ¯x λ0 + N 2 − τ 2 (λ0µ2 0 + Nx2) (45) = exp − τ(λ0 + N) 2 µ − λ0µ0 + N ¯x λ0 + N 2 + τ 2 (λ0µ0 + N ¯x)2 λ0 + N − (λ0µ2 0 + Nx2) (46) = exp − τ(λ0 + N) 2 µ − λ0µ0 + N ¯x λ0 + N 2 × exp τ 2 (λ0µ0 + N ¯x)2 λ0 + N − (λ0µ2 0 + Nx2) (47) (48) 3
- 4. Therefore, exp − τ 2 N n=1 (xn − µ)2 × exp − λ0τ 2 (µ − µ0)2 dµ (49) = 2π τ(λ0 + N) 1/2 exp τ 2 (λ0µ0 + N ¯x)2 λ0 + N − (λ0µ2 0 + Nx2) (50) p(X, τ; µ0, λ0, a0, b0) = p(X, µ, τ; µ0, λ0, a0, b0)dµ (51) = τ 2π N/2 λ0τ 2π 1/2 1 Γ(a0) ba0 0 τa0−1 exp(−b0τ) × 2π τ(λ0 + N) 1/2 exp τ 2 (λ0µ0 + N ¯x)2 λ0 + N − (λ0µ2 0 + Nx2) (52) = 1 2π N/2 λ0 λ0 + N 1/2 1 Γ(a0) ba0 0 τa0+N/2−1 × exp − b0 + 1 2 (λ0µ0 + N ¯x)2 λ0 + N − (λ0µ2 0 + Nx2) τ (53) = 1 2π N/2 λ0 λ0 + N 1/2 1 Γ(a0) ba0 0 τa0+N/2−1 × exp − b0 + λ0µ0 + N ¯x 2 λ0µ0 + N ¯x λ0 + N − λ0µ2 0 + Nx2 λ0µ0 + N ¯x τ (54) (55) Therefore, the evidence is given by p(X; µ0, λ0, a0, b0) = p(X, τ; µ0, λ0, a0, b0)dτ (56) = 1 2π N/2 λ0 λ0 + N 1/2 1 Γ(a0) ba0 0 τa0+N/2−1 × exp − b0 + λ0µ0 + N ¯x 2 λ0µ0 + N ¯x λ0 + N − λ0µ2 0 + Nx2 λ0µ0 + N ¯x τ dτ (57) = 1 2π N/2 λ0 λ0 + N 1/2 Γ(a0 + N/2) Γ(a0) ba0 0 b0 + λ0µ0 + N ¯x 2 λ0µ0 + N ¯x λ0 + N − λ0µ2 0 + Nx2 λ0µ0 + N ¯x a0+N/2 (58) Therefore, the true posterior is given by p(µ, τ|X; µ0, λ0, a0, b0) = p(X, µ, τ; µ0, λ0, a0, b0) p(X; µ0, λ0, a0, b0) = τ 2π N/2 exp − τ 2 N n=1 (xn − µ)2 × λ0τ 2π 1/2 exp − λ0τ 2 (µ − µ0)2 × 1 Γ(a0) ba0 0 τa0−1 exp(−b0τ) 1 2π N/2 λ0 λ0 + N 1/2 Γ(a0 + N/2) Γ(a0) ba0 0 b0 + λ0µ0 + N ¯x 2 λ0µ0 + N ¯x λ0 + N − λ0µ2 0 + Nx2 λ0µ0 + N ¯x a0+N/2 = λ0 + N 2π 1/2 exp − τ 2 N n=1 (xn − µ)2 − λ0τ 2 (µ − µ0)2 × τa0+(N−1)/2 exp(−b0τ) Γ(a0 + N/2) b0 + λ0µ0 + N ¯x 2 λ0µ0 + N ¯x λ0 + N − λ0µ2 0 + Nx2 λ0µ0 + N ¯x a0+N/2 (59) 4