![An Alternative Expression for the Log-Likelihood Ratio of the Multivariate Normal Distribution
Cole Arora
April 1, 2014
Theorem Suppose prior distribution P(i) = gi for all i ∈ {1, . . . , m} and multivariate normal model X|i ∼ Nk(µi
, Σ) for the k-vector of data X =
[ X1 ··· Xk ] , where the mean vector µi
= E[X|i] = [ E[X1|i] ··· E[Xk|i] ] = [ µi
1 ··· µi
k ] and the common, symmetric covariance matrix Σ is positive definite.
Then the log-likelihood ratio, computed posterior to making the observation x = [ x1 ··· xk ] , has the form
log
f(x|j)
f(x|i)
= x −
µi
+ µj
2
Σ−1
(µj
− µi
). (1)
Proof The left-hand side of Equation (1) can be expanded using the definition of the joint probability density function of the multivariate normal
distribution — which exists only because Σ is positive definite — as follows:
log
f(x|j)
f(x|i)
= log
1√
(2π)k|Σ|
exp −1
2 (x − µj
) Σ−1
(x − µj
)
1√
(2π)k|Σ|
exp −1
2 (x − µi) Σ−1
(x − µi)
= log exp
1
2
(x − µi
) Σ−1
(x − µi
) −
1
2
(x − µj
) Σ−1
(x − µj
)
=
1
2
(x − µi
) Σ−1
(x − µi
) −
1
2
(x − µj
) Σ−1
(x − µj
). (2)
For expediency, let
Σ−1
=
Σ−1
11 Σ−1
12 · · · Σ−1
1k
Σ−1
21 Σ−1
22 · · · Σ−1
2k
...
...
...
...
Σ−1
k1 Σ−1
k2 · · · Σ−1
kk
.
1](https://image.slidesharecdn.com/multivariatenormalproof-140401220010-phpapp01/85/Multivariate-normal-proof-1-320.jpg)
Multivariate normal proof
- 1. An Alternative Expression for the Log-Likelihood Ratio of the Multivariate Normal Distribution Cole Arora April 1, 2014 Theorem Suppose prior distribution P(i) = gi for all i ∈ {1, . . . , m} and multivariate normal model X|i ∼ Nk(µi , Σ) for the k-vector of data X = [ X1 ··· Xk ] , where the mean vector µi = E[X|i] = [ E[X1|i] ··· E[Xk|i] ] = [ µi 1 ··· µi k ] and the common, symmetric covariance matrix Σ is positive definite. Then the log-likelihood ratio, computed posterior to making the observation x = [ x1 ··· xk ] , has the form log f(x|j) f(x|i) = x − µi + µj 2 Σ−1 (µj − µi ). (1) Proof The left-hand side of Equation (1) can be expanded using the definition of the joint probability density function of the multivariate normal distribution — which exists only because Σ is positive definite — as follows: log f(x|j) f(x|i) = log 1√ (2π)k|Σ| exp −1 2 (x − µj ) Σ−1 (x − µj ) 1√ (2π)k|Σ| exp −1 2 (x − µi) Σ−1 (x − µi) = log exp 1 2 (x − µi ) Σ−1 (x − µi ) − 1 2 (x − µj ) Σ−1 (x − µj ) = 1 2 (x − µi ) Σ−1 (x − µi ) − 1 2 (x − µj ) Σ−1 (x − µj ). (2) For expediency, let Σ−1 = Σ−1 11 Σ−1 12 · · · Σ−1 1k Σ−1 21 Σ−1 22 · · · Σ−1 2k ... ... ... ... Σ−1 k1 Σ−1 k2 · · · Σ−1 kk . 1
- 2. Substitution for x, µi , µj , and Σ−1 in Equation (2) gives 1 2 x1 − µi 1 · · · xk − µi k Σ−1 11 Σ−1 12 · · · Σ−1 1k Σ−1 21 Σ−1 22 · · · Σ−1 2k ... ... ... ... Σ−1 k1 Σ−1 k2 · · · Σ−1 kk x1 − µi 1 ... xk − µi k − 1 2 x1 − µj 1 · · · xk − µj k Σ−1 11 Σ−1 12 · · · Σ−1 1k Σ−1 21 Σ−1 22 · · · Σ−1 2k ... ... ... ... Σ−1 k1 Σ−1 k2 · · · Σ−1 kk x1 − µj 1 ... xk − µj k = 1 2 k a=1 (xa − µi a)(Σ−1 a1 ) · · · k a=1 (xa − µi a)(Σ−1 ak ) x1 − µi 1 ... xk − µi k − 1 2 k a=1 (xa − µj a)(Σ−1 a1 ) · · · k a=1 (xa − µj a)(Σ−1 ak ) x1 − µj 1 ... xk − µj k = 1 2 k b=1 k a=1 (xa − µi a)(Σ−1 ab )(xb − µi b) − 1 2 k b=1 k a=1 (xa − µj a)(Σ−1 ab )(xb − µj b) = 1 2 k b=1 k a=1 (Σ−1 ab ) (xa − µi a)(xb − µi b) − (xa − µj a)(xb − µj b) = 1 2 k b=1 k a=1 (Σ−1 ab )(xaxb − xaµi b − xbµi a + µi aµi b − xaxb + xaµj b + xbµj a − µj aµj b) = 1 2 k b=1 k a=1 (Σ−1 ab )(−xaµi b − xbµi a + µi aµi b + xaµj b + xbµj a − µj aµj b) = 1 2 k b=1 k a=1 (Σ−1 ab ) xa(µj b − µi b) + xb(µj a − µi a) + µi aµi b − µj aµj b . (3) There are two key (and somewhat trivial) points that must be realized at this point about the terms under double summation; failure to do so makes finishing the proof in this way impossible. The first point: k b=1 k a=1 xa(µj b − µi b)(Σ−1 ab ) = k b=1 k a=1 xb(µj a − µi a)(Σ−1 ba ) = k b=1 k a=1 xb(µj a − µi a)(Σ−1 ab ), where the first equality simply represents a renaming of the indices, and where the second equality holds because the matrix inverse of a symmetric matrix 2
- 3. is symmetric. So from this point forward, it is justified to replace xb(µj a −µi a) with xa(µj b −µi b) in the summand of Equation (3). The second point is similar: k b=1 k a=1 (µi bµj a − µi aµj b)(Σ−1 ab ) = k b=1 k a=1 (µi bµj a)(Σ−1 ab ) − k b=1 k a=1 (µi aµj b)(Σ−1 ab ) = k b=1 k a=1 (µi bµj a)(Σ−1 ab ) − k b=1 k a=1 (µi bµj a)(Σ−1 ba ) = k b=1 k a=1 (µi bµj a)(Σ−1 ab ) − k b=1 k a=1 (µi bµj a)(Σ−1 ab ) = 0, so the addition of the quantity (µi bµj a − µi aµj b) into the right-most term in the summand of Equation (3) would change absolutely nothing, which turns out to be useful. Using these two insights, Equation (3) can be rewritten as follows: 1 2 k b=1 k a=1 (Σ−1 ab ) 2xa(µj b − µi b) + µi bµj a − µi aµj b + µi aµi b − µj aµj b = k b=1 k a=1 xa − µi a + µj a 2 (Σ−1 ab )(µj b − µi b) = k a=1 xa − µi a + µj a 2 (Σ−1 a1 ) · · · k a=1 xa − µi a + µj a 2 (Σ−1 ak ) µj 1 − µi 1 ... µj k − µi k = x1 − µi 1 + µj 1 2 · · · xk − µi k + µj k 2 Σ−1 11 Σ−1 12 · · · Σ−1 1k Σ−1 21 Σ−1 22 · · · Σ−1 2k ... ... ... ... Σ−1 k1 Σ−1 k2 · · · Σ−1 kk µj 1 − µi 1 ... µj k − µi k = x − µi + µj 2 Σ−1 (µj − µi ). 3