Reading the Lindley-Smith 1973 paper on linear Bayes estimators

Introduction Exchangeability General bayesian linear model Examples Estimation with unknown Covariance References
. ..... .... ...... ...............

.
.
Bayes Estimates for the Linear Model
.
.. .

.
Reading Seminar in Statistical Classics
Director: C. P. Robert

Presenter: Kaniav Kamary

12 Novembre, 2012

. ..... .... ...... ...............

Outline

.
. . Introduction
1
The Model and the bayesian methods

.
. . Exchangeability
2

.
. . General bayesian linear model
3

.
. . Examples
4

.
. . Estimation with unknown Covariance
5

. ..... .... ...... ...............

The Model and the bayesian methods

The linear model :

Structure of the linear model:

E(y ) = Aθ

y : a vector of the random variables
A: a known design Matrix
Θ: unknown parameters
For estimating Θ:
The usual estimate by the method of least squares.
Unsatisfactory or inadmissibility in demensions greater
than two.
Improved estimates with knowing prior information about
the parameters in the bayesian framework

. ..... .... ...... ...............

Outline

.
. . Introduction
1

.
. . Exchangeability
2
un example

.
3

.
. . Examples
4

.
5

. ..... .... ...... ...............

un example

The concept of exchangeability

In general linear model suppose A = I :
E(yi ) = Θi for i = 1, 2, . . . , n and yi ∼ N(θi , σ 2 ) iid
The distribution of θi is exchangeable if:
The prior opinion of θi is the same of that of θj or any other θk
where i, j, k = 1, 2, . . . , n.
In the other hand:
A sequence θ1 , . . . , θn of random variables is said to be
exchangeable if for all k = 2, 3, . . .

θ1 , . . . , θn ∼ θπ(1) , θπ(2) , θπ(k)
=
for all π ∈ S(k ) where S(k ) is the group of permutation of
1, 2, . . . , k

. ..... .... ...... ...............

un example

The concept of exchangeability. . .

One way for obtaining an exchangeable distribution p(Θ):

∏
n
p(Θ) = p(θi | µ)dQ(µ) (1)
i=1

p(Θ): exchangeable prior knowledge described by a mixture
Q(µ): arbitrary probability distribution for each µ
µ: the hyperparameters
A linear structure to the parameters:

E(θi ) = µ

. ..... .... ...... ...............

un example

Estimate of Θ

If θi ∼ N(µ, τ 2 ):
a closer parallelism between the two stage for y and Θ
By assuming that µ have a uniform distribution over the real line
then: yi y.
2 + τ2
θi∗ = σ
1 1
(2)
σ 2 + τ2
∑n
i=1 yi
where y. = n and θi∗ = E(θi | y ).

. ..... .... ...... ...............

un example

The features of θi∗

ˆ
A weighted averages of yi = θi , overall mean y. and
inversely proportional to the variances of yi and θi
A biased estimate of θi
Use the estimates of τ 2 and σ 2
An admissible estimate with known σ 2 , τ 2
A bayes estimates as substitution for the usual
least-squares estimates

. ..... .... ...... ...............

un example

The features of θi∗ . . .

.
Judging the merit of θi with one of the other estimates .
..
The condition that the average M.S.E for θi ∗ to be less than that

ˆ
for θi is: ∑
(θi − θ. )2
< 2τ 2 + σ 2 (3)
n−1
∑
s2 = (θi −θ. ) is an usual estimate for τ 2 . Hence, the chance of
2
n−1
unequal (3) being satisﬁed is high for n as law as 4 and rapidly
tends to 1 as n increases.
.
.. .

.

. ..... .... ...... ...............

Outline

.
. . Introduction
1

.
. . Exchangeability
2

.
3
The posterior distribution of the parameters

.
. . Examples
4

.
5

. ..... .... ...... ...............


The structure of the model

Let:
Y : a column vector
.
Lemma .
..
Suppose Y ∼ N(A1 Θ1 , C1 ) and Θ1 ∼ N(A2 Θ2 , C2 ) that Θ1 is a
vector of P1 parameters, that Θ2 is a vector of P2
hyperparameters.
Then (a): Y ∼ N(A1 A2 Θ2 , C1 + A1 C2 AT ),
1
and (b): Θ1 | Y ∼ N(Bb, B) where:
−1 −1
B −1 = AT C1 A1 + C2
1
−1 −1
. b = AT C1 y + C2 A2 Θ2
1 (4)
.. .

.

. ..... .... ...... ...............


The posterior distribution with three stages

.
Theorem .
..
With the assumptions of the Lemma, suppose that given Θ3 ,

Θ2 ∼ N(A3 Θ3 , C3 )

then for i = 1, 2, 3:

Θ1 | {Ai }, {Ci }, Θ3 , Y ∼ N(Dd, D)

with
−1 −1
D −1 = AT C1 A1 + {C2 + A2 C3 AT }
1 2 (5)
and
T −1 T −1 −1
. d = A1 C1 y + A1 C1 A1 + {C2 + A2 C3 A2 }
T
A2 A3 Θ3 (6)
.. .

.

. ..... .... ...... ...............


The properties
.
Result of the Lemma .
..
For any matrices A1 , A2 , C1 and C2 of appropriate dimensions
and for witch the inverses stated, we have:
−1 −1 −1 −1 −1
C1 − C1 A1 (AT C1 A1 + C2 )−1 AT C1 = (C1 + A1 C2 AT )
1 1 1
. (7)
.. .

.
.
Properties of the bayesian estimation .
..
The E(Θ1 | {Ai }, {Ci }, Θ3 , Y ) is:
A weighed average of the least-squares estimates
−1 −1 −1
(AT C1 A1 ) AT C1 y .
1 1
A weithed average of the prior mean A2 A3 Θ3 .
It may be regarded as a point estimate of Θ1 to replace the
usual least-squares estimate.
.
.. .

.

. ..... .... ...... ...............


Results of the Theorem

.
Corollary1 .
..
An alternative expression for D −1 :

T −1 −1 −1 T −1 −1 T −1 −1
. A1 C1 A1 + C2 − C2 A2 {A2 C2 A2 + C3 } A2 C2 (8)
.. .

.
.
Corollary2 .
..
−1
If C3 = 0, the posterior distribution of Θ1 is N(D0 d0 , D0 ) with:

−1 −1 −1 −1 −1 −1 T −1
D0 = AT C1 A1 + C2 − C2 A2 {AT C2 A2 }
1 2 A2 C2 (9)

and
−1
. d0 = AT C1 y
1 (10)
.. .

.

. ..... .... ...... ...............

Outline

.
. . Introduction
1

.
. . Exchangeability
2

.
3

.
. . Examples
4
Two-factor Experimental Designs
Exchangeability Between Multiple Regression Equation
Exchangeability within Multiple Regression Equation

.
5

. ..... .... ...... ...............


The structure of the Two-factor Experimental Designs

The usual model of n observations with the errors
independent N(0, σ 2 ):
E(yij ) = µ + αi + βj , 1 ≤ i ≤ t, 1 ≤ j ≤ b
ΘT
1 = (µ, α1 , . . . , αt , β1 , . . . , βb ) (11)
yij : an observation in the ith treatment and the jth block.
The exchangeable prior knowledge of {αi } and {βj } but
independent
αi ∼ N(0, σα ), βj ∼ N(0, σβ ), µ ∼ N(w, σµ )
2 2 2

The vague prior knowledge of µ and σµ → ∞
2
−1
C2 : the diagonal matrix that leading diagonal of C2 is
−2 −2 −2 −2
(0, σα , . . . , σα , σβ , . . . , σβ )
C1 : the unit matrix times σ 2

. ..... .... ...... ...............


Bayesian estimate of the parameters

With substituting the assumptions stated and C3 = 0 in to (5)
and (6), then:
−1
D −1 = σ −2 AT A1 + C2
1
d = σ −2 AT y
1 (12)

Hence Θ∗ , the bayes estimate Dd, satisﬁes the equation as
1
following
−1
(AT A1 + σ 2 C2 )Θ∗ = AT y
1 1 1 (13)
by solving (13),

µ = y..
−1
αi∗ = (bσα + σ 2 )
2
bσα (yi. − y.. )
2

−1
βj∗ = (tσβ + σ 2 )
2
tσβ (y.j − y.. )
2
(14)

. ..... .... ...... ...............


The structure of the Multiple Regression Equation

The usual model for p regressor variables where
j = 1, 2, . . . , m:
yj ∼ N(Xj βj , Inj σj2 ) (15)
A1 : a diagonal matrix with xj as the jth diagonal submatrix

ΘT = (β1 , β2 , . . . , βm )
1
T T T

Suppose variables X and Y were related with the usual linear
regression structure and

βj ∼ N(ξ, Σ), Θ2 = ξ

A2 : a matrix of order mp × p, all of whose p × p submatrices
are unit matrices

. ..... .... ...... ...............


Bayesian estimation for the parameters of the Multiple Regression
Equation. . .

The equation for the bayes estimates βj∗ is
 
σ1 −2 X1 T X1 + Σ−1 ··· 0
 . . 
 .
. σ2 −2 X2 T X2 + Σ−1 .
. 
0 ··· σm −2 Xm T Xm + Σ−1
     
β1 ∗ β. ∗ σ1 −2 X1 T y
 β2 ∗   β. ∗   σ2 −2 X1 T y 
  −1    
× . −Σ  . = .  (16)
 .
.   .
.   .
. 
βm ∗ β. ∗ σm −2 X1 T y
∑ βi ∗
where β. ∗ = m .

. ..... .... ...... ...............


Bayesian estimation for the parameters of the Multiple Regression
Equation

By solving equation (16) for βj ∗ , the bayes estimate is
−1
βj ∗ = (σj −2 Xj T Xj + Σ−1 ) (σj −2 Xj T y + Σ−1 β. ∗ ) (17)

Noting that D0 −1 , given in Corollary 2 (9) and the matrix
Lemma 7, we obtain a weighted form of (17) with β. ∗ replaced
∑
by wj βj ∗ :
−1
∑m
−1 −1
wi = { (σj −2 Xj T Xj + Σ−1 ) σj −2 Xj T Xj } (σi −2 Xi T Xi + Σ−1 ) σi −2 Xi T Xi
j=1
(18)

. ..... .... ...... ...............

Exchangeability within Multiple Regression Equation

the model and bayes estimates of the parameters

A single multiple regression:

y ∼ N(X β, In σ 2 ) (19)

The individual regression coefﬁcients in β T = (β1 , β2 , . . . , βp )
are exchangeable and βj ∼ N(ξ, σβ 2 ).
.
bayes estimate with two possibilities .
..
σ2
to suppose vague prior knowledge for ξ with k = σ2
β

β ∗ = {Ip + k (X T X )−1 (Ip − p−1 )}−1 β
ˆ (20)

to put ξ = 0, reﬂecting a feeling that the βi are small

β ∗ = {Ip + k (X T X )−1 }−1 β
ˆ (21)
.
.. .

.

. ..... .... ...... ...............

Outline

.
. . Introduction
1

.
. . Exchangeability
2

.
3

.
. . Examples
4

.
5
Exposition and method
Two-factor Experimental Designs(unknown Covariance)
Exch between Multiple Regression(unknown Covariance)
Exch within Multiple Regression(unknown Covariance)

. ..... .... ...... ...............


Method

θ: the parameters of interest in the general model
ϕ: the nuisance parameters
Ci : the unknown dispersion matrices
.
The method and its defect .
..
assign a joint prior distribution to θ and ϕ
provide the joint posterior distribution p(θ, ϕ | y )
integrating the joint posterior with respect to ϕ and leaving
the posterior for θ
for using loss function, necessity another integration for
calculate the mean
require the constant of proportionality in bayes’s formula
for calculating the mean
the
. above argument is technically most complex to execute.
.. .

.

. ..... .... ...... ...............


Solution

For simpliﬁed the method:
considering an approximation
using the mode of the posterior distribution in place of the
mean
using the mode of the joint distribution rather than that of
the θ-margin
taking the estimates derived in section 2 and replace the
unknown values of the nuisances parameters by their
modal estimates

. ..... .... ...... ...............


Solution. . .

The modal value:
∂ ∂
p(θ, ϕ | y ) = 0, p(θ, ϕ | y ) = 0
∂θ ∂ϕ

assuming that p(ϕ | y ) ̸= 0 as

∂
p(θ | y , ϕ) = 0 (22)
∂θ
The approximation is good if:
the samples are large
the resulting posterior distributions approximately normal

. ..... .... ...... ...............


The prior distributions for σ 2 , σα 2 and σβ 2 are invers-χ2 .

νλ να λα νβ λ β
2
∼ χν 2 , 2
∼ χνα 2 , ∼ χνβ 2
σ σα σβ 2

With assuming the three variances independent.
The joint distribution of all quantities:
−1 −1
(σ 2 ) 2
(n+ν+2)
× exp {νλ + S 2 (µ, α, β)}
2σ 2

−1
−1 ∑2
×(σα )
2 2
(t+να +2)
exp 2σ2 {να λα + αi }
2 −1 −1
α
∑2
×(σβ ) 2 (b+νβ +2) exp 2σ2 {νβ λβ + βj }
β

. ..... .... ...... ...............


Estimates of the parameters of the model

To ﬁnd the modal estimates:
reversing the roles of θ and ϕ with supposing µ, α and β
known
{νλ + S 2 (µ∗ , α∗ , β ∗ )}
s2 =
(n + ν + 2)
∑
{να λα + αi ∗2 }
sα 2 =
(t + να + 2)
∑
{νβ λβ + βj ∗2 }
sβ 2 = (23)
(b + νβ + 2)

solving (13) with trial value of σ 2 , σα and σβ
2 2

inserting the value in to (23)

. ..... .... ...... ...............


Suppositions of the model. . .

In the model (15), suppose σj 2 = σ 2 with νλ ∼ χν 2 and Σ−1 has
σ2
a Wishart distribution with ρ degree of freedom and matrix R
independent of σ 2 .
The joint distribution of all the quantities:

−1 ∑
m
−1n
(σ 2 ) 2 × exp{ (yj − Xj βj )T (yj − Xj βj )}
2σ 2
j=1

−1 ∑
m
−1m
× (| Σ |) 2 exp{ (βj − ξ)T Σ−1 (βj − ξ)}
2
j=1
−1
−1(ρ−p−1)
× (| Σ |) 2 tr Σ−1 R}
exp{
2
−1(ν+2) −νλ
× (σ 2 ) 2 exp{ 2 } (24)
2σ

. ..... .... ...... ...............


The joint posterior distribution

The joint posterior density for β, σ 2 and Σ−1 :

−1 ∑
m
−1(n+ν+2) T
(σ )2 2 × exp{ 2 {m−1 νλ + (yj − Xj βj ) (yj − Xj βj )}}
2σ
j=1
−1(m+ρ−p−2)
× (| Σ |) 2

−1 ∑ m
× exp{ tr Σ−1 {R + (βj − β. )(βj − β. )T }} (25)
2
j=1

∑m
where β. = m−1 j=1 βj .

. ..... .... ...... ...............


The modal estimates

.
The estimates of the parameters .
.. ∑m −1 ∗ T ∗
2 j=1 {m νλ + (yj − Xj βj ) (yj − Xj βj )}
s = (26)
(n + ν + 2)
and ∑m
{R + ∗ − β.∗ )(βj∗ − β.∗ )T }
∗ j=1 (βj
Σ = (27)
. (m + ρ − p − 2)
.. .

.

. ..... .... ...... ...............


The modal estimates . . .

The posterior distribution of the βj ’s, free of σ 2 and Σ:

∑
n
1
{ {m−1 νλ + (yj − Xj βj )T (yj − Xj βj )}}− 2 (n+ν)
j=1
− 1 (m+ρ−1)
∑
m 2

×| R + (βj − β. )(βj − β. )T | (28)
j=1

The mode of this distribution can be used in place of the modal
values for the wider distribution.

. ..... .... ...... ...............


Application
.
an application in an educational context .
..
data from the American Collage Testing Program 1968, 1969
prediction of grade-point average at 22 collages
the results of 4 tests (English, Mathematics, Social Studies,
Natural Sciences),p = 5, m = 22, and nj varying from 105 to 739

Table: Comparison of predictive efﬁciency

reduction the error by under 2 per cent by using the bayesian
method in the ﬁrst row but 9 per cent with the quarter sample
.

.

. ..... .... ...... ...............


Assumptions of the model regression

In the model 19 and βj ∼ N(ξ, σβ 2 ),
suppose
νλ νβ λ β
2
∼ χν 2 , ∼ χνβ 2
σ σβ 2
The posterior distribution of β, σ 2 and σβ 2 :

−1(n+ν+2) −1
(σ 2 ) 2 × exp{ {νλ + (y − X β)T (y − X β)}}
2σ 2
−1(p+νβ +1)
× (σβ 2 ) 2

−1 ∑ p
× exp{ 2
{νβ λβ + (βj − β. )2 }} (29)
2σβ
j=1

∑p
that β. = p−1 j=1 βj .

. ..... .... ...... ...............


The modal estimation. . .

The modal equations:
−1
β ∗ = {Ip + k ∗ X T X (Ip − p−1 Jp )}−1 β
ˆ
{νλ + (y − X β ∗ )T (y − X β ∗ )}
s2 =
(n + ν + 2)
∑p
{νβ λβ + j=1 (βj ∗ − β. ∗ )2 }
sβ 2 = (30)
(p + νβ + 1)

where k ∗ = s2
sβ ∗ .

. ..... .... ...... ...............


Comparison between the methods of the estimates

The main difference lies in the choice of k
in absolute value, the least-squares procedure produce
regression estimates too large, of incorrect sign and
unstable with respect to small changes in the data
The ridge method avoid some of these undesirable
features
The bayesian method reaches the same conclusion but
has the added advantage of dispensing with the rather
arbitrary choice of k and allows the data to estimate it

Table: 10-factor multiple regression example

. ..... .... ...... ...............


A brief explanation of a recent paper

On overview of the Bayesian Linear Model with unknown
Variance:
Yn×p = Xp×1 + ξ
The bayesian approache to ﬁtting the linear model consists of
three steps (S.Kuns, 2009)[4]:
assign priors to all unknown parameters
write down the likelihood of the data given the parameters
determine the posterior distribution of the parameters
given the data using bayes’ theorem
If Y ∼ N(X β, k −1 ) then a conjugate prior distribution for the
parameters is: β, k ∼ NG(β0 , Σ0 , a, b). In other word:
p−2 −1
f (β, k ) = CK a+ 2 exp k {(β − β0 )T Σ−1 (β − β0 ) + 2b}
0
2
ba
where C = p 1
(2π) 2 |Σ0 | 2 Γ(a)

. ..... .... ...... ...............


A brief explanation of a recent paper...

The posterior distribution is:
∗ + p −1 −1
f (β, k | Y ) ∝ k a 2 exp{ k ((β − β ∗ )T (Σ∗ )−1 (β − β ∗ ) + 2b∗ )}
2

β ∗ = (Σ0 −1 + X T X )−1 ((Σ0 −1 β0 + X T y )
Σ∗ = (Σ0 −1 + X T X )−1
n
a∗ = a +
2
1
b∗ = b + (β0 T Σ0 −1 β0 + y T y − (β ∗ )T (Σ∗ )−1 β ∗ )
2
And β | y follows a multivariate t-distribution:
−1
(ν+p)
1 2
f (β | y ) ∝ (1 + (β − β ∗ )T (Σ∗ )−1 (β − β ∗ ))
ν

. ..... .... ...... ...............

References

L. D. Brown, On the Admissibility of Invariant Estimators of One
or More Location Parameters, The Annals of Mathematical
Statistics, Vol. 37, No. 5 (Oct., 1966), pp. 1087-1136.
A. E. Hoerl, R. W. kennard, Ridge Regression: Biased
Estimation for Nonorthogonal Problems, Technometrics, Vol.
12, No. 1. (Feb., 1970), pp. 55-67.
T. Bouche, Formation LaTex, (2007).
S. Kunz, The Bayesian Linear Model with unknown Variance,
Seminar for Statistics. ETH Zurich, (2009).
V. Roy, J. P. Hobert, On Monte Carlo methods for Bayesian
multivariate regression models with heavy-tailed errors, (2009).

. ..... .... ...... ...............

References

Thank you
for your Attention

. ..... .... ...... ...............

Result
.
Proof. .
..
To prove (a), suppose y = A1 Θ1 + u, Θ1 = A2 Θ2 + v that:

u ∼ N(0, C1 )

and v ∼ N(0, C2 ) Then y = A1 A2 Θ2 + A1 v + u that:

A1 v + u ∼ N(0, C1 + A1 C2 AT )
1

To prove (b), by using the Bayesian Theorem:
1
p(Θ1 | Y ) ∝ e− 2 Q
Q = (Θ1 − Bb)T B −1 (Θ1 − Bb)
−1 −1
+ y T C1 y + Θ2 T A2 T C2 A2 Θ2 − bT Bb (31)

.

.

Reading the Lindley-Smith 1973 paper on linear Bayes estimators

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Reading the Lindley-Smith 1973 paper on linear Bayes estimators (20)

More from Christian Robert (20)

Recently uploaded (20)

Reading the Lindley-Smith 1973 paper on linear Bayes estimators