![1
Vectors of Random Variables
1.1 NOTATION
Matrices and vectors are denoted by boldface letters A and a, respectively,
and scalars by italics. Random variables are represented by capital letters
and their values by lowercase letters (e.g., Y and y, respectively). This use
of capitals for random variables, which seems to be widely accepted, is par-
ticularly useful in regression when distinguishing between fixed and random
regressor (independent) variables. However, it does cause problems because
a vector of random variables, Y" say, then looks like a matrix. Occasionally,
because of a shortage of letters, aboldface lowercase letter represents a vector
of random variables.
IfX and Yare randomvariables, then the symbols E[Y), var[Y], cov[X, Y),
and E[XIY = y) (or, more briefly, E[XIY)) represent expectation, variance,
covariance, and conditional expectation, respectively.
The n x n matrix with diagonal elements d1 , d2 , •.• ,dn and zeros elsewhere
is denoted by diag(d1 , d2 , •.. , dn ), and when all the di's are unity we have the
identity Il].atrix In.
Ifa is an n x 1 column vector with elements al, a2, . .. , an, we write a = (ai),
and the length or norm of ais denoted by Iiali. Thus
lIall = Va'a = (a~ + a~ + ... + a~y/2.
The vector with elements all equal to unity is represented by In, and the set
of all vectors having n elements is denoted by lRn .
If the m x n matrix A has elements aij, we write A = (aij), and the
sum of the diagonal elements, called the trace of A, is denoted by tr(A)
(= a11 + a22 + ... + akk, where k is the smaller of m and n). The transpose
1](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-23-320.jpg)
![4 VECTORS OF RANDOM VARIABLES
1.3 LINEAR REGRESSION MODELS
If we denote the response variable by Y and the explanatory variables by
Xl, X 2 , ... , X K , then a general model relating these variables is
although, for brevity, we will usually drop the conditioning part and write
E[Y]. In this book we direct our attention to the important class of linear
models, that is,
which is linear in the parameters {3j. This restriction to linearity is not as re-
strictive as one might think. For example, many functions of several variables
are approximately linear over sufficiently small regions, or they may be made
linear by a suitable transformation. Using logarithms for the gravitational
model, we get the straight line
logF == loga - (3 log d. (1.1)
For the linear model, the Xi could be functions of other variables z, w, etc.;
for example, Xl == sin z, X2 == logw, and X3 == zw. We can also have Xi == Xi,
which leads to a polynomial model; the linearity refers to the parameters,
not the variables. Note that "categorical" models can be included under our
umbrella by using dummy (indicator) x-variables. For example, suppose that
we wish to compare the means of two populations, say, JLi = E[Ui] (i = 1,2).
Then we can combine the data into the single model
E[Y] - JLl + (JL2 - JLl)X
- {30 + {3lX,
where X = awhen Y is a Ul observation and X = 1 when Y is a U2 observation.
Here JLl = {30 and JL2 == {30 +{3l, the difference being {3l' We can extend this
idea to the case of comparing m means using m - 1 dummy variables.
In a similar fashion we can combine two straight lines,
(j = 1,2),
using a dummy X2 variable which takes the value 0 if the observation is from
the first line, and 1 otherwise. The combined model is
E[Y] al + I'lXl + (a2 - al)x2 + (')'2 - I'I)XlX2
{30 +{3lXl + {32 x 2 + {33 x 3, (1.2)
say, where X3 == Xl X2. Here al == {30, a2 = {30 +{32, 1'1 == {3l, and 1'2 == {3l +{33'](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-26-320.jpg)
![EXPECTATION AND COVARIANCE OPERATORS 5
In the various models considered above, the explanatory variables mayor
may not be random. For example, dummy variables are nonrandom. With
random X-variables, we carry out the regression conditionally on their ob-
served values, provided that they are measured exactly (or at least with suf-
ficient accuracy). We effectively proceed as though the X-variables were not
random at all. When measurement errors cannot be ignored, the theory has
to be modified, as we shall see in Chapter 9.
1.4 EXPECTATION AND COVARIANCE OPERATORS
In this book we focus on vectors and matrices, so we first need to generalize
the ideas of expectation, covariance, and variance, which we do in this section.
Let Zij (i = 1,2, ... ,mj j = 1,2, ... ,n) be a set of random variables
with expected values E[Zij]. Expressing both the random variables and their
expectations in matrix form, we can define the general expectation operator
of the matrix Z = (Zij) as follows:
Definition 1.1
E[Z] = (E[Zij]).
THEOREM 1.1 If A = (aij), B = (bij ), and C = (Cij) are l x m, n x p,
and l x p matrices, respectively, of constants, then
E[AZB + C] = AE[Z]B + C.
Proof· Let W = AZB + Cj then Wij = 2::."=1 2:;=1 airZrsbsj +Cij and
E [AZB + C] = (E[Wij]) = (~~airE[Zrs]bsj + Cij )
= ((AE[Z]B)ij) + (Cij)
= AE[Z]B + C. 0
In this proof we note that l, m, n, and p are any positive integers, and
the matrices of constants can take any values. For example, if X is an m x 1
vector, tlien E[AX] = AE[X]. Using similar algebra, we can prove that if A
and B are m x n matrices of constants, and X and Yare n x 1 vectors of
random variables, then
E[AX + BY] = AE[X] + BE[Y].
In a similar manner we can generalize the notions of covariance and variance
for vectors. IT X and Yare m x 1 and n x 1 vectors of random variables, then
we define the generalized covariance operator Cov as follows:](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-27-320.jpg)
![6 VECTORS OF RANDOM VARIABLES
Definition 1.2
Cov[X, Y] = (COV[Xi , ¥j]).
THEOREM 1.2 If E[X) = a and E[Y) =(3, then
Cov[X, Y] = E [(X - a)(Y - (3)'].
Proof·
Cov[X, Y) = (COV[Xi, Yj])
= {E[(Xi - ai)(Yj - .aj)]}
=E {[(Xi - ai)(Yj - .aj)]}
=E [(X - a)(Y - (3)']. o
Definition 1.3 When Y = X, Cov[X, X], written as Var[X], is called the
variance (variance-covariance 01' dispersion) matrix of X. Thus
Var[X] - (cov[Xi , Xj])
var[X1] cov[XI , X 2 ] cov[XI,Xn]
COV[X2,XI) var[X2) cov[X2 ,Xn ]
(1.3)
cov[Xn,Xd cov[Xn,X2] var[Xn]
Since cov[Xi, X j ] = cov[Xj , Xi], the matrix above is symmetric. We note
that when X = Xl we write Var[X] = var[Xd.
From Theorem 1.2 with Y = X we have
Var[X] = E [(X - a)(X - a)'] , (1.4)
which, on expanding, leads to
Var[X] = E[XX') - aa'. (1.5)
These last two equations are natural generalizations of univariate results.
EXAMPLE 1.4 If a is any n x 1 vector of constants, then
Var[X - a) = Var[X].
This follows from the fact that Xi - ai - E[Xi - ail = Xi - E[Xi ], so that
o](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-28-320.jpg)
![EXPECTATION AND COVARIANCE OPERATORS 7
THEOREM 1.3 If X and Yare mx1 and n x1 vectors of random variables,
and A and B are l x m and p x n matrices of constants, respectively, then
Cov[AX, BY] = A Cov[X, Y]B'.
Proof. Let U = AX and V = BY. Then, by Theorems 1.2 and 1.1,
Cov[AX, BY] = Cov[U, V]
=E [(U - E[U]) (V - E[V])']
=E [(AX - Aa)(BY - B,8)']
=E [A(X - a)(Y - ,8)'B']
=AE [(X - a)(Y - ,8)'] B'
=A Cov[X, Y]B'.
From the theorem above we have the special cases
Cov[AX, Y] = A Cov[X, Y] and Cov[X, BY] = Cov[X, Y]B'.
(1.6)
o
Of particular importance is the following result, obtained by setting B = A
and Y = X:
Var[AX] = Cov[AX, AX] = ACov[X,X]A' = A Var[X]A'. (1.7)
EXAMPLE 1.5 If X, Y, U, and V are any (not necessarily distinct) n xl
vectors of random variables, then for all real numbers a, b, c, and d (including
zero),
Cov[aX + bY,eU + dV]
- ac Cov[X, U] +ad Cov[X, V] + be Cov[Y, U] + bd CoYlY,V].
(1.8)
To prove this result, we simply multiply out
E [(aX + bY - aE[X]- bE[Y])(cU +dV - cE[U] - dE[V])']
= E [(a(X - E[X]) + b(Y - E[Y])) (c(U - E[U]) + d(V - E[V]))'].
If we set U = X and V = Y, c = a and d = b, we get
Var[aX + bY] Cov[aX + bY,aX + bY]
- a2
Var[X] + ab(Cov[X, Y] + CoylY, X])
+b2
Var[Y]. (1.9)
o](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-29-320.jpg)
![8 VECTORS OF RANDOM VARIABLES
In Chapter 2 we make frequent use of the following theorem.
THEOREM 1.4 If X is a vector of random variables such that no element
of X is a linear combination of the remaining elements ri. e., there do not exist
a (=1= 0) and b such that a'X = b for all values of X = xj, then Var[X) is a
positive-definite matrix (see A.4).
Proof. For any vector e, we have
o < var[e'X)
e'Var[X)e [by equation (1.7»).
Now equality holds if and only if e'X is a constant, that is, if and only if
e'X = d (e =1= 0) or e = O. Because the former possibility is ruled out, e = 0
and Var[X) is positive-definite. 0
EXAMPLE 1.6 If X and Y are m x 1 and n x 1 vectors of random variables
such that no element of X is a linear combination of the remaining elements,
then there exists an n x m. matrix M such that Cov[X, Y - MX) == O. To
find M, we use the previous results to get
Cov[X, Y - MX) Cov[X, Y) - Cov[X, MX)
Cov[X, Y) - Cov[X, X]M'
Cov[X, Y) - Var[X)M'. (1.10)
By Theorem lA, Var[X] is positive-definite and therefore nonsingular (AA.1).
Hence (1.10) is zero for
M' = (Vai[X])-1 Cov[X, Y). o
EXAMPLE 1.7 We now give an example of a singular variance matrix by
using the two-cell multinomial distribution to represent a binomial distribu-
tion as follows:
(X X ) n! "'1 "'2 1
pr I = Xl, 2 = X2 = , ,PI P2 , PI +P2 == ,Xl + X2 = n.
Xl· X2'
IT X = (XI ,X2)', then
Var[X) = ( npl(l- PI)
-npIP2
which has rank 1 as P2 = 1 - Pl'
EXERCISES 1a
o
1. Prove that if a is a vector of constants with the same dimension as the
random vector X, then
E[(X - a)(X - a)') = Var[X] + (E[X] - a)(E[X] - a)'.](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-30-320.jpg)
![MEAN AND VARIANCE OF QUADRATIC FORMS 9
If Var[X] = E = ((J'ij), deduce that
E[IIX - aWl = L (J'ii + IIE[X] - aW·
i
2. If X and Y are m x 1 and n x 1 vectors of random variables, and a and
bare m x 1 and n x 1 vectors of constants, prove that
Cov[X - a, Y - b] = Cov[X, Y].
3. Let X = (XI ,X2 , ••• ,Xn)' be a vector of random variables, and let
YI = Xl, Yi = Xi - Xi - l (i = 2,3, ... , n). If the Yi are mutually
independent random variables, each with unit variance, find Var[X].
4. If Xl, X 2 , ••• , Xn are random variables satisfying Xi+l = pXi (i =
1,2, ... , n - 1), where p is a constant, and var[Xd = (J'2, find Var[X].
1.5 MEAN AND VARIANCE OF QUADRATIC FORMS
Quadratic forms play a major role in this book. In particular, we will fre-
quently need to find the expected value of a quadratic form using the following
theorem.
THEOREM 1.5 Let X = (Xi) be an n x 1 vector of random variables, and
let A be an n x n symmetric matrix. If E[X) = J1, and Var[X) = E = ((J'ij) ,
then
Proof·
E[X'AX) = tr(AE) + J1,'AJ1,.
E[X'AX) = tr(E[X'AX))
=E[tr(X'AX))
=E[tr(AXX')) [by A.1.2)
= tr(E[AXX'))
= tr(AE[XX'J)
= tr [A( Var[X) + J1,J1,')) [by (1.5))
= tr(AE) + tr(AJ.LJ1,')
=tr(AE) + J1,'AJ1, [by A.1.2). o
We can deduce two special cases. First, by setting Y = X - b and noting
that Var[Y) = Var[X) (by Example 1.4), we have
E[(X - b)'A(X - b)) = tr(AE) + (J1, - b)'A(J1, - b). (1.11)](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-31-320.jpg)
![10 VECTORS OF RANDOM VARIABLES
Second, if ~ = 0-2In (a common situation in this book), then tr(A~) =
0-2
tr(A). Thus in this case we have the simple rule
E[X'AX] = 0-2(sum of coefficients of Xl) + (X'AX)x=l'. (1.12)
EXAMPLE 1.8 If Xl, X 2 , • •• ,Xn are independently and identically dis-
tributed with mean J.t and variance 0-2, then we can use equation (1.12) to
find the expected value of
Q = (Xl - X 2)2 + (X2 - X3)2 + ... + (Xn-l - Xn)2.
To do so, we first write
n n-l
Q = X'AX = 2 LX; - xl - X~ - 2 L XiXi+l.
i=l i=l
Then, since COV[Xi' Xj] = 0 (i f= j), ~ = 0-2In and from the squared terms,
tr(A) = 2n - 2. Replacing each Xi by J.t in the original expression for Q, we
see that the second term of. E[X'AX] is zero, so that E[Q] = 0-2(2n - 2). 0
EXAMPLE 1.9 Suppose that the elements of X = (Xl ,X2, ... ,Xn)' have
a common mean J.t and X has variance matrix ~ with o-ii = 0-2 and o-ij = p0-2
(i f= j). Then, when p = 0, we know that Q = Ei(Xi - X)2 has expected
value 0-2 (n - 1). To find its expected value when p f= 0, we express Q in the
form X'AX, where A = [(Oij - n-l
)] and
1 -1
-n -n-1 -n-1
1 p p
-1 1 -1 -1
P 1 p
A~ 0-2 -n -n -n
-n-1
-n-1 1 -1
-n P P 1
0-2(1 - p)A.
Once again the second term in E[Q] is zero, so that
E[Q] = tr(A~) = 0-2(1- p) tr(A) =0-2(1- p)(n - 1). 0
THEOREM 1.6 Let Xl, X 2, ... ,Xn be independent random variables with
means (h, B2, ... ,Bn, common variance J.t2, and common third and fourth mo-
ments about their means, J.t3 and J.t4, respectively (i.e., J.tr = E[(Xi - Bit]).
If A is any n x n symmetric matrix and a is a column vector of the diagonal
elements of A, then
var[X'AX] = (J.t4 - 3J.t~)a' a + 2J.t~ tr(A2) + 4J.t2(J'A 2(J + 4J.t3(J'Aa.
(This result is stated without proof in Atiqullah {1962}.}
Proof. We note that E[X] = (J, Var[X] = J.t2In, and
Var[X'AX] = E[(X'AX)2]- (E[X'AX])2. (1.13)](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-32-320.jpg)
![MEAN AND VARIANCE OF QUADRATIC FORMS 11
Now
X'AX = (X - O)'A(X - 0) + 20'A(X - fJ) + O'AfJ,
so that squaring gives
(X'AX)2 = [(X - 0)'A(X - 0)]2 +4[0'A(X - 0)]2 + (0' AfJ)2
+ 20'AO[(X -0)'A(X - 0) +40'AOO'A(X - 0)]
+40'A(X - O)(X - O)'A(X - 0).
Setting Y =X - 0, we have E[Y] = 0 and, using Theorem 1.5,
E[(X'AX)2] = E[(Y'Ay)2] +4E[(O'Ay)2] + (O'AO?
+ 20'AOJ.L2 tr(A) + 4E[O'AYY'AY].
As a first step in evaluating the expression above we note that
(Y'Ay)2 = 2:2:2:2>ijaklYiYjYkll.
i j k I
Since the l'i are mutually independent with the same first four moments about
the origin, we have
Hence
i = j = k = l,
i =j, k =lj i = k, j =lj i =l,j = k,
otherwise.
E[(Y'Ay)2] - J.L4 L:>~i + J.L~ L (L aiiakk + 2:atj + L aijaji)
i i k#-i #i #i
- (J.L4 - 3J.L~)a'a + J.L~ [tr(A)2 + 2tr(A2)] , (1.14)
since A is symmetric and Ei Ej a~j = tr(A2). Also,
say, and
so that
and
(O'Ay)2 = (b'y)2 = LLbibjYiYj,
i j
fJ'Ayy'AY = LLLbiajkYiYjYk,
i j k
E[(O'Ay)2] =J.L2 L b~ =J.L2b'b = J.L20'A20
i
E[O'AYY'AY] = J.L3 L biaii = J.L3b'a = J.L30'Aa.
i](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-33-320.jpg)
![12 VECTORS OF RANDOM VARIABLES
Finally, collecting all the terms and substituting into equation (1.13) leads to
the desired result. 0
EXERCISES Ib
1. Suppose that Xl, X 2 , and X3 are random variables with common mean
fl, and variance matrix
Var[X] = u2
( ~
1 o
1
I
'4
2. If Xl, X 2 , ••• , Xn are independent random variables with common mean
fl, and variances u?, u~, ... , u;, prove that I:i(Xi - X)2 I[n(n -1)] is an
unbiased estimate of var[X].
3. Suppose that in Exercise 2 the variances are known. Let X w = I:iWiXi
be an unbiased estimate of fl, (Le., I:iWi = 1).
(a) Prove that var[Xw] is minimized when Wi <X l/ur Find this min-
imum variance Vrnin.
(b) Let S! = L:iWi (Xi - Xw)2/(n - 1). If WW; = a (i = 1,2, ... , n),
prove that E[S!] is an unbiased estimate of Vrnin.
4. The random variables Xl, X 2 , ••• , Xn have a common nonzero mean fl"
a common variance u 2 , and the correlation between any pair of random
variables is p.
(a) Find var[X] and hence prove that -1I(n - 1) < P < 1.
(b) If
Q = a ~X; +b (~Xi)2
is an unbiased estimate of u 2 , find a and b. Hence show that, in
this case,
Q _ n (Xi _ X)2
-~(I-p)(n-1)"
5. Let Xl, X 2 , • •• , Xn be independently distributed as N(fl" ( 2
). Define](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-34-320.jpg)
![14 VECTORS OF RANDOM VARIABLES
then Xl = (X1, ... ,Xr )' andX2 = (Xr +1,' .. 'Xn )' are statistically indepen-
dent. An equivalent result is that Xl and X 2 are independent if and only if
we have the factorization
Mx(t) = a(tI, ... ,tr)b(tr+l, ... ,tn)
for some functions a(·) and b(·).
EXAMPLE 1.11 Suppose that the joint distribution of the vectors of ran-
dom variables X and Y have a joint m.g.f. which exists in an interval contain-
ing the origin. Then if X and Yare independent, so are any (measurable)
functions of them. This follows from the fact that if c(·) and d(·) are suitable
vector functions,
E[exp{s'c(X) + s'd(Y)} = E[exp{s'c(X)}]E[exp{s'd(Y)}] = a(s)b(t),
say. This result is, in fact, true for any X and Y, even if their m.g.f.'s do not
exist, and can be proved using characteristic functions. 0
Another route .that we shall use for proving independence is via covariance.
It is well known that cov[X, Y] = 0 does not in general imply that X and
Y are independent. However, in one important special case, the bivariate
normal distribution, X and Y are independent if and only if cov[X, Y] = O. A
generalization of this result applied to the multivariate normal distribution is
given in Chapter 2. For more than two variables we find that for multivariate
normal distributions, the variables are mutually independent if and only if
they are pairwise independent. Bowever, pairwise independence does not
necessarily imply mutual independence, as we see in the following example.
EXAMPLE 1.12 Suppose that Xl, X 2 , and X3 have joint density function
(27r) -3/2 exp [- ~xt + x~ + xm
x {I + XIX2X3 exp [-Hx~ + x~ + x~)]}
-00 < Xi < 00 (i = 1,2,3).
Then the second term in the braces above is an odd function of X3, so that
its integral over -00 < X3 < 00 is zero. Hence
(27r)-1 exp [-~(x~ + xm
!I(Xd!z(X2),
and Xl and X 2 are independent N(O,l) variables. Thus although Xl, X 2 ,
and X3 are pairwise independent, they are not mutually independent, as](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-36-320.jpg)
![MOMENT GENERATING FUNCTIONS AND INDEPENDENCE 15
EXERCISES Ie
1. If X and Y are random variables with the same variance, prove that
cov[X +Y, X - Y] = O. Give a counterexample which shows that zero
covariance does not necessarily imply independence.
2. Let X and Y be discrete random variables taking values 0 or 1 only,
and let pr(X = i, Y = j) =Pij (i = 1, OJ j = 1,0). Prove that X and Y
are independent if and only if cov[X, Y] = o.
3. If X is a random variable with a density function symmetric about zero
and having zero mean, prove that cov[X, X2] = O.
4. If X, Y and Z have joint density function
f(x,y,z) = i(1 + xyz) (-1 < x,y,z < 1),
prove that they are pairwise independent but not mutually independent.
MISCElLANEOUS EXERCISES I
1. If X and Y are random variables, prove that
var[X) = Ey{ var[XJYJ} + vary{E[XJYJ}.
Generalize this result to vectors X and Y of random variables.
(
523)
Var[X) = 2 3 0 .
303
(a) Find the variance of Xl - 2X2 + X 3 •
(b) Find the variance matrix of Y = (Yi, }2)', where Yl = Xl + X 2
and Y2 = Xl +X2 +X3 •
3. Let Xl, X2, . .. , Xn be random variables with a common mean f.L. Sup-
pose that cov[Xi , Xj) = 0 for all i and j such that j > i + 1. If
i=l
and](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-37-320.jpg)
![16 VECTORS OF RANDOM VARIABLES
prove that
E [3Ql - Q2] = var[X].
n(n - 3)
4. Given a random sample Xl ,X2,X3 from the distribution with density
function
f(x) = ~
find the variance of (Xl - X 2)2 + (X2 - X3)2 + (X3 - Xl)2.
5. If Xl, ... , Xn are independently and identically distributed as N(O, 0"2),
and A and B are any n x n symmetric matrices, prove that
Cov[X'AX, X'BX] = 20"4 tr(AB).](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-38-320.jpg)
![2
Multivariate Normal Distribution
2.1 DENSITY FUNCTION
Let E be a positive-definite n x n matrix and I-L an n-vector. Consider the
(positive) function
where k is a constant. Since E (and hence E-l by A.4.3) is positive-definite,
the quadratic form (y -I-L),E-l(y - I-L) is nonnegative and the function f is
bounded, taking its maximum value of k-1 at y = I-L.
Because E is positive-definite, it has a symmetric positive-definite square
root El/2, which satisfies (El/2)2 = E (by A.4.12).
Let z = E-1/2(y - I-L), so that y = El/2z + I-L. The Jacobian of this
transformation is
J = det (8Yi
) = det(El/2) = [det(EW/2.
8zj
Changing the variables in the integral, we get
L:···L:exp[-~(y -1-L)'E-1(y -I-L)] dYl·· ·dYn
L:...L:exp(_~z'El/2E-lEl/2z)IJI dZl ... dZn
L:...L:exp(-~z'z)IJI dz1 ··• dZn
17](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-39-320.jpg)
![18 MULTIVARIATE tVOF?MAL DISTRIBUTION
,(,,, ,"CO
- PI [11 exp(-~zf) dZi
i=:I. J -(X)
n
i=l
(27r)n/2 det(:E)1/2.
Since f > 0, it follows that if k = (27r)n/2 det(:E)1/2, then (2.1) represents a
density function.
Definition 2.1 The distribution corresponding to the density (2.1) is called
the multivariate normal distribution.
THEOREM 2.1 If a random vector Y has density (2.1), then E[Y] = I-L
and Var[Y] = :E.
Proof. Let Z = :E-1/2(y - I-L). Repeating the argument above, we see, using
the change-of-variable formula, that Z has density
f[y(z)lIJI
(2.2)
(2.3)
The factorization of the joint density function in (2.2) implies that the Zi are
mutually independent normal variables and Zi '" N(O, 1). Thus E[Z] = 0 and
Var[Z] = In, so that
E[Y] = E[:E1/2Z + I-L] = :E1
/
2E[Z] + f-L =I-L
and
Var[Y] = Var[:E1
/
2Z + I-L] = Var[:E1/2Z] = :El/2In:El/2 = :E. 0
We shall use the notation Y ,...., Nn(I-L,:E) to indicate that Y has the density
(2.1). When n = 1 we drop the subscript.
EXAMPLE 2.1 Let Zl, .. " Zn be independent N(O,l) random variables.
The density of Z = (Zr, ... , Zn)' is the product of the univariate densities
given by (2.2), so that by (2.3) the density of Z is of the form (2.1) with
I-L = 0 and :E =In [Le., Z '" Nn(O, In)]. 0
We conclude that if Y '" Nn(I-L,:E) and Y = :E1
/
2
Z + f-L, then Z =
:E-1
/
2(y - f-L) and Z '" Nn(O,In). The distribution of Z is the simplest and
most fundamental example of the multivariate normal. Just as any univariate
normal can be obtained by rescaling and translating a standard normal with](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-40-320.jpg)
![DENSITY FUNCTION 19
mean zero and variance 1, so can any multivariate normal be thought of as
a rescaled and translated Nn(O, In). Multiplying by :E1
/
2
is just a type of
rescaling of the the elements of Z, and adding J1, is just a translation by J1,.
EXAMPLE 2.2 Consider the function
1
f(x,y) = 2 (1 2)1
7r - P 2 (1a;(1y
X exp {_ 1 2 [(X - ~a;)2 _ 2p (X - f-La;)(Y - f-Ly) + (y - ~y)2]}
2(1- P ) (1a; (1a;(1y (1y
where (1a; > 0, (1y > 0, and Ipi < 1. Then f is of the form (2.1) with
The density f above is the density of the bivariate normal distribution. 0
EXERCISES 2a
1. Show that
f(Yl,Y2) = k-
1
exp[-H2y~ +y~ + 2YIY2 - 22Yl - 14Y2 + 65)]
is the density of a bivariate normal random vector Y = (Y1 , Y2)'.
(a) Find k.
(b) Find E[Y] and Var[Y].
2. Let U have density 9 and let Y = A(U + c), where A is nonsingular.
Show that the density f of Y satisfies
f(y) = g(u)/I det(A)I,
where y = A(u +c).
3. (a) Show that the 3 x 3 matrix
E~O !n
is positive-definite for p > - t.
(b) Find :E1/ 2 when](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-41-320.jpg)
![20 MULTIVARIATE NORMAL DISTRIBUTION
2.2 MOMENT GENERATING FUNCTIONS
We can use the results of Section 2.1 to calculate the moment generating
fUnction (m.gJ.) of the multivariate normal. First, if Z ,...., Nn(O, In), then, by
the independence of the Zi'S, the m.gJ. of Z is
E[exp(t'Z)] E[exp (ttiZi )]
- E [fieXp(tiZi)]
n
- IIE [exp(tiZi)]
i=l
n
- IIexp(~t;)
i=l
exp(~t't). (2.4)
Now if Y '" Nn(l-t, E), we can write Y = E1/2Z + I-t, where Z '" Nn(O, In).
Hence using (2.4) and putting s = E1/2t, we get
E[exp(t'Y)] - E[exp{t'(E1/2Z + I-t)}]
E[exp(s'Z)) exp(t'I-t)
- exp( ~s's) exp(t'I-t)
- exp( ~t'E1/2 E1/2t + t'I-t)
- exp(t'I-t + ~t'Et). (2.5)
Another well-known result for the univariate normal is that if Y '" N(p"a2 ),
then aY + b is N(ap, + b, a2
(
2
) provided that a ::f. O. A similar result is true
for the multivariate normal, as we see below.
THEOREM 2.2 Let Y '" Nn(/L, E), C be an m x n matrix of rank m, and
d be an m x 1 vector. Then CY + d '" Nm(CI-t + d, CEC').
Proof. The m.gJ. of CY + d is
E{exp[t'(CY + d)]} E{exp[(C't)'Y + t'd]}
exp[(C't)'/L + ~(C't)'EC't + t'd]
- exp[t'(C/L + d) + ~t'CEC't).
Since C:EC' is positive-definite, the equation above is the moment generating
function of Nm(CI-t + d, CEC'). We stress that C must be of full rank to
ensure that CEC' is positive-definite (by A.4.5), since we have only defined
the multivariate normal for positive-definite variance matrices. 0](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-42-320.jpg)
![MOMENT GENERATING FUNCTIONS 21
COROLLARY If Y = AZ + 1-£, where A is an n x n nonsingular matrix,
then Y "" Nn(l-£, AA').
Proof. We replace Y, 1-£, E and d by Z, 0, In and 1-£, respectively, in Theorem
2.2. 0
EXAMPLE 2.3 Suppose that Y "" Nn(O, In) and that T is an orthogonal
matrix. Then, by Theorem 2.2, Z = T'Y is Nn(O, In), since T'T = In. 0
In subsequent chapters, we shall need to deal with random vectors of the
form CY, where Y is multivariate normal but the matrix C is not of full rank.
For example, the vectors of fitted values and residuals in a regression are of
this form. In addition, the statement and proof of many theorems become
much simpler if we admit the possibility of singular variance matrices. In
particular we would like the Corollary above to hold in some sense when C
does not have full row rank.
Let Z "" Nm(O, 1m), and let A be an n x m matrix and 1-£ an n x 1 vector.
By replacing El/2 by A in the derivation of (2.5), we see that the m.g.f. of
Y =AZ + 1-£ is exp(t'1-£ + ~t'Et), with E =AA'. Since distributions having
the same m.g.f. are identical, the distribution of Y depends on A only through
AA'. We note that E[Y] = AE[Z] + 1-£ = 1-£ and Var[Y] = A Var[Z]A' =
AA'. These results motivate us to introduce the following definition.
Definition 2.2 A random n x 1 vector Y with mean 1-£ and variance matrix E
has a multivariate normal distribution if it has the same distribution as AZ +
1-£, where A is any n x m matrix satisfying E = AA' and Z "" Nm(O, 1m). We
write Y "" AZ +1-£ to indicate that Y and AZ +1-£ have the same distribution.
We need to prove that whenE is positive-definite, the new definition is
equivalent to the old. As noted above, the distribution is invariant to the
choice of A, as long as E = AA'. If E is of full rank (or, equivalently, is
positive-definite), then there exists a nonsingular A with E = AA', by AA.2.
If Y is multivariate normal by Definition 2.1, then Theorem 2.2 shows that
Z = A -1 (Y - 1-£) is Nn(O, In), so Y is multivariate normal in the sense of
Definition 2.2. Conversely, if Y is multivariate normal by Definition 2.2, then
its m.g.f. is given by (2.5). But this is also the m.g.f. of a random vector
having dellsity (2.1), so by the uniqueness of the m.g.f.'s, Y must also have
density (2.1).
If E is of rank m < n, the probability distribution of Y cannot be expressed
in terms of a density function. In both cases, irrespective of whether E is
positive-definite or just positive-semidefinite, we saw above that the m.g.f. is
exp (t'1-£ + ~t'Et) . (2.6)
We write Y "" Nm(l-£, E) as before. When E has less than full rank, Y is
sometimes said to have a singular distribution. From now on, no assumption
that E is positive-definite will be made unless explicitly stated.](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-43-320.jpg)
![22 MULTIVARIATE NORMAL DISTRIBUTiON
EXAMPLE 2.4 Let Y '" N(/-t, (52) and put yl = (Y, -Y). The variance-
covariance matrix of Y is
Put Z = (Y - /-t)/(5. Then
1
-1
-1 )
1 .
Y = ( _~ ) Z + ( ~ ) = AZ + ~
and
E = AA/.
Thus Y has a multivariate normal distribution. o
EXAMPLE 2.5 We can show that Theorem 2.2 remains true for random
vectors having this extended definition of the multivariate normal without the
restriction on the rank of A. If Y '" Nn(~' E), then Y '" AZ +~. Hence
CY '" CAZ + C~ = HZ + b, say, and CY is multivariate normal with
E[CY] = b = C~ and Var[CY] = BB' =CANC' = CEC/. 0
EXAMPLE 2.6 Under the extended definition, a constant vector has a
multivariate normal distribution. (Take A to be a matrix of zeros.) In par-
ticular, if A is a zero row vector, a scalar constant has a (univariate) normal
distribution under this definition, so that we regard constants (with zero vari-
ance) as being normally distributed. 0
EXAMPLE 2.7 (Marginal distributions) Suppose that Y '" Nn(~, E) and
we partition Y, ~ and E conformably as
Then Y 1 '" Np(~l' Ell). We see this by writing Y 1 = BY, where B = (Ip, 0).
Then B~ = ~l and BEB' = Eu , so the result follows from Theorem 2.2.
Clearly, Y 1 can be any subset of Y. In other words, the marginal distributions
of the multivariate normal are multivariate normal. 0
Our final result in this section is a characterization of the multivariate
normal.
THEOREM 2.3 A random vector Y with variance-covariance matrix E and
mean vector ~ has a Nn(~, E) distribution if and only if a/y has a univariate
normal distribution for every vector a.
Proof. First, assume that Y '" Nn(~, E). Then Y '" AZ + ~, so that a/y '"
al
AZ + al
~ = (A/a)'Z + a/~. This has a (univariate) normal distribution in
the sense of Definition 2.2.](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-44-320.jpg)
![MOMENT GENERATING FUNCTIONS 23
Conversely, assume that t'Y is a univariate normal random variable for all
t. Its mean is t'I-t and the variance is t'Et. Then using the formula for the
m.g.f. of the univariate normal, we get
E{exp[s(t'Y)]} = exp[s(t'I-t) + ~s2(t'Et)].
Putting s = 1 shows that the m.g.f. of Y is given by (2.6), and thus Y ,....,
Nn(J-t,E). 0
We have seen in Example 2.7 that the multivariate normal has normal
marginalsj and in particular the univariate marginals are normal. However,
the converse is not true, as the following example shows. Consider the function
which is nonnegative (since 1 + ye-y2
> 0) and integrates to 1 (since the
integral r~: ye-y2
/
2 dy has value 0). Thus f is a joint density, but it is not
bivariate normal. However,
1 1 1+00
I'<'L exp(- ~y~) x I'<'L exp(- ~y~) dY2
y 27r y 27r -00
1 1 1+00
+ I'<'LYl exp(- ~Yn x I'<'L Y2 exp(- ~y~) dY2
y 27r y 27r -00
~ exp(- ~y~),
so that the marginals are N(O, 1). In terms of Theorem 2.3, to prove that Y
is bivariate normal, we must show that a'Y is bivariate normal for all vectors
a, not just for the vectors (1,0) and (0,1). Many other examples such as
this are known; see, for example, Pierce and Dykstra [1969], Joshi [1970], and
Kowalski [1970].
EXERCISES 2b
1. Find the moment generating function of the bivariate normal distribu-
tion given in Example 2.2.
2. If Y,""" Nn(J-t, E), show that Yi '" N(P,i, au).
3. Suppose that Y '" N3 (1-t, E), where
~~(n andE~U~D
Find the joint distribution of Zl =Y1 + Y2 + Y3 and Z2 =Yl - Y2.
4. Given Y '" Nn(I-t,In), find the joint density of a'Y and b'Y, where
a'b =0, and hence show that a'Y and b'Y are independent.](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-45-320.jpg)
![24 MULTIVARIATE NORMAL DISTRIBUTION
5. Let (Xi, Yi), i = 1,2, ... ,n, be a random sample from a bivariate normal
distribution. Find the joint distribution of (X, Y).
6. If Yl and Y2 are random variables such that Yi + Y2 and Yl - Y2 are
independent N(O, 1) random variables, show that Yl and Y2 have a
bivariate normal distribution. Find the mean and variance matrix of
Y = (Yl, Y2 )'.
7. Let Xl and X 2 have joint density
Show that Xl and X 2 have N(O, 1) marginal distributions.
(Joshi [1970])
8. Suppose that Yl , Y2 , • •• , Yn are independently distributed as N(O,l).
Calculate the m.g.f. of the random vector
(Y, Yl - Y, Y2 - Y, ... ,Yn - Y)
and hence show that Y is independent of 'Ei(Yi _ y)2.
(Hogg and Craig [1970])
9. Let Xl, X 2 , and X3 be LLd. N(O,l). Let
Yl (Xl + X 2 + X 3 )/V3,
- (Xl - X2 )/v'2,
(Xl + X 2 - 2X3)/V6.
Show that Yl , Y2 and Y3 are LLd. N(O,l). (The transformation above
is a special case of the so-called Helmert transformation.)
2.3 STATISTICAL INDEPENDENCE
For any pair of random variables, independence implies that the pair are
uncorrelated. For the normal distribution the converse is also true, as we now
show.
THEOREM 2.4 Let Y '" Nn(J.L, I:.) and partition Y, f-t and I:. as in Example
2.7. Then Yland Y 2 are independent if and only if I:.12 = 0.
Proof. The m.g.f. of Y is exp (t'f-t + ~t' I:.t). Partition t conformably with Y.
Then the exponent in the m.g.f. above is
t~f-tl + t~f-t2 + ~t~I:.lltl + ~t~I:.22t2 + t~I:.12t2. (2.7)](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-46-320.jpg)
![STATISTICAL INDEPENDENCE 25
If E12 = 0, the exponent can be written as a function of just tl plus a function
of just t 2, so the m.g.f. factorizes into a term in tl alone times a term in t2
alone. This implies that Y I" and Y2 are independent.
Conversely, if Y1 and Y2 are independent, then
where M is the m.g.f. of Y. By (2.7) this implies that t~E12t2 = 0 for
all tl and t 2, which in turn implies that E12 = O. [This follows by setting
tl = (1,0, ... ,0)', etc.] 0
We use this theorem to prove our next result.
THEOREM 2.5 Let Y ....., Nn(/-L, E) and define U = AY, V = BY. Then
U and V are independent if and only if Cov[U, V] = AEB' = O.
Proof. Consider
Then, by Theorem 2.2, the random vector W is multivariate normal with
variance-covariance matrix
Var[W] = ( ~ ) Var[Y) (A',B') = ( ~~1: AEB' )
REB' .
Thus, by Theorem 2.4, U and V are independent if and only if AEB' = O. 0
EXAMPLE 2.8 Let Y ....., N n (/-L,0'2In ) and let In be an n-vector of 1's.
Then the sample mean Y = n-1
Zi Yi is independent of the sample variance
8 2 = (n - 1)-1 L:i(Yi - y)2. To see this, let I n = Inl~ be the n x n matrix
of 1's. Then Y = n-ll~Y (= AY, say) and
~-Y
Y2 -Y
Yn-Y
say. Now
A~B' -II' 21 (I -IJ) 2 -11 2 -11 0
~ = n nO' n n - n n = 0' n n - 0' n n = ,
so by Theorem 2.5, Y is independent of (~ - Y, ... ,Yn - Y),
independent of 8 2.
and hence
o
EXAMPLE 2.9 Suppose that Y ....., Nn(/-L, E) with E positive-definite, and
Y is partitioned into two subvectors y' = (Yi, Y~), where Y1 has di~ension](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-47-320.jpg)
![DISTRIBUTION OF QUADRATIC FORMS 27
3. Given Y ,...., N3 (J-L, E), where
E~U'U PO)
1 P ,
P 1
for what value(s) of P are Yi + Y2 + Y3 and Y1 - Y2 - Y3 statistically
independent?
2.4 DISTRIBUTION OF QUADRATIC FORMS
Quadratic forms in normal variables arise frequently in the theory of regression
in connection with various tests of hypotheses. In this section we prove some
simple results concerning the distribution of such quadratic forms.
Let Y ,..., Nn(J-L, E), where E is positive-definite. We are interested in the
distribution of random variables of the form yl AY = L:?=1 L:;=1 aij Y,;1j.
Note that we can always assume that the matrix A is symmetric, since if
not we can replace aij with ~(aij + aji) without changing the value of the
quadratic form. Since A is symmetric, we can diagonalize it with an orthog-
onal transformation; that is, there is an orthogonal matrix T and a diagonal
matrix D with
TIAT = D = diag(d1 , ... , dn ). (2.9)
The diagonal elements di are the eigenvalues of A and can be any real num-
bers.
We begin by assuming that the random vector in the quadratic form has a
Nn(O,In) distribution. The general case can be reduced to this through the
usual transformations. By Example 2.3, if T is an orthogonal matrix and Y
has an Nn(O, In) distribution, so does Z = T/y. Thus we can write
n
y/AY =y/TDT/y =Z/DZ =L:diZl,
i=l
(2.10)
so the distribution of ylAY is a linear combination of independent X~ random
variables. Given the values of di , it is possible to calculate the distribution,
at least numerically. Farebrother [1990] describes algorithms for this.
There is an important special case that allows us to derive the distribution
of the quadratic form exactly, without recourse to numerical methods. If r of
the eigenvalues di are 1 and the remaining n - r zero, then the distribution
is the sum of r independent X~'s, which is X~. We can recognize when the
eigenvalues are zero or 1 using the following theorem.
THEOREM 2.6 Let A be a symmetric matrix. Then A has r eigenvalues
equal to 1 and the rest zero if and only if A2 = A and rank A = r.](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-49-320.jpg)
![DISTRIBUTION OF QUADRATIC FORMS 29
so that Y'(In - A)Y is X~-r' 0
EXAMPLE 2.12 Suppose that A and B are symmetric, Y '" Nn(O, In),
and Y'AY and Y'BY are both chi-squared. Then Y'AY and Y'BY are
independent if and only if AB = O.
To prove this, suppose first that AB = O. Since A and B are idempo-
tent, we can write the quadratic forms as Y'AY = YA'AY = IIAYl12 and
Y'BY = IIBYII2. By Theorem 2.5, AY and BY are independent, which
implies that the quadratic forms are independent.
Conversely, suppose that the quadratic forms are independent. Then their
sum is the sum of independent chi-squareds, which implies that Y'(A + B)Y
is also chi-squared. Thus A + B must be idempotent and
A + B = (A + B)2 = A2 + AB + BA + B2 = A + AB + BA + B,
so that
AB +BA = O.
Multiplying on the left by A gives AB + ABA = 0, while multiplying on the
right by A gives ABA + BA = OJ hence AB = BA = O. 0
EXAMPLE 2.13 (Hogg and Craig [1958, 1970]) Let Y '" Nn (8, 0"2In) and
let Qi = (Y - 8)'Pi(Y - 8)/0"2 (i = 1,2). We will show that if Qi '" X~. and
QI - Q2 > 0, then QI - Q2 and Q2 are independently distributed as X;1-r2
and X~2' respectively.
We begin by noting that if Qi '" X~i' then P~ =Pi (Theorem 2.7). Also,
QI - Q2 > 0 implies that PI - P 2 is positive-semidefinite and therefore
idempotent (A.6.5). Hence, by Theorem 2.7, QI - Q2 '" X~, where
r rank(Pl - P 2)
tr(PI - P 2)
- trPI - trP2
rankPI - rankP2
Also, by A.6.5, P IP 2 = P 2P I = P 2, and (PI - P 2 )P2 = O. Therefore,
since Z = (Y - 8)/0"2", Nn(O,In), we have, by Example 2.12, that QI - Q2
[= Z'(PI - P 2)Z] is independent of Q2 (= Z'P2Z). 0
We can use these results to study the distribution of quadratic forms when
the variance-covariance matrix :E is any positive-semidefinite matrix. Suppose
that Y is now Nn(O, :E), where :E is of rank 8 (8 < n). Then, by Definition 2.2
(Section 2.2), Y has the same distribution as RZ, where :E = RR' and R is
n x 8 of rank 8 (A.3.3). Thus the distribution of Y'AY is that of Z'R'ARZ,](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-51-320.jpg)
![30 MULTlVARIATE NORMAL DISTRjBUTJON
which, by T~'leOTem 2.7, will be X~ if and only if RIAR is idempotent of rank
r. However, this is not a very useful condition. A better one is contained in
our next theorem.
THEOREM 2.8 Suppose that Y '" Nn(O, ~), and A is symmetric. Then
yl AY is X~ if and only if r of the eigenvalues of A~ are 1 and the rest are
zero.
Proof. We assume that Y'AY = Z'R'ARZ is X~. Then R'AR is symmetric
and idempotent with r unit eigenvalues and the rest zero (by A.6.1), and its
rank equals its trace (A.6.2). Hence, by (A.1.2),
r =rank(R'AR) =tr(R'AR) =tr(ARR') =tr(A~).
Now, by (A.7.1), R'AR and ARR' = A~ have the same eigenvalues, with
possibly different multiplicities. Hence the eigenvalues of A~ are 1 or zero.
As the trace of any square matrix equals the sum of its eigenvalues (A.1.3), r
of the eigenvalues of A~ must be 1 and the rest zero. The converse argument
is just the reverse of the' one above. 0
For nonsymmetric matrices, idempotence implies that the eigenvalues are
zero or 1, but the converse is not true. However, when ~ (and hence R) has
full rank, the fact that R'AR is idempotent implies that A~ is idempotent.
This is because the equation
R'ARR'AR = R'AR
can be premultiplied by (R')-land postmultiplied by R' to give
A~A~=A~.
Thus we have the following corollary to Theorem 2.8.
COROLLARY Let Y '" Nn(O, ~), where :E is positive-definite, and sup-
pose that A is symmetric. Then Y'AY is X~ if and only A:E is idempotent
and has rank r.
For other necessary and sufficient conditions, see Good [1969, 1970] and
Khatri [1978].
Our final theorem concerns a very special quadratic form that arises fre-
quently in statistics.
THEOREM 2.9 Suppose that Y '" NnUL, :E), where :E is positive-definite.
Then Q = (Y - 1-£)':E-1(y - 1-£) <'oJ X;.
Proof. Making the transformation Y = :E1/2
Z +1-£ considered in Theorem 2.1,
we get
n
Q = Z':EI/2:E-l:EI/2Z = Z'Z = L Z?'
i=l](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-52-320.jpg)
![DISTRIBUTION OF QUADRATIC FORMS 31
Since the Zl's are independent x~ variables, Q '" X~. 0
EXERCISES 2d
1. Show that the m.g.f. for (2.10) is n~(1- 2tdi )-1/2.
2. Let Y '" Nn(O, In) and let A be symmetric.
(a) Show that the m.g.f. of Y'AY is [det(ln - 2tA)]-1/2.
(b) If A is idempotent ofrank r, show that the m.g.f. is (1 - 2t)-r/2.
(c) Find the m.g.f. if Y '" Nn(O, ~).
3. If Y '" N 2 (0, 12 ), find values of a and b such that
aCYl - y2)2 + b(Yl + y2)2 '" X~·
4. Suppose that Y '" N3 (0, In). Show that
t [(Yl - y2)2 + (Y2 - y3 )2 + (Y3 - Yl)2]
has a X~ distribution. Does some multiple of
(Yl - y2 )2 + (Y2 - y3 )2 + ... + (Yn-l - Yn)2 + (Yn - Yd
have a chi-squared distribution for general n?
5. Let Y '" Nn(O, In) and let A and B be symmetric. Show that the joint
m.g.f. of Y'AY and Y'BY is [det(ln - 2sA - 2tB)]-1/2. Hence show
that the two quadratic forms are independent if AB = 0.
MISCELLANEOUS EXERCISES 2
1. Suppose that e '" N 3 (0, (1"213) and that Yo is N(O, (1"5), independently of
the c:/s. Define
}i = p}i-l +C:i (i = 1,2,3).
(a) Find the variance-covariance matrix of Y = (Yl , Y2 , Y3 )'.
(b) What is the distribution of Y?
2. Let Y '" Nn(O, In), and put X = AY, U =BY and V =CY. Suppose
that Cov[X, U] = °and Cov[X, V] = 0. Show that X is independent
ofU + V.
3. If Yl , Y2 , ••• , Yn is a random sample from N(IL, (1"2), prove that Y is
independent of L:~;ll (}i - }i+l)2.
4. If X and Y are n-dimensional vectors with independent multivariate
normal distributions, prove that aX +bY is also multivariate normal.](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-53-320.jpg)
![CHAPTER VI.
DISCOVERY OF THE GOLD PLACERS.
By the treaty concluded between the United States and Mexico, in
1847, the territory of Upper California became the property of the
United States. Little thought the Mexican government of the value of
the land they were ceding, further than its commercial importance;
and, doubtless, little thought the buyers of the territory, that its soil
was pregnant with a wealth untold, and that its rivers flowed over
golden beds.
This territory, now belonging to the American Union, embraces an
area of 448,961 square miles. It extends along the Pacific coast,
from about the thirty-second parallel of north latitude, a distance of
near seven hundred miles, to the forty-second parallel, the southern
boundary of Oregon. On the east, it is bounded by New Mexico.
During the long period which transpired between its discovery and
its cession to the United States, this vast tract of country was
frequently visited by men of science, from all parts of the world.
Repeated examinations were made by learned and enterprising
officers and civilians; but none of them discovered the important
fact, that the mountain torrents of the Sierra Nevada were
constantly pouring down their golden sands into the valleys of the
Sacramento and San Joaquin. The glittering particles twinkled
beneath their feet, in the ravines which they explored, or glistened
in the watercourses which they forded, yet they passed them by
unheeded. Not a legend or tradition was heard among the white
settlers, or the aborigines, that attracted their curiosity. A nation's
ransom lay within their grasp, but, strange to say, it escaped their
notice—it flashed and sparkled all in vain.[1]](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-61-320.jpg)
![The Russian American Company had a large establishment at Ross
and Bodega, ninety miles north of San Francisco, founded in the
year 1812; and factories were also established in the territory by the
Hudson Bay Company. Their agents and employes ransacked the
whole country west of the Sierra Nevada, or Snowy Mountain, in
search of game. In 1838, Captain Sutter, formerly an officer in the
Swiss Guards of Charles X., King of France, emigrated from the state
of Missouri to Upper California, and obtained from the Mexican
government a conditional grant of thirty leagues square of land,
bounded on the west by the Sacramento river. Having purchased the
stock, arms, and ammunition of the Russian establishment, he
erected a dwelling and fortification on the left bank of the
Sacramento, about fifty miles from its mouth, and near what was
termed, in allusion to the new settlers, the American Fork. This
formed the nucleus of a thriving settlement, to which Captain Sutter
gave the name of New Helvetia. It is situated at the head of
navigation for vessels on the Sacramento, in latitude 38° 33' 45"
north, and longitude 121° 20' 05" west. During a residence of ten
years in the immediate vicinity of the recently discovered placéras,
or gold regions, Captain Sutter was neither the wiser nor the richer
for the brilliant treasures that lay scattered around him.[2]
In the year 1841, careful examinations of the Bay of San Francisco,
and of the Sacramento River and its tributaries, were made by
Lieutenant Wilkes, the commander of the Exploring Expedition; and
a party under Lieutenant Emmons, of the navy, proceeded up the
valley of the Willamette, crossed the intervening highlands, and
descended the Sacramento. In 1843-4, similar examinations were
made by Captain, afterwards Lieutenant-Colonel Fremont, of the
Topographical Engineers, and in 1846, by Major Emory, of the same
corps. None of these officers made any discoveries of minerals,
although they were led to conjecture, as private individuals who had
visited the country had done, from its volcanic formation and
peculiar geological features, that they might be found to exist in
considerable quantities.[3]](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-62-320.jpg)
![the earth being rarely disturbed more than eighteen inches below
the surface. The value of the gold dust obtained by each man, per
day, is said to have ranged from ten to fifty dollars, and sometimes
even to have far exceeded that. The natural consequence of this
state of things was, that the price of labor, and, indeed, of every
thing, rose immediately from ten to twenty fold.[4]
As may readily be conjectured, every stream and ravine in the valley
of the Sacramento was soon explored. Gold was found on every one
of its tributaries; but the richest earth was discovered near the Rio
de los Plumas, or Feather River,[5] and its branches, the Yuba and
Bear rivers, and on Weber's creek, a tributary of the American Fork.
Explorations were also made in the valley of the San Joaquin, which
resulted in the discovery of gold on the Cosumnes and other
streams, and in the ravines of the Coast Range, west of the valley,
as far down as Ciudad de los Angeles.
In addition to the gold mines, other important discoveries were
made in Upper California. A rich vein of quicksilver was opened at
New Almaden, near Santa Clara, which, with imperfect machinery,—
the heat by which the metal is made to exude from the rock being
applied by a very rude process,—yielded over thirty per cent. This
mine—one of the principal advantages to be derived from which will
be, that the working of the silver mines scattered through the
territory must now become profitable—is superior to those of
Almaden, in Old Spain, and second only to those of Idria, near
Trieste, the richest in the world.
Lead mines were likewise discovered in the neighborhood of
Sonoma, and vast beds of iron ore near the American Fork, yielding
from eighty-five to ninety per cent. Copper, platina, tin, sulphur, zinc,
and cobalt, were discovered every where; coal was found to exist in
large quantities in the Cascade range of Oregon, of which the Sierra
Nevada is a continuation; and in the vicinity of all this mineral
wealth, there are immense quarries of marble and granite, for
building purposes.](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-64-320.jpg)
![Colonel Mason had succeeded Colonel Fremont in the post of
governor of California and military commandant. A regiment of New
York troops, under the command of Colonel Stevenson, had been
ordered to California before the conclusion of the treaty of peace,
and formed the principal part of the military force in the territory.
Colonel Mason expressed the opinion, in his official despatch, that
"there is more gold in the country drained by the Sacramento and
San Joaquin rivers, than will pay the cost of the [late] war with
Mexico a hundred times over." Should this even prove to be an
exaggeration, there can be little reason to doubt, when we take into
consideration all the mineral resources of the country, that the
territory of California is by far the richest acquisition made by this
government since its organization.
The appearance of the mines, at the period of Governor Mason's
visit, three months after the discovery, he thus graphically describes:
"At the urgent solicitation of many gentlemen, I delayed there [at
Sutter's Fort] to participate in the first public celebration of our
national anniversary at that fort, but on the 5th resumed the
journey, and proceeded twenty-five miles up the American Fork to a
point on it now known as the Lower Mines, or Mormon Diggins. The
hill-sides were thickly strewn with canvas tents and bush arbors; a
store was erected, and several boarding shanties in operation. The
day was intensely hot, yet about two hundred men were at work in
the full glare of the sun, washing for gold—some with tin pans, some
with close-woven Indian baskets, but the greater part had a rude
machine, known as the cradle. This is on rockers, six or eight feet
long, open at the foot, and at its head has a coarse grate, or sieve;
the bottom is rounded, with small cleats nailed across. Four men are
required to work this machine; one digs the ground in the bank close
by the stream; another carries it to the cradle and empties it on the
grate; a third gives a violent rocking motion to the machine; while a
fourth dashes on water from the stream itself.](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-65-320.jpg)
!["The sieve keeps the coarse stones from entering the cradle, the
current of water washes off the earthy matter, and the gravel is
gradually carried out at the foot of the machine, leaving the gold
mixed with a heavy, fine black sand above the first cleats. The sand
and gold, mixed together, are then drawn off through auger holes
into a pan below, are dried in the sun, and afterward separated by
blowing off the sand. A party of four men thus employed at the
lower mines, averaged $100 a day. The Indians, and those who have
nothing but pans or willow baskets, gradually wash out the earth
and separate the gravel by hand, leaving nothing but the gold mixed
with sand, which is separated in the manner before described. The
gold in the lower mines is in fine bright scales, of which I send
several specimens.
"From the mill [where the gold was first discovered], Mr. Marshall
guided me up the mountain on the opposite or north bank of the
south fork, where, in the bed of small streams or ravines, now dry, a
great deal of coarse gold has been found. I there saw several parties
at work, all of whom were doing very well; a great many specimens
were shown me, some as heavy as four or five ounces in weight,
and I send three pieces, labeled No. 5, presented by a Mr. Spence.
You will perceive that some of the specimens accompanying this,
hold mechanically pieces of quartz; that the surface is rough, and
evidently moulded in the crevice of a rock. This gold cannot have
been carried far by water, but must have remained near where it
was first deposited from the rock that once bound it. I inquired of
many people if they had encountered the metal in its matrix, but in
every instance they said they had not; but that the gold was
invariably mixed with washed gravel, or lodged in the crevices of
other rocks. All bore testimony that they had found gold in greater
or less quantities in the numerous small gullies or ravines that occur
in that mountainous region.
"On the 7th of July I left the mill, and crossed to a stream emptying
into the American Fork, three or four miles below the saw-mill. I
struck this stream (now known as Weber's creek) at the washings of](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-66-320.jpg)
![workshops have been established, and a considerable amount of
business is transacted there. Captain Sutter is held in very great
respect by the people of the settlement and those stopping at his
house on the road to the placers. Several versions of the account of
the discovery of the gold mines have been circulated, but the true
one, in the Captain's own words, is given in a work recently
published.[6] The account is here inserted, both on account of the
interest connected with the discovery, and in order to correct wrong
versions of the matter.
"I was sitting one afternoon," said the Captain, "just after my siesta,
engaged, by-the-bye, in writing a letter to a relation of mine at
Lucerne, when I was interrupted by Mr. Marshall—a gentleman with
whom I had frequent business transactions—bursting hurriedly into
the room. From the unusual agitation in his manner, I imagined that
something serious had occurred, and, as we involuntarily do in this
part of the world, I at once glanced to see if my rifle was in its
proper place. You should know that the mere appearance of Mr.
Marshall at that moment in the fort was quite enough to surprise
me, as he had but two days before left the place to make some
alterations in a mill for sawing pine planks, which he had just run up
for me, some miles higher up the Americanos. When he had
recovered himself a little, he told me that, however great my
surprise might be at his unexpected reappearance, it would be much
greater when I heard the intelligence he had come to bring me.
'Intelligence,' he added, 'which, if properly profited by, would put
both of us in possession of unheard-of wealth—millions and millions
of dollars, in fact.' I frankly own, when I heard this, that I thought
something had touched Marshall's brain, when suddenly all my
misgivings were put an end to by his flinging on the table a handful
of scales of pure virgin gold. I was fairly thunderstruck, and asked
him to explain what all this meant, when he went on to say, that,
according to my instructions, he had thrown the mill-wheel out of
gear, to let the whole body of the water in the dam find a passage
through the tail-race, which was previously too narrow to allow the
water to run off in sufficient quantity, whereby the wheel was](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-78-320.jpg)
![framers. These rules provided for the modus operandi of procuring
the gold, supplying the party with necessaries, attending to the sick,
and the division of the fruits of their labor.
One of the most frequented placers of California is called the
Stanislaus mine, situated near the Stanislaus River. It was one of the
first places worked to any extent by the gold-seekers, but not
satisfying the expectations of some of the most greedy, it has since
been partially abandoned. A description of this mine, and of the
living and operations of its workers in the winter of 1848-49, will
give a good general idea of the toils and privations endured by the
early gold-seekers in that region, and, also, of their mode of
procuring the precious metal at most of the mines. We extract from
a recently published work, distinguished for minuteness of detail and
accuracy of description.[7]
"The mine was a deep ravine, embosomed amidst lofty hills,
surmounted by, and covered with pine, and having, in the bottom
itself, abundance of rock, mud, and sand. Halliday and I encamped
at the very lowest part of the ravine, at a little distance from Don
Emanuel's party; a steep rock which towered above our heads
affording us shelter, and a huge, flat stone beneath our feet
promising a fair substitute for a dry bed. Here then we stretched our
macheers and blankets, and arranged our saddles and bags, so as to
make ourselves as comfortable and warm as possible, although, in
spite of our precautions and contrivances, and of a tolerably good
fire, our encampment was bitterly cold, and we lay exposed to a
heavy dew. We had given up our horses into the charge of the
Indians, and I saw to their being safely placed in the cavallard,
whilst Halliday went to chop wood; a task I was too weak to
perform. I cannot say we slept; we might more correctly be said to
have had a long and most uncomfortable doze, and when morning
broke, we were shivering with cold, and shook the dew in a shower
from our clothes. I consulted with my companion, and urged upon
him the prudence of our setting to work to construct ourselves a sort
of log cabin; otherwise I felt certain, from the experience of the past](https://image.slidesharecdn.com/1269675-250526055353-2c2da242/85/Linear-Regression-Analysis-2nd-Edition-Wiley-Series-In-Probability-And-Statistics-2nd-Edition-George-A-F-Seber-82-320.jpg)
Linear Regression Analysis 2nd Edition Wiley Series In Probability And Statistics 2nd Edition George A F Seber
- 1. Linear Regression Analysis 2nd Edition Wiley Series In Probability And Statistics 2nd Edition George A F Seber download https://ebookbell.com/product/linear-regression-analysis-2nd- edition-wiley-series-in-probability-and-statistics-2nd-edition- george-a-f-seber-2539350 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Methods And Applications Of Linear Models Regression And The Analysis Of Variance Wiley Series In Probability And Statistics 2nd Edition Ronald R Hocking https://ebookbell.com/product/methods-and-applications-of-linear- models-regression-and-the-analysis-of-variance-wiley-series-in- probability-and-statistics-2nd-edition-ronald-r-hocking-4876026 Regression Modeling Strategies With Applications To Linear Models Logistic And Ordinal Regression And Survival Analysis 2nd Edition Frank E Harrell https://ebookbell.com/product/regression-modeling-strategies-with- applications-to-linear-models-logistic-and-ordinal-regression-and- survival-analysis-2nd-edition-frank-e-harrell-5234906 Regression Modeling Strategies With Applications To Linear Models Logistic And Ordinal Regression And Survival Analysis 2nd Edition Frank E Harrell https://ebookbell.com/product/regression-modeling-strategies-with- applications-to-linear-models-logistic-and-ordinal-regression-and- survival-analysis-2nd-edition-frank-e-harrell-11104940 Linear Regression Analysis Theory And Computing 1st Edition Xin Yan https://ebookbell.com/product/linear-regression-analysis-theory-and- computing-1st-edition-xin-yan-43136366
- 3. Linear Regression With Python A Tutorial Introduction To The Mathematics Of Regression Analysis James V Stone https://ebookbell.com/product/linear-regression-with-python-a- tutorial-introduction-to-the-mathematics-of-regression-analysis-james- v-stone-189793490 Introduction To Linear Regression Analysis Wiley Series In Probability And Statistics 6th Edition Montgomery https://ebookbell.com/product/introduction-to-linear-regression- analysis-wiley-series-in-probability-and-statistics-6th-edition- montgomery-54899286 Formulas Useful For Linear Regression Analysis And Related Matrix Theory Its Only Formulas But We Like Them 1st Edition Puntanen Simo https://ebookbell.com/product/formulas-useful-for-linear-regression- analysis-and-related-matrix-theory-its-only-formulas-but-we-like- them-1st-edition-puntanen-simo-4261008 Introduction To Linear Regression Analysis 5th Edition Douglas C Montgomery https://ebookbell.com/product/introduction-to-linear-regression- analysis-5th-edition-douglas-c-montgomery-5702756 Introduction To Linear Regression Analysis 4th Edition Douglas C Montgomery https://ebookbell.com/product/introduction-to-linear-regression- analysis-4th-edition-douglas-c-montgomery-6975244
- 8. WILEY SERIES IN PROBABILITY AND STATISTICS Established by WALTER A. SHEWHART and SAMUEL S. WILKS Editors: David J. Balding, Peter Bloomfield, Noel A. C. Cressie, Nicholas I. Fisher, lain M. Johnstone, J. B. Kadane, Louise M. Ryan, David W. Scott, Adrian F. M. Smith, JozeJL. Teugels Editors Emeriti: Vic Barnett, J. Stuart Hunter, David G. Kendall A complete list ofthe titles in this series appears at the end ofthis volume.
- 9. Linear Regression Analysis Second Edition GEORGE A. F. SEBER ALANJ.LEE Department of Statistics University of Auckland Auckland, New Zealand ~WILEY ~INTERSCIENCE A JOHN WILEY & SONS PUBLICATION
- 10. Copyright © 2003 by John Wiley & Sons, Inc. All rights reserved. Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada.' No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Section 107 or 108 ofthe 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment ofthe appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4744, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, e-mail: permreq@wiley.com. Limit of LiabilitylDisclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representation or warranties with respect to the accuracy or completeness of the contents ofthis book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss ofprofit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services please contact our Customer Care Department within the U.S. at 877-762-2974, outside the U.S. at 317-572-3993 or fax 317-572-4002. Wiley also publishes its books in a variety ofelectronic formats. Some content that appears in print, however, may not be available in electronic format. Library ofCongress Cataloging-in-Publication Data Is Available ISBN 0-471-41540-5 Printed in the United States ofAmerica. 10 9 8 7 6 5 4 3 2 1
- 11. Contents Preface xv 1 Vectors of Random Variables 1 1.1 Notation 1 1.2 Statistical Models 2 1.3 Linear Regression Models 4 1.4 Expectation and Covariance Operators 5 Exercises la 8 1.5 Mean and Variance of Quadratic Forms 9 Exercises 1b 12 1.6 Moment Generating Functions and Independence 13 Exercises lc 15 Miscellaneous Exercises 1 15 2 Multivariate Normal Distribution 17 2.1 Density Function 17 Exercises 2a 19 2.2 Moment Generating Functions 20 Exercises 2b 23 2.3 Statistical Independence 24 v
- 12. VI CONTENTS Exercises 2c 26 2.4 Distribution of Quadratic Forms 27 Exercises 2d 31 Miscellaneous Exercises 2 31 3 Linear Regression: Estimation and Distribution Theory 35 3.1 Least Squares Estimation 35 Exercises 3a 41 3.2 Properties of Least Squares Estimates 42 Exercises 3b 44 3.3 Unbiased Estimation of (72 44 Exercises 3c 47 3.4 Distribution Theory 47 Exercises 3d 49 3.5 Maximum Likelihood Estimation 49 3.6 Orthogonal Columns in the Regression Matrix 51 Exercises 3e 52 3.7 Introducing Further Explanatory Variables 54 3.7.1 General Theory 54 3.7.2 One Extra Variable 57 Exercises 3f 58 3.8 Estimation with Linear Restrictions 59 3.8.1 Method of Lagrange Multipliers 60 3.8.2 Method of Orthogonal Projections 61 Exercises 3g 62 3.9 Design Matrix of Less Than Full Rank 62 3.9.1 Least Squares Estimation 62 Exercises 3h 64 3.9.2 Estimable Functions 64 Exercises 3i 65 3.9.3 Introducing Further Explanatory Variables 65 3.9.4 Introducing Linear Restrictions 65 Exercises 3j 66 3.10 Generalized Least Squares 66 Exercises 3k 69 3.11 Centering and Scaling the Explanatory Variables 69 3.11.1 Centering 70 3.11.2 Scaling 71
- 13. CONTENTS VII Exercises 31 72 3.12 Bayesian Estimation 73 Exercises 3m 76 3.13 Robust Regression 77 3.13.1 M-Estimates 78 3.13.2 Estimates Based on Robust Location and Scale Measures 3.13.3 Measuring Robustness 3.13.4 Other Robust Estimates Exercises 3n Miscellaneous Exercises 3 4 Hypothesis Testing 4.1· Introduction 4.2 Likelihood Ratio Test 4.3 F-Test 4.3.1 Motivation 4.3.2 Derivation Exercises 4a 4.3.3 Some Examples 4.3.4 The Straight Line Exercises 4b 4.4 Multiple Correlation Coefficient Exercises 4c 4.5 Canonical Form for H Exercises 4d 4.6 Goodness-of-Fit Test 4.7 F-Test and Projection Matrices Miscellaneous Exercises 4 5 Confidence Intervals and Regions 5.1 Simultaneous Interval Estimation 5.1.1 Simultaneous Inferences 5.1.2 Comparison of Methods 5.1.3 Confidence Regions 5.1.4 Hypothesis Testing and Confidence Intervals 5.2 Confidence Bands for the Regression Surface 5.2.1 Confidence Intervals 5.2.2 Confidence Bands 80 82 88 93 93 97 97 98 99 99 99 102 103 107 109 110 113 113 114 115 116 117 119 119 119 124 125 127 129 129 129
- 14. VIII CONTENTS 5.3 Prediction Intervals and Bands for the Response 5.3.1 Prediction Intervals 5.3.2 Simultaneous Prediction Bands 5.4 Enlarging the Regression Matrix Miscellaneous Exercises 5 131 131 133 135 136 6 Straight-Line Regression 139 6.1 The Straight Line 139 6.1.1 Confidence Intervals for the Slope and Intercept 139 6.1.2 Confidence Interval for the x-Intercept 6.1.3 Prediction Intervals and Bands 6.1.4 Prediction Intervals for the Response 6.1.5 Inverse Prediction (Calibration) Exercises 6a 6.2 Straight Line through the Origin 6.3 Weighted Least Squares for the Straight Line 6.3.1 Known Weights 6.3.2 Unknown Weights Exercises 6b 6.4 Comparing Straight Lines 6.4.1 General Model 6.4.2 Use of Dummy Explanatory Variables Exercises 6c 6.5 Two-Phase Linear Regression 6.6 Local Linear Regression Miscellaneous Exercises 6 7 Polynomial Regression 7.1 Polynomials in One Variable 7.1.1 Problem of Ill-Conditioning 7.1.2 Using Orthogonal Polynomials 7.1.3 Controlled Calibration 7.2 Piecewise Polynomial Fitting 7.2.1 Unsatisfactory Fit 7.2.2 Spline Functions 7.2.3 Smoothing Splines 7.3 Polynomial Regression in Several Variables 7.3.1 Response Surfaces 140 141 145 145 148 149 150 150 151 153 154 154 156 157 159 162 163 165 165 165 166 172 172 172 173 176 180 180
- 15. 8 9 CONTENTS IX 7.3.2 Multidimensional Smoothing Miscellaneous Exercises 7 Analysis of Variance 8.1 Introduction 8.2 One-Way Classification 8.2.1 General Theory 8.2.2 Confidence Intervals 8.2.3 Underlying Assumptions Exercises 8a 8.3 Two-Way Classification (Unbalanced) 8.3.1 Representation as a Regression Model 8.3.2 Hypothesis Testing 8.3.3 Procedures for Testing the Hypotheses 8.3.4 Confidence Intervals Exercises 8b 8.4 Two-Way Classification (Balanced) Exercises 8c 8.5 Two-Way Classification (One Observation per Mean) 8.5.1 Underlying Assumptions 8.6 Higher-Way Classifications with Equal Numbers per Mean 8.6.1 Definition of Interactions 8.6.2 Hypothesis Testing 8.6.3 Missing Observations Exercises 8d 8.7 Designs with Simple Block Structure 8.8 Analysis of Covariance Exercises 8e Miscellaneous Exercises 8 Departures from Underlying Assumptions 9.1 Introduction 9.2 Bias 9.2.1 Bias Due to Underfitting 9.2.2 Bias Due to Overfitting Exercises 9a 9.3 Incorrect Variance Matrix Exercises 9b 184 185 187 187 188 188 192 195 196 197 197 197 201 204 205 206 209 211 212 216 216 217 220 221 221 222 224 225 227 227 228 228 230 231 231 232
- 16. x CONTENTS 9.4 Effect of Outliers 233 9.5 Robustness of the F-Test to Nonnormality 235 9.5.1 Effect of the Regressor Variables 235 9.5.2 Quadratically Balanced F-Tests 236 Exercises 9c 239 9.6 Effect of Random Explanatory Variables 240 9.6.1 Random Explanatory Variables Measured without Error 240 9.6.2 Fixed Explanatory Variables Measured with Error 241 9.6.3 Round-off Errors 245 9.6.4 Some Working Rules 245 9.6.5 Random Explanatory Variables Measured with Error 246 9.6.6 Controlled Variables Model 248 9.7 Collinearity 249 9.7.1 Effect on the Variances of the Estimated Coefficients 249 9.7.2 Variance Inflation Factors 254 9.7.3 Variances and Eigenvalues 255 9.7.4 Perturbation Theory 255 9.7.5 Collinearity and Prediction 261 Exercises 9d 261 Miscellaneous Exercises 9 262 10 Departures from Assumptions: Diagnosis and Remedies 265 10.1 Introduction 265 10.2 Residuals and Hat Matrix Diagonals 266 Exercises lOa 270 10.3 Dealing with Curvature 271 10.3.1 Visualizing Regression Surfaces 271 10.3.2 Transforming to Remove Curvature 275 10.3.3 Adding and Deleting Variables 277 Exercises lOb 279 10.4 Nonconstant Variance and Serial Correlation 281 10.4.1 Detecting Nonconstant Variance 281 10.4.2 Estimating Variance Functions 288 10.4.3 Transforming to Equalize Variances 291 10.4.4 Serial Correlation and the Durbin-Watson Test 292 Exercises 10c 294 10.5 Departures from Normality 295 10.5.1 Normal Plotting 295
- 17. 11 10.5.2 Transforming the Response 10.5.3 Transforming Both Sides Exercises 10d 10.6 Detecting and Dealing with Outliers 10.6.1 Types of Outliers 10.6.2 Identifying High-Leverage Points 10.6.3 Leave-One-Out Case Diagnostics 10.6.4 Test for Outliers 10.6.5 Other Methods Exercises lOe 10.7 Diagnosing Collinearity 10.7.1 Drawbacks of Centering 10.7.2 Detection of Points Influencing Collinearity 10.7.3 Remedies for Collinearity Exercises 10f Miscellaneous Exercises 10 Computational Algorithms for Fitting a Regression 11.1 Introduction 11.1.1 Basic Methods 11.2 Direct Solution of the Normal Equations 11.2.1 Calculation of the Matrix XIX 11.2.2 Solving the Normal Equations Exercises 11a 11.3 QR Decomposition 11.3.1 Calculation of Regression Quantities CONTENTS 11.3.2 Algorithms for the QR and WU Decompositions Exercises 11b 11.4 Singular Value Decomposition !l.4.1 Regression Calculations Using the SVD 11.4.2 Computing the SVD 11.5 Weighted Least Squares 11.6 Adding and Deleting Cases and Variables 11.6.1 Updating Formulas 11.6.2 Connection with the Sweep Operator 11.6.3 Adding and Deleting Cases and Variables Using QR 11.7 Centering the Data 11.8 Comparing Methods XI 297 299 300 301 301 304 306 310 311 314 315 316 319 320 326 327 329 329 329 330 330 331 337 338 340 341 352 353 353 354 355 356 356 357 360 363 365
- 18. xii CONTENTS 11.8.1 Resources 11.8.2 Efficiency 11.8.3 Accuracy 11.8.4 Two Examples 11.8.5 Summary Exercises 11c 11.9 Rank-Deficient Case 11.9.1 Modifying the QR Decomposition 365 366 369 372 373 374 376 376 11.9.2 Solving the Least Squares Problem 378 11.9.3 Calculating Rank in the Presence of Round-off Error 378 11.9.4 Using the Singular Value Decomposition 379 11.10 Computing the Hat Matrix Diagonals 379 11.10.1 Using the Cholesky Factorization 380 11.10.2Using the Thin QR Decomposition 380 11.11 Calculating Test Statistics 380 11.12 Robust Regression Calculations 382 11.12.1 Algorithms for L1 Regression 382 11.12.2Algorithms for M- and GM-Estimation 384 11.12.3 Elemental Regressions 385 11.12.4Algorithms for High-Breakdown Methods 385 Exercises 11d 388 Miscellaneous Exercises 11 389 12 Prediction and Model Selection 391 12.1 Introduction 391 12.2 Why Select? 393 Exercises 12a 399 12.3 Choosing the Best Subset 399 12.3.1 Goodness-of-Fit Criteria 400 12.3.2 Criteria Based on Prediction Error 401 12.3.3 Estimating Distributional Discrepancies 407 12.3.4 Approximating Posterior Probabilities 410 Exercises 12b 413 12.4 Stepwise Methods 413 12.4.1 Forward Selection 414 12.4.2 Backward Elimination 416 12.4.3 Stepwise Regression 418 Exercises 12c 420
- 19. CONTENTS xiii 12.5 Shrinkage Methods 420 12.5.1 Stein Shrinkage 420 12.5.2 Ridge Regression 423 12.5.3 Garrote and Lasso Estimates 425 Exercises 12d 12.6 Bayesian Methods 12.6.1 Predictive Densities 12.6.2 Bayesian Prediction 427 428 428 431 12.6.3 Bayesian Model Averaging 433 Exercises 12e 433 12.7 Effect of Model Selection on Inference 434 12.7.1 Conditional and Unconditional Distributions 434 12.7.2 Bias 436 12.7.3 Conditional Means and Variances 437 12.7.4 Estimating Coefficients Using Conditional Likelihood 437 12.7.5 Other Effects of Model Selection 438 Exercises 12f 438 12.8 Computational Considerations 439 12.8.1 Methods for All Possible Subsets 439 12.8.2 Generating the Best Regressions 442 12.8.3 All Possible Regressions Using QR Decompositions 446 Exercises 12g 12.9 Comparison of Methods 12.9.1 Identifying the Correct Subset 12.9.2 Using Prediction Error as a Criterion Exercises 12h Miscellaneous Exercises 12 Appendix A Some Matrix Algebra A.l Trace and Eigenvalues A.2 Rank A.3 Positive-Semidefinite Matrices A.4 Positive-Definite Matrices A.5 Permutation Matrices A.6 Idempotent Matrices A.7 Eigenvalue Applications A.8 Vector Differentiation A.9 Patterned Matrices 447 447 447 448 456 456 457 457 458 460 461 464 464 465 466 466
- 20. xiv COIIJTENTS A.lO Generalized bversc 469 A.l1 Some Useful Results 471 A.12 Singular Value Decomposition 471 A.13 Some Miscellaneous Statistical Results 472 A.14 Fisher Scoring 473 Appendix B Orthogonal Projections 475 B.1 Orthogonal Decomposition of Vectors 475 B.2 Orthogonal Complements 477 B.3 Projections on Subspaces 477 Appendix C Tables 479 C.1 Percentage Points of the Bonferroni t-Statistic 480 C.2 Distribution of the Largest Absolute Value of k Student t Variables 482 C.3 Working-Hotelling Confidence Bands for Finite Intervals 489 Outline Solutions to Selected Exercises 491 References 531 Index 549
- 21. Preface Since pUblication of the first edition in 1977, there has been a steady flow of books on regression ranging over the pure-applied spectrum. Given the success of the first edition in both English and other languages (Russian and Chinese), we have therefore decided to maintain the same theoretical approach in this edition, so we make no apologies for a lack of data! However, since 1977 there have major advances in computing, especially in the use of powerful sta- tistical packages, so our emphasis has changed. Although we cover much the same topics, the book has been largely rewritten to reflect current thinking. Of course, some theoretical aspects of regression, such as least squares and maximum likelihood are almost set in stone. However, topics such as analysis of covariance which, in the past, required various algebraic techniques can now be treated as a special case of multiple linear regression using an appropriate package. We now list some of the major changes. Chapter 1 has been reorganized with more emphasis on moment generating functions. In Chapter 2 we have changed our approach to the multivariate normal distribution and the ensuing theorems about quadratics. Chapter 3 has less focus on the dichotomy of full-rank and less-than-full-rank models. Fitting models using Bayesian and robust methods are also included. Hypothesis testing again forms the focus of Chapter 4. The methods of constructing simultaneous confidence intervals have been updated in Chapter 5. In Chapter 6, on the straight line, there is more emphasis on modeling and piecewise fitting and less on algebra. New techniques of smoothing, such as splines and loess, are now considered in Chapters 6 and 7. Chapter 8, on analysis of variance and covariance, has xv
- 22. xvi Preface been updated, and the thorny problem of the two-way unbalanced model is addressed in detail. Departures from the underlying assumptions as well as the problem of collinearity are addressed in Chapter 9, and in Chapter 10 we discuss diagnostics and strategies for detecting and coping with such departures. Chapter 11 is a major update on the computational aspects, and Chapter 12 presents a comprehensive approach to the problem of model selection. There are some additions to the appendices and more exercises have been added. One of the authors (GAFS) has been very encouraged by positive comments from many people, and he would like to thank those who have passed on errors found in the first edition. We also express our thanks to those reviewers of our proposed table of contents for their useful comments and suggestions. Auckland, New Zealand November 2002 GEORGE A. F. SEBER ALAN J. LEE
- 23. 1 Vectors of Random Variables 1.1 NOTATION Matrices and vectors are denoted by boldface letters A and a, respectively, and scalars by italics. Random variables are represented by capital letters and their values by lowercase letters (e.g., Y and y, respectively). This use of capitals for random variables, which seems to be widely accepted, is par- ticularly useful in regression when distinguishing between fixed and random regressor (independent) variables. However, it does cause problems because a vector of random variables, Y" say, then looks like a matrix. Occasionally, because of a shortage of letters, aboldface lowercase letter represents a vector of random variables. IfX and Yare randomvariables, then the symbols E[Y), var[Y], cov[X, Y), and E[XIY = y) (or, more briefly, E[XIY)) represent expectation, variance, covariance, and conditional expectation, respectively. The n x n matrix with diagonal elements d1 , d2 , •.• ,dn and zeros elsewhere is denoted by diag(d1 , d2 , •.. , dn ), and when all the di's are unity we have the identity Il].atrix In. Ifa is an n x 1 column vector with elements al, a2, . .. , an, we write a = (ai), and the length or norm of ais denoted by Iiali. Thus lIall = Va'a = (a~ + a~ + ... + a~y/2. The vector with elements all equal to unity is represented by In, and the set of all vectors having n elements is denoted by lRn . If the m x n matrix A has elements aij, we write A = (aij), and the sum of the diagonal elements, called the trace of A, is denoted by tr(A) (= a11 + a22 + ... + akk, where k is the smaller of m and n). The transpose 1
- 24. 2 VECTORS OF RANDOM VARIABLES of A is represented by A' = (a~j)' where a~j = aji. If A is square, its determinant is written det(A), and if A is nonsingular its inverse is denoted by A -1. The space spanned by the columns of A, called the column space of A, is denoted by C(A). The null space or kernel of A (= {x: Ax = O}) is denoted by N(A). We say that Y '" N(B, (7"2) if Y is normally distributed with mean B and variance (7"2: Y has a standard normal distribution if B = 0 and (7"2 = 1. The t- and chi-square distributions with k degrees of freedom are denoted by tk and X~, respectively, and the F-distribution with m and n degrees offreedom is denoted by Fm,n' Finally we mention the dot and bar notation, representing sum and average, respectively; for example, J ai· = Laij j=l and In the case of a single subscript, we omit the dot. Some knowledge of linear' algebra by the reader is assumed, and for a short review course several books are available (see, e.g., Harville [1997)). However, a number of matrix results are included in Appendices A and B at the end of this book, and references to these appendices are denoted by, e.g., A.2.3. 1.2 STATISTICAL MODELS A major activity in statistics is the building of statistical models that hope- fully reflect the important aspects of the object of study with some degree of realism. In particular, the aim of regression analysis is to construct math- ematical models which describe or explain relationships that may exist be- tween variables. The simplest case is when there are just two variables, such as height and weight, income and intelligence quotient (IQ), ages of husband and wife at marriage, population size and time, length and breadth of leaves, temperature and pressure of a certain volume of gas, and so on. If we have n pairs of observations (Xi, Yi) (i = 1,2, . .. ,n), we can plot these points, giving a scatter diagram, and endeavor to fit a smooth curve through the points in such a way that the points are as close to the curve as possible. Clearly, we would not expect an exact fit, as at least one of the variables is subject to chance fluctuations due to factors outside our control. Even if there is an "exact" relationship between such variables as temperature and pressure, fluctuations would still show up in the scatter diagram because of errors of measurement. The simplest two-variable regression model is the straight line, and it is assumed that the reader has already come across the fitting of such a model. Statistical models are fitted for a variety of reasons. One important reason is that of trying to uncover causes by studying relationships between vari-
- 25. STATISTICAL MODELS 3 abIes. Usually, we are interested in just one variable, called the response (or predicted or dependent) variable, and we want to study how it depends on a set of variables called the explanatory variables (or regressors or indepen- dent variables). For example, our response variable might be the risk of heart attack, and the explanatory variables could include blood pressure, age, gen- der, cholesterol level, and so on. We know that statistical relationships do not necessarily imply causal relationships, but the presence of any statistical relationship does give us a starting point for further research. Once we are confident that a statistical relationship exists, we can then try to model this relationship mathematically and then use the model for prediction. For a given person, we can use their values of the explanatory variables to predict their risk of a heart attack. We need, however, to be careful when making predictions outside the usual ranges of the explanatory variables, as the model ~ay not be valid there. A second reason for fitting models, over and above prediction and expla- nation, is to examine and test scientific hypotheses, as in the following simple examples. EXAMPLE 1.1 Ohm's law states that Y = rX, where X amperes is the current through a resistor of r ohms and Y volts is the voltage across the resistor. This give us a straight line through the origin so that a linear scatter diagram will lend support to the law. 0 EXAMPLE 1.2 The theory of gravitation states that the force of gravity F between two objects is given by F = a/df3. Here d is the distance between the objects and a is a constant related to the masses of the two objects. The famous inverse square law states that (3 = 2. We might want to test whether this is consistent with experimental measurements. 0 EXAMPLE 1.3 Economic theory uses a production function, Q = aLf3K"I , to relate Q (production) to L (the quantity of labor) and K (the quantity of capital). Here a, (3, and 'Y are constants that depend on the type of goods and the market involved. We might want to estimate these parameters for a particular, market and use the relationship to predict the effects of infusions of capital on the behavior of that market. 0 From these examples we see that we might use models developed from the- oretical considerations to (a) check up on the validity of the theory (as in the Ohm's law example), (b) test whether a parameter has the value predicted from the theory, under the assumption that the model is true (as in the grav- itational example and the inverse square law), and (c) estimate the unknown constants, under the assumption of a valid model, and then use the model for prediction purposes (as in the economic example).
- 26. 4 VECTORS OF RANDOM VARIABLES 1.3 LINEAR REGRESSION MODELS If we denote the response variable by Y and the explanatory variables by Xl, X 2 , ... , X K , then a general model relating these variables is although, for brevity, we will usually drop the conditioning part and write E[Y]. In this book we direct our attention to the important class of linear models, that is, which is linear in the parameters {3j. This restriction to linearity is not as re- strictive as one might think. For example, many functions of several variables are approximately linear over sufficiently small regions, or they may be made linear by a suitable transformation. Using logarithms for the gravitational model, we get the straight line logF == loga - (3 log d. (1.1) For the linear model, the Xi could be functions of other variables z, w, etc.; for example, Xl == sin z, X2 == logw, and X3 == zw. We can also have Xi == Xi, which leads to a polynomial model; the linearity refers to the parameters, not the variables. Note that "categorical" models can be included under our umbrella by using dummy (indicator) x-variables. For example, suppose that we wish to compare the means of two populations, say, JLi = E[Ui] (i = 1,2). Then we can combine the data into the single model E[Y] - JLl + (JL2 - JLl)X - {30 + {3lX, where X = awhen Y is a Ul observation and X = 1 when Y is a U2 observation. Here JLl = {30 and JL2 == {30 +{3l, the difference being {3l' We can extend this idea to the case of comparing m means using m - 1 dummy variables. In a similar fashion we can combine two straight lines, (j = 1,2), using a dummy X2 variable which takes the value 0 if the observation is from the first line, and 1 otherwise. The combined model is E[Y] al + I'lXl + (a2 - al)x2 + (')'2 - I'I)XlX2 {30 +{3lXl + {32 x 2 + {33 x 3, (1.2) say, where X3 == Xl X2. Here al == {30, a2 = {30 +{32, 1'1 == {3l, and 1'2 == {3l +{33'
- 27. EXPECTATION AND COVARIANCE OPERATORS 5 In the various models considered above, the explanatory variables mayor may not be random. For example, dummy variables are nonrandom. With random X-variables, we carry out the regression conditionally on their ob- served values, provided that they are measured exactly (or at least with suf- ficient accuracy). We effectively proceed as though the X-variables were not random at all. When measurement errors cannot be ignored, the theory has to be modified, as we shall see in Chapter 9. 1.4 EXPECTATION AND COVARIANCE OPERATORS In this book we focus on vectors and matrices, so we first need to generalize the ideas of expectation, covariance, and variance, which we do in this section. Let Zij (i = 1,2, ... ,mj j = 1,2, ... ,n) be a set of random variables with expected values E[Zij]. Expressing both the random variables and their expectations in matrix form, we can define the general expectation operator of the matrix Z = (Zij) as follows: Definition 1.1 E[Z] = (E[Zij]). THEOREM 1.1 If A = (aij), B = (bij ), and C = (Cij) are l x m, n x p, and l x p matrices, respectively, of constants, then E[AZB + C] = AE[Z]B + C. Proof· Let W = AZB + Cj then Wij = 2::."=1 2:;=1 airZrsbsj +Cij and E [AZB + C] = (E[Wij]) = (~~airE[Zrs]bsj + Cij ) = ((AE[Z]B)ij) + (Cij) = AE[Z]B + C. 0 In this proof we note that l, m, n, and p are any positive integers, and the matrices of constants can take any values. For example, if X is an m x 1 vector, tlien E[AX] = AE[X]. Using similar algebra, we can prove that if A and B are m x n matrices of constants, and X and Yare n x 1 vectors of random variables, then E[AX + BY] = AE[X] + BE[Y]. In a similar manner we can generalize the notions of covariance and variance for vectors. IT X and Yare m x 1 and n x 1 vectors of random variables, then we define the generalized covariance operator Cov as follows:
- 28. 6 VECTORS OF RANDOM VARIABLES Definition 1.2 Cov[X, Y] = (COV[Xi , ¥j]). THEOREM 1.2 If E[X) = a and E[Y) =(3, then Cov[X, Y] = E [(X - a)(Y - (3)']. Proof· Cov[X, Y) = (COV[Xi, Yj]) = {E[(Xi - ai)(Yj - .aj)]} =E {[(Xi - ai)(Yj - .aj)]} =E [(X - a)(Y - (3)']. o Definition 1.3 When Y = X, Cov[X, X], written as Var[X], is called the variance (variance-covariance 01' dispersion) matrix of X. Thus Var[X] - (cov[Xi , Xj]) var[X1] cov[XI , X 2 ] cov[XI,Xn] COV[X2,XI) var[X2) cov[X2 ,Xn ] (1.3) cov[Xn,Xd cov[Xn,X2] var[Xn] Since cov[Xi, X j ] = cov[Xj , Xi], the matrix above is symmetric. We note that when X = Xl we write Var[X] = var[Xd. From Theorem 1.2 with Y = X we have Var[X] = E [(X - a)(X - a)'] , (1.4) which, on expanding, leads to Var[X] = E[XX') - aa'. (1.5) These last two equations are natural generalizations of univariate results. EXAMPLE 1.4 If a is any n x 1 vector of constants, then Var[X - a) = Var[X]. This follows from the fact that Xi - ai - E[Xi - ail = Xi - E[Xi ], so that o
- 29. EXPECTATION AND COVARIANCE OPERATORS 7 THEOREM 1.3 If X and Yare mx1 and n x1 vectors of random variables, and A and B are l x m and p x n matrices of constants, respectively, then Cov[AX, BY] = A Cov[X, Y]B'. Proof. Let U = AX and V = BY. Then, by Theorems 1.2 and 1.1, Cov[AX, BY] = Cov[U, V] =E [(U - E[U]) (V - E[V])'] =E [(AX - Aa)(BY - B,8)'] =E [A(X - a)(Y - ,8)'B'] =AE [(X - a)(Y - ,8)'] B' =A Cov[X, Y]B'. From the theorem above we have the special cases Cov[AX, Y] = A Cov[X, Y] and Cov[X, BY] = Cov[X, Y]B'. (1.6) o Of particular importance is the following result, obtained by setting B = A and Y = X: Var[AX] = Cov[AX, AX] = ACov[X,X]A' = A Var[X]A'. (1.7) EXAMPLE 1.5 If X, Y, U, and V are any (not necessarily distinct) n xl vectors of random variables, then for all real numbers a, b, c, and d (including zero), Cov[aX + bY,eU + dV] - ac Cov[X, U] +ad Cov[X, V] + be Cov[Y, U] + bd CoYlY,V]. (1.8) To prove this result, we simply multiply out E [(aX + bY - aE[X]- bE[Y])(cU +dV - cE[U] - dE[V])'] = E [(a(X - E[X]) + b(Y - E[Y])) (c(U - E[U]) + d(V - E[V]))']. If we set U = X and V = Y, c = a and d = b, we get Var[aX + bY] Cov[aX + bY,aX + bY] - a2 Var[X] + ab(Cov[X, Y] + CoylY, X]) +b2 Var[Y]. (1.9) o
- 30. 8 VECTORS OF RANDOM VARIABLES In Chapter 2 we make frequent use of the following theorem. THEOREM 1.4 If X is a vector of random variables such that no element of X is a linear combination of the remaining elements ri. e., there do not exist a (=1= 0) and b such that a'X = b for all values of X = xj, then Var[X) is a positive-definite matrix (see A.4). Proof. For any vector e, we have o < var[e'X) e'Var[X)e [by equation (1.7»). Now equality holds if and only if e'X is a constant, that is, if and only if e'X = d (e =1= 0) or e = O. Because the former possibility is ruled out, e = 0 and Var[X) is positive-definite. 0 EXAMPLE 1.6 If X and Y are m x 1 and n x 1 vectors of random variables such that no element of X is a linear combination of the remaining elements, then there exists an n x m. matrix M such that Cov[X, Y - MX) == O. To find M, we use the previous results to get Cov[X, Y - MX) Cov[X, Y) - Cov[X, MX) Cov[X, Y) - Cov[X, X]M' Cov[X, Y) - Var[X)M'. (1.10) By Theorem lA, Var[X] is positive-definite and therefore nonsingular (AA.1). Hence (1.10) is zero for M' = (Vai[X])-1 Cov[X, Y). o EXAMPLE 1.7 We now give an example of a singular variance matrix by using the two-cell multinomial distribution to represent a binomial distribu- tion as follows: (X X ) n! "'1 "'2 1 pr I = Xl, 2 = X2 = , ,PI P2 , PI +P2 == ,Xl + X2 = n. Xl· X2' IT X = (XI ,X2)', then Var[X) = ( npl(l- PI) -npIP2 which has rank 1 as P2 = 1 - Pl' EXERCISES 1a o 1. Prove that if a is a vector of constants with the same dimension as the random vector X, then E[(X - a)(X - a)') = Var[X] + (E[X] - a)(E[X] - a)'.
- 31. MEAN AND VARIANCE OF QUADRATIC FORMS 9 If Var[X] = E = ((J'ij), deduce that E[IIX - aWl = L (J'ii + IIE[X] - aW· i 2. If X and Y are m x 1 and n x 1 vectors of random variables, and a and bare m x 1 and n x 1 vectors of constants, prove that Cov[X - a, Y - b] = Cov[X, Y]. 3. Let X = (XI ,X2 , ••• ,Xn)' be a vector of random variables, and let YI = Xl, Yi = Xi - Xi - l (i = 2,3, ... , n). If the Yi are mutually independent random variables, each with unit variance, find Var[X]. 4. If Xl, X 2 , ••• , Xn are random variables satisfying Xi+l = pXi (i = 1,2, ... , n - 1), where p is a constant, and var[Xd = (J'2, find Var[X]. 1.5 MEAN AND VARIANCE OF QUADRATIC FORMS Quadratic forms play a major role in this book. In particular, we will fre- quently need to find the expected value of a quadratic form using the following theorem. THEOREM 1.5 Let X = (Xi) be an n x 1 vector of random variables, and let A be an n x n symmetric matrix. If E[X) = J1, and Var[X) = E = ((J'ij) , then Proof· E[X'AX) = tr(AE) + J1,'AJ1,. E[X'AX) = tr(E[X'AX)) =E[tr(X'AX)) =E[tr(AXX')) [by A.1.2) = tr(E[AXX')) = tr(AE[XX'J) = tr [A( Var[X) + J1,J1,')) [by (1.5)) = tr(AE) + tr(AJ.LJ1,') =tr(AE) + J1,'AJ1, [by A.1.2). o We can deduce two special cases. First, by setting Y = X - b and noting that Var[Y) = Var[X) (by Example 1.4), we have E[(X - b)'A(X - b)) = tr(AE) + (J1, - b)'A(J1, - b). (1.11)
- 32. 10 VECTORS OF RANDOM VARIABLES Second, if ~ = 0-2In (a common situation in this book), then tr(A~) = 0-2 tr(A). Thus in this case we have the simple rule E[X'AX] = 0-2(sum of coefficients of Xl) + (X'AX)x=l'. (1.12) EXAMPLE 1.8 If Xl, X 2 , • •• ,Xn are independently and identically dis- tributed with mean J.t and variance 0-2, then we can use equation (1.12) to find the expected value of Q = (Xl - X 2)2 + (X2 - X3)2 + ... + (Xn-l - Xn)2. To do so, we first write n n-l Q = X'AX = 2 LX; - xl - X~ - 2 L XiXi+l. i=l i=l Then, since COV[Xi' Xj] = 0 (i f= j), ~ = 0-2In and from the squared terms, tr(A) = 2n - 2. Replacing each Xi by J.t in the original expression for Q, we see that the second term of. E[X'AX] is zero, so that E[Q] = 0-2(2n - 2). 0 EXAMPLE 1.9 Suppose that the elements of X = (Xl ,X2, ... ,Xn)' have a common mean J.t and X has variance matrix ~ with o-ii = 0-2 and o-ij = p0-2 (i f= j). Then, when p = 0, we know that Q = Ei(Xi - X)2 has expected value 0-2 (n - 1). To find its expected value when p f= 0, we express Q in the form X'AX, where A = [(Oij - n-l )] and 1 -1 -n -n-1 -n-1 1 p p -1 1 -1 -1 P 1 p A~ 0-2 -n -n -n -n-1 -n-1 1 -1 -n P P 1 0-2(1 - p)A. Once again the second term in E[Q] is zero, so that E[Q] = tr(A~) = 0-2(1- p) tr(A) =0-2(1- p)(n - 1). 0 THEOREM 1.6 Let Xl, X 2, ... ,Xn be independent random variables with means (h, B2, ... ,Bn, common variance J.t2, and common third and fourth mo- ments about their means, J.t3 and J.t4, respectively (i.e., J.tr = E[(Xi - Bit]). If A is any n x n symmetric matrix and a is a column vector of the diagonal elements of A, then var[X'AX] = (J.t4 - 3J.t~)a' a + 2J.t~ tr(A2) + 4J.t2(J'A 2(J + 4J.t3(J'Aa. (This result is stated without proof in Atiqullah {1962}.} Proof. We note that E[X] = (J, Var[X] = J.t2In, and Var[X'AX] = E[(X'AX)2]- (E[X'AX])2. (1.13)
- 33. MEAN AND VARIANCE OF QUADRATIC FORMS 11 Now X'AX = (X - O)'A(X - 0) + 20'A(X - fJ) + O'AfJ, so that squaring gives (X'AX)2 = [(X - 0)'A(X - 0)]2 +4[0'A(X - 0)]2 + (0' AfJ)2 + 20'AO[(X -0)'A(X - 0) +40'AOO'A(X - 0)] +40'A(X - O)(X - O)'A(X - 0). Setting Y =X - 0, we have E[Y] = 0 and, using Theorem 1.5, E[(X'AX)2] = E[(Y'Ay)2] +4E[(O'Ay)2] + (O'AO? + 20'AOJ.L2 tr(A) + 4E[O'AYY'AY]. As a first step in evaluating the expression above we note that (Y'Ay)2 = 2:2:2:2>ijaklYiYjYkll. i j k I Since the l'i are mutually independent with the same first four moments about the origin, we have Hence i = j = k = l, i =j, k =lj i = k, j =lj i =l,j = k, otherwise. E[(Y'Ay)2] - J.L4 L:>~i + J.L~ L (L aiiakk + 2:atj + L aijaji) i i k#-i #i #i - (J.L4 - 3J.L~)a'a + J.L~ [tr(A)2 + 2tr(A2)] , (1.14) since A is symmetric and Ei Ej a~j = tr(A2). Also, say, and so that and (O'Ay)2 = (b'y)2 = LLbibjYiYj, i j fJ'Ayy'AY = LLLbiajkYiYjYk, i j k E[(O'Ay)2] =J.L2 L b~ =J.L2b'b = J.L20'A20 i E[O'AYY'AY] = J.L3 L biaii = J.L3b'a = J.L30'Aa. i
- 34. 12 VECTORS OF RANDOM VARIABLES Finally, collecting all the terms and substituting into equation (1.13) leads to the desired result. 0 EXERCISES Ib 1. Suppose that Xl, X 2 , and X3 are random variables with common mean fl, and variance matrix Var[X] = u2 ( ~ 1 o 1 I '4 2. If Xl, X 2 , ••• , Xn are independent random variables with common mean fl, and variances u?, u~, ... , u;, prove that I:i(Xi - X)2 I[n(n -1)] is an unbiased estimate of var[X]. 3. Suppose that in Exercise 2 the variances are known. Let X w = I:iWiXi be an unbiased estimate of fl, (Le., I:iWi = 1). (a) Prove that var[Xw] is minimized when Wi <X l/ur Find this min- imum variance Vrnin. (b) Let S! = L:iWi (Xi - Xw)2/(n - 1). If WW; = a (i = 1,2, ... , n), prove that E[S!] is an unbiased estimate of Vrnin. 4. The random variables Xl, X 2 , ••• , Xn have a common nonzero mean fl" a common variance u 2 , and the correlation between any pair of random variables is p. (a) Find var[X] and hence prove that -1I(n - 1) < P < 1. (b) If Q = a ~X; +b (~Xi)2 is an unbiased estimate of u 2 , find a and b. Hence show that, in this case, Q _ n (Xi _ X)2 -~(I-p)(n-1)" 5. Let Xl, X 2 , • •• , Xn be independently distributed as N(fl" ( 2 ). Define
- 35. MOMENT GENERATING FUNCTIONS AND INDEPENDENCE 13 and n-1 1 "" 2 Q = 2(n _ 1) ~(Xi+1 - Xi) . •=1 (a) Prove that var(82 ) = 20- 4 j(n - 1). (b) Show that Q is an unbiased estimate of 0-2 • (c) Find the variance of Q and hence show that as n -+ 00, the effi- ciency of Q relative to 8 2 is ~. 1.6 MOMENT GENERATING FUNCTIONS AND INDEPENDENCE If X and tare n x 1 vectors of random variables and constants, respectively, then the moment generating function (m.g.f.) of X is defined to be Mx(t) = E[exp(t'X»). A key result about m.g.f.'s is that if Mx(t) exists for all Iltll < to (to> 0) (i.e., in an interval containing the origin), then it determines the distribution uniquely. Fortunately, most of the common distributions have m.g.f. 's, one important exception being the t-distribution (with some of its moments being infinite, including the Cauchy distribution with 1 degree offreedom). We give an example where this uniqueness is usefully exploited. It is assumed that the reader is familiar with the m.g.f. of X~: namely, (1- 2t)-r/2. EXAMPLE 1.10 Suppose that Qi '" X~i for i = 1,2, and Q = Q1 - Q2 is statistically independent of Q2. We now show that Q '" X~, where r = r1 -r2. Writing (1 - 2t)-rl/2 E[exp(tQ1») E[exp(tQ + tQ2») - E[exp(tQ»)E[exp(tQ2») E[exp(tQ»)(l - 2t)-1/2, we have E[exp(tQ») = (1 - 2t)-h-r2)/2, which is the m.g.f. of X~. o Moment generating functions also provide a convenient method for proving results about statistical independence. For example, if Mx(t) exists and Mx(t) = MX(t1, ... , tr, 0, ... , O)Mx(O, ... , 0, tr+1' ... ' tn),
- 36. 14 VECTORS OF RANDOM VARIABLES then Xl = (X1, ... ,Xr )' andX2 = (Xr +1,' .. 'Xn )' are statistically indepen- dent. An equivalent result is that Xl and X 2 are independent if and only if we have the factorization Mx(t) = a(tI, ... ,tr)b(tr+l, ... ,tn) for some functions a(·) and b(·). EXAMPLE 1.11 Suppose that the joint distribution of the vectors of ran- dom variables X and Y have a joint m.g.f. which exists in an interval contain- ing the origin. Then if X and Yare independent, so are any (measurable) functions of them. This follows from the fact that if c(·) and d(·) are suitable vector functions, E[exp{s'c(X) + s'd(Y)} = E[exp{s'c(X)}]E[exp{s'd(Y)}] = a(s)b(t), say. This result is, in fact, true for any X and Y, even if their m.g.f.'s do not exist, and can be proved using characteristic functions. 0 Another route .that we shall use for proving independence is via covariance. It is well known that cov[X, Y] = 0 does not in general imply that X and Y are independent. However, in one important special case, the bivariate normal distribution, X and Y are independent if and only if cov[X, Y] = O. A generalization of this result applied to the multivariate normal distribution is given in Chapter 2. For more than two variables we find that for multivariate normal distributions, the variables are mutually independent if and only if they are pairwise independent. Bowever, pairwise independence does not necessarily imply mutual independence, as we see in the following example. EXAMPLE 1.12 Suppose that Xl, X 2 , and X3 have joint density function (27r) -3/2 exp [- ~xt + x~ + xm x {I + XIX2X3 exp [-Hx~ + x~ + x~)]} -00 < Xi < 00 (i = 1,2,3). Then the second term in the braces above is an odd function of X3, so that its integral over -00 < X3 < 00 is zero. Hence (27r)-1 exp [-~(x~ + xm !I(Xd!z(X2), and Xl and X 2 are independent N(O,l) variables. Thus although Xl, X 2 , and X3 are pairwise independent, they are not mutually independent, as
- 37. MOMENT GENERATING FUNCTIONS AND INDEPENDENCE 15 EXERCISES Ie 1. If X and Y are random variables with the same variance, prove that cov[X +Y, X - Y] = O. Give a counterexample which shows that zero covariance does not necessarily imply independence. 2. Let X and Y be discrete random variables taking values 0 or 1 only, and let pr(X = i, Y = j) =Pij (i = 1, OJ j = 1,0). Prove that X and Y are independent if and only if cov[X, Y] = o. 3. If X is a random variable with a density function symmetric about zero and having zero mean, prove that cov[X, X2] = O. 4. If X, Y and Z have joint density function f(x,y,z) = i(1 + xyz) (-1 < x,y,z < 1), prove that they are pairwise independent but not mutually independent. MISCElLANEOUS EXERCISES I 1. If X and Y are random variables, prove that var[X) = Ey{ var[XJYJ} + vary{E[XJYJ}. Generalize this result to vectors X and Y of random variables. ( 523) Var[X) = 2 3 0 . 303 (a) Find the variance of Xl - 2X2 + X 3 • (b) Find the variance matrix of Y = (Yi, }2)', where Yl = Xl + X 2 and Y2 = Xl +X2 +X3 • 3. Let Xl, X2, . .. , Xn be random variables with a common mean f.L. Sup- pose that cov[Xi , Xj) = 0 for all i and j such that j > i + 1. If i=l and
- 38. 16 VECTORS OF RANDOM VARIABLES prove that E [3Ql - Q2] = var[X]. n(n - 3) 4. Given a random sample Xl ,X2,X3 from the distribution with density function f(x) = ~ find the variance of (Xl - X 2)2 + (X2 - X3)2 + (X3 - Xl)2. 5. If Xl, ... , Xn are independently and identically distributed as N(O, 0"2), and A and B are any n x n symmetric matrices, prove that Cov[X'AX, X'BX] = 20"4 tr(AB).
- 39. 2 Multivariate Normal Distribution 2.1 DENSITY FUNCTION Let E be a positive-definite n x n matrix and I-L an n-vector. Consider the (positive) function where k is a constant. Since E (and hence E-l by A.4.3) is positive-definite, the quadratic form (y -I-L),E-l(y - I-L) is nonnegative and the function f is bounded, taking its maximum value of k-1 at y = I-L. Because E is positive-definite, it has a symmetric positive-definite square root El/2, which satisfies (El/2)2 = E (by A.4.12). Let z = E-1/2(y - I-L), so that y = El/2z + I-L. The Jacobian of this transformation is J = det (8Yi ) = det(El/2) = [det(EW/2. 8zj Changing the variables in the integral, we get L:···L:exp[-~(y -1-L)'E-1(y -I-L)] dYl·· ·dYn L:...L:exp(_~z'El/2E-lEl/2z)IJI dZl ... dZn L:...L:exp(-~z'z)IJI dz1 ··• dZn 17
- 40. 18 MULTIVARIATE tVOF?MAL DISTRIBUTION ,(,,, ,"CO - PI [11 exp(-~zf) dZi i=:I. J -(X) n i=l (27r)n/2 det(:E)1/2. Since f > 0, it follows that if k = (27r)n/2 det(:E)1/2, then (2.1) represents a density function. Definition 2.1 The distribution corresponding to the density (2.1) is called the multivariate normal distribution. THEOREM 2.1 If a random vector Y has density (2.1), then E[Y] = I-L and Var[Y] = :E. Proof. Let Z = :E-1/2(y - I-L). Repeating the argument above, we see, using the change-of-variable formula, that Z has density f[y(z)lIJI (2.2) (2.3) The factorization of the joint density function in (2.2) implies that the Zi are mutually independent normal variables and Zi '" N(O, 1). Thus E[Z] = 0 and Var[Z] = In, so that E[Y] = E[:E1/2Z + I-L] = :E1 / 2E[Z] + f-L =I-L and Var[Y] = Var[:E1 / 2Z + I-L] = Var[:E1/2Z] = :El/2In:El/2 = :E. 0 We shall use the notation Y ,...., Nn(I-L,:E) to indicate that Y has the density (2.1). When n = 1 we drop the subscript. EXAMPLE 2.1 Let Zl, .. " Zn be independent N(O,l) random variables. The density of Z = (Zr, ... , Zn)' is the product of the univariate densities given by (2.2), so that by (2.3) the density of Z is of the form (2.1) with I-L = 0 and :E =In [Le., Z '" Nn(O, In)]. 0 We conclude that if Y '" Nn(I-L,:E) and Y = :E1 / 2 Z + f-L, then Z = :E-1 / 2(y - f-L) and Z '" Nn(O,In). The distribution of Z is the simplest and most fundamental example of the multivariate normal. Just as any univariate normal can be obtained by rescaling and translating a standard normal with
- 41. DENSITY FUNCTION 19 mean zero and variance 1, so can any multivariate normal be thought of as a rescaled and translated Nn(O, In). Multiplying by :E1 / 2 is just a type of rescaling of the the elements of Z, and adding J1, is just a translation by J1,. EXAMPLE 2.2 Consider the function 1 f(x,y) = 2 (1 2)1 7r - P 2 (1a;(1y X exp {_ 1 2 [(X - ~a;)2 _ 2p (X - f-La;)(Y - f-Ly) + (y - ~y)2]} 2(1- P ) (1a; (1a;(1y (1y where (1a; > 0, (1y > 0, and Ipi < 1. Then f is of the form (2.1) with The density f above is the density of the bivariate normal distribution. 0 EXERCISES 2a 1. Show that f(Yl,Y2) = k- 1 exp[-H2y~ +y~ + 2YIY2 - 22Yl - 14Y2 + 65)] is the density of a bivariate normal random vector Y = (Y1 , Y2)'. (a) Find k. (b) Find E[Y] and Var[Y]. 2. Let U have density 9 and let Y = A(U + c), where A is nonsingular. Show that the density f of Y satisfies f(y) = g(u)/I det(A)I, where y = A(u +c). 3. (a) Show that the 3 x 3 matrix E~O !n is positive-definite for p > - t. (b) Find :E1/ 2 when
- 42. 20 MULTIVARIATE NORMAL DISTRIBUTION 2.2 MOMENT GENERATING FUNCTIONS We can use the results of Section 2.1 to calculate the moment generating fUnction (m.gJ.) of the multivariate normal. First, if Z ,...., Nn(O, In), then, by the independence of the Zi'S, the m.gJ. of Z is E[exp(t'Z)] E[exp (ttiZi )] - E [fieXp(tiZi)] n - IIE [exp(tiZi)] i=l n - IIexp(~t;) i=l exp(~t't). (2.4) Now if Y '" Nn(l-t, E), we can write Y = E1/2Z + I-t, where Z '" Nn(O, In). Hence using (2.4) and putting s = E1/2t, we get E[exp(t'Y)] - E[exp{t'(E1/2Z + I-t)}] E[exp(s'Z)) exp(t'I-t) - exp( ~s's) exp(t'I-t) - exp( ~t'E1/2 E1/2t + t'I-t) - exp(t'I-t + ~t'Et). (2.5) Another well-known result for the univariate normal is that if Y '" N(p"a2 ), then aY + b is N(ap, + b, a2 ( 2 ) provided that a ::f. O. A similar result is true for the multivariate normal, as we see below. THEOREM 2.2 Let Y '" Nn(/L, E), C be an m x n matrix of rank m, and d be an m x 1 vector. Then CY + d '" Nm(CI-t + d, CEC'). Proof. The m.gJ. of CY + d is E{exp[t'(CY + d)]} E{exp[(C't)'Y + t'd]} exp[(C't)'/L + ~(C't)'EC't + t'd] - exp[t'(C/L + d) + ~t'CEC't). Since C:EC' is positive-definite, the equation above is the moment generating function of Nm(CI-t + d, CEC'). We stress that C must be of full rank to ensure that CEC' is positive-definite (by A.4.5), since we have only defined the multivariate normal for positive-definite variance matrices. 0
- 43. MOMENT GENERATING FUNCTIONS 21 COROLLARY If Y = AZ + 1-£, where A is an n x n nonsingular matrix, then Y "" Nn(l-£, AA'). Proof. We replace Y, 1-£, E and d by Z, 0, In and 1-£, respectively, in Theorem 2.2. 0 EXAMPLE 2.3 Suppose that Y "" Nn(O, In) and that T is an orthogonal matrix. Then, by Theorem 2.2, Z = T'Y is Nn(O, In), since T'T = In. 0 In subsequent chapters, we shall need to deal with random vectors of the form CY, where Y is multivariate normal but the matrix C is not of full rank. For example, the vectors of fitted values and residuals in a regression are of this form. In addition, the statement and proof of many theorems become much simpler if we admit the possibility of singular variance matrices. In particular we would like the Corollary above to hold in some sense when C does not have full row rank. Let Z "" Nm(O, 1m), and let A be an n x m matrix and 1-£ an n x 1 vector. By replacing El/2 by A in the derivation of (2.5), we see that the m.g.f. of Y =AZ + 1-£ is exp(t'1-£ + ~t'Et), with E =AA'. Since distributions having the same m.g.f. are identical, the distribution of Y depends on A only through AA'. We note that E[Y] = AE[Z] + 1-£ = 1-£ and Var[Y] = A Var[Z]A' = AA'. These results motivate us to introduce the following definition. Definition 2.2 A random n x 1 vector Y with mean 1-£ and variance matrix E has a multivariate normal distribution if it has the same distribution as AZ + 1-£, where A is any n x m matrix satisfying E = AA' and Z "" Nm(O, 1m). We write Y "" AZ +1-£ to indicate that Y and AZ +1-£ have the same distribution. We need to prove that whenE is positive-definite, the new definition is equivalent to the old. As noted above, the distribution is invariant to the choice of A, as long as E = AA'. If E is of full rank (or, equivalently, is positive-definite), then there exists a nonsingular A with E = AA', by AA.2. If Y is multivariate normal by Definition 2.1, then Theorem 2.2 shows that Z = A -1 (Y - 1-£) is Nn(O, In), so Y is multivariate normal in the sense of Definition 2.2. Conversely, if Y is multivariate normal by Definition 2.2, then its m.g.f. is given by (2.5). But this is also the m.g.f. of a random vector having dellsity (2.1), so by the uniqueness of the m.g.f.'s, Y must also have density (2.1). If E is of rank m < n, the probability distribution of Y cannot be expressed in terms of a density function. In both cases, irrespective of whether E is positive-definite or just positive-semidefinite, we saw above that the m.g.f. is exp (t'1-£ + ~t'Et) . (2.6) We write Y "" Nm(l-£, E) as before. When E has less than full rank, Y is sometimes said to have a singular distribution. From now on, no assumption that E is positive-definite will be made unless explicitly stated.
- 44. 22 MULTIVARIATE NORMAL DISTRIBUTiON EXAMPLE 2.4 Let Y '" N(/-t, (52) and put yl = (Y, -Y). The variance- covariance matrix of Y is Put Z = (Y - /-t)/(5. Then 1 -1 -1 ) 1 . Y = ( _~ ) Z + ( ~ ) = AZ + ~ and E = AA/. Thus Y has a multivariate normal distribution. o EXAMPLE 2.5 We can show that Theorem 2.2 remains true for random vectors having this extended definition of the multivariate normal without the restriction on the rank of A. If Y '" Nn(~' E), then Y '" AZ +~. Hence CY '" CAZ + C~ = HZ + b, say, and CY is multivariate normal with E[CY] = b = C~ and Var[CY] = BB' =CANC' = CEC/. 0 EXAMPLE 2.6 Under the extended definition, a constant vector has a multivariate normal distribution. (Take A to be a matrix of zeros.) In par- ticular, if A is a zero row vector, a scalar constant has a (univariate) normal distribution under this definition, so that we regard constants (with zero vari- ance) as being normally distributed. 0 EXAMPLE 2.7 (Marginal distributions) Suppose that Y '" Nn(~, E) and we partition Y, ~ and E conformably as Then Y 1 '" Np(~l' Ell). We see this by writing Y 1 = BY, where B = (Ip, 0). Then B~ = ~l and BEB' = Eu , so the result follows from Theorem 2.2. Clearly, Y 1 can be any subset of Y. In other words, the marginal distributions of the multivariate normal are multivariate normal. 0 Our final result in this section is a characterization of the multivariate normal. THEOREM 2.3 A random vector Y with variance-covariance matrix E and mean vector ~ has a Nn(~, E) distribution if and only if a/y has a univariate normal distribution for every vector a. Proof. First, assume that Y '" Nn(~, E). Then Y '" AZ + ~, so that a/y '" al AZ + al ~ = (A/a)'Z + a/~. This has a (univariate) normal distribution in the sense of Definition 2.2.
- 45. MOMENT GENERATING FUNCTIONS 23 Conversely, assume that t'Y is a univariate normal random variable for all t. Its mean is t'I-t and the variance is t'Et. Then using the formula for the m.g.f. of the univariate normal, we get E{exp[s(t'Y)]} = exp[s(t'I-t) + ~s2(t'Et)]. Putting s = 1 shows that the m.g.f. of Y is given by (2.6), and thus Y ,...., Nn(J-t,E). 0 We have seen in Example 2.7 that the multivariate normal has normal marginalsj and in particular the univariate marginals are normal. However, the converse is not true, as the following example shows. Consider the function which is nonnegative (since 1 + ye-y2 > 0) and integrates to 1 (since the integral r~: ye-y2 / 2 dy has value 0). Thus f is a joint density, but it is not bivariate normal. However, 1 1 1+00 I'<'L exp(- ~y~) x I'<'L exp(- ~y~) dY2 y 27r y 27r -00 1 1 1+00 + I'<'LYl exp(- ~Yn x I'<'L Y2 exp(- ~y~) dY2 y 27r y 27r -00 ~ exp(- ~y~), so that the marginals are N(O, 1). In terms of Theorem 2.3, to prove that Y is bivariate normal, we must show that a'Y is bivariate normal for all vectors a, not just for the vectors (1,0) and (0,1). Many other examples such as this are known; see, for example, Pierce and Dykstra [1969], Joshi [1970], and Kowalski [1970]. EXERCISES 2b 1. Find the moment generating function of the bivariate normal distribu- tion given in Example 2.2. 2. If Y,""" Nn(J-t, E), show that Yi '" N(P,i, au). 3. Suppose that Y '" N3 (1-t, E), where ~~(n andE~U~D Find the joint distribution of Zl =Y1 + Y2 + Y3 and Z2 =Yl - Y2. 4. Given Y '" Nn(I-t,In), find the joint density of a'Y and b'Y, where a'b =0, and hence show that a'Y and b'Y are independent.
- 46. 24 MULTIVARIATE NORMAL DISTRIBUTION 5. Let (Xi, Yi), i = 1,2, ... ,n, be a random sample from a bivariate normal distribution. Find the joint distribution of (X, Y). 6. If Yl and Y2 are random variables such that Yi + Y2 and Yl - Y2 are independent N(O, 1) random variables, show that Yl and Y2 have a bivariate normal distribution. Find the mean and variance matrix of Y = (Yl, Y2 )'. 7. Let Xl and X 2 have joint density Show that Xl and X 2 have N(O, 1) marginal distributions. (Joshi [1970]) 8. Suppose that Yl , Y2 , • •• , Yn are independently distributed as N(O,l). Calculate the m.g.f. of the random vector (Y, Yl - Y, Y2 - Y, ... ,Yn - Y) and hence show that Y is independent of 'Ei(Yi _ y)2. (Hogg and Craig [1970]) 9. Let Xl, X 2 , and X3 be LLd. N(O,l). Let Yl (Xl + X 2 + X 3 )/V3, - (Xl - X2 )/v'2, (Xl + X 2 - 2X3)/V6. Show that Yl , Y2 and Y3 are LLd. N(O,l). (The transformation above is a special case of the so-called Helmert transformation.) 2.3 STATISTICAL INDEPENDENCE For any pair of random variables, independence implies that the pair are uncorrelated. For the normal distribution the converse is also true, as we now show. THEOREM 2.4 Let Y '" Nn(J.L, I:.) and partition Y, f-t and I:. as in Example 2.7. Then Yland Y 2 are independent if and only if I:.12 = 0. Proof. The m.g.f. of Y is exp (t'f-t + ~t' I:.t). Partition t conformably with Y. Then the exponent in the m.g.f. above is t~f-tl + t~f-t2 + ~t~I:.lltl + ~t~I:.22t2 + t~I:.12t2. (2.7)
- 47. STATISTICAL INDEPENDENCE 25 If E12 = 0, the exponent can be written as a function of just tl plus a function of just t 2, so the m.g.f. factorizes into a term in tl alone times a term in t2 alone. This implies that Y I" and Y2 are independent. Conversely, if Y1 and Y2 are independent, then where M is the m.g.f. of Y. By (2.7) this implies that t~E12t2 = 0 for all tl and t 2, which in turn implies that E12 = O. [This follows by setting tl = (1,0, ... ,0)', etc.] 0 We use this theorem to prove our next result. THEOREM 2.5 Let Y ....., Nn(/-L, E) and define U = AY, V = BY. Then U and V are independent if and only if Cov[U, V] = AEB' = O. Proof. Consider Then, by Theorem 2.2, the random vector W is multivariate normal with variance-covariance matrix Var[W] = ( ~ ) Var[Y) (A',B') = ( ~~1: AEB' ) REB' . Thus, by Theorem 2.4, U and V are independent if and only if AEB' = O. 0 EXAMPLE 2.8 Let Y ....., N n (/-L,0'2In ) and let In be an n-vector of 1's. Then the sample mean Y = n-1 Zi Yi is independent of the sample variance 8 2 = (n - 1)-1 L:i(Yi - y)2. To see this, let I n = Inl~ be the n x n matrix of 1's. Then Y = n-ll~Y (= AY, say) and ~-Y Y2 -Y Yn-Y say. Now A~B' -II' 21 (I -IJ) 2 -11 2 -11 0 ~ = n nO' n n - n n = 0' n n - 0' n n = , so by Theorem 2.5, Y is independent of (~ - Y, ... ,Yn - Y), independent of 8 2. and hence o EXAMPLE 2.9 Suppose that Y ....., Nn(/-L, E) with E positive-definite, and Y is partitioned into two subvectors y' = (Yi, Y~), where Y1 has di~ension
- 48. 26 MU1.TlVARIATE nORMAL DISTRIBUTION T. Partition ~ and j;' similarly. Then the conditional distribution of Yl given Y 2 = Y2 is NAILl -:- :£12:E2"l(Y2 - J-L2),:Ell - :Elz:E2"l:E2d· To derive this, put U 1 Y1 - J-Ll - :E12:E2"21(Y2 - J-L2), U 2 Y 2 - J-L2· Then so that U is multivariate normal with mean 0 and variance matrix A:EA' given by Hence, U 1 and U 2 are independent, with joint density of the form g(Ul, U2) = gl(Ul)g2(U2). Now consider the conditional density function of Y 1 given Y 2: (2.8) and write Ul Yl - J-Ll - :E12:E2"21(Y2- J-L2), u2 - Y2 - J-L2· By Exercises 2a, No.2, h(Y2) = g2(U2) and f(Yl, Y2) = g1(U1)g2(U2), so that from (2.8), f1!2(YlIY2) = g1 (ud = gl (Yl - J-L1 - :E12:E2"l (Y2 - J-L2)). The result now follows from the fact that g1 is the density of the Nr(O,:Ell - :E12:E2"l:E21) distribution. 0 EXERCISES 2c 1. IfY1 , Y2 , ••• , Yn have a multivariate normal distribution and are pairwise independent, are they mutually independent? 2. Let Y '" Nn(p,ln, :E), where :E = (1- p)In + pJn and p> -l/(n - 1). When p = 0, Y and Li(Yi - y)2 are independent, by Example 2.8. Are they independent when p f= O?
- 49. DISTRIBUTION OF QUADRATIC FORMS 27 3. Given Y ,...., N3 (J-L, E), where E~U'U PO) 1 P , P 1 for what value(s) of P are Yi + Y2 + Y3 and Y1 - Y2 - Y3 statistically independent? 2.4 DISTRIBUTION OF QUADRATIC FORMS Quadratic forms in normal variables arise frequently in the theory of regression in connection with various tests of hypotheses. In this section we prove some simple results concerning the distribution of such quadratic forms. Let Y ,..., Nn(J-L, E), where E is positive-definite. We are interested in the distribution of random variables of the form yl AY = L:?=1 L:;=1 aij Y,;1j. Note that we can always assume that the matrix A is symmetric, since if not we can replace aij with ~(aij + aji) without changing the value of the quadratic form. Since A is symmetric, we can diagonalize it with an orthog- onal transformation; that is, there is an orthogonal matrix T and a diagonal matrix D with TIAT = D = diag(d1 , ... , dn ). (2.9) The diagonal elements di are the eigenvalues of A and can be any real num- bers. We begin by assuming that the random vector in the quadratic form has a Nn(O,In) distribution. The general case can be reduced to this through the usual transformations. By Example 2.3, if T is an orthogonal matrix and Y has an Nn(O, In) distribution, so does Z = T/y. Thus we can write n y/AY =y/TDT/y =Z/DZ =L:diZl, i=l (2.10) so the distribution of ylAY is a linear combination of independent X~ random variables. Given the values of di , it is possible to calculate the distribution, at least numerically. Farebrother [1990] describes algorithms for this. There is an important special case that allows us to derive the distribution of the quadratic form exactly, without recourse to numerical methods. If r of the eigenvalues di are 1 and the remaining n - r zero, then the distribution is the sum of r independent X~'s, which is X~. We can recognize when the eigenvalues are zero or 1 using the following theorem. THEOREM 2.6 Let A be a symmetric matrix. Then A has r eigenvalues equal to 1 and the rest zero if and only if A2 = A and rank A = r.
- 50. 28 MULTIVARIATE NORMAL DISTRIBUTION Proof. See A.6.1. o Matrices A satisfying A2 = A are called idempotent. Thus, if A is sym- metric, idempotent, and has rank r, we have shown that the distribution of ylAY must be X~. The converse is also true: If A is symmetric and ylAY is X~, then A must be idempotent and have rank r. To prove this by The- orem 2.6, all we need to show is that r of the eigenvalues of A are 1 and the rest are zero. By (2.10) and Exercises 2d, No.1, the m.g.f. of Y'AY is n~=1 (1 - 2di t)-1/2. But since Y'AY is X~, the m.g.f. must also equal (1 - 2t)-r/2. Thus n II(I- 2di t) = (1- 2W, i=1 so by the unique factorization of polynomials, r of the di are 1 and the rest are zero. We summarize these results by stating them as a theorem. THEOREM 2.7 Let Y '" Nn(O, In) and let A be a symmetric matrix. Then ylAY is X~ if and only irA is idempotent of rank r. EXAMPLE 2.10 Let Y '" NnUL, cr2In) and let 8 2 be the sample variance as defined in Example 2.8. Then (n - 1)82/ cr2 '" X~-1' To .see this, recall that (n - 1)82 /cr2 can be written as cr-2yl(In - n-1 J n )Y. Now define Z = cr-1 (y - {tIn), so that Z'" Nn(O,In). Then we have (n - 1)82 /cr2 = ZI(In - n-1 J n )Z, where the matrix In - n-1 J n is symmetric and idempotent, as can be veri- fied by direct multiplication. To calculate its rank, we use the fact that for symmetric idempotent matrices, the rank and trace are the same (A.6.2). We get so the result follows from Theorem 2.7. tr(In - n-1 Jn ) tr(In) - n-1 tr(Jn) n -1, o Our next two examples illustrate two very important additional properties of quadratic forms, which will be useful in Chapter 4. EXAMPLE 2.11 Suppose that A is symmetric and Y '" Nn(O, In). Then if Y'AY is X~, the quadratic form Y' (In - A)Y is X~-r' This follows because A must be idempotent, which implies that (In - A) is also idempotent. (Check by direct multiplication.) Furthermore, rank(In - A) = tr(In - A) = tr(In) - tr(A) = n - r,
- 51. DISTRIBUTION OF QUADRATIC FORMS 29 so that Y'(In - A)Y is X~-r' 0 EXAMPLE 2.12 Suppose that A and B are symmetric, Y '" Nn(O, In), and Y'AY and Y'BY are both chi-squared. Then Y'AY and Y'BY are independent if and only if AB = O. To prove this, suppose first that AB = O. Since A and B are idempo- tent, we can write the quadratic forms as Y'AY = YA'AY = IIAYl12 and Y'BY = IIBYII2. By Theorem 2.5, AY and BY are independent, which implies that the quadratic forms are independent. Conversely, suppose that the quadratic forms are independent. Then their sum is the sum of independent chi-squareds, which implies that Y'(A + B)Y is also chi-squared. Thus A + B must be idempotent and A + B = (A + B)2 = A2 + AB + BA + B2 = A + AB + BA + B, so that AB +BA = O. Multiplying on the left by A gives AB + ABA = 0, while multiplying on the right by A gives ABA + BA = OJ hence AB = BA = O. 0 EXAMPLE 2.13 (Hogg and Craig [1958, 1970]) Let Y '" Nn (8, 0"2In) and let Qi = (Y - 8)'Pi(Y - 8)/0"2 (i = 1,2). We will show that if Qi '" X~. and QI - Q2 > 0, then QI - Q2 and Q2 are independently distributed as X;1-r2 and X~2' respectively. We begin by noting that if Qi '" X~i' then P~ =Pi (Theorem 2.7). Also, QI - Q2 > 0 implies that PI - P 2 is positive-semidefinite and therefore idempotent (A.6.5). Hence, by Theorem 2.7, QI - Q2 '" X~, where r rank(Pl - P 2) tr(PI - P 2) - trPI - trP2 rankPI - rankP2 Also, by A.6.5, P IP 2 = P 2P I = P 2, and (PI - P 2 )P2 = O. Therefore, since Z = (Y - 8)/0"2", Nn(O,In), we have, by Example 2.12, that QI - Q2 [= Z'(PI - P 2)Z] is independent of Q2 (= Z'P2Z). 0 We can use these results to study the distribution of quadratic forms when the variance-covariance matrix :E is any positive-semidefinite matrix. Suppose that Y is now Nn(O, :E), where :E is of rank 8 (8 < n). Then, by Definition 2.2 (Section 2.2), Y has the same distribution as RZ, where :E = RR' and R is n x 8 of rank 8 (A.3.3). Thus the distribution of Y'AY is that of Z'R'ARZ,
- 52. 30 MULTlVARIATE NORMAL DISTRjBUTJON which, by T~'leOTem 2.7, will be X~ if and only if RIAR is idempotent of rank r. However, this is not a very useful condition. A better one is contained in our next theorem. THEOREM 2.8 Suppose that Y '" Nn(O, ~), and A is symmetric. Then yl AY is X~ if and only if r of the eigenvalues of A~ are 1 and the rest are zero. Proof. We assume that Y'AY = Z'R'ARZ is X~. Then R'AR is symmetric and idempotent with r unit eigenvalues and the rest zero (by A.6.1), and its rank equals its trace (A.6.2). Hence, by (A.1.2), r =rank(R'AR) =tr(R'AR) =tr(ARR') =tr(A~). Now, by (A.7.1), R'AR and ARR' = A~ have the same eigenvalues, with possibly different multiplicities. Hence the eigenvalues of A~ are 1 or zero. As the trace of any square matrix equals the sum of its eigenvalues (A.1.3), r of the eigenvalues of A~ must be 1 and the rest zero. The converse argument is just the reverse of the' one above. 0 For nonsymmetric matrices, idempotence implies that the eigenvalues are zero or 1, but the converse is not true. However, when ~ (and hence R) has full rank, the fact that R'AR is idempotent implies that A~ is idempotent. This is because the equation R'ARR'AR = R'AR can be premultiplied by (R')-land postmultiplied by R' to give A~A~=A~. Thus we have the following corollary to Theorem 2.8. COROLLARY Let Y '" Nn(O, ~), where :E is positive-definite, and sup- pose that A is symmetric. Then Y'AY is X~ if and only A:E is idempotent and has rank r. For other necessary and sufficient conditions, see Good [1969, 1970] and Khatri [1978]. Our final theorem concerns a very special quadratic form that arises fre- quently in statistics. THEOREM 2.9 Suppose that Y '" NnUL, :E), where :E is positive-definite. Then Q = (Y - 1-£)':E-1(y - 1-£) <'oJ X;. Proof. Making the transformation Y = :E1/2 Z +1-£ considered in Theorem 2.1, we get n Q = Z':EI/2:E-l:EI/2Z = Z'Z = L Z?' i=l
- 53. DISTRIBUTION OF QUADRATIC FORMS 31 Since the Zl's are independent x~ variables, Q '" X~. 0 EXERCISES 2d 1. Show that the m.g.f. for (2.10) is n~(1- 2tdi )-1/2. 2. Let Y '" Nn(O, In) and let A be symmetric. (a) Show that the m.g.f. of Y'AY is [det(ln - 2tA)]-1/2. (b) If A is idempotent ofrank r, show that the m.g.f. is (1 - 2t)-r/2. (c) Find the m.g.f. if Y '" Nn(O, ~). 3. If Y '" N 2 (0, 12 ), find values of a and b such that aCYl - y2)2 + b(Yl + y2)2 '" X~· 4. Suppose that Y '" N3 (0, In). Show that t [(Yl - y2)2 + (Y2 - y3 )2 + (Y3 - Yl)2] has a X~ distribution. Does some multiple of (Yl - y2 )2 + (Y2 - y3 )2 + ... + (Yn-l - Yn)2 + (Yn - Yd have a chi-squared distribution for general n? 5. Let Y '" Nn(O, In) and let A and B be symmetric. Show that the joint m.g.f. of Y'AY and Y'BY is [det(ln - 2sA - 2tB)]-1/2. Hence show that the two quadratic forms are independent if AB = 0. MISCELLANEOUS EXERCISES 2 1. Suppose that e '" N 3 (0, (1"213) and that Yo is N(O, (1"5), independently of the c:/s. Define }i = p}i-l +C:i (i = 1,2,3). (a) Find the variance-covariance matrix of Y = (Yl , Y2 , Y3 )'. (b) What is the distribution of Y? 2. Let Y '" Nn(O, In), and put X = AY, U =BY and V =CY. Suppose that Cov[X, U] = °and Cov[X, V] = 0. Show that X is independent ofU + V. 3. If Yl , Y2 , ••• , Yn is a random sample from N(IL, (1"2), prove that Y is independent of L:~;ll (}i - }i+l)2. 4. If X and Y are n-dimensional vectors with independent multivariate normal distributions, prove that aX +bY is also multivariate normal.
- 54. 32 MULTIVARIATE NORMAL DISTRIBUTION 5. If Y '" Nn(O, In) and a is a nonzero vector, show that the conditional distribution of Y'Y given a'Y = 0 is X;-l' 6. Let Y '" Nn(f.Lln, :E), where :E = (1- p)In + plnl~ and p > -1/(n -1). Show that 2:i(Yi - y)2 /(1- p) is X;-l' 7. Let Vi, i = 1, ... , n, be independent Np(/L,:E) random vectors. Show that is an unbiased estimate of :E. 8. Let Y '" Nn(O, In) and let A and B be symmetric idempotent matrices with AB = BA = 0. Show that Y'AY, Y'BY and Y'(In - A - B)Y have independent chi-square distributions. 9. Let (Xi,Yi), i = 1,2, ... ,n, be a random sample from a bivariate normal distribution, with means f.Ll and J-t2, variances a? and a~, and correlation p, and let (a) Show that W has a N2n(/L,:E) distribution, where (b) Find the conditional distribution of X given Y. 10. If Y '" N2 (O, :E), where :E = (aij), prove that ( Y':E-ly _ Yl) '"X~. all ). 11. Let aD, al, ... ,an be independent N(O, ( 2 ) random variables and define Yi =ai + c/Jai-l (i = 1,2, ... ,n). Show that Y = (Yl , Y2 , • •• ,Yn )' has a multivariate normal distribution and find its variance-covariance matrix. (The sequence Yl , Y2 , ••• is called a moving average process of order one and is a commonly used model in time series analysis.) 12. Suppose that Y rv Na(O, In). Find the m.gJ. of 2(Y1 Y2 - Y2 Y3 - YaYl)' Hence show that this random variable has the same distribution as that of 2Ul - U2 - U3 , where the U;'s are independent xi random variables.
- 55. Other documents randomly have different content
- 56. all the acts and proceedings of the governor. The legislative power was vested in the governor and a council of seven persons, who were to be appointed by the governor at first, and hold their office for two years; afterwards they were to be elected by the people. All the laws of Mexico, and the municipal officers existing in the territory before the conquest, were continued until altered by the governor and council. On the 15th of August, 1846, Commodore Stockton adopted a tariff of duties on all goods imported from foreign parts, of fifteen per cent. ad valorem, and a tonnage duty of fifty cents per ton on all foreign vessels. On the 15th of September, when the elections were held, Walter Colton, the chaplain of the frigate Congress, was elected Alcalde of Monterey. In the mean time, a newspaper called the "Californian," had been established by Messrs. Colton and Semple. This was the first newspaper issued in California. Early in September, Commodore Stockton withdrew his forces from Los Angeles, and proceeded with his squadron to San Francisco. Scarcely had he arrived when he received intelligence that all the country below Monterey was in arms and the Mexican flag again hoisted. The Californians invested the "City of the Angels," on the 23d of September. That place was guarded by thirty riflemen under Captain Gillespie, and the Californians investing it numbered 300. Finding himself overpowered, Captain Gillespie capitulated on the 30th, and thence retired with all the foreigners aboard of a sloop-of- war, and sailed for Monterey. Lieutenant Talbot, who commanded only nine men at Santa Barbara, refused to surrender, and marched out with his men, arms in hand. The frigate Savannah was sent to relieve Los Angeles, but she did not arrive till after the above events had occurred. Her crew, numbering 320 men, landed at San Pedro and marched to meet the Californians. About half way between San Pedro and Los Angeles, about fifteen miles from their ship, the sailors found the enemy drawn up on a plain. The Californians were mounted on fine horses, and with artillery, had every advantage.
- 57. The sailors were forced to retreat with a loss of five killed and six wounded. Commodore Stockton came down in the Congress to San Pedro, and then marched for the "City of the Angels," the men dragging six of the ship's guns. At the Rancho Sepulvida, a large force of the Californians was posted. Commodore Stockton sent one hundred men forward to receive the fire of the enemy and then fall back upon the main body without returning it. The main body was formed in a triangle, with the guns hid by the men. By the retreat of the advance party, the enemy were decoyed close to the main force, when the wings were extended and a deadly fire opened upon the astonished Californians. More than a hundred were killed, the same number wounded, and their whole force routed. About a hundred prisoners were taken, many of whom were at the time on parole and had signed an obligation not to take up arms during the war. Commodore Stockton soon mounted his men and prepared for operations on shore. Skirmishes followed, and were continually occurring until January, 1847, when a decisive action occurred. General Kearny had arrived in California, after a long and painful march overland, and his co-operation was of great service to Stockton. The Americans left San Diego on the 29th of December, to march to Los Angeles. The Californians determined to meet them on their route, and decide the fate of the country in a general battle. The American force amounted to six hundred men, and was composed of detachments from the ships Congress, Savannah, Portsmouth and Cyane, aided by General Kearny, with sixty men on foot, from the first regiment of United States dragoons, and Captain Gillespie with sixty mounted riflemen. The troops marched one hundred and ten miles in ten days, and, on the 8th of January, they found the Californians in a strong position on the high bank of the San Gabriel river, with six hundred mounted men and four pieces of artillery, prepared to dispute the passage of the river. The Americans waded through the water, dragging their guns with them, exposed to a galling fire from the enemy, without returning a shot. When they
- 58. reached the opposite shore, the Californians charged upon them, but were driven back. They then charged up the bank and succeeded in driving the Californians from their post. Stockton, with his force, continued his march, and the next day, in crossing the plains of Mesa, the enemy made another attempt to save their capital. They were concealed with their artillery in a ravine, until the Americans came within gun-shot, when they opened a brisk fire upon their right flank, and at the same time charged both their front and rear. But the guns of the Californians were soon silenced, and the charge repelled. The Californians then fled, and the next morning the Americans entered Los Angeles without opposition. The loss of the Americans in killed and wounded did not exceed twenty, while that of their opponents reached between seventy and eighty. These two battles decided the contest in California. General Flores, governor and commandant-general of the Californians, as he styled himself, immediately after the Americans entered Los Angeles, made his escape and his troops dispersed. The territory became again tranquil, and the civil government was soon in operation again in the places where it had been interrupted by the revolt. Commodore Stockton and General Kearny having a misunderstanding about their respective powers, Colonel Fremont exercised the duties of governor and commander-in-chief of California, declining to obey the orders of General Kearny. The account of the adventures and skirmishes with which the small force of United States troops under General Kearny met, while on their march to San Diego, in Upper California, is one of the most interesting to which the contest gave birth. The party, which consisted of one hundred men when it started from Santa Fé, reached Warner's rancho, the frontier settlement in California, on the Sonoma route, on the 2d of December, 1846. They continued their march, and on the 5th were met by a small party of volunteers, under Captain Gillespie, sent out by Commodore Stockton to meet them, and inform them of the revolt of the Californians. The party encamped for the night at Stokes's rancho, about forty miles from
- 59. San Diego. Information was received that an armed party of Californians was at San Pasqual, three leagues from Stokes's rancho. A party of dragoons was sent out to reconnoitre, and they returned by two o'clock on the morning of the 6th. Their information determined General Kearny to attack the Californians before daylight, and arrangements were accordingly made. Captain Johnson was given the command of an advance party of twelve dragoons, mounted upon the best horses in possession of the party. Then followed fifty dragoons, under Captain Moore, mounted mostly on the tired mules they had ridden from Santa Fé—a distance of 1050 miles. Next came about twenty volunteers, under Captain Gibson. Then followed two mountain howitzers, with dragoons to manage them, under charge of Lieutenant Davidson. The remainder of the dragoons and volunteers were placed under command of Major Swords, with orders to follow on the trail with the baggage. As the day of December 6th dawned, the enemy at San Pasqual were seen to be already in the saddle, and Captain Johnson, with his advance guard, made a furious charge upon them; he being supported by the dragoons, the Californians at length gave way. They had kept up a continual fire from the first appearance of the dragoons, and had done considerable execution. Captain Johnson was shot dead in his first charge. The enemy were pursued by Captain Moore and his dragoons, and they retreated about half a mile, when seeing an interval between the small advance party of Captain Moore and the main force coming to his support, they rallied their whole force, and charged with their lances. For five minutes they held the ground, doing considerable execution, until the arrival of the rest of the American party, when they broke and fled. The troops of Kearny lost two captains, a lieutenant, two sergeants, two corporals, and twelve privates. Among the wounded were General Kearny, Lieutenant Warner, Captains Gillespie and Gibson, one sergeant, one bugleman, and nine privates. The Californians carried off all their wounded and dead except six.
- 60. On the 7th the march was resumed, and, near San Bernardo, Kearny's advance encountered and defeated a small party of the Californians who had taken post on a hill. At San Bernardo, the troops remained till the morning of the 11th, when they were joined by a party of sailors and marines, under Lieutenant Gray. They then proceeded upon their march, and on the 12th, arrived at San Diego; having thus completed a march of eleven hundred miles through an enemy's country, with but one hundred men. The force of General Kearny having joined that of Commodore Stockton, the expedition against Los Angeles, of which we have given an account in this chapter, was successfully consummated, and tranquillity restored in California. General Kearny and Commodore Stockton returned to the United States in January, 1847, leaving Colonel Fremont to exercise the office of governor and military commandant of California. No further events of an importance worth recording occurred till the treaty of peace between the United States and Mexico.
- 61. CHAPTER VI. DISCOVERY OF THE GOLD PLACERS. By the treaty concluded between the United States and Mexico, in 1847, the territory of Upper California became the property of the United States. Little thought the Mexican government of the value of the land they were ceding, further than its commercial importance; and, doubtless, little thought the buyers of the territory, that its soil was pregnant with a wealth untold, and that its rivers flowed over golden beds. This territory, now belonging to the American Union, embraces an area of 448,961 square miles. It extends along the Pacific coast, from about the thirty-second parallel of north latitude, a distance of near seven hundred miles, to the forty-second parallel, the southern boundary of Oregon. On the east, it is bounded by New Mexico. During the long period which transpired between its discovery and its cession to the United States, this vast tract of country was frequently visited by men of science, from all parts of the world. Repeated examinations were made by learned and enterprising officers and civilians; but none of them discovered the important fact, that the mountain torrents of the Sierra Nevada were constantly pouring down their golden sands into the valleys of the Sacramento and San Joaquin. The glittering particles twinkled beneath their feet, in the ravines which they explored, or glistened in the watercourses which they forded, yet they passed them by unheeded. Not a legend or tradition was heard among the white settlers, or the aborigines, that attracted their curiosity. A nation's ransom lay within their grasp, but, strange to say, it escaped their notice—it flashed and sparkled all in vain.[1]
- 62. The Russian American Company had a large establishment at Ross and Bodega, ninety miles north of San Francisco, founded in the year 1812; and factories were also established in the territory by the Hudson Bay Company. Their agents and employes ransacked the whole country west of the Sierra Nevada, or Snowy Mountain, in search of game. In 1838, Captain Sutter, formerly an officer in the Swiss Guards of Charles X., King of France, emigrated from the state of Missouri to Upper California, and obtained from the Mexican government a conditional grant of thirty leagues square of land, bounded on the west by the Sacramento river. Having purchased the stock, arms, and ammunition of the Russian establishment, he erected a dwelling and fortification on the left bank of the Sacramento, about fifty miles from its mouth, and near what was termed, in allusion to the new settlers, the American Fork. This formed the nucleus of a thriving settlement, to which Captain Sutter gave the name of New Helvetia. It is situated at the head of navigation for vessels on the Sacramento, in latitude 38° 33' 45" north, and longitude 121° 20' 05" west. During a residence of ten years in the immediate vicinity of the recently discovered placéras, or gold regions, Captain Sutter was neither the wiser nor the richer for the brilliant treasures that lay scattered around him.[2] In the year 1841, careful examinations of the Bay of San Francisco, and of the Sacramento River and its tributaries, were made by Lieutenant Wilkes, the commander of the Exploring Expedition; and a party under Lieutenant Emmons, of the navy, proceeded up the valley of the Willamette, crossed the intervening highlands, and descended the Sacramento. In 1843-4, similar examinations were made by Captain, afterwards Lieutenant-Colonel Fremont, of the Topographical Engineers, and in 1846, by Major Emory, of the same corps. None of these officers made any discoveries of minerals, although they were led to conjecture, as private individuals who had visited the country had done, from its volcanic formation and peculiar geological features, that they might be found to exist in considerable quantities.[3]
- 63. As is often the case, chance at length accomplished what science had failed to do. In the winter of 1847-8, a Mr. Marshall commenced the construction of a saw-mill for Captain Sutter, on the north branch of the American Fork, and about fifty miles above New Helvetia, in a region abounding with pine timber. The dam and race were completed, but on attempting to put the mill in motion, it was ascertained that the tail-race was too narrow to permit the water to escape with perfect freedom. A strong current was then passed in, to wash it wider and deeper, by which a large bed of mud and gravel was thrown up at the foot of the race. Some days after this occurrence, Mr. Marshall observed a number of brilliant particles on this deposit of mud, which attracted his attention. On examining them, he became satisfied that they were gold, and communicated the fact to Captain Sutter. It was agreed between them, that the circumstance should not be made public for the present; but, like the secret of Midas, it could not be concealed. The Mormon emigrants, of whom Mr. Marshall was one, were soon made acquainted with the discovery, and in a few weeks all California was agitated with the startling information. Business of every kind was neglected, and the ripened grain was left in the fields unharvested. Nearly the whole population of Upper California became infected with the mania, and flocked to the mines. Whalers and merchant vessels entering the ports were abandoned by their crews, and the American soldiers and sailors deserted in scores. Upon the disbandment of Colonel Stevenson's regiment, most of the men made their way to the mineral regions. Within three months after the discovery, it was computed that there were near four thousand persons, including Indians, who were mostly employed by the whites, engaged in washing for gold. Various modes were adopted to separate the metal from the sand and gravel —some making use of tin pans, others of close-woven Indian baskets, and others still, of a rude machine called the cradle, six or eight feet long, and mounted on rockers, with a coarse grate, or sieve, at one end, but open at the other. The washings were mainly confined to the low wet grounds, and the margins of the streams—
- 64. the earth being rarely disturbed more than eighteen inches below the surface. The value of the gold dust obtained by each man, per day, is said to have ranged from ten to fifty dollars, and sometimes even to have far exceeded that. The natural consequence of this state of things was, that the price of labor, and, indeed, of every thing, rose immediately from ten to twenty fold.[4] As may readily be conjectured, every stream and ravine in the valley of the Sacramento was soon explored. Gold was found on every one of its tributaries; but the richest earth was discovered near the Rio de los Plumas, or Feather River,[5] and its branches, the Yuba and Bear rivers, and on Weber's creek, a tributary of the American Fork. Explorations were also made in the valley of the San Joaquin, which resulted in the discovery of gold on the Cosumnes and other streams, and in the ravines of the Coast Range, west of the valley, as far down as Ciudad de los Angeles. In addition to the gold mines, other important discoveries were made in Upper California. A rich vein of quicksilver was opened at New Almaden, near Santa Clara, which, with imperfect machinery,— the heat by which the metal is made to exude from the rock being applied by a very rude process,—yielded over thirty per cent. This mine—one of the principal advantages to be derived from which will be, that the working of the silver mines scattered through the territory must now become profitable—is superior to those of Almaden, in Old Spain, and second only to those of Idria, near Trieste, the richest in the world. Lead mines were likewise discovered in the neighborhood of Sonoma, and vast beds of iron ore near the American Fork, yielding from eighty-five to ninety per cent. Copper, platina, tin, sulphur, zinc, and cobalt, were discovered every where; coal was found to exist in large quantities in the Cascade range of Oregon, of which the Sierra Nevada is a continuation; and in the vicinity of all this mineral wealth, there are immense quarries of marble and granite, for building purposes.
- 65. Colonel Mason had succeeded Colonel Fremont in the post of governor of California and military commandant. A regiment of New York troops, under the command of Colonel Stevenson, had been ordered to California before the conclusion of the treaty of peace, and formed the principal part of the military force in the territory. Colonel Mason expressed the opinion, in his official despatch, that "there is more gold in the country drained by the Sacramento and San Joaquin rivers, than will pay the cost of the [late] war with Mexico a hundred times over." Should this even prove to be an exaggeration, there can be little reason to doubt, when we take into consideration all the mineral resources of the country, that the territory of California is by far the richest acquisition made by this government since its organization. The appearance of the mines, at the period of Governor Mason's visit, three months after the discovery, he thus graphically describes: "At the urgent solicitation of many gentlemen, I delayed there [at Sutter's Fort] to participate in the first public celebration of our national anniversary at that fort, but on the 5th resumed the journey, and proceeded twenty-five miles up the American Fork to a point on it now known as the Lower Mines, or Mormon Diggins. The hill-sides were thickly strewn with canvas tents and bush arbors; a store was erected, and several boarding shanties in operation. The day was intensely hot, yet about two hundred men were at work in the full glare of the sun, washing for gold—some with tin pans, some with close-woven Indian baskets, but the greater part had a rude machine, known as the cradle. This is on rockers, six or eight feet long, open at the foot, and at its head has a coarse grate, or sieve; the bottom is rounded, with small cleats nailed across. Four men are required to work this machine; one digs the ground in the bank close by the stream; another carries it to the cradle and empties it on the grate; a third gives a violent rocking motion to the machine; while a fourth dashes on water from the stream itself.
- 66. "The sieve keeps the coarse stones from entering the cradle, the current of water washes off the earthy matter, and the gravel is gradually carried out at the foot of the machine, leaving the gold mixed with a heavy, fine black sand above the first cleats. The sand and gold, mixed together, are then drawn off through auger holes into a pan below, are dried in the sun, and afterward separated by blowing off the sand. A party of four men thus employed at the lower mines, averaged $100 a day. The Indians, and those who have nothing but pans or willow baskets, gradually wash out the earth and separate the gravel by hand, leaving nothing but the gold mixed with sand, which is separated in the manner before described. The gold in the lower mines is in fine bright scales, of which I send several specimens. "From the mill [where the gold was first discovered], Mr. Marshall guided me up the mountain on the opposite or north bank of the south fork, where, in the bed of small streams or ravines, now dry, a great deal of coarse gold has been found. I there saw several parties at work, all of whom were doing very well; a great many specimens were shown me, some as heavy as four or five ounces in weight, and I send three pieces, labeled No. 5, presented by a Mr. Spence. You will perceive that some of the specimens accompanying this, hold mechanically pieces of quartz; that the surface is rough, and evidently moulded in the crevice of a rock. This gold cannot have been carried far by water, but must have remained near where it was first deposited from the rock that once bound it. I inquired of many people if they had encountered the metal in its matrix, but in every instance they said they had not; but that the gold was invariably mixed with washed gravel, or lodged in the crevices of other rocks. All bore testimony that they had found gold in greater or less quantities in the numerous small gullies or ravines that occur in that mountainous region. "On the 7th of July I left the mill, and crossed to a stream emptying into the American Fork, three or four miles below the saw-mill. I struck this stream (now known as Weber's creek) at the washings of
- 67. Sunol and Co. They had about thirty Indians employed, whom they payed in merchandise. They were getting gold of a character similar to that found in the main fork, and doubtless in sufficient quantities to satisfy them. I send you a small specimen, presented by this company, of their gold. From this point, we proceeded up the stream about eight miles, where we found a great many people and Indians —some engaged in the bed of the stream, and others in the small side valleys that put into it. These latter are exceedingly rich, and two ounces were considered an ordinary yield for a day's work. A small gutter not more than a hundred yards long, by four feet wide and two or three feet deep, was pointed out to me as the one where two men—William Daly and Parry McCoon—had, a short time before, obtained $17,000 worth of gold. Captain Weber informed me that he knew that these two men had employed four white men and about a hundred Indians, and that, at the end of one week's work, they paid off their party, and had left $10,000 worth of this gold. Another small ravine was shown me, from which had been taken upward of $12,000 worth of gold. Hundreds of similar ravines, to all appearances, are as yet untouched. I could not have credited these reports, had I not seen, in the abundance of the precious metal, evidence of their truth. "Mr. Neligh, an agent of Commodore Stockton, had been at work about three weeks in the neighborhood, and showed me, in bags and bottles, over $2000 worth of gold; and Mr. Lyman, a gentleman of education, and worthy of every credit, said he had been engaged, with four others, with a machine, on the American Fork, just below Sutter's mill; that they worked eight days, and that his share was at the rate of fifty dollars a day; but hearing that others were doing better at Weber's place, they had removed there, and were then on the point of resuming operations. I might tell of hundreds of similar instances; but, to illustrate how plentiful the gold was in the pockets of common laborers, I will mention a single occurrence which took place in my presence when I was at Weber's store. This store was nothing but an arbor of bushes, under which he had exposed for sale goods and groceries suited to his customers. A man came in,
- 68. picked up a box of Seidlitz powders, and asked the price. Captain Weber told him it was not for sale. The man offered an ounce of gold, but Captain Weber told him it only cost fifty cents, and he did not wish to sell it. The man then offered an ounce and a half, when Captain Weber had to take it. The prices of all things are high, and yet Indians, who before hardly knew what a breech cloth was, can now afford to buy the most gaudy dresses. "The country on either side of Weber's creek is much broken up by hills, and is intersected in every direction by small streams or ravines, which contain more or less gold. Those that have been worked are barely scratched; and although thousands of ounces have been carried away, I do not consider that a serious impression has been made upon the whole. Every day was developing new and richer deposits; and the only impression seemed to be, that the metal would be found in such abundance as seriously to depreciate in value. "On the 8th of July, I returned to the lower mines, and on the following day to Sutter's, where, on the 19th, I was making preparations for a visit to the Feather, Yuba, and Bear Rivers, when I received a letter from Commander A. R. Long, United States Navy, who had just arrived at San Francisco from Mazatlan with a crew for the sloop-of-war Warren, with orders to take that vessel to the squadron at La Paz. Captain Long wrote to me that the Mexican Congress had adjourned without ratifying the treaty of peace, that he had letters from Commodore Jones, and that his orders were to sail with the Warren on or before the 20th of July. In consequence of these, I determined to return to Monterey, and accordingly arrived here on the 17th of July. Before leaving Sutter's, I satisfied myself that gold existed in the bed of the Feather River, in the Yuba and Bear, and in many of the smaller streams that lie between the latter and the American Fork; also, that it had been found in the Cosumnes to the south of the American Fork. In each of these streams the gold is found in small scales, whereas in the intervening mountains it occurs in coarser lumps.
- 69. "Mr. Sinclair, whose rancho is three miles above Sutter's, on the north side of the American, employs about fifty Indians on the north fork, not far from its junction with the main stream. He had been engaged about five weeks when I saw him, and up to that time his Indians had used simply closely woven willow baskets. His net proceeds (which I saw) were about $16,000 worth of gold. He showed me the proceeds of his last week's work—fourteen pounds avoirdupois of clean-washed gold. "The principal store at Sutter's Fort, that of Brannan and Co., had received in payment for goods $36,000 (worth of this gold) from the 1st of May to the 10th of July. Other merchants had also made extensive sales. Large quantities of goods were daily sent forward to the mines, as the Indians, heretofore so poor and degraded, have suddenly become consumers of the luxuries of life. I before mentioned that the greater part of the farmers and rancheros had abandoned their fields to go to the mines. This is not the case with Captain Sutter, who was carefully gathering his wheat, estimated at 40,000 bushels. Flour is already worth at Sutter's thirty-six dollars a barrel, and soon will be fifty. Unless large quantities of breadstuffs reach the country, much suffering will occur; but as each man is now able to pay a large price, it is believed the merchants will bring from Chili and Oregon a plentiful supply for the coming winter. "The most moderate estimate I could obtain from men acquainted with the subject, was, that upward of four thousand men were working in the gold district, of whom more than one-half were Indians; and that from $30,000 to $50,000 worth of gold, if not more, was daily obtained. The entire gold district, with very few exceptions of grants made some years ago by the Mexican authorities, is on land belonging to the United States. It was a matter of serious reflection with me, how I could secure to the government certain rents or fees for the privilege of procuring this gold; but upon considering the large extent of country, the character of the people engaged, and the small scattered force at my command, I resolved not to interfere, but to permit all to work
- 70. freely, unless broils and crimes should call for interference. I was surprised to hear that crime of any kind was very unfrequent, and that no thefts or robberies had been committed in the gold district. "All live in tents, in bush arbors, or in the open air; and men have frequently about their persons thousands of dollars worth of this gold, and it was to me a matter of surprise that so peaceful and quiet state of things should continue to exist. Conflicting claims to particular spots of ground may cause collisions, but they will be rare, as the extent of country is so great, and the gold so abundant, that for the present there is room enough for all. Still the government is entitled to rents for this land, and immediate steps should be devised to collect them, for the longer it is delayed the more difficult it will become. One plan I would suggest is, to send out from the United States surveyors with high salaries, bound to serve specified periods. "The discovery of these vast deposits of gold has entirely changed the character of Upper California. Its people, before engaged in cultivating their small patches of ground, and guarding their herds of cattle and horses, have all gone to the mines, or are on their way thither. Laborers of every trade have left their work benches, and tradesmen their shops. Sailors desert their ships as fast as they arrive on the coast, and several vessels have gone to sea with hardly enough hands to spread a sail. Two or three are now at anchor in San Francisco with no crew on board. Many desertions, too, have taken place from the garrisons within the influence of these mines; twenty-six soldiers have deserted from the post of Sonoma, twenty- four from that of San Francisco, and twenty-four from Monterey. For a few days the evil appeared so threatening, that great danger existed that the garrisons would leave in a body; and I refer you to my orders of the 25th of July, to show the steps adopted to meet this contingency. I shall spare no exertions to apprehend and punish deserters, but I believe no time in the history of our country has presented such temptations to desert as now exist in California.
- 71. "The danger of apprehension is small, and the prospect of high wages certain; pay and bounties are trifles, as laboring men at the mines can now earn in one day more than double a soldier's pay and allowances for a month, and even the pay of a lieutenant or captain cannot hire a servant. A carpenter or mechanic would not listen to an offer of less than fifteen or twenty dollars a day. Could any combination of affairs try a man's fidelity more than this? I really think some extraordinary mark of favor should be given to those soldiers who remain faithful to their flag throughout this tempting crisis. "Many private letters have gone to the United States, giving accounts of the vast quantity of gold recently discovered, and it may be a matter of surprise why I have made no report on this subject at an earlier date. The reason is, that I could not bring myself to believe the reports that I heard of the wealth of the gold district until I visited it myself. I have no hesitation now in saying that there is more gold in the country drained by the Sacramento and San Joaquin Rivers than will pay the cost of the present war with Mexico a hundred times over. No capital is required to obtain this gold, as the laboring man wants nothing but his pick and shovel and tin pan, with which to dig and wash the gravel; and many frequently pick gold out of the crevices of the rocks with their butcher knives, in pieces of from one to six ounces. "Mr. Dye, a gentleman residing in Monterey, and worthy of every credit, has just returned from Feather River. He tells me that the company to which he belonged worked seven weeks and two days, with an average of fifty Indians (washers,) and that their gross product was two hundred and seventy-three pounds of gold. His share (one seventh,) after paying all expenses, is about thirty-seven pounds, which he brought with him and exhibited in Monterey. I see no laboring man from the mines who does not show his two, three, or four pounds of gold. A soldier of the artillery company returned here a few days ago from the mines, having been absent on furlough twenty days. He made by trading and working, during that
- 72. time, $1500. During these twenty days he was travelling ten or eleven days, leaving but a week in which he made a sum of money greater than he receives in pay, clothes, and rations, during a whole enlistment of five years. These statements appear incredible, but they are true. "Gold is also believed to exist on the eastern slope of the Sierra Nevada; and when at the mines, I was informed by an intelligent Mormon that it had been found near the Great Salt Lake by some of his fraternity. Nearly all the Mormons are leaving California to go to the Salt Lake, and this they surely would not do unless they were sure of finding gold there in the same abundance as they now do on the Sacramento. "The gold 'placer' near the mission of San Fernando has long been known, but has been little wrought for want of water. This is a spur which puts off from the Sierra Nevada (see Fremont's map,) the same in which the present mines occur. There is, therefore, every reason to believe, that in the intervening spaces, of five hundred miles (entirely unexplored) there must be many hidden and rich deposits. The 'placer' gold is now substituted as the currency of this country; in trade it passes freely at $16 per ounce; as an article of commerce its value is not yet fixed. The only purchase I made was of the specimen No. 7, which I got of Mr. Neligh at $12 the ounce. That is about the present cash value in the country, although it has been sold for less. The great demand for goods and provisions, made by this sudden development of wealth, has increased the amount of commerce at San Francisco very much, and it will continue to increase." The Californian, published at San Francisco on the 14th of August, furnishes the following interesting account of the Gold Region: "It was our intention to present our readers with a description of the extensive gold, silver, and iron mines, recently discovered in the Sierra Nevada, together with some other important items, for the good of the people, but we are compelled to defer it for a future
- 73. number. Our prices current, many valuable communications, our marine journal, and other important matters, have also been crowded out. But to enable our distant readers to draw some idea of the extent of the gold mine, we will confine our remarks to a few facts. The country from the Ajuba to the San Joaquin rivers, a distance of about one hundred and twenty miles, and from the base toward the summit of the mountains, as far as Snow Hill, about seventy miles, has been explored, and gold found on every part. There are now probably 3000 people, including Indians, engaged collecting gold. The amount collected by each man who works, ranges from $10 to $350 per day. The publisher of this paper, while on a tour alone to the mining district, collected, with the aid of a shovel, pick and tin pan, about twenty inches in diameter, from $44 to $128 a day—averaging $100. The gross amount collected will probably exceed $600,000, of which amount our merchants have received about $250,000 worth for goods sold; all within the short space of eight weeks. The largest piece of gold known to be found weighed four pounds. "Labor has ever been high in California, but previous to the discovery of the placera gold, the rates ranged from $1 to $3 per day. Since that epoch common labor cannot be obtained, and if to be had, for no less price than fifty cents per hour, and that the most common. Carpenters and other mechanics have been offered $15 a day, but it has been flatly refused. Many of our enterprising citizens were largely engaged in building, and others wish to commence on dwellings, warehouses, and the like, but all have had to suspend for the lack of that all important class of community, the working men." The following extracts from the published journal of a physician in California, give accounts of the reception of the news of the gold discovery in San Francisco, with its consequent effects. "May 8th.—Captain Fulsom called at Sweeting's to-day. He had seen a man this morning, who reported that he had just come from a river called the American Fork, about one hundred miles in the interior, where he had been gold washing. Captain Fulsom saw the
- 74. gold he had with him; it was about twenty-three ounces weight, and in small flakes. The man stated that he was eight days getting it, but Captain Fulsom hardly believed this. He says that he saw some of this gold a few weeks since, and thought it was only 'mica,' but good judges have pronounced it to be genuine metal. He talks, however, of paying a visit to the place where it is reported to come from. After he was gone, Bradley stated that the Sacramento settlements, which Malcolm wished to visit, were in the neighborhood of the American Fork, and that we might go there together; he thought the distance was only one hundred and twenty miles. "May 10th.—Yesterday and to-day nothing has been talked of but the new gold 'placer,' as people call it. It seems that four other men had accompanied the person Captain Fulsom saw yesterday, and that they had each realized a large quantity of gold. They left the 'diggings' on the American Fork, (which it seems is the Rio de los Americanos, a tributary to the Sacramento) about a week ago, and stopped a day or two at Sutter's Fort, a few miles this side of the diggings, on their way; from there they had travelled by boat to San Francisco. The gold they brought has been examined by the first Alcalde here and by all the merchants in the place. Bradley showed us a lump weighing a quarter of an ounce, which he had bought of one of the men, and for which he gave him three dollars and a half. I have no doubt in my own mind about its being genuine gold. Several parties, we hear, are already made up to visit the diggings; and, according to the newspaper here, a number of people have actually started off with shovels, mattocks, and pans, to dig the gold themselves. It is not likely, however, that this will be allowed, for Captain Fulsom has already written to Colonel Mason about taking possession of the mine on behalf of the government, it being, as he says, on public land. "May 17th.—This place is now in a perfect furor of excitement; all the work-people have struck. Walking through the town to-day, I observed that laborers were employed only upon about half-a-dozen of the fifty new buildings which were in the course of being run up.
- 75. The majority of the mechanics at this place are making preparations for moving off to the mines, and several hundred people of all classes—lawyers, store-keepers, merchants, &c.,—are bitten with the fever; in fact, there is a regular gold mania springing up. I counted no less than eighteen houses which were closed, the owners having left." The mania continued to increase, and within a few months all the principal towns were nearly emptied of their population. Gold was the universal object, and splendid and rapid fortune the universal hope. No occupation seemed to offer such a prospect as that of digging gold, and, accordingly, those who were not able to bear the fatigues of such work, or were at the head of any sort of business in the different towns, had to pay enormous prices for the labor of subordinates who performed the meanest services. The prices of all agricultural and manufactured products became treble the previous rates. Soon came the first waves of the tide of emigration that was to flood the placers of the gold region. The first influx consisted of Mexicans of the province of Sonoma, Chilians, and some few Chinese. These, principally took possession of the southern mines, or those on the San Joaquin and its tributaries. Some few stopped at San Francisco, and secured lots of ground which they knew would become very valuable in a short time, and erected temporary stores and dwellings. This gave the impulse to the progress of the town, and it soon advanced rapidly in size and population. Then came the emigration from the Atlantic States of the Union, and the whole territory felt the progressive and enterprising spirit of the gold- seekers. The Americans generally took possession of the mines upon the northern tributaries of the Sacramento River; but as their numbers increased they pushed towards the southern mines, and frequent collisions with the foreigners were the consequence. Finally, a great number of the latter were compelled to leave the country.
- 77. CHAPTER VII. ADVENTURES OF SOME OF THE MINERS, AND INCIDENTS CONNECTED WITH MINING. The adventures of the eager gold-seekers in the region of their hopes, among the washings and the diggings of the placers, cannot but be interesting. The toil to which the men have to submit if they would obtain any thing like a satisfaction to their desires, is of a very irksome character. In the summer season, the heat is intense, and the principal part of the labor of washing and digging must be performed exposed to the full blaze of the sun. In the "dry diggings," the miners suffer greatly from the want of water. Most of the provisions having to be transported from the towns on the Sacramento and San Joaquin, soon grow unwholesome from exposure to the sultry air of the day and the damp air of the night. This diet, conjointly with the exposure of the miners, tends to produce intermittent fever and dysentery. The miners generally reside in huts of a rude construction, or in canvas tents, which afford but poor protection from the changes of the weather. The most prominent man in the neighborhood of the "diggins," is Captain Sutter, the Daniel Boone of that part of the country. He was formerly an officer in the Swiss guards of Charles X. of France. After the revolution of 1830, in that country, he came to the United States. Emigrating to California, he obtained a grant of land from the Mexican government, and founded the settlement known as Sutter's Fort. Upon his land, the first discovery of the richness of the soil was made, and his house and the settlement around it has been, ever since, the resort of persons going to and from the placers, and a depot for provisions and articles used by the miners. Stores and
- 78. workshops have been established, and a considerable amount of business is transacted there. Captain Sutter is held in very great respect by the people of the settlement and those stopping at his house on the road to the placers. Several versions of the account of the discovery of the gold mines have been circulated, but the true one, in the Captain's own words, is given in a work recently published.[6] The account is here inserted, both on account of the interest connected with the discovery, and in order to correct wrong versions of the matter. "I was sitting one afternoon," said the Captain, "just after my siesta, engaged, by-the-bye, in writing a letter to a relation of mine at Lucerne, when I was interrupted by Mr. Marshall—a gentleman with whom I had frequent business transactions—bursting hurriedly into the room. From the unusual agitation in his manner, I imagined that something serious had occurred, and, as we involuntarily do in this part of the world, I at once glanced to see if my rifle was in its proper place. You should know that the mere appearance of Mr. Marshall at that moment in the fort was quite enough to surprise me, as he had but two days before left the place to make some alterations in a mill for sawing pine planks, which he had just run up for me, some miles higher up the Americanos. When he had recovered himself a little, he told me that, however great my surprise might be at his unexpected reappearance, it would be much greater when I heard the intelligence he had come to bring me. 'Intelligence,' he added, 'which, if properly profited by, would put both of us in possession of unheard-of wealth—millions and millions of dollars, in fact.' I frankly own, when I heard this, that I thought something had touched Marshall's brain, when suddenly all my misgivings were put an end to by his flinging on the table a handful of scales of pure virgin gold. I was fairly thunderstruck, and asked him to explain what all this meant, when he went on to say, that, according to my instructions, he had thrown the mill-wheel out of gear, to let the whole body of the water in the dam find a passage through the tail-race, which was previously too narrow to allow the water to run off in sufficient quantity, whereby the wheel was
- 79. prevented from efficiently performing its work. By this alteration the narrow channel was considerably enlarged, and a mass of sand and gravel carried off by the force of the torrent. Early in the morning after this took place, he (Mr. Marshall) was walking along the left bank of the stream, when he perceived something which he at first took for a piece of opal—a clear transparent stone, very common here—glittering on one of the spots laid bare by the sudden crumbling away of the bank. He paid no attention to this; but while he was giving directions to the workmen, having observed several similar glittering fragments, his curiosity was so far excited, that he stooped down and picked one of them up. 'Do you know,' said Mr. Marshall to me, 'I positively debated within myself two or three times whether I should take the trouble to bend my back to pick up one of the pieces, and had decided on not doing so, when, further on, another glittering morsel caught my eye—the largest of the pieces now before you. I condescended to pick it up, and to my astonishment found that it was a thin scale of what appears to be pure gold.' He then gathered some twenty or thirty similar pieces, which on examination convinced him that his suppositions were right. His first impression was, that this gold had been lost or buried there by some early Indian tribe—perhaps some of those mysterious inhabitants of the West, of whom we have no account, but who dwelt on this continent centuries ago, and built those cities and temples, the ruins of which are scattered about these solitary wilds. On proceeding, however, to examine the neighboring soil, he discovered that it was more or less auriferous. This at once decided him. He mounted his horse, and rode down to me as fast as it would carry him, with the news. "At the conclusion of Mr. Marshall's account," continued Captain Sutter, "and when I had convinced myself, from the specimens he had brought with him, that it was not exaggerated, I felt as much excited as himself. I eagerly inquired if he had shown the gold to the work people at the mill, and was glad to hear that he had not spoken to a single person about it. We agreed," said the Captain, smiling, "not to mention the circumstance to any one, and arranged
- 80. to set off early the next day for the mill. On our arrival, just before sundown, we poked the sand about in various places, and before long succeeded in collecting between us, more than an ounce of gold, mixed up with a good deal of sand. I stayed at Mr. Marshall's that night, and the next day we proceeded some little distance up the South Fork, and found that gold existed along the whole course, not only in the bed of the main stream, where the water had subsided, but in every little dried-up creek and ravine. Indeed, I think it is more plentiful in these latter places, for I myself, with nothing more than a small knife, picked out from a dry gorge, a little way up the mountain, a solid lump of gold which weighed nearly an ounce and a half. "On our return to the mill, we were astonished by the work-people coming up to us in a body, and showing us small flakes of gold similar to those we had ourselves procured. Marshall tried to laugh the matter off with them, and to persuade them that what they had found was only some shining mineral of trifling value; but one of the Indians, who had worked at the gold mine in the neighborhood of La Paz, in Lower California, cried out, 'Oro! oro!' We were disappointed enough at this discovery, and supposed that the work-people had been watching our movements, although we thought we had taken every precaution against being observed by them. I heard, afterwards, that one of them, a sly Kentuckian, had dogged us about, and that, looking on the ground to see if he could discover what we were in search of, he had lighted on some flakes of gold himself. "The next day I rode back to the Fort, organized a laboring party, set the carpenters to work on a few necessary matters, and the next day, accompanied them to a point of the Fork, where they encamped for the night. By the following morning I had a party of fifty Indians fairly at work. The way we first managed was to shovel the soil into small buckets, or into some of our famous Indian baskets; then wash all the light earth out, and pick away the stones; after this, we dried the sand on pieces of canvas, and with long reeds blew away
- 81. all but the gold. I have now some rude machines in use, and upwards of one hundred men employed, chiefly Indians, who are well fed, and who are allowed whisky three times a day. "The report soon spread. Some of the gold was sent to San Francisco, and crowds of people flocked to the diggings. Added to this, a large emigrant party of Mormons entered California across the Rocky Mountains, just as the affair was first made known. They halted at once, and set to work on a spot some thirty miles from here, where a few of them still remain. When I was last up to the diggings, there were full eight hundred men at work, at one place and another, with perhaps something like three hundred more passing backwards and forwards between here and the mines. I at first imagined that the gold would soon be exhausted by such crowds of seekers, but subsequent observations have convinced me that it will take many years to bring about such a result, even with ten times the present number of people employed. "What surprises me," continued the Captain, "is, that this country should have been visited by so many scientific men, and that not one of them should have ever stumbled upon the treasures; that scores of keen eyed trappers should have crossed this valley in every direction, and tribes of Indians have dwelt in it for centuries, and yet that this gold should have never been discovered. I myself have passed the very spot above a hundred times during the last ten years, but was just as blind as the rest of them, so I must not wonder at the discovery not having been made earlier." The plan of operations adopted by most of the miners who were not Indians or Californians, was to form bands of three, five or ten, under the command of one of the number, whose name the party took, and by which it was afterwards known. Some larger companies were formed in the United States, and repaired to California, and their operations were of course, on a more extensive scale; they having all the necessary equipments of gold-washers and miners. Written rules were generally drawn up for the government of the parties, varying in particulars according to the peculiar views of the
- 82. framers. These rules provided for the modus operandi of procuring the gold, supplying the party with necessaries, attending to the sick, and the division of the fruits of their labor. One of the most frequented placers of California is called the Stanislaus mine, situated near the Stanislaus River. It was one of the first places worked to any extent by the gold-seekers, but not satisfying the expectations of some of the most greedy, it has since been partially abandoned. A description of this mine, and of the living and operations of its workers in the winter of 1848-49, will give a good general idea of the toils and privations endured by the early gold-seekers in that region, and, also, of their mode of procuring the precious metal at most of the mines. We extract from a recently published work, distinguished for minuteness of detail and accuracy of description.[7] "The mine was a deep ravine, embosomed amidst lofty hills, surmounted by, and covered with pine, and having, in the bottom itself, abundance of rock, mud, and sand. Halliday and I encamped at the very lowest part of the ravine, at a little distance from Don Emanuel's party; a steep rock which towered above our heads affording us shelter, and a huge, flat stone beneath our feet promising a fair substitute for a dry bed. Here then we stretched our macheers and blankets, and arranged our saddles and bags, so as to make ourselves as comfortable and warm as possible, although, in spite of our precautions and contrivances, and of a tolerably good fire, our encampment was bitterly cold, and we lay exposed to a heavy dew. We had given up our horses into the charge of the Indians, and I saw to their being safely placed in the cavallard, whilst Halliday went to chop wood; a task I was too weak to perform. I cannot say we slept; we might more correctly be said to have had a long and most uncomfortable doze, and when morning broke, we were shivering with cold, and shook the dew in a shower from our clothes. I consulted with my companion, and urged upon him the prudence of our setting to work to construct ourselves a sort of log cabin; otherwise I felt certain, from the experience of the past
- 83. night, our sojourn at the mines would be likely to prove fatal to one or both of us. He was, however, far too eager to try his fortune at digging to listen to my proposal, at which he even smiled, probably at the bare idea of weather, privation, or toil, being able to affect his powerful frame. I saw him presently depart up the ravine, shouldering a pick, and glancing now and then at his knife, whilst I proceeded in search of materials for constructing a temporary place of shelter. "As my strength was unequal to the task of felling timber, I endeavored to procure four poles, intending to sink them into the ground, and to stretch on the top of them a bed-tick I had reserved for the purpose. The contrivance was a sorry one at the best, but shelter was indispensable; and great was my disappointment— though I procured the timber after a painful search—to find that the rocks presented an insuperable obstacle to my employing it as I intended. My efforts to sink the poles proved utterly futile, and I was at last compelled to renounce the attempt in despair. I then packed up our goods into as close a compass as possible; and, having requested one of the Spaniards in Don Emanuel's party to keep watch over them, departed to explore the ravine. "Within a few paces of our encampment there was a large area of ground, probably half a mile square, the surface of which consisted of dark soil and slate, and was indented with innumerable holes of every possible dimension, from six inches to as many feet or more, wide and deep. In all of these lay abundance of water, of which large quantities are to be found a little beneath the surface, the ravine being supplied with it in great abundance by the rains that pour down from the hills during the wet season. To the extreme right of our camp, the ground assumed a more rocky character; and, from the vast deposit of stagnant water, did not seem to offer many attractions to the miners. Yet there was scarcely a spot in any of these places where the crow-bar, the pick, or the jack-knife, had not been busy: evidence that the whole locality must have been
- 84. extremely rich in the precious metal, or it would not have been so thoroughly worked. "In crossing the ravine, I was obliged to leap from one mound of earth to another, to avoid plunging ancle-deep in mud and water. It was wholly deserted in this part, though formerly so much frequented; and, with the exception of a few traders, who, having taken up their station here when times were good, had not yet made arrangements for removing to a more productive place, not a soul was to be seen. "I walked on until I reached the trading-post of Mr. Anderson, formerly our interpreter in the Lower Country, whom I felt delighted to meet with again. His shed was situated in one of the dampest parts of the mine, and consisted of a few upright poles, traversed by cross-pieces, and covered in with raw hides and leaves, but yet much exposed at the sides to the wind and the weather. He had a few barrels of flour and biscuit, which he retailed at two dollars a pound; for he made no difference between the price of the raw and the prepared material. The flour would go further, it was true; but then the biscuit required no cooking on the part of the miner, whose time was literally money, and whose interest therefore it was to economize it in every possible manner. He also sold unprepared coffee and sugar at six Yankee shillings a pound; dried beef at one dollar and a half; and pork, which was regarded as a great delicacy here, at two dollars for the same weight. The various articles of which his stock-in-trade consisted he had brought all the way from Monterey at considerable labor and expense; but, by the exercise of extraordinary tact, perseverance, and industry, he had succeeded in establishing a flourishing business. "I discovered, however, that he possessed another resource—by which his gains were marvellously increased—in the services of seven or eight Indians, whom he kept constantly at work, in the rear of his shed, digging gold, and whose labor he remunerated with provisions, and occasional presents of articles of trifling value to him, but highly esteemed by the Indians. They were watched by an
- 85. American overseer, who was employed by him, to assist in the general business, particularly in slaughtering; for, as beef was scarce, he used to send his man in quest of cows and oxen; which he killed, cut up, salted and dried, in his shed, and watching the most favorable moment for the operation—namely, when meat could not be procured at the 'diggins'—never failed to realize his own price for it. "Proceeding higher up the ravine, I observed a large tent erected on the slope of a hill, within a few yards of the bottom, where the gold is usually found. It was surrounded by a trench, the clay from which, as it was dug up, had apparently been thrown out against the canvas, forming a kind of embankment, rendering it at once water and weather-proof. I ventured into it, encountering on my way an immense piece of raw beef, suspended from the ridge-pole. Upon some stones in front, inclosing a small fire, stood a frying-pan, filled with rich looking beef collops, that set my mouth watering, and severely tested my honesty; for, although acorns are all very well in their way, and serve to stay the cravings of the stomach for awhile, I did not find my appetite any the less sharp, notwithstanding the quantity I had eaten. But I resisted the temptation, and penetrated further into the tent. At one side of it lay a crow-bar, and an old saddle that had seen rough service; yet not a soul appeared, and my eyes were again ogling the collops, whilst an inward voice whispered how imprudent it was to leave them frizzling there, when, all at once, a little man, in a 'hickory shirt,' with his face all bedaubed with pot-black and grease, darted out of some dark corner, flourishing in one hand a long bowie knife, and in the other three by no means delicate slices of fat pork, which he at once dropped into the frying- pan, stooping down on one knee, and becoming immediately absorbed in watching the interesting culinary process then going on in it. "I came up next with a group of three Sonomeans, or inhabitants of Sonoma, busily engaged on a small sandy flat—the only one I had observed—at the bottom of the ravine. There was no water near,
- 86. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com