![14 BACKGROUND
square or tall) and constructs
X = QR
where R is upper triangular, so that ri,j = 0 for i j, and Q is an orthogonal matrix,
QTQ = I. Note that the columns of Q are said to be orthonormal (i.e., orthogonal and
normalized); however, the matrix itself is said to be orthogonal.
The singular value decomposition (SVD) takes a matrix, X, of any size and shape and
replaces it with the product,
X = USVT
where S is a diagonal matrix with non-negative diagonal elements. By convention, the
elements of S, the singular values, are arranged in decreasing order along the diagonal.
Thus, s1,1 ≥ s2,2 ≥ · · · ≥ sn,n. U and V are orthogonal matrices containing the left and
right singular vectors, respectively.
Throughout the book, the approximate computational cost of algorithms will be given
in flops. A flop, or floating point operation, represents either the addition or the mul-
tiplication of two floating point numbers. Note that a flop was originally defined as a
floating point multiplication optionally followed by a floating point addition. Thus, the
first edition of Golub and Van Loan (1989) used this earlier definition, whereas subse-
quent editions used the later definition. Indeed, they Golub and Van Loan (1989) noted
that supercomputer manufacturers were delighted by this change in terminology, since
it gave the appearance of an overnight doubling in performance. In any case, the pre-
cise definition is not important to this text, so long as it is applied consistently; the flop
count will be used only as a first-order measurement of the computation requirements of
algorithms.
2.2 GAUSSIAN RANDOM VARIABLES
Gaussian random variables, and signals derived from them, will play a central role in
much of the development to follow. A Gaussian random variable has the probability
density (Bendat and Piersol, 1986; Papoulis, 1984)
f (x) =
1
√
2πσ2
exp
−
(x − µ)2
2σ2
(2.1)
that is completely defined by two parameters: the mean, µ, and variance, σ2. By con-
vention, the mean, variance, and other statistical moments, are denoted by symbols sub-
scripted by the signal they describe. Thus,
µx = E[x]
σ2
x = E
(x − µx)2
Figure 2.1 shows a single realization of a Gaussian signal, the theoretical probability
density function (PDF) of the process that generated the signal, and an estimate of the
PDF derived from the single realization.](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-30-320.jpg)
![GAUSSIAN RANDOM VARIABLES 15
0 0.1 0.2 0.3 0.4 0.5
−5
0
5
10
Range
Time (s)
m + s
m − s
m
(A)
(B) (C)
−10 −5 0 5 10
0
0.1
0.2
m
m + s
m − s
Probability
Range
−10 −5 0 5 10
0
0.1
0.2
Range
m
m + s
m − s
Figure 2.1 Gaussian random variable, x, with mean µx = 2 and standard deviation σx = 3.
(A) One realization of the random process: x(t). (B) The ideal probability distribution of the
sequence x. (C) An estimate of the PDF obtained from the realization shown in A.
2.2.1 Products of Gaussian Variables
The probability density, f (x) defined in equation (2.1), of a Gaussian random variable is
completely specified by its first two moments; all higher-order moments can be derived
from them. Furthermore, filtering a Gaussian signal with a linear system produces an out-
put that is also Gaussian (Bendat and Piersol, 1986; Papoulis, 1984). Consequently, only
the first two moments are required for linear systems analysis. However, the situation is
more complex for nonlinear systems since the response to a Gaussian input is not Gaus-
sian; rather it is the sum of products of one or more Gaussian signals. Consequently, the
expected value of the product of n zero-mean, Gaussian random variables, E[x1x2 . . . xn],
is an important higher-order statistic that will be used frequently in subsequent chapters.
The expected value of the product of an odd number of zero-mean Gaussian signals
will be zero. However, the result is more complex for the product of an even number of
Gaussian variables. The expected value may be obtained as follows (Bendat and Piersol,
1986):
1. Form every distinct pair of random variables and compute the expected value of
each pair. For example, when n is 4, the expected values would be
E[x1x2], E[x1x3], E[x1x4], E[x2x3], E[x2x4], E[x3x4]
2. Form all possible distinct combinations, each involving n/2 of these pairs, such
that each variable is included exactly once in each combination. For n = 4, there](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-31-320.jpg)
![16 BACKGROUND
are three combinations
(x1x2)(x3x4), (x1x3)(x2x4), (x1x4)(x2x3)
3. For each combination, compute the product of the expected values of each pair,
determined from step 1. Sum the results of all combinations to get the expected
value of the overall product. Thus, for n = 4, the combinations are
E[x1x2x3x4] = E[x1x2]E[x3x4] + E[x1x3]E[x2x4] + E[x1x4]E[x2x3] (2.2)
Similarly, when n is 6:
E[x1x2x3x4x5x6] = E[x1x2]E[x3x4]E[x5x6] + E[x1x2]E[x3x5]E[x4x6] + · · ·
For the special case where all signals are identically distributed, this yields the relation
E[x1 · x2 · . . . · xn] = (1 · 3 · 5 · . . . · n − 1)E[xixj ]n/2
=
n!
2n/2(n/2)!
E[xixj ]n/2
i = j (2.3)
2.3 CORRELATION FUNCTIONS
Correlation functions describe the sequential structures of signals. In signal analysis, they
can be used to detect repeated patterns within a signal. In systems analysis, they are used
to analyze relationships between signals, often a system’s input and output.
2.3.1 Autocorrelation Functions
The autocorrelation function characterizes the sequential structure of a signal, x(t), by
describing its relation to a copy of itself shifted by τ time units. The correlation will be
maximal at τ = 0 and change as the lag, τ, is increased. The change in correlation as a
function of lag characterizes the sequential structure of the signal.
Three alternative correlation functions are used commonly: the autocorrelation func-
tion, the autocovariance function, and the autocorrelation-coefficient function.
To illustrate the relationships between these functions, consider a random signal, x(t),
and let x0(t) be a zero-mean, unit variance random variable derived from x(t),
x0(t) =
x(t) − µx
σx
(2.4)
The autocorrelation function of the signal, x(t), is defined by
φxx(τ) = E[x(t − τ)x(t)] (2.5)
Figure 2.2 illustrates time records of several typical signals together with their autocorre-
lations. Evidently, the autocorrelations reveal structures not apparent in the time records
of the signals.](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-32-320.jpg)
![CORRELATION FUNCTIONS 17
−10
0
10
(A) (B)
0
1
−4
0
4
(C) (D)
0
1
0 5
−10
0
10
Time (s)
(G) (H)
−1 0 1
0
1
Lag (s)
−1
0
1
(E) (F)
−1
0
1
Figure 2.2 Time signals (left column) and their autocorrelation coefficient functions (right col-
umn). (A, B) Low-pass filtered white-noise signal. (C, D) Low-pass filtered white noise with a
lower cutoff. The resulting signal is smoother and the autocorrelation peak is wider. (E, F) Sine
wave. The autocorrelation function is also a sinusoid. (G, H) Sine wave buried in white noise. The
sine wave is more visible in the autocorrelation than in the time record.
Substituting equation (2.4) into equation (2.5) gives
φxx(τ) = E[(σxx0(t − τ) + µx)(σxx0(t) + µx)] (2.6)
= σ2
x E[x0(t − τ)x0(t)] + µ2
x (2.7)](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-33-320.jpg)
![18 BACKGROUND
Thus, the autocorrelation, φxx, at any lag, τ, depends on both the mean, µx, and the
variance, σ2
x , of the signal.
In many applications, particularly where systems have been linearized about an oper-
ating point, signals will not have zero means. It is common practice in these cases
to remove the mean before calculating the correlation, resulting in the autocovariance
function:
Cxx(τ) = E[(x(t − τ) − µx)(x(t) − µx)]
= φxx(τ) − µ2
x (2.8)
Thus, if µx = 0, the autocorrelation and autocovariance functions will be identical.
At zero lag, the value of the autocovariance function is
Cxx(0) = E[(x(t) − µx)2
]
= σ2
x
which is the signal’s variance. Dividing the autocovariance by the variance gives the
autocorrelation coefficient function,
rxx(τ) =
Cxx(τ)
Cxx(0)
= E[x0(t − τ)x0(t)] (2.9)
The values of the autocorrelation coefficient function may be interpreted as correla-
tions in the statistical sense. Thus the autocorrelation coefficient function ranges from 1
(i.e., complete positive correlation) to 0 (i.e., no correlation), through −1 (i.e., complete
negative correlation).
It is not uncommon, though it can be very confusing, for the autocovariance function
and the autocorrelation coefficient function to be referred to as the autocorrelation func-
tion. The relationships between the different autocorrelation functions are illustrated in
Figure 2.3.
Finally, note that all the autocorrelation formulations are even and thus are symmetric
about zero lag:
φxx(−τ) = φxx(τ)
Cxx(−τ) = Cxx(τ) (2.10)
rxx(−τ) = rxx(τ)
2.3.2 Cross-Correlation Functions
Cross-correlation functions measure the sequential relation between two signals. It is
important to remember that two signals may each have considerable sequential structure
yet have no correlation with each other.
The cross-correlation function between two signals x(t) and y(t) is defined by
φxy(τ) = E[x(t − τ)y(t)] (2.11)](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-34-320.jpg)
![CORRELATION FUNCTIONS 19
−4
0
4
Signal
0
1
Correlation
0
1
Covariance
0
1
Coefficient
−4
0
4
0
5
0
1
0
1
−40
0
40
0
100
0
100
0
1
0 1
−400
0
400
Time (s)
−10 10
0
10 K
Lag (ms)
−10 10
0
10 K
Lag (ms)
−10 10
0
1
Lag (ms)
Figure 2.3 Examples of autocorrelation functions. The first column shows the four time signals,
the second column shows their autocorrelation functions, the third column shows the corresponding
autocovariance functions, and the fourth column the equivalent autocorrelation coefficient functions.
First Row: Low-pass filtered sample of white, Gaussian noise with zero mean and unit variance.
Second Row: The signal from the first row with an offset of 2 added. Note that the mean is clearly
visible in the autocorrelation function but not in the auto-covariance or autocorrelation coefficient
function. Third Row: The signal from the top row multiplied by 10. Bottom Row: The signal from
the top row multiplied by 100. Scaling the signal changes the values of the autocorrelation and
autocovariance function but not of the autocorrelation coefficient function.
As before, removing the means of both signals prior to the computation gives the
cross-covariance function,
Cxy(τ) = E[(x(t − τ) − µx)(y(t) − µy)]
= φxy(τ) − µxµy (2.12)
where µx is the mean of x(t), and µy is the mean of y(t). Notice that if either µx = 0
or µx = 0, the cross-correlation and the cross-covariance functions will be identical.
The cross-correlation coefficient function of two signals, x(t) and y(t), is defined by
rxy(τ) =
Cxy(τ)
Cxx(0)Cyy(0)
(2.13)
The value of the cross-correlation coefficient function at zero lag, rxy(0), will be unity
only if the two signals are identical to within a scale factor (i.e., x(t) = ky(t)). In this](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-35-320.jpg)
![CORRELATION FUNCTIONS 21
In contrast, for the cross-correlation function
φwz(τ) = E[w(t)z(t + τ)]
= E
(x(t − τ) + n(t − τ))(y(t) + v(t))
= φxy(τ) + φxv(τ) + φny(τ) + φnv(τ)
and φxv ≡ φny ≡ φnv ≡ 0, since the noise signals are independent of each other by
assumption. Thus,
φwz(τ) = φxy(τ)
and estimates of the cross-correlation with additive noise will be unbiased.
2.3.4 Estimates of Correlation Functions
Provided that x(t) and y(t) are realizations of ergodic processes,∗ the expected value in
equation (2.11) may be replaced with an infinite time-average:
φxy(τ) = lim
T →∞
1
2T
T
−T
x(t − τ)y(t) dt (2.16)
In any practical application, x(t) and y(t) will be finite-length, discrete-time signals.
Thus, it can be assumed that x(t) and y(t) have been sampled every t units from
t = 0, t , . . . , (N − 1)t , giving the samples x(i) and y(i) for i = 1, 2, . . . , N.
By using rectangular integration, the cross-correlation function (2.16) can be approxi-
mated as
φ̂xy(τ) =
1
N − τ
N
i=τ
x(i − τ)y(i) (2.17)
This is an unbiased estimator, but its variance increases with lag τ. To avoid this, it is
common to use the estimator:
φ̂xy(τ) =
1
N
N
i=τ
x(i − τ)y(i) (2.18)
which is biased, because it underestimates correlation function values at long lags, but
its variance does not increase with lag τ.
Similar estimators of the auto- and cross-covariance and correlation coefficient func-
tions may be constructed. Note that if N is large with respect to the maximum lag, the
biased and unbiased estimates will be very similar.
∗Strictly speaking, a deterministic signal, such as a sinusoid, is nonstationary and is certainly not ergodic.
Nevertheless, computations based on time averages are routinely employed with both stochastic and determin-
istic signals. Ljung (1999) defines a class of quasi-stationary signals, together with an alternate expected value
operator, to get around this technicality.](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-37-320.jpg)
![MEAN-SQUARE PARAMETER ESTIMATION 25
the cross-covariance function is maximal. For example, the delay between two signals
measured at two points along a nerve can be determined from the cross-covariance
function (Heetderks and Williams, 1975). It should be remembered that dynamics can
also give rise to delayed peaks in the cross-correlation function.
2.3.7 Higher-Order Correlation Functions
Second-order cross-correlation functions will be represented by φ with three subscripts.
Thus,
φxxy(τ1, τ2) = E[x(t − τ1)x(t − τ2)y(t)] (2.27)
while the second-order autocorrelation function is
φxxx(τ1, τ2) = E[x(t − τ1)x(t − τ2)x(t)] (2.28)
Note that there is some confusion in the literature about the terminology for this function.
Some authors (Korenberg, 1988) use the nomenclature “second-order,” as does this book;
others have used the term “third-order” (Marmarelis and Marmarelis, 1978) to describe
the same relation.
2.4 MEAN-SQUARE PARAMETER ESTIMATION
Given a model structure and a set of input–output measurements, it is often desirable
to identify the parameter vector that generates the output that “best approximates” the
measured output. Consider a model, M, with a parameter set, θ, whose response to the
input u(t) will be written
ŷ(θ, t) = M(θ, u(t))
A common definition for the “best approximation” is that which minimizes the mean-
square error between the measured output, z(t), and the model output, ŷ(θ, t):
MSE(M, θ, u(t)) = E
z(t) − ŷ(θ, t)
2
(2.29)
If the signals are ergodic, then this expectation can be evaluated using a time average
over a single record. In discrete time, this results in the summation:
VN (M, θ, u(t)) =
1
N
N
t=1
z(t) − ŷ(θ, t)
2
(2.30)
which is often referred to as the “mean-square error,” even though it is computed using
a time average rather than an ensemble average.
Note that the MSE depends on the model structure, M, the parameter vector, θ, and
the test input, u(t). For a particular structure, the goal is to find the parameter vector, θ̂,
that minimizes (2.30). That is (Ljung, 1999):
θ̂ = arg min
θ
VN (θ, u(t)) (2.31)](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-41-320.jpg)
![MEAN-SQUARE PARAMETER ESTIMATION 27
where v is a zero-mean, white Gaussian noise sequence. First, construct the regression
matrix
U =
1 u1 u2
1 u3
1
1 u2 u2
2 u3
2
.
.
.
.
.
.
.
.
.
.
.
.
1 uN u2
N u3
N
(2.34)
Then, rewrite the expression for z as the matrix equation
z = Uθ + v (2.35)
where θ =
c(0) c(1) c(2) c(3)
T
, and use equations (2.33) to solve for θ.
2.4.2 Properties of Estimates
The solution of equations (2.33) gives the model parameters that provide the minimum
mean-square error solution. It is important to know how close to the true parameters
these estimated values are likely to be. Consider the following conditions:
1. The model structure is correct; that is, there is a parameter vector such that the
system can be represented exactly as y = Uθ.
2. The output, z = y + v, contains additive noise, v(t), that is zero-mean and statisti-
cally independent of the input, u(t).
If conditions 1 and 2 hold, then the expected value of the estimated parameter vector is
E[θ̂] = E[(UT
U)−1
UT
z]
= θ + E[(UT
U)−1
UT
v] (2.36)
However, since v is independent of u, it will be independent of all the columns of U
and so
E[θ̂] = θ (2.37)
That is, the least-squares parameter estimate is unbiased (Beck and Arnold, 1977).
The covariance matrix for these parameter estimates is
Cθ̂
= E[(θ̂ − E[θ̂])(θ̂ − E[θ̂])T
] (2.38)
= (UT
U)−1
UT
E[vvT
]U(UT
U)−1
(2.39)
Usually, equation (2.39) cannot be evaluated directly since E[vvT ], the covariance
matrix of the measurement noise, is not known. However, if the measurement noise is
white, then the noise covariance in equation (2.39) reduces to E[vvT ] = σ2
v IN, where
σ2
v is the variance of v(t), and IN is an N ×N identity matrix. Furthermore, an unbiased
estimate of the noise variance (Beck and Arnold, 1977) is given by
σ̂2
v =
1
N − M
N
t=1
(z(t) − ŷ(t))2
(2.40)](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-43-320.jpg)
![28 BACKGROUND
where M is the number of model parameters. Thus the covariance matrix for the param-
eter estimates, Cθ̂
, reduces to
Cθ̂
= σ̂2
v (UT
U)−1
(2.41)
which can be evaluated directly.
2.4.2.1 Condition of the Hessian Instead of using probabilistic considerations,
as in the previous section, it is also instructive to examine the numerical properties of
the estimate. Thus, instead of taking an expected value over a hypothetical ensemble of
records, consider the sensitivity of the computed estimate, θ̂, to changes in the single,
finite-length, input–output record, [u(t) z(t)].
From the normal equations (2.33), it is evident that the error in the parameter esti-
mate is
θ̃ = θ − θ̂
= θ − (UT
U)−1
UT
z
= −(UT
U)−1
UT
v (2.42)
where v(t) is the noise in the measured output, z = Uθ + v.
The error is the product of two terms. The first term, (UT U)−1, is the inverse of the
Hessian, denoted H, an M × M matrix containing the second-order derivatives of the
cost function with respect to the parameters.
H(i, j) =
∂2VN (θ)
∂θi∂θj
= UT
U (2.43)
Furthermore, if the measurement noise is white, the inverse of the Hessian is proportional
to the parameter covariance matrix, Cθ̂
, given in equation (2.41).
Let ν = UT v be the second term in equation (2.42). Substituting these two expres-
sions gives
θ̃ = −H−1
ν
Now, consider the effect of a small change in ν on the parameter estimate. The Hessian
is a non-negative definite matrix, so its singular value decomposition (SVD) (Golub and
Van Loan, 1989) can be written as
H = VSVT
where V = [v1 v2 . . . vM] is an orthogonal matrix, VT V = I, and S is a diagonal matrix,
S = diag[s1, s2 . . . , sM], where s1 ≥ s2 ≥ · · · ≥ sM ≥ 0. Using this, the Hessian can be
expanded as
H =
M
i=1
sivivi
T](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-44-320.jpg)
![30 BACKGROUND
2. The columns of U will not be orthogonal. This is most easily seen by examining
the Hessian, UT U, which will have the form
H = N
1 E[u] E[u2] . . . E[uq]
E[u] E[u2] E[u3] . . . E[uq+1]
E[u2] E[u3] E[u4] . . . E[uq+2]
.
.
.
.
.
.
.
.
.
...
.
.
.
E[uq] E[uq+1] E[uq+2] . . . E[u2q]
Since H is not diagonal, the columns of U will not be orthogonal to each other.
Note that the singular values of U can be viewed as the lengths of the semiaxes of
a hyperellipsoid defined by the columns of U. Thus, nonorthogonal columns will
stretch this ellipse in directions more nearly parallel to multiple columns and will
shrink it in other directions, increasing the ratio of the axis lengths, and hence the
condition number of the estimation problem (Golub and Van Loan, 1989).
2.5.2 Orthogonal Polynomials
Ideally, the regressors should be mutually orthogonal, so the Hessian will be diagonal
with elements of similar size. This can be achieved by replacing the power series basis
functions with another polynomial function P(q)(u)
m(u) =
Q
q=0
c(q)
P(q)
(u) (2.45)
chosen to make the estimation problem well-conditioned. The objective is to find a
polynomial function that makes the Hessian diagonal. That is, the (i, j)th element of the
Hessian should be
H(i, j) = N · E[P(i+1)
(u)P(j+1)
(u)]
= δi,j
which demonstrates that the terms of P(q)(u) must be orthonormal.
The expected value of a function, g, of a random variable, u, with probability density
f (u), is given by (Bendat and Piersol, 1986; Papoulis, 1984)
E[g(u)] =
∞
−∞
f (u)g(u) du
Consequently, the expected values of the elements of the Hessian will be
E[P(i)
(u)P(j)
(u)] =
∞
−∞
f (u)P(i)
(u)P(j)
(u) du
This demonstrates that a particular polynomial basis function, P(q)(u), will be orthogonal
only for a particular input probability distribution. Thus each polynomial family will
be orthogonal for a particular input distribution. Figure 2.5 shows the basis functions
corresponding to three families of polynomials: the ordinary power series, as well as the
Hermite and Tchebyshev families of orthogonal polynomials to be discussed next.](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-46-320.jpg)
![POLYNOMIALS 31
0
1
2
Power
0
1
2
Hermite
−1
0
1
Tcheb
10
0
10
−5
0
5
−1
0
1
0
50
100
−10
0
10
−1
0
1
−1 k
0
1 k
−20
0
20
−1
0
1
0
5 k
10 k
−50
0
50
−1
0
1
−10 0 10
−100 k
0
100 k
−3 0 3
−50
0
50
−1 0 1
−1
0
1
Order
0
Order
1
Order
2
Order
3
Order
4
Order
5
Figure 2.5 Power series, Hermite and Tchebyshev polynomials of orders 0 through 5. Left Col-
umn: Power series polynomials over the arbitrarily chosen domain [−10 10]. Middle Column:
Hermite polynomials over the domain [−3 3] corresponding to most of the range of the unit-
variance, normal random variable. Right Column: Tchebyshev polynomials over their full domain
[−1 1].
2.5.3 Hermite Polynomials
The Hermite polynomials H(q)(u) result when the orthogonalization is done for inputs
with a zero-mean, unit-variance, Gaussian distribution. Thus, Hermite polynomials are
constructed such that
∞
−∞
exp(−u2
)H(i)
(u)H(j)
(u) du = 0 for i = j (2.46)](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-47-320.jpg)
![32 BACKGROUND
Using equation (2.46) and the results for the expected value of products of Gaussian
variables (2.3), the Hermite polynomials can be shown to be
H(n)
(u) = n!
n/2
m=0
(−1)m
m!2m(n − 2m)!
u(n−2m)
(2.47)
The first four elements are
H(0)
(u) = 1
H(1)
(u) = u
H(2)
(u) = u2
− 1
H(3)
(u) = u3
− 3u
The Hermite polynomials may also be generated using the recurrence relation,
H(k+1)
(u) = uH(k)
(u) − kH(k−1)
(u) (2.48)
Note that these polynomials are only orthogonal for zero-mean, unit variance, Gaus-
sian inputs. Consequently, input data are usually transformed to zero mean and unit
variance before fitting Hermite polynomials. The transformation is retained as part of the
polynomial representation and used to transform any other inputs that may be applied to
the polynomial.
2.5.4 Tchebyshev Polynomials
A Gaussian random variable can take on any value between −∞ and +∞ (although the
probability of attaining the extreme values is negligible). Real data, on the other hand,
always have a finite range since they are limited by the dynamic range of the recording
apparatus, if nothing else. Thus, it is logical to develop a set of polynomials that are
orthogonal over a finite range.
The Tchebyshev polynomials, T , are orthogonalized over the range [−1 1] for the
probability distribution (1 − u2)−1/2.
1
−1
T (i)(u)T (j)(u)
√
1 − u2
du =
π
2 δi,j for i = 0, j = 0
π for i = j = 0
(2.49)
This probability density tends to infinity at ±1, and thus does not correspond to the PDF
of any realistic data set. Thus, the Tchebyshev polynomials will not be exactly orthogonal
for any data. However, all Tchebyshev polynomial basis functions are bounded between
−1 and +1 for inputs between −1 and +1 (see Figure 2.5). In addition, each goes through
all of its local extrema (which are all ±1) in this range. Thus, although the regressors
will not be exactly orthogonal, they will have similar variances, and so the estimation
problem should remain well-scaled. Hence, Tchebyshev polynomials are used frequently
since they usually lead to well-conditioned regressions.
The general expression for the order-q Tchebyshev polynomial is
T (q)
(u) =
q
2
q/2
m=0
(−1)m
q − m
q − m
m
(2u)q−2m
(2.50)](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-48-320.jpg)
![POLYNOMIALS 33
The first four Tchebyshev polynomials are
T (0)
(u) = 1
T (1)
(u) = u
T (2)
(u) = 2u2
− 1
T (3)
(u) = 4u3
− 3u
The Tchebyshev polynomials are also given by the recursive relationship
T (q+1)
(u) = 2uT (q)
(u) − T (q−1)
(u) (2.51)
In practice, input data are transformed to [−1 1] prior to fitting the coefficients, and
the scale factor is retained as part of the polynomial representation.
2.5.5 Multiple-Variable Polynomials
Polynomials in two or more variables will also be needed to describe nonlinear systems.
To establish the notation for this section, consider the types of terms involved in a
multivariable power series. The order-zero term will be a constant, as in the single
variable case. Similarly, the first-order terms will include only a single input, raised to
the first power, and thus will have the same form as their single-input counterparts,
M(1)
(uk) = uk
However, the second-order terms will involve products of two inputs, either two copies
of the same signal or two distinct signals. Thus, the second-order terms will be of two
types,
M(2)
(uk, uk) = u2
k
M(2)
(uj , uk) = uj uk
For example, the second-order terms in a three-input, second-order polynomial will
involve three single-input terms, u2
1, u2
2, and u2
3, and three two-input terms, u1u2, u1u3,
and u2u3. To remain consistent with the notation for single-variable polynomials, let
M(2)
(uk) = M(2)
(uk, uk) = u2
k
Similarly, the order-q terms will involve from one to q inputs, raised to powers such
that the sum of their exponents is q. For example, the third-order terms in a three-input
polynomial are
u3
1 u3
2 u3
3 u2
1u2 u2
1u3
u2
2u1 u2
2u3 u2
3u1 u2
3u2 u1u2u3
2.5.5.1 Orthogonal Multiple-Input Polynomials Next, consider the construction
of orthogonal, multiple-input polynomials. Let u1, u2, . . . , un be mutually orthonormal,
zero-mean Gaussian random variables. These may be measured signals that just happen
to be orthonormal, but will more likely result from orthogonalizing and normalizing a
set of more general (Gaussian) signals with a QR factorization or SVD.](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-49-320.jpg)
![34 BACKGROUND
For Gaussian inputs, it is reasonable to start with Hermite polynomials. The order-
zero term will be a constant, since it is independent of the inputs. Similarly, since the
first-order terms include only one input, they must be equivalent to their single-input
counterparts:
H(1)
(uk) = uk (2.52)
Next, consider the first type of second-order term, involving a second-degree function
of a single input. These terms will have the form
H(2)
(uk, uk) = H(2)
(uk)
= u2
k − 1 (2.53)
and will be orthogonal to order-zero terms since
E[H(2)
(uk, uk)H(0)
(uj )] = E[(u2
k − 1) · 1]
and E[u2
k − 1] = 0. They will also be orthogonal to all first-order terms, since
E[H(2)
(uk, uk)H(1)
(uj )] = E[(u2
k − 1)uj ]
= E[u2
kuj ] − E[uj ]
Both terms in this expression are products of odd numbers of zero-mean Gaussian random
variables, and so their expected values will be zero. Furthermore, any two second-order,
single-input terms will be orthogonal. To see why, note that
E[H(2)
(uk, uk)H(2)
(uj , uj )] = E[(u2
k − 1)(u2
j − 1)]
= E[u2
ku2
j ] − E[u2
j ] − E[u2
k] + 1
The inputs are orthonormal, by assumption, and so E[u2
k] = E[u2
j ] = 1. Furthermore,
from equation (2.2) we obtain
E[u2
ku2
j ] = E[u2
k]E[u2
j ] + 2E[ukuj ]2
= 1
and so H(2)(uk, uk) and H(2)(uj , uj ) will be orthogonal (for j = k).
Now, consider second-order terms involving different inputs. The corresponding Her-
mite polynomial is simply the product of corresponding first-order terms,
H(2)
(uj , uk) = H(1)
(uj )H(1)
(uk) j = k
= uj uk (2.54)
Since E[ukuj ] = 0, this term will be orthogonal to the constant, order-zero term. It
will also be orthogonal to any odd-order term because the resulting expectations will
involve an odd number of Gaussian random variables. It remains to show that this term
is orthogonal to the other second-order Hermite polynomial terms.
First consider the two-input terms,
E[H(2)
(ui, uj )H(2)
(uk, ul)] = E[uiuj ukul]
= E[uiuj ]E[ukul] + E[uiuk]E[uj ul]
+ E[uiul]E[uj uk]](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-50-320.jpg)
![NOTES AND REFERENCES 35
Provided that we have neither i = k, j = l nor i = l, j = k (in which case the terms
would be identical), each pair of expectations will contain at least one orthogonal pair
of signals, and therefore equal zero. Thus, any distinct pair of two-input second-order
terms will be orthogonal to each other.
Next consider the orthogonality of the two-input term to a single-input, second-order
Hermite polynomial.
E[H(2)
(ui, uj )H(2)
(uk, uk)] = E[uiuj (u2
k − 1)]
= E[uiuj u2
k] − E[uiuj ]
= E[uiuj ]E[u2
k] − 2E[uiuk]E[uj uk] − E[uiuj ]
Since ui and uj are orthogonal, the first and third terms will be zero. Similarly, uk
is orthogonal to at least one of ui and uj , so the second term will also be zero. Thus,
equations (2.53) and (2.54) define two-input polynomials orthogonalized for orthonormal,
zero-mean Gaussian inputs.
Similarly, there will be three types of third-order terms corresponding to the number
of distinct inputs. The single-input terms are the same as the third-order, single-input
Hermite polynomial term,
H(3)
(uk, uk, uk) = H(3)
(uk)
= u3
k − 3uk (2.55)
Terms with distinct inputs are generated using products of lower-order single-input Her-
mite polynomials,
H(3)
(uj , uj , uk) = H(2)
(uj )H(1)
(uk) j = k
= (u2
j − 1)uk (2.56)
H(3)
(ui, uj , uk) = H(1)
(ui)H(1)
(uj )H(1)
(uk) i = j = k
= uiuj uk (2.57)
Higher-order terms are constructed in a similar fashion resulting in an orthogonal series
known as the Grad–Hermite (Barrett, 1963) polynomial series. The same approach may
be used to construct multiple-input polynomials based on the Tchebyshev polynomials.
2.6 NOTES AND REFERENCES
1. Bendat and Piersol (1986) and Papoulis (1984) provide more detailed discussions
of probability theory and first-order correlation functions.
2. More information concerning higher-order correlation functions can be found in
Marmarelis and Marmarelis (1978).
3. There are many texts dealing with linear regression. In particular, Beck and Arnold
(1977) and Seber (1977) are recommended. Discussions more relevant to system
identification can be found in Ljung (1999) and Söderström and Stoica (1989).](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-51-320.jpg)
![36 BACKGROUND
4. Beckmann (1973) contains a detailed discussion of several orthogonal polynomial
systems, including both the Tchebyshev and Hermite polynomials. The classic
reference for multiple-input, orthogonal polynomials is Barrett (1963).
2.7 PROBLEMS
1. Let x and y be zero-mean Gaussian random variables with variances σ2
x and σ2
y , and
covariance E[xy] = σ2
xy. Evaluate
E[(xy)2
] E[x3
y] E[x3
y2
]
2. Let x(t) and n(t) be mutually independent zero-mean Gaussian signals with variances
σ2
x and σ2
n , respectively. Let y(t) = 3x(t − τ) + n(t). What is the cross-correlation
between x(t) and y(t)? What is the cross-correlation between x(t) and z(t) = x3(t −
τ) + n(t)?
3. Show that if x is a zero-mean unit variance Gaussian random variable, then the Hessian
associated with fitting Hermite polynomials is given by
H(i, j) = i!δi,j
where δi,j is a Kronecker delta.
4. Prove the recurrence relation (2.51).
5. Prove the recurrence relation (2.48).
2.8 COMPUTER EXERCISES
1. Use the MATLAB NLID toolbox to create a RANDV object describing a zero-mean
Gaussian random variable with a standard deviation of 0.5. Use it to generate a
NLDAT object containing 1000 points with this distribution. Suppose that this signal
is an unrectified EMG, measured in millivolts, and was sampled at 1000 Hz. Set the
domain and range of your NLDAT object accordingly.
Plot the NLDAT object (signal) as a function of time, and check that the axes are labeled
properly. Use PDF to construct a histogram for this signal. Construct the theoretical
probability density, using PDF, and your RANDV object. Vary the number of bins used to
compute the histogram, and compare the result with the theoretical distribution. How
does the histogram change if you use 10,000 data points, or 100?
How many of the 1000 points in your original signal are greater than 1 (i.e., 2σ).
How many are less than −1?
2. Use the RANDV object from question 1 to generate an NLDAT object containing white
Gaussian noise. Use equation (2.3), and the variance of the original signal, to predict
the expected value of the fourth power of this signal. Check the validity of this
prediction by raising the signal to the fourth power and computing its mean. Try
changing the length of the signal. How many points are necessary before equation (2.3)](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-52-320.jpg)
![down the street. This man who was calling for volunteers, says he,
The guns are spiked, we will all go; and they all went off again.
That is my knowledge of the mob.
By Mr. Lindsey:
Q. What stores of ammunition were here at that time that the mob
could have got if they had entered?
A. We have here many buildings full of ordinance stores. We had for
years, and have yet, something like thirty-six or forty thousand stand
of arms. Don't put these down as the exact figures. We had a great
many thousand stand of arms, and two magazines full of powder
and ammunition, prepared and partially prepared for service; that is,
the powder in the shells, the powder in the cartridges, two large
magazines full. We have there, in fact, two of them full, and another
partially so in the upper park. Besides these arms I speak of, we
have many thousand stands of arms, revolvers, carbines, muskets,
and all sorts of things. We have many large warehouses here. There
is one there, [indicating,] and here is one, [indicating,] and one on
the other side of the street; above that are the magazines. We have
got a great deal of property here, valuable property, too, but we had
no small arm ammunition except some of the old style ammunition—
a lot of the old style paper cartridges which I had broken up. The
arms we had are mostly loaders, except fifty breech-loading
muskets, and my men here are armed with caliber fifty. A year
before the riot began I was impressed with the dangerous position
of this place, and I drew the attention of the authorities in
Washington to it. There is a map showing the arsenal, [indicating.]
That is Butler street. There, you see, are four buildings called
temporary magazines. Those are wooden buildings. There are a
great mass of breech-loading ammunition in there, partially prepared
for service. There is one magazine, and there is the other one. There
is Penn avenue—it is called a pike there. A man might have thrown a
lighted cigar over and set fire to this place. I drew the attention of](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-70-320.jpg)
![my chief to it, and called particular attention to this dangerous place.
He saw the importance of it, and ordered me to break up the
ammunition and otherwise get rid of it. Fortunately, all that was
cleared out before the riot began. These magazines were all full, and
the small arm ammunition I had broken up. Here the shops are
below the work-shops, on a plateau just below this, and here is the
road over which you came. Here is a sort of open space, and
nothing but a low wall here with a picket. Right opposite, there is
another gate leading into the upper park. My men were here, and
this part is utterly defenseless, and in that place were a number of
cannon. The mob would have cleaned me out here. There is not a
man there, but a man in charge of the magazine, and twenty men,
you see, would be a small force to defend it. It is not a fortified
place, it is the same as houses surrounded by a wall with a wooden
picket fence. The mob could push it over and come in, and there
would be no trouble about it.
Q. Not a very strong fortification?
A. It is utterly defenseless; but, at the same time, I was not afraid
the mob would do me any injury.
Q. How many cannon had you that they could have taken and
moved off?
A. I don't know how many are in that shed. I have five or six pieces
which I call in current service. Those pieces are mounted. Those are
six-pounder guns, and there is plenty of ammunition which could be
used for that purpose in those magazines. I had one of them on this
side, [indicating,] and one on the other side. As mobs generally do,
they always come where the danger is. Here was those six-pounder
guns, with canister. The only hostile demonstration they made was
to rush for the gate, but I merely raised my hands, and says, that
won't do, and they stopped instantly. To show what the state of
affairs was here, and my information of what was going on,
sometime on Saturday night I received this communication from](https://image.slidesharecdn.com/1024296-250522180930-9f37be0e/85/Identification-Of-Nonlinear-Physiological-Systems-Westwick-D-71-320.jpg)
Identification Of Nonlinear Physiological Systems Westwick D
- 1. Identification Of Nonlinear Physiological Systems Westwick D download https://ebookbell.com/product/identification-of-nonlinear- physiological-systems-westwick-d-2048592 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Identification Of Nonlinear Systems Using Neural Networks And Polynomial Models A Blockoriented Approach 1st Edition Andrzej Janczak https://ebookbell.com/product/identification-of-nonlinear-systems- using-neural-networks-and-polynomial-models-a-blockoriented- approach-1st-edition-andrzej-janczak-2159368 Parameter Identification And Monitoring Of Mechanical Systems Under Nonlinear Vibration 1st Edition Jc Jauregui https://ebookbell.com/product/parameter-identification-and-monitoring- of-mechanical-systems-under-nonlinear-vibration-1st-edition-jc- jauregui-4977682 Identification Of Ancient Olive Oil Processing Methods Based On Olive Remains Peter Warnock https://ebookbell.com/product/identification-of-ancient-olive-oil- processing-methods-based-on-olive-remains-peter-warnock-49990180 Identification Of Problembased Learning Components And Their Effects On Graduates Competencies 1st Edition Bhina Patria https://ebookbell.com/product/identification-of-problembased-learning- components-and-their-effects-on-graduates-competencies-1st-edition- bhina-patria-50558510
- 3. Identification Of Novel Modulators Towards High Cell Density And Highproducing Chinese Hamster Ovary Suspension Cell Cultures As Well As Their Application In Biopharmaceutical Protein Production 1st Edition Beat Thalmann https://ebookbell.com/product/identification-of-novel-modulators- towards-high-cell-density-and-highproducing-chinese-hamster-ovary- suspension-cell-cultures-as-well-as-their-application-in- biopharmaceutical-protein-production-1st-edition-beat- thalmann-51672066 Identification Of Trees And Shrubs In Winter Using Buds And Twigs Translated Bernd Schulz https://ebookbell.com/product/identification-of-trees-and-shrubs-in- winter-using-buds-and-twigs-translated-bernd-schulz-58258670 Identification Of Cleaner Production Improvement Opportunities 1st Edition Kenneth L Mulholland https://ebookbell.com/product/identification-of-cleaner-production- improvement-opportunities-1st-edition-kenneth-l-mulholland-2170894 Identification Of Pathological Conditions In Human Skeletal Remains Second Edition 2nd Edition Donald J Ortner https://ebookbell.com/product/identification-of-pathological- conditions-in-human-skeletal-remains-second-edition-2nd-edition- donald-j-ortner-2198476 Identification Of Textile Fibers Woodhead Publishing Series In Textiles Max M Houck https://ebookbell.com/product/identification-of-textile-fibers- woodhead-publishing-series-in-textiles-max-m-houck-2367268
- 7. IEEE Press Series in Biomedical Engineering The focus of our series is to introduce current and emerging technologies to biomedical and electrical engineer- ing practitioners, researchers, and students. This series seeks to foster interdisciplinary biomedical engineering education to satisfy the needs of the industrial and academic areas. This requires an innovative approach that overcomes the difficulties associated with the traditional textbook and edited collections. Metin Akay, Series Editor Dartmouth College Advisory Board Thomas Budinger Simon Haykin Richard Robb Ingrid Daubechies Murat Kunt Richard Satava Andrew Daubenspeck Paul Lauterbur Malvin Teich Murray Eden Larry McIntire Herbert Voigt James Greenleaf Robert Plonsey Lotfi Zadeh Editorial Board Eric W. Abel Gabor Herman Kris Ropella Dan Adam Helene Hoffman Joseph Rosen Peter Adlassing Donna Hudson Christian Roux Berj Bardakjian Yasemin Kahya Janet Rutledge Erol Basar Michael Khoo Wim L. C. Rutten Katarzyna Blinowska Yongmin Kim Alan Sahakian Bernadette Bouchon-Meunier Andrew Laine Paul S. Schenker Tom Brotherton Rosa Lancini G. W. Schmid-Schönbein Eugene Bruce Swamy Laxminarayan Ernest Stokely Jean-Louis Coatrieux Richard Leahy Ahmed Tewfik Sergio Cerutti Zhi-Pei Liang Nitish Thakor Maurice Cohen Jennifer Linderman Michael Unser John Collier Richard Magin Eugene Veklerov Steve Cowin Jaakko Malmivuo Al Wald Jerry Daniels Jorge Monzon Bruce Wheeler Jaques Duchene Michael Neuman Mark Wiederhold Walter Greenleaf Banu Onaral William Williams Daniel Hammer Keith Paulsen Andy Yagle Dennis Healy Peter Richardson Yuan-Ting Zhang A list of books in the IEEE Press Series in Biomedical Engineering can be found on page 262.
- 8. IDENTIFICATION OF NONLINEAR PHYSIOLOGICAL SYSTEMS DAVID T. WESTWICK ROBERT E. KEARNEY A JOHN WILEY & SONS, INC., PUBLICATION
- 9. IEEE Press 445 Hoes Lane Piscataway, NJ 08854 IEEE Press Editorial Board Stamatios V. Kartalopoulos, Editor in Chief M. Ajay R. J. Herrick M. S. Newman J .B. Anderson R. F. Hoyt M. Padgett J. Baker D. Kirk W. D. Reeve J. E. Brewer R. Leonardi S. Tewksbury M. E. El-Hawary G. Zobrist Kenneth Moore, Director of IEEE Press Catherine Faduska, Senior Acquisitions Editor Christina Kuhnen, Associate Acquisitions Editor IEEE Engineering in Medicine and Biology Society, Sponsor EMB-S Liaison to IEEE Press, Metin Akay Technical Reviewers Metin Akay Robert F. Kirsch John A. Daubenspeck Copyright c 2003 by the Institute of Electrical and Electronics Engineers. All rights reserved. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax 978-750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, e-mail: permreq@wiley.com. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services please contact our Customer Care Department within the U.S. at 877-762-2974, outside the U.S. at 317-572-3993 or fax 317-572-4002. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print, however, may not be available in electronic format. Library of Congress Cataloging-in-Publication Data: Westwick, D. T. (David T.) Identification of nonlinear physiological systems / D.T. Westwick, R.E. Kearney. p. cm.—(IEEE Press series on biomedical engineering) “IEEE Engineering in Medicine and Biology Society, Sponsor.” Includes bibliographical references and index. ISBN 0-471-27456-9 (cloth) 1. Physiology—Mathematical models. 2. Nonlinear systems. I. Kearney, Robert E., 1947- II. IEEE Engineering in Medicine and Biology Society. III. Title. IV. IEEE Press series in biomedical engineering. QP33.6.M36W475 2003 612.015118—dc21 2003043255 Printed in the United States of America 10 9 8 7 6 5 4 3 2 1
- 10. CONTENTS Preface xi 1 Introduction 1 1.1 Signals / 1 1.1.1 Domain and Range / 2 1.1.2 Deterministic and Stochastic Signals / 2 1.1.3 Stationary and Ergodic Signals / 3 1.2 Systems and Models / 3 1.2.1 Model Structure and Parameters / 4 1.2.2 Static and Dynamic Systems / 5 1.2.3 Linear and Nonlinear Systems / 6 1.2.4 Time-Invariant and Time-Varying Systems / 7 1.2.5 Deterministic and Stochastic Systems / 7 1.3 System Modeling / 8 1.4 System Identification / 8 1.4.1 Types of System Identification Problems / 9 1.4.2 Applications of System Identification / 11 1.5 How Common are Nonlinear Systems? / 11 2 Background 13 2.1 Vectors and Matrices / 13 2.2 Gaussian Random Variables / 14 2.2.1 Products of Gaussian Variables / 15 2.3 Correlation Functions / 16 v
- 11. vi CONTENTS 2.3.1 Autocorrelation Functions / 16 2.3.2 Cross-Correlation Functions / 18 2.3.3 Effects of Noise / 20 2.3.4 Estimates of Correlation Functions / 21 2.3.5 Frequency Domain Expressions / 22 2.3.6 Applications / 23 2.3.7 Higher-Order Correlation Functions / 25 2.4 Mean-Square Parameter Estimation / 25 2.4.1 Linear Least-Squares Regression / 26 2.4.2 Properties of Estimates / 27 2.5 Polynomials / 29 2.5.1 Power Series / 29 2.5.2 Orthogonal Polynomials / 30 2.5.3 Hermite Polynomials / 31 2.5.4 Tchebyshev Polynomials / 32 2.5.5 Multiple-Variable Polynomials / 33 2.6 Notes and References / 35 2.7 Problems / 36 2.8 Computer Exercises / 36 3 Models of Linear Systems 39 3.1 Linear Systems / 39 3.2 Nonparametric Models / 40 3.2.1 Time Domain Models / 41 3.2.2 Frequency Domain Models / 43 3.3 Parametric Models / 46 3.3.1 Parametric Frequency Domain Models / 46 3.3.2 Discrete-Time Parametric Models / 48 3.4 State-Space Models / 52 3.4.1 Example: Human Ankle Compliance—Discrete-Time, State-Space Model / 54 3.5 Notes and References / 54 3.6 Theoretical Problems / 55 3.7 Computer Exercises / 56 4 Models of Nonlinear Systems 57 4.1 The Volterra Series / 57 4.1.1 The Finite Volterra Series / 59 4.1.2 Multiple-Input Systems / 62 4.1.3 Polynomial Representation / 64 4.1.4 Convergence Issues(†) / 65
- 12. CONTENTS vii 4.2 The Wiener Series / 67 4.2.1 Orthogonal Expansion of the Volterra Series / 68 4.2.2 Relation Between the Volterra and Wiener Series / 70 4.2.3 Example: Peripheral Auditory Model—Wiener Kernels / 71 4.2.4 Nonwhite Inputs / 73 4.3 Simple Block Structures / 73 4.3.1 The Wiener Model / 73 4.3.2 The Hammerstein Model / 77 4.3.3 Sandwich or Wiener–Hammerstein Models / 79 4.3.4 NLN Cascades / 83 4.3.5 Multiple-Input Multiple-Output Block Structured Models / 87 4.4 Parallel Cascades / 87 4.4.1 Approximation Issues(†) / 89 4.5 The Wiener–Bose Model / 91 4.5.1 Similarity Transformations and Uniqueness / 92 4.5.2 Approximation Issues(†) / 94 4.5.3 Volterra Kernels of the Wiener–Bose Model / 94 4.5.4 Wiener Kernels of the Wiener–Bose Model / 95 4.5.5 Relationship to the Parallel Cascade Model / 97 4.6 Notes and References / 100 4.7 Theoretical Problems / 100 4.8 Computer Exercises / 101 5 Identification of Linear Systems 103 5.1 Introduction / 103 5.1.1 Example: Identification of Human Joint Compliance / 103 5.1.2 Model Evaluation / 105 5.2 Nonparametric Time Domain Models / 107 5.2.1 Direct Estimation / 107 5.2.2 Least-Squares Regression / 108 5.2.3 Correlation-Based Methods / 109 5.3 Frequency Response Estimation / 115 5.3.1 Sinusoidal Frequency Response Testing / 115 5.3.2 Stochastic Frequency Response Testing / 116 5.3.3 Coherence Functions / 117 5.4 Parametric Methods / 119 5.4.1 Regression / 119 5.4.2 Instrumental Variables / 120 5.4.3 Nonlinear Optimization / 121 5.5 Notes and References / 122 5.6 Computer Exercises / 122
- 13. viii CONTENTS 6 Correlation-Based Methods 125 6.1 Methods for Functional Expansions / 125 6.1.1 Lee–Schetzen Cross-Correlation / 125 6.1.2 Colored Inputs / 140 6.1.3 Frequency Domain Approaches / 144 6.2 Block-Structured Models / 149 6.2.1 Wiener Systems / 150 6.2.2 Hammerstein Models / 155 6.2.3 LNL Systems / 162 6.3 Problems / 167 6.4 Computer Exercises / 167 7 Explicit Least-Squares Methods 169 7.1 Introduction / 169 7.2 The Orthogonal Algorithms / 169 7.2.1 The Orthogonal Algorithm / 171 7.2.2 The Fast Orthogonal Algorithm / 173 7.2.3 Variance of Kernel Estimates / 180 7.2.4 Example: Fast Orthogonal Algorithm Applied to Simulated Fly Retina Data / 182 7.2.5 Application: Dynamics of the Cockroach Tactile Spine / 186 7.3 Expansion Bases / 187 7.3.1 The Basis Expansion Algorithm / 190 7.3.2 The Laguerre Expansion / 191 7.3.3 Limits on α / 192 7.3.4 Choice of α and P / 194 7.3.5 The Laguerre Expansion Technique / 195 7.3.6 Computational Requirements / 195 7.3.7 Variance of Laguerre Kernel Estimates / 195 7.3.8 Example: Laguerre Expansion Kernels of the Fly Retina Model / 196 7.4 Principal Dynamic Modes / 198 7.4.1 Example: Principal Dynamic Modes of the Fly Retina Model / 200 7.4.2 Application: Cockroach Tactile Spine / 201 7.5 Problems / 205 7.6 Computer Exercises / 205 8 Iterative Least-Squares Methods 207 8.1 Optimization Methods / 207 8.1.1 Gradient Descent Methods / 208
- 14. CONTENTS ix 8.1.2 Identification of Block-Structured Models / 209 8.1.3 Second-Order Optimization Methods / 212 8.1.4 Jacobians for Other Block Structures / 216 8.1.5 Optimization Methods for Parallel Cascade Models / 219 8.1.6 Example: Using a Separable Volterra Network / 220 8.2 Parallel Cascade Methods / 223 8.2.1 Parameterization Issues / 226 8.2.2 Testing Paths for Significance / 228 8.2.3 Choosing the Linear Elements / 230 8.2.4 Parallel Wiener Cascade Algorithm / 242 8.2.5 Longer Cascades / 242 8.2.6 Example: Parallel Cascade Identification / 243 8.3 Application: Visual Processing in the Light-Adapted Fly Retina / 246 8.4 Problems / 249 8.5 Computer Exercises / 250 References 251 Index 259 IEEE Press Series in Biomedical Engineering 262
- 15. PREFACE Since it first appeared in 1978, Advanced Methods in Physiological Modeling: The White Noise Approach by P. Z. Marmarelis and M. Z. Marmarelis has been the standard ref- erence for the field of nonlinear system identification, especially as applied in biomed- ical engineering and physiology. Despite being long out of print, Marmarelis and Mar- marelis is still, in many cases, the primary reference. Over the years, dramatic advances have been made in the field, many of which became practical only with the advent of widespread computing power. Many of these newer developments have been described in the three volumes of the series Advanced Methods in Physiological Modeling, edited by V. Z. Marmarelis. While these volumes have been an invaluable resource to many researchers, helping them to stay abreast of recent developments, they are all collections of research articles. As a resource for someone starting out in the field, they are some- what lacking. It is difficult for a newcomer to the field to see the relationships between myriad contributions. Choosing which approach is best for a given application can be an arduous task, at best. This textbook developed out of a review article (Westwick and Kearney, 1998) on the same subject. The goal of the review article was to bring the various analyses that have been developed by several groups of researchers into a common notation and framework, and thus to elucidate the relationships between them. The aim of this book was to go one step farther and to provide this common framework along with the background necessary to bring the next generation of systems physiologists into the fold. In this book, we have attempted to provide the student with an overview of many of the techniques currently in use, and some of the earlier methods as well. Everything is presented in a common notation and from a consistent theoretical framework. We hope that the relationships between the methods and their relative strengths and weaknesses will become apparent to the reader. The reader should be well-equipped to make an informed decision as to which techniques to try, when faced with an identification or modeling problem. xi
- 16. xii PREFACE We have assumed that readers of this book have a background in linear signals and systems equivalent to that given by a junior year signals and systems course. Back- ground material beyond that level is summarized, with references given to more detailed, pedagogical treatments. Each chapter has several theoretical problems, which can be solved with pencil and paper. In addition, most of the chapters conclude with some computer exercises. These are intended to give the reader practical experience with the tools described in the text. These computer exercises make use of MATLAB∗ and the nonlinear system identifica- tion (NLID) toolbox (Kearney and Westwick, 2003). More information regarding the NLID toolbox can be found at www.bmed.mcgill.ca. In addition to implementing all of the system identification tools as MATLAB m-files, the toolbox also contains the data and model structures used to generate the examples that run throughout the text. Although our primary goal is to educate informed users of these techniques, we have included several theoretical sections dealing with issues such as the generality of some model structures, convergence of series-based models, and so on. These sections are marked with a dagger, †, and they can be skipped by readers interested primarily in practical application of these methods, with little loss in continuity. The dedication in Marmarelis and Marmarelis reads “To an ambitious breed: Systems Physiologists.” We feel that the sentiment reflected in those words is as true today as it was a quarter century ago. The computers are (much) faster, and they will undoubtedly be faster still in a few years. As a result, the problems that we routinely deal with today would have been inconceivable when M M was first published. However, with increased computational abilities come more challenging problems. No doubt, this trend will continue. We hope that it is an interesting ride. DAVID T. WESTWICK ROBERT E. KEARNEY Calgary, Alberta, Canada Montreal, Quebec, Canada May, 2003 ∗MATLAB is a registered trademark of the MathWorks, Inc.
- 17. CHAPTER 1 INTRODUCTION The term “Biomedical Engineering” can refer to any endeavor in which techniques from engineering disciplines are used to solve problems in the life sciences. One such under- taking is the construction of mathematical models of physiological systems and their subsequent analysis. Ideally the insights gained from analyzing these models will lead to a better understanding of the physiological systems they represent. System identification is a discipline that originated in control engineering; it deals with the construction of mathematical models of dynamic systems using measurements of their inputs and outputs. In control engineering, system identification is used to build a model of the process to be controlled; the process model is then used to construct a controller. In biomedical engineering, the goal is more often to construct a model that is detailed enough to provide insight into how the system operates. This text deals with system identification methods that are commonly used in biomedical engineering. Since many physiological systems are highly nonlinear, the text will focus on methods for nonlinear systems and their application to physiological systems. This chapter will introduce the concepts of signals, systems, system modeling, and identification. It also provides a brief overview of the system identification problem and introduces some of the notation and terminology to be used in the book. The reader should be acquainted with most of the material covered in this chapter. If not, pedagogical treatments can be found in most undergraduate level signals and systems texts, such as that by Kamen (1990). 1.1 SIGNALS The concept of a signal seems intuitively clear. Examples would include speech, a televi- sion picture, an electrocardiogram, the price of the NASDAQ index, and so on. However, formulating a concise, mathematical description of what constitutes a signal is somewhat involved. Identification of Nonlinear Physiological Systems, By David T. Westwick and Robert E. Kearney ISBN 0-471-27456-9 c 2003 Institute of Electrical and Electronics Engineers 1
- 18. 2 INTRODUCTION 1.1.1 Domain and Range In the examples above, two sets of values were required to describe each “signal”; these will be termed the domain and range variables of the signal. Simply put, a signal may be viewed as a function that assigns a value in the range set for each value in the domain set; that is, it represents a mapping from the domain to the range. For example, with speech, the domain is time while the range could be one of a variety of variables: the air pressure near the speaker’s mouth, the deflection of the listener’s ear drum, or perhaps the voltage produced by a microphone. This concept can be defined formally by describing a signal, s(t), as a mapping from a domain set, T , which is usually time, to a range set, Y. Thus, s : T → Y where t ∈ T is a member of the domain set, usually time. In continuous time, T is the real line; in discrete time, it is the set of integers. In either case, the value of the signal is in the range set, Y. The range of the signal is given by applying the mapping to the domain set, and is therefore s(T ). The above definition really describes a function. A key point regarding the domain set of a signal is the notion that it is ordered and thus has a direction. Thus, if x1 and x2 are members of the domain set, there is some way of stating x1 x2, or the reverse. If time is the domain, t1 t2 is usually taken to mean that t1 is later than t2. The analysis in this book will focus on signals with one-dimensional domains—usually time. However, most of the ideas can be extended to signals with domains having two dimensions (e.g., X-ray images), three dimensions (e.g., MRI images), or more (e.g., time-varying EEG signals throughout the brain). 1.1.2 Deterministic and Stochastic Signals A signal is deterministic if its future values can be generated based on a set of known rules and parameters, perhaps expressed as a mathematical equation. For example, the sinusoid yd(t) = cos(2πf t + φ) can be predicted exactly, provided that its frequency f and phase φ are known. In contrast, if yr(k) is generated by repeatedly tossing a fair, six-sided die, there is no way to predict the kth value of the output, even if all other output values are known. These represent two extreme cases: yd(t) is purely deterministic while yr(k) is completely random, or stochastic. The die throwing example is an experiment where each repetition of the experiment produces a single random variable: the value of the die throw. On the other hand, for a stochastic process the result of each experiment will be a signal whose value at each time is a random variable. Just as a single throw of a die produces a single realization of a random variable, a random signal is a single realization of a stochastic process. Each experiment produces a different time signal or realization of the process. Conceptually, the stochastic process is the ensemble of all possible realizations. In reality, most signals fall between these two extremes. Often, a signal may be deterministic but there may not be enough information to predict it. In these cases, it
- 19. SYSTEMS AND MODELS 3 may be necessary to treat the deterministic signal as if it were a single realization of some underlying stochastic process. 1.1.3 Stationary and Ergodic Signals The statistical properties of a random variable, such as its mean and variance, are deter- mined by integrating the probability distribution function (PDF) over all possible range values. Thus, if f (x) is the PDF of a random variable x, its mean and variance are given by µx = ∞ −∞ xf (x) dx σ2 x = ∞ −∞ (x − µx)2 f (x) dx Similar integrals are used to compute higher-order moments. Conceptually, these integrals can be viewed as averages taken over an infinite ensemble of all possible realizations of the random variable, x. The value of a random signal at a point in time, considered as a random variable, will have a PDF, f (x, t), that depends on the time, t. Thus, any statistic obtained by integrating over the PDF will be a function of time. Alternately, the integrals used to compute the statistics can be viewed as averages taken over an infinite ensemble of realizations of the stochastic process, at a particular point in time. If the PDF, and hence statistics, of a stochastic process is independent of time, then the process is said to be stationary. For many practical applications, only a single realization of a stochastic process will be available; therefore, averaging must be done over time rather than over an ensemble of realizations. Thus, the mean of a stochastic process would be estimated as µ̂x = 1 T T 0 x(t) dt Many stochastic process are ergodic, meaning that the ensemble and time averages are equal. 1.2 SYSTEMS AND MODELS Figure 1.1 shows a block diagram of a system in which the “black box,” N, transforms the input signal, u(t), into the output y(t). This will be written as y(t) = N(u(t)) (1.1) to indicate that when the input u(t) is applied to the system N, the output y(t) results. Note that the domain of the signals need not be time, as shown here. For example, if the system operates on images, the input and output domains could be two- or three-dimensional spatial coordinates. This book will focus mainly on single-input single-output (SISO) systems whose domain is time. Thus u(t) and y(t) will be single-valued functions of t. For multiple-input
- 20. 4 INTRODUCTION Input(s) u(t) N Output(s) y(t) Figure 1.1 Block diagram of a “black box” system, which transforms the input(s) u(t), into the output(s), y(t). The mathematical description of the transformation is represented by the operator N. multiple-output (MIMO) systems, Figure 1.1, equation (1.1), and most of the develop- ment to follow will not change; the input and output simply become vector-valued func- tions of their domains. For example, a multidimensional input signal may be written as a time-dependent vector, u(t) = u1(t) u2(t) . . . un(t) (1.2) 1.2.1 Model Structure and Parameters Using M to indicate a mathematical model of the physical system, N, the model output can be written as ŷ(t) = M(u(t)) (1.3) where the caret, or “hat,” indicates that ŷ(t) is an estimate of the system output, y(t). In general, a model will depend on a set of parameter parameters contained in the parameter vector θ. For example, if the model, M(θ), was a third-degree polynomial, ŷ(θ, t) = M(θ, u(t)) = c(0) + c(1) u(t) + c(2) u2 (t) + c(3) u3 (t) (1.4) the parameter vector, θ, would contain the polynomial coefficients, θ = c(0) c(1) c(2) c(3) T Note that in equation (1.4) the dependence of the output, ŷ(θ, t), on the parameter vector, θ, is shown explicitly. Models are often classified as being either parametric or nonparametric. A parametric model generally has relatively few parameters that often have direct physical interpreta- tions. The polynomial in equation (1.4) is an example of a parametric model. The model structure comprises the constant, linear, quadratic and third-degree terms; the parameters are the coefficients associated with each term. Thus each parameter is related to a par- ticular behavior of the system; for example, the parameter c(2) defines how the output varies with the square of the input. In contrast, a nonparametric model is described by a curve or surface defined by its values at a collection of points in its domain, as illustrated in Figure 1.2. Thus, a set of samples of the curve defined by equation (1.4) would be a nonparametric model of the same system. Here, the model structure would contain the domain values, and the “parameters” would be the corresponding range values. Thus, a nonparametric model usually has a large number of parameters that do not in themselves have any direct physical interpretation.
- 21. SYSTEMS AND MODELS 5 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 −4 −2 0 2 4 Input: u(t) Output: y(t) = N(u(t)) Figure 1.2 A memoryless nonlinear system. A parametric model of this system is y(t) = −3 − u(t) + u2(t) − 0.5u3(t). A nonparametric model of the same system could include a list of some of the domain and range values, say those indicated by the dots. The entire curve is also a nonparametric model of the system. While the parametric model is more compact, the nonparametric model is more flexible. 1.2.2 Static and Dynamic Systems In a static, or memoryless, system, the current value of the output depends only on the current value of the input. For example, a full-wave rectifier is a static system since its output, y(t) = |u(t)|, depends only on the instantaneous value of its input, u(t). On the other hand, in a dynamic system, the output depends on some or all of the input history. For example, the output at time t of the delay operator, y(t) = u(t − τ) depends only on the value of the input at the previous time, t − τ. In contrast, the output of the peak-hold operation y(t) = max τ≤t (u(τ)) retains the largest value of the past input and consequently depends on the entire history of the input. Dynamic systems can be further classified according to whether they respond to the past or future values of the input, or both. The delay and peak-hold operators are both examples of causal systems, systems whose outputs depend on previous, but not future, values of their inputs. Systems whose outputs depend only on future values of their inputs are said to be anti-causal or anticipative. If the output depends on both the past and future inputs, the system said to be noncausal or mixed causal anti-causal. Although physical systems are causal, there are a number of situations where noncausal system descriptions are needed. For example, behavioral systems may display a predictive ability if the input signal is deterministic or a preview is available. For example, the dynamics of a tracking experiment may show a noncausal component if the subject is permitted to see future values of the input as well as its current value. Sometimes, feedback can produce behavior that appears to be noncausal. Consider the system in Figure 1.3. Suppose that the experimenter can measure the signals labeled u(t) and y(t), but not w1(t) and w2(t). Let both N1 and N2 be causal systems that include delays. The effect of w1(t) will be measured first in the “input,” u(t), and then later in the
- 22. 6 INTRODUCTION N1 N2 w1(t) w2(t) u(t) y(t) Figure 1.3 A feedback loop with two inputs. Depending on the relative power of the inputs w1(t) and w2(t), the system N1, or rather the relationship between u(t) and y(t), may appear to be either causal, anti-causal, or noncausal. “output,” y(t). However, the effect of the other input, w2(t), will be noted in y(t) first, followed by u(t). Thus, the delays in the feedback loop create what appears to be non- causal system behavior. Of course the response is not really noncausal, it merely appears so because neither u(t) nor y(t) was directly controlled. Thus, inadequate experimental design can lead to the appearance of noncausal relationships between signals. In addition, as will be seen below, there are cases where it is advantageous to reverse the roles of the input and output. In the resulting analysis, a noncausal system description must be used to describe the inverse system. 1.2.3 Linear and Nonlinear Systems Consider a system, N, and let y(t) be the response of the system due to the input u(t). Thus, y(t) = N(u(t)) Let c be a constant scalar. Then if the response to the input c · u(t) satisfies N(c · u(t)) = c · y(t) (1.5) for any constant c, the system is said to obey the principle of proportionality or to have the scaling property. Consider two pairs of inputs and their corresponding outputs, y1(t) = N(u1(t)) y2(t) = N(u2(t)) If the response to the input u1(t) + u2(t) is given by N(u1(t) + u2(t)) = y1(t) + y2(t) (1.6) then the operator N is said to obey the superposition property. Systems that obey both superposition and scaling are said to be linear. Nonlinear systems do not obey superposition and scaling. In many cases, a system will obey the superposition and scaling properties approximately, provided that the inputs
- 23. SYSTEMS AND MODELS 7 lie within a restricted class. In such cases, the system is said to be operating within its “linear range.” 1.2.4 Time-Invariant and Time-Varying Systems If the relationship between the input and output does not depend on the absolute time, then the system is said to be time-invariant. Thus, if y(t) is the response to the input u(t) generated by a time-invariant system, its response due to u(t − τ), for any real τ, will be y(t − τ). Thus, a time-invariant system must satisfy N(u(t)) = y(t) ⇒ N(u(t − τ)) = y(t − τ) ∀τ ∈ IR (1.7) Systems for which equation (1.7) does not hold are said to be time-varying. 1.2.5 Deterministic and Stochastic Systems In a deterministic system, the output, y(t), depends only on the input, u(t). In many applications, the output measurement is corrupted by additive noise, z(t) = y(t) + v(t) = N(u(t)) + v(t) (1.8) where v(t) is independent of the input, u(t). Although the measured output, z(t), has both deterministic and random components, the system (1.8) is still referred to as deterministic, since the “true” output, y(t), is a deterministic function of the input. Alternatively, the output may depend on an unmeasurable process disturbance, w(t), y(t) = N(u(t), w(t)) (1.9) where w(t) is a white, Gaussian signal that cannot be measured. In this case, the system is said to be stochastic, since there is no “noise free” deterministic output. The process noise term, w(t), can be thought of as an additional input driving the dynamics of the system. Measurement noise, in contrast, only appears additively in the final output. Clearly, it is possible for a system to have both a process disturbance and measurement noise, as illustrated in Figure 1.4, leading to the relation z(t) = y(t) + v(t) = N(u(t), w(t)) + v(t) (1.10) u(t) N w(t) y(t) z(t) v(t) Figure 1.4 Block diagram of a system including a process disturbance, w(t), and measurement noise, v(t).
- 24. 8 INTRODUCTION 1.3 SYSTEM MODELING In many cases, a mathematical model of a system can be constructed from “first princi- ples.” Consider, for example, the problem of modeling a spring. As a first approxima- tion, it might be assumed to obey Hooke’s law and have no mass so that it could be described by y = −ku (1.11) where the output, y, is the force produced, the input, u, is the displacement, and k is the spring constant. If the spring constant were known, then equation (1.11) would constitute a mathematical model of the system. If the spring constant, k, was unknown, it could be estimated experimentally. Whether or not the assumptions hold, equation (1.11) is a model of the system (but not necessarily a good model). If it yields satisfactory predictions of the system’s behavior, then, and only then, can it be considered to be a good model. If it does not predict well, then the model must be refined, perhaps by considering the mass of the spring and using Newton’s second law to give y(t) = −ku(t) + m d2u(t) dt2 (1.12) Other possibilities abound; the spring might be damped, behave nonlinearly, or have significant friction. The art of system modeling lies in determining which terms are likely to be significant, and in limiting the model to relevant terms only. Thus, even in this simple case, constructing a mathematical model based on “first principles” can become unwieldy. For complex systems, the approach can become totally unmanageable unless there is a good understanding of which effects should and should not be incorporated into the model. 1.4 SYSTEM IDENTIFICATION The system identification approach to constructing a mathematical model of the system is much different. It assumes a general form, or structure, for the mathematical model and then determines the parameters from experimental data. Often, a variety of model structures are evaluated, and the most successful ones are retained. For example, consider the spring system described in the previous section. If it were assumed to be linear, then a linear differential equation model, such as dny(t) dtn + an−1 dn−1y(t) dtn−1 + · · · + a1 dy(t) dt + a0y(t) = bm dmu(t) dtm + bm−1 dm−1u(t) dtm−1 + · · · + b1 du(t) dt + b0u(t) (1.13) could be postulated. It would then be necessary to perform an experiment, record u(t) and y(t), compute their derivatives, and determine the coefficients a0 . . . an−1 and b0 . . . bm. Under ideal conditions, many of the coefficients would be near zero and could be removed from the model. Thus, if the system could be described as a massless linear spring, then equation (1.13) would reduce to equation (1.11) once all extraneous terms were removed.
- 25. SYSTEM IDENTIFICATION 9 The scheme outlined in the previous paragraph is impractical for a number of reasons. Most importantly, numerical differentiation amplifies high-frequency noise. Thus, the numerically computed derivatives of the input and output, particularly the high-order derivatives, will be dominated by high-frequency noise that will distort the parameter estimates. Thus, a more practical approach to estimating the system dynamics from input–output measurements is required. First, note that a system need not be represented as a differential equation. There are many possible parametric and nonparametric representations or model structures for both linear and nonlinear systems. Parameters for many of these model structures can be estimated reliably from measured data. In general, the model structure will be represented by an operator, M, having some general mathematical form capable of representing a wide variety of systems. The model itself will depend on a list of parameters, the vector θ. From this viewpoint, the system output may be written as y(t, θ) = M(θ, u(t)) (1.14) where it is assumed that the model structure, M, and parameter vector, θ, exactly rep- resent the physical system. Thus, the physical system, N, can be replaced with an exact model, M(θ). The objective of system identification is to find a suitable model structure, M, and corresponding parameter vector, θ, given measurements of the input and output. Then, the identified model will have a parameter vector, θ̂, and generate ŷ(t) = M(θ̂, u(t)) (1.15) where ŷ(t) is an estimate of the system output, y(t). Similarly, M(θ̂, u(t)) represents the model structure chosen together with a vector of estimated parameters. The system iden- tification problem is then to choose the model structure, M, and find the corresponding parameter vector, θ̂, that produces the model output, given by equation (1.15), that best predicts the measured system output. Often, instead of having the system output, y(t), only a noise corrupted measurement will be available. Usually, this measurement noise is assumed to be additive, random, and statistically independent of the system’s inputs and outputs. The goal, then, is to find the model, M(θ̂, u(t)), whose output, ŷ(t, θ̂), “best approximates” the measured output, z(t). The relationship between the system, model, and the various signals, is depicted in Figure 1.5. 1.4.1 Types of System Identification Problems Figure 1.6 gives a more complete view of a typical system identification experiment. First, the desired test input, labeled µ(t), is applied to an actuator. In some applications, such as the study of biomechanics, the actuator dynamics may restrict the types of test input which can be applied. In addition, the actuator may be influenced by the noise term, n1(t). Thus, instead of using the desired input, µ(t), in the identification, it is desirable to measure the actuator output, u(t), and use it as the input instead. Note, however, that measurements of the input, û(t), may contain noise, n2(t). Many system identification methods are based, either directly or indirectly, on solving an ordinary least-squares problem. Such formulations are well suited to dealing with
- 26. 10 INTRODUCTION u(t) N y(t) z(t) v(t) M(θ) ŷ(t, θ) Figure 1.5 The deterministic system identification problem in the presence of measurement noise. µ(t) Actuator dynamics n1(t) u(t) û(t) n2(t) N w(t) y(t) z(t) v(t) Figure 1.6 A more realistic view of the system being identified, including the actuator, which transforms the ideal input, µ, into the applied input, u(t), which may contain the effects of the process noise term, n1(t). Furthermore, the measured input, û(t), may contain noise, n2(t). As before, the plant may be affected by process noise, w(t), and the output may contain additive noise, v(t). noise in the output signals. However, to deal with noise at the input, it is necessary to adopt a “total least-squares” or “errors in the variables” framework, both of which are much more computationally demanding. To avoid this added complexity, identification experiments are usually designed to minimize the noise in the input measurements. In some cases, it may be necessary to adopt a noncausal system description so that the measurement with the least noise may be treated as the input. Throughout this book it will be assumed that n2(t) is negligible, unless otherwise specified. The system may also include an unmeasurable process noise input, w(t), and the measured output may also contain additive noise, v(t). Given this framework, there are three broad categories of system identification problem: • Deterministic System Identification Problem. Find the relationship between u(t) and y(t), assuming that the process noise, w(t), is zero. The measured output, z(t), may contain additive noise, v(t). The identification of deterministic systems is generally pursued with the objective of gaining insight into the system function and is the problem of primary interest in this text. • Stochastic System Identification Problem. Find the relationship between w(t) and y(t), given only the system output, z(t), and assumptions regarding the statistics of w(t). Usually, the exogenous input, u(t), is assumed to be zero or constant. This formulation is used where the inputs are not available to the experimenter, or
- 27. HOW COMMON ARE NONLINEAR SYSTEMS? 11 where it is not evident which signals are inputs and which are outputs. The myriad approaches to this problem have been reviewed by Brillinger (1975) and Caines (1988). • Complete System Identification Problem. Given both the input and the output, esti- mate both the stochastic and deterministic components of the model. This problem formulation is used when accurate output predictions are required, for example in model-based control systems (Ljung, 1999; Söderström and Stoica, 1989). 1.4.2 Applications of System Identification There are two general areas of application for models produced by system identification that will be referred to as “control” and “analysis.” In “control” applications, the identified model will be used to design a controller, or perhaps be embedded in a control system. Here, the chief requirements are that the model be compact and easy to manipulate, so that it produces output predictions with little computational overhead. Many control applications use the model “online” to pre- dict future outputs from the histories of both the input and output. Such predictions commonly extend only one time-step into the future. At each time-step the model uses the previous output measurement to correct its estimate of the model’s trajectory. Such one-step-ahead predictions are often all that is required of a control model. As a result, low-order, linear parametric models of the complete system (i.e., both stochastic and deterministic parts) are often adequate. Since the model’s output trajectory is corrected at each sample, the model need not be very accurate. The effects of missing dynam- ics, or weak nonlinearities, can usually be removed by modeling them as process noise. Similarly, more severe nonlinearities can handled using an adaptive, time-varying linear model. Here, the measured output is used to correct the model by varying its parameters on-line, to track gradual changes in the linearized model. In “analysis” applications the model is usually employed for off-line simulation of the system to gain insight into its functioning. For these applications, the model must be simulated as a free run—that is, without access to past output measurements. With no access to prediction errors, and hence no means to reconstruct process noise, the model must be entirely deterministic. Moreover, without the recursive corrections used in on-line models, an off-line model must be substantially more accurate than the on-line models typically used in control applications. Thus, in these applications it is critical for the nonlinearities to be described exactly. Moreover, since simulations are done off-line, there is less need to minimize the mathematical/arithmetic complexity of the model and consequently large, nonlinear models may be employed. 1.5 HOW COMMON ARE NONLINEAR SYSTEMS? Many physiological systems are highly nonlinear. Consider, for example, a single joint and its associated musculature. First, the neurons that transmit signals to and from the muscles fire with an “all or nothing” response. The geometry of the tendon insertions is such that lever arms change with joint angle. The muscle fibers themselves have nonlinear force–length and force–velocity properties as well as being only able exert force in one direction. Nevertheless, this complex system is often represented using a simple linear model.
- 28. 12 INTRODUCTION In many biomedical engineering applications, the objective of an identification experi- ment is to gain insight into the functioning of the system. Here, the nonlinearities may play a crucial role in the internal functioning of the system. While it may be possible to linearize the system about one or more operating points, linearization will discard important information about the nonlinearities. Thus, while a controller may perform adequately using a linearized model, the model would provide little insight into the functional organization of the system. Thus, in biomedical applications, it is both common and important to identify nonlinear systems explicitly. For these reasons, nonlinear system analysis techniques have been applied to a wide variety of biomedical systems. Some of these applications include: • Sensory Systems. These include primitive sensory organs such as the cockroach tactile spine (French and Korenberg, 1989, 1991; French and Marmarelis, 1995; French and Patrick, 1994; French et al., 2001), as well as more evolved sensors such as the auditory system (Carney and Friedman, 1996; Eggermont, 1993; Shi and Hecox, 1991) and the retina (Citron et al., 1988; Juusola et al., 1995; Korenberg and Paarmann, 1989; Naka et al., 1988; Sakuranaga et al., 1985a). • Reflex Loops. Nonlinear system identification techniques have been used to study reflex loops in the control of limb (Kearney and Hunter, 1988; Kearney et al., 1997; Westwick and Kearney, 2001; Zhang and Rymer, 1997) and eye position (the vestibulo-ocular reflex) (Galiana et al., 1995, 2001). • Organ Systems. Similarly, nonlinear feedback loops have been investigated in mod- els of heart rate variability (Chon et al., 1996) and in renal auto-regulation (Chon et al., 1998; Marmarelis et al., 1993, 1999). • Tissue Mechanics. Biological tissues themselves can exhibit nonlinearities. Strips of lung tissue (Maksym et al., 1998; Yuan et al., 1999) and the whole lung (Maksym and Bates, 1997; Zhang et al., 1999) have been shown to include nonlinearities. Skeletal muscle (Crago, 1992; Hunt et al., 1998; Munih et al., 2000) also has strongly nonlinear behavior. Given the prevalence of nonlinearities in physiological systems, along with the requirement in many biomedical engineering applications to deal explicitly with those nonlinearities, the need for nonlinear system identification methods is clear. Sample applications included in Chapters 6–8 will present results from some of the studies cited above.
- 29. CHAPTER 2 BACKGROUND This chapter will review a number of important mathematical results and establish the notation to be used throughout the book. Material is drawn from diverse areas, some of which are not well known and thus extensive references are provided with each section. 2.1 VECTORS AND MATRICES Many of the techniques presented in this text use numerical methods derived from linear algebra. This section presents a brief overview of some important results to be used in the chapters that follow. For a more thorough treatment, the reader should consult the canonical reference by Golub and Van Loan (1989). Vectors will be represented using lowercase, boldface letters. The same letter, in lightface type, will be used for the elements of the vector, subscripted with its position in the vector. Thus, an M element vector will be written as follows: θ = θ1 θ2 . . . θM T where the superscript T denotes transposition (i.e., θ is a column vector). Bold uppercase letters will denote matrices. Depending on the context, individual elements of a matrix will be referenced by placing the row and column number in parentheses or by using a lowercase, lightface letter with a double subscript. Thus, both R(i,j) and ri,j represent the element in the ith row and jth column of the matrix R. MATLAB’s convention will be used for referencing rows and columns of a matrix, so that R(:, j) will be the jth column of R. Two matrix decompositions will be used extensively: the QR factorization and the singular value decomposition (SVD). The QR factorization takes a matrix X (i.e., either Identification of Nonlinear Physiological Systems, By David T. Westwick and Robert E. Kearney ISBN 0-471-27456-9 c 2003 Institute of Electrical and Electronics Engineers 13
- 30. 14 BACKGROUND square or tall) and constructs X = QR where R is upper triangular, so that ri,j = 0 for i j, and Q is an orthogonal matrix, QTQ = I. Note that the columns of Q are said to be orthonormal (i.e., orthogonal and normalized); however, the matrix itself is said to be orthogonal. The singular value decomposition (SVD) takes a matrix, X, of any size and shape and replaces it with the product, X = USVT where S is a diagonal matrix with non-negative diagonal elements. By convention, the elements of S, the singular values, are arranged in decreasing order along the diagonal. Thus, s1,1 ≥ s2,2 ≥ · · · ≥ sn,n. U and V are orthogonal matrices containing the left and right singular vectors, respectively. Throughout the book, the approximate computational cost of algorithms will be given in flops. A flop, or floating point operation, represents either the addition or the mul- tiplication of two floating point numbers. Note that a flop was originally defined as a floating point multiplication optionally followed by a floating point addition. Thus, the first edition of Golub and Van Loan (1989) used this earlier definition, whereas subse- quent editions used the later definition. Indeed, they Golub and Van Loan (1989) noted that supercomputer manufacturers were delighted by this change in terminology, since it gave the appearance of an overnight doubling in performance. In any case, the pre- cise definition is not important to this text, so long as it is applied consistently; the flop count will be used only as a first-order measurement of the computation requirements of algorithms. 2.2 GAUSSIAN RANDOM VARIABLES Gaussian random variables, and signals derived from them, will play a central role in much of the development to follow. A Gaussian random variable has the probability density (Bendat and Piersol, 1986; Papoulis, 1984) f (x) = 1 √ 2πσ2 exp − (x − µ)2 2σ2 (2.1) that is completely defined by two parameters: the mean, µ, and variance, σ2. By con- vention, the mean, variance, and other statistical moments, are denoted by symbols sub- scripted by the signal they describe. Thus, µx = E[x] σ2 x = E (x − µx)2 Figure 2.1 shows a single realization of a Gaussian signal, the theoretical probability density function (PDF) of the process that generated the signal, and an estimate of the PDF derived from the single realization.
- 31. GAUSSIAN RANDOM VARIABLES 15 0 0.1 0.2 0.3 0.4 0.5 −5 0 5 10 Range Time (s) m + s m − s m (A) (B) (C) −10 −5 0 5 10 0 0.1 0.2 m m + s m − s Probability Range −10 −5 0 5 10 0 0.1 0.2 Range m m + s m − s Figure 2.1 Gaussian random variable, x, with mean µx = 2 and standard deviation σx = 3. (A) One realization of the random process: x(t). (B) The ideal probability distribution of the sequence x. (C) An estimate of the PDF obtained from the realization shown in A. 2.2.1 Products of Gaussian Variables The probability density, f (x) defined in equation (2.1), of a Gaussian random variable is completely specified by its first two moments; all higher-order moments can be derived from them. Furthermore, filtering a Gaussian signal with a linear system produces an out- put that is also Gaussian (Bendat and Piersol, 1986; Papoulis, 1984). Consequently, only the first two moments are required for linear systems analysis. However, the situation is more complex for nonlinear systems since the response to a Gaussian input is not Gaus- sian; rather it is the sum of products of one or more Gaussian signals. Consequently, the expected value of the product of n zero-mean, Gaussian random variables, E[x1x2 . . . xn], is an important higher-order statistic that will be used frequently in subsequent chapters. The expected value of the product of an odd number of zero-mean Gaussian signals will be zero. However, the result is more complex for the product of an even number of Gaussian variables. The expected value may be obtained as follows (Bendat and Piersol, 1986): 1. Form every distinct pair of random variables and compute the expected value of each pair. For example, when n is 4, the expected values would be E[x1x2], E[x1x3], E[x1x4], E[x2x3], E[x2x4], E[x3x4] 2. Form all possible distinct combinations, each involving n/2 of these pairs, such that each variable is included exactly once in each combination. For n = 4, there
- 32. 16 BACKGROUND are three combinations (x1x2)(x3x4), (x1x3)(x2x4), (x1x4)(x2x3) 3. For each combination, compute the product of the expected values of each pair, determined from step 1. Sum the results of all combinations to get the expected value of the overall product. Thus, for n = 4, the combinations are E[x1x2x3x4] = E[x1x2]E[x3x4] + E[x1x3]E[x2x4] + E[x1x4]E[x2x3] (2.2) Similarly, when n is 6: E[x1x2x3x4x5x6] = E[x1x2]E[x3x4]E[x5x6] + E[x1x2]E[x3x5]E[x4x6] + · · · For the special case where all signals are identically distributed, this yields the relation E[x1 · x2 · . . . · xn] = (1 · 3 · 5 · . . . · n − 1)E[xixj ]n/2 = n! 2n/2(n/2)! E[xixj ]n/2 i = j (2.3) 2.3 CORRELATION FUNCTIONS Correlation functions describe the sequential structures of signals. In signal analysis, they can be used to detect repeated patterns within a signal. In systems analysis, they are used to analyze relationships between signals, often a system’s input and output. 2.3.1 Autocorrelation Functions The autocorrelation function characterizes the sequential structure of a signal, x(t), by describing its relation to a copy of itself shifted by τ time units. The correlation will be maximal at τ = 0 and change as the lag, τ, is increased. The change in correlation as a function of lag characterizes the sequential structure of the signal. Three alternative correlation functions are used commonly: the autocorrelation func- tion, the autocovariance function, and the autocorrelation-coefficient function. To illustrate the relationships between these functions, consider a random signal, x(t), and let x0(t) be a zero-mean, unit variance random variable derived from x(t), x0(t) = x(t) − µx σx (2.4) The autocorrelation function of the signal, x(t), is defined by φxx(τ) = E[x(t − τ)x(t)] (2.5) Figure 2.2 illustrates time records of several typical signals together with their autocorre- lations. Evidently, the autocorrelations reveal structures not apparent in the time records of the signals.
- 33. CORRELATION FUNCTIONS 17 −10 0 10 (A) (B) 0 1 −4 0 4 (C) (D) 0 1 0 5 −10 0 10 Time (s) (G) (H) −1 0 1 0 1 Lag (s) −1 0 1 (E) (F) −1 0 1 Figure 2.2 Time signals (left column) and their autocorrelation coefficient functions (right col- umn). (A, B) Low-pass filtered white-noise signal. (C, D) Low-pass filtered white noise with a lower cutoff. The resulting signal is smoother and the autocorrelation peak is wider. (E, F) Sine wave. The autocorrelation function is also a sinusoid. (G, H) Sine wave buried in white noise. The sine wave is more visible in the autocorrelation than in the time record. Substituting equation (2.4) into equation (2.5) gives φxx(τ) = E[(σxx0(t − τ) + µx)(σxx0(t) + µx)] (2.6) = σ2 x E[x0(t − τ)x0(t)] + µ2 x (2.7)
- 34. 18 BACKGROUND Thus, the autocorrelation, φxx, at any lag, τ, depends on both the mean, µx, and the variance, σ2 x , of the signal. In many applications, particularly where systems have been linearized about an oper- ating point, signals will not have zero means. It is common practice in these cases to remove the mean before calculating the correlation, resulting in the autocovariance function: Cxx(τ) = E[(x(t − τ) − µx)(x(t) − µx)] = φxx(τ) − µ2 x (2.8) Thus, if µx = 0, the autocorrelation and autocovariance functions will be identical. At zero lag, the value of the autocovariance function is Cxx(0) = E[(x(t) − µx)2 ] = σ2 x which is the signal’s variance. Dividing the autocovariance by the variance gives the autocorrelation coefficient function, rxx(τ) = Cxx(τ) Cxx(0) = E[x0(t − τ)x0(t)] (2.9) The values of the autocorrelation coefficient function may be interpreted as correla- tions in the statistical sense. Thus the autocorrelation coefficient function ranges from 1 (i.e., complete positive correlation) to 0 (i.e., no correlation), through −1 (i.e., complete negative correlation). It is not uncommon, though it can be very confusing, for the autocovariance function and the autocorrelation coefficient function to be referred to as the autocorrelation func- tion. The relationships between the different autocorrelation functions are illustrated in Figure 2.3. Finally, note that all the autocorrelation formulations are even and thus are symmetric about zero lag: φxx(−τ) = φxx(τ) Cxx(−τ) = Cxx(τ) (2.10) rxx(−τ) = rxx(τ) 2.3.2 Cross-Correlation Functions Cross-correlation functions measure the sequential relation between two signals. It is important to remember that two signals may each have considerable sequential structure yet have no correlation with each other. The cross-correlation function between two signals x(t) and y(t) is defined by φxy(τ) = E[x(t − τ)y(t)] (2.11)
- 35. CORRELATION FUNCTIONS 19 −4 0 4 Signal 0 1 Correlation 0 1 Covariance 0 1 Coefficient −4 0 4 0 5 0 1 0 1 −40 0 40 0 100 0 100 0 1 0 1 −400 0 400 Time (s) −10 10 0 10 K Lag (ms) −10 10 0 10 K Lag (ms) −10 10 0 1 Lag (ms) Figure 2.3 Examples of autocorrelation functions. The first column shows the four time signals, the second column shows their autocorrelation functions, the third column shows the corresponding autocovariance functions, and the fourth column the equivalent autocorrelation coefficient functions. First Row: Low-pass filtered sample of white, Gaussian noise with zero mean and unit variance. Second Row: The signal from the first row with an offset of 2 added. Note that the mean is clearly visible in the autocorrelation function but not in the auto-covariance or autocorrelation coefficient function. Third Row: The signal from the top row multiplied by 10. Bottom Row: The signal from the top row multiplied by 100. Scaling the signal changes the values of the autocorrelation and autocovariance function but not of the autocorrelation coefficient function. As before, removing the means of both signals prior to the computation gives the cross-covariance function, Cxy(τ) = E[(x(t − τ) − µx)(y(t) − µy)] = φxy(τ) − µxµy (2.12) where µx is the mean of x(t), and µy is the mean of y(t). Notice that if either µx = 0 or µx = 0, the cross-correlation and the cross-covariance functions will be identical. The cross-correlation coefficient function of two signals, x(t) and y(t), is defined by rxy(τ) = Cxy(τ) Cxx(0)Cyy(0) (2.13) The value of the cross-correlation coefficient function at zero lag, rxy(0), will be unity only if the two signals are identical to within a scale factor (i.e., x(t) = ky(t)). In this
- 36. 20 BACKGROUND case, the cross-correlation coefficient function will be the same as the autocorrelation coefficient function of either signal. As with the autocorrelation coefficient function, the values of the cross-correlation coefficient function can be interpreted as correlations in the statistical sense, ranging from complete positive correlation (1) through 0 to complete negative correlation (−1). Furthermore, the same potential for confusion exists; the cross-covariance and cross- correlation coefficient functions are often referred to simply as cross-correlations. The various cross-correlation formulations are neither even nor odd, but do satisfy the interesting relations: φxy(τ) = φyx(−τ) (2.14) and φxy(τ) ≤ φxx(0)φyy(0) (2.15) Finally, consider the cross-correlation between x(t) and y(t) where y(t) = αx(t − τ0) + v(t) that is, y(t) is a delayed, scaled version of x(t) added to an uncorrelated noise signal, v(t). Then, φxy(τ) = αφxx(τ − τ0) That is, the cross-correlation function is simply the autocorrelation of the input signal, x(t), displaced by the delay τ0, and multiplied by the gain α. As a result, the lag at which the cross-correlation function reaches its maximum provides an estimate of the delay. 2.3.3 Effects of Noise Frequently, the auto- and cross-correlations of the signals x(t) and y(t) must be estimated from measurements containing additive noise. Let w(t) = x(t) + n(t) z(t) = y(t) + v(t) where n(t) and v(t) are independent of x(t) and y(t) and of each other. First, consider the autocorrelation of one signal, φzz(τ) = E (y(t − τ) + v(t − τ))(y(t) + v(t)) = φyy(τ) + φyv(τ) + φvy(τ) + φvv(τ) The terms φyv ≡ φvy ≡ 0 will disappear because y and v are independent. However, the remaining term, φvv, is the autocorrelation of the noise sequence and will not be zero. As a result, additive noise will bias autocorrelation estimates, φzz(τ) = φyy(τ) + φvv(τ)
- 37. CORRELATION FUNCTIONS 21 In contrast, for the cross-correlation function φwz(τ) = E[w(t)z(t + τ)] = E (x(t − τ) + n(t − τ))(y(t) + v(t)) = φxy(τ) + φxv(τ) + φny(τ) + φnv(τ) and φxv ≡ φny ≡ φnv ≡ 0, since the noise signals are independent of each other by assumption. Thus, φwz(τ) = φxy(τ) and estimates of the cross-correlation with additive noise will be unbiased. 2.3.4 Estimates of Correlation Functions Provided that x(t) and y(t) are realizations of ergodic processes,∗ the expected value in equation (2.11) may be replaced with an infinite time-average: φxy(τ) = lim T →∞ 1 2T T −T x(t − τ)y(t) dt (2.16) In any practical application, x(t) and y(t) will be finite-length, discrete-time signals. Thus, it can be assumed that x(t) and y(t) have been sampled every t units from t = 0, t , . . . , (N − 1)t , giving the samples x(i) and y(i) for i = 1, 2, . . . , N. By using rectangular integration, the cross-correlation function (2.16) can be approxi- mated as φ̂xy(τ) = 1 N − τ N i=τ x(i − τ)y(i) (2.17) This is an unbiased estimator, but its variance increases with lag τ. To avoid this, it is common to use the estimator: φ̂xy(τ) = 1 N N i=τ x(i − τ)y(i) (2.18) which is biased, because it underestimates correlation function values at long lags, but its variance does not increase with lag τ. Similar estimators of the auto- and cross-covariance and correlation coefficient func- tions may be constructed. Note that if N is large with respect to the maximum lag, the biased and unbiased estimates will be very similar. ∗Strictly speaking, a deterministic signal, such as a sinusoid, is nonstationary and is certainly not ergodic. Nevertheless, computations based on time averages are routinely employed with both stochastic and determin- istic signals. Ljung (1999) defines a class of quasi-stationary signals, together with an alternate expected value operator, to get around this technicality.
- 38. 22 BACKGROUND 2.3.5 Frequency Domain Expressions The discrete Fourier transform of a discrete-time signal of length N is defined as U(f ) = F(u(t)) = N t=1 u(t)e−2πjf t/N (2.19) Note that in this definition, f takes on integer values f = 0, 1, . . . , N − 1. If t is the sampling increment in the time domain, then the sampling increment in the frequency domain will be f = 1 Nt The inverse Fourier transform is given by∗, u(t) = F−1 (U(f )) = 1 N N f =1 U(f )e2πjf t/N (2.20) Direct computation of either equation (2.19) or equation (2.20) requires about 4N2 flops, since the computations involve complex numbers. However, if the fast Fourier transform (FFT) algorithm (Bendat and Piersol, 1986; Oppenheim and Schafer, 1989; Press et al., 1992) is used, the cost can be reduced to about N log2(N) flops (Oppenheim and Schafer, 1989). Note that these figures are for real valued signals. FFTs of complex- valued signals will require twice as many flops. The Fourier transform of the autocorrelation function is called the power spectrum. For a discrete-time signal, Suu(f ) = N−1 τ=0 φuu(τ)e−2πjf τ/N = E N−1 τ=0 u(t)u(t − τ)e−2πjf τ/N (2.21) This expression, like the definition of the autocorrelation, is in terms of an expected value. To estimate the power spectrum, u(t) is treated as if it were ergodic, and the expected value is replaced with a time average. There are several ways to do this. One possibility is to estimate the correlation in the time domain using a time average (2.18) and then Fourier transform the result: F φ̂uu(τ) = 1 N N τ=0 N i=0 u(i − τ)u(i)e−j2πτf/N Multiplying by e−j2π(i−i)f/N = 1 and then simplifying gives F φ̂uu(τ) = 1 N N τ=0 u(i − τ)ej2π(i−τ)f/N N i=0 u(i)e−j2πif/N = 1 N U∗ (f )U(f ) (2.22) ∗Some authors (Ljung, 1999) include a factor of 1/ √ N in both the forward and inverse Fourier transform.
- 39. CORRELATION FUNCTIONS 23 Taking the inverse Fourier transform of (2.22) yields (2.18), the biased estimate of the cross-correlation. In practice, correlation functions are often computed this way, using the FFT to transform to and from the frequency domain. An alternative approach, the averaged periodogram (Oppenheim and Schafer, 1989), implements the time average differently. Here, the signal is divided into D segments of length ND, and the ensemble of segments is averaged to estimate the expected value. Thus, the averaged periodogram spectral estimate is Ŝuu(f ) = 1 DND D d=1 U∗ d (f )Ud(f ) (2.23) where Ud(f ) is the Fourier transform of the dth segment of ND points of the signal u(t), and the asterisk, U∗(f ), denotes the complex conjugate. It is common to overlap the data blocks to increase the number of blocks averaged. Since the FFT assumes that the data are periodic, it is also common to window the blocks before transforming them. The proper selection of the window function, and of the degree of overlap between windows, is a matter of experience as well as trial and error. Further details can be found in Bendat and Piersol (1986). The averaged periodogram (2.23) is the most commonly used nonparametric spectral estimate, and it is implemented in MATLAB’s spectrum command. Parametric spectral estimators have also been developed and are described in Percival and Walden (1993). 2.3.5.1 The Cross-Spectrum The Fourier transform of the cross-correlation func- tion is called the cross-spectrum, Suy(f ) = N−1 τ=0 φuy(τ)e−2πjf τ/N (2.24) Replacing the autocorrelation with the cross-correlation in the preceding derivations gives the Fourier transform of the cross-correlation, F φ̂uy(τ) = 1 N U∗ (f )Y(f ) (2.25) and an averaged periodogram estimate of the cross-spectrum, Ŝuy(f ) = 1 DND D d=1 U∗ d (f )Yd(f ) (2.26) 2.3.6 Applications The variance of a biased correlation estimate (2.18) is proportional to 1/N and thus decreases as N increases. Furthermore, the effect of the bias is a scaling by a factor of N/(N − τ), which decreases as N increases with respect to τ. Thus, in general the length of a correlation function should be much shorter than the data length from which it is estimated; that is, N τ. As a rule of thumb, correlation functions should be no more than one-fourth the length of the data and should never exceed one-half of the data length. Autocovariance functions determined from stochastic signals tend to “die out” at longer lags. The lag at which the autocovariance function has decreased to values that
- 40. 24 BACKGROUND cannot be distinguished from zero provides a subjective measure of the extent of the sequential structure in a signal, or its “memory.” In contrast, if there is an underlying periodic component, the autocorrelation function will not die out but will oscillate at the frequency of the periodic component. If there is substantial noise in the original signal, the periodicity may be much more evident in the correlation function than in the original data. This periodicity will be evident in the autocorrelation function at large lags, after the contribution from the noise component has decayed to zero. An example of this is presented in the bottom row of Figure 2.2. A common use for the cross-covariance function, as illustrated in the middle panel of Figure 2.4, is to estimate the delay between two signals. The delay is the lag at which −3 0 3 (A) −3 0 3 (B) (C) −0.5 0 0.5 1 −3 0 3 (D) −3 0 3 (E) (F) −0.5 0 0.5 1 0 5 −3 0 3 Time (s) (G) 0 5 −3 0 3 Time (s) (H) (I) −0.2 0 0.2 −0.5 0 0.5 1 Lag (s) Figure 2.4 Examples of the cross-correlation coefficient function. Top Row: The signals in A and B are uncorrelated with each other. Their cross-correlation coefficient function, C, is near zero at all lags. Middle Row: E is a delayed, scaled version of D with additive noise. The peak in the cross-correlation coefficient function, F, indicates the delay between input and output. Bottom Row: G is a low-pass filtered version of H, also with additive noise. The filtering appears as ringing in I.
- 41. MEAN-SQUARE PARAMETER ESTIMATION 25 the cross-covariance function is maximal. For example, the delay between two signals measured at two points along a nerve can be determined from the cross-covariance function (Heetderks and Williams, 1975). It should be remembered that dynamics can also give rise to delayed peaks in the cross-correlation function. 2.3.7 Higher-Order Correlation Functions Second-order cross-correlation functions will be represented by φ with three subscripts. Thus, φxxy(τ1, τ2) = E[x(t − τ1)x(t − τ2)y(t)] (2.27) while the second-order autocorrelation function is φxxx(τ1, τ2) = E[x(t − τ1)x(t − τ2)x(t)] (2.28) Note that there is some confusion in the literature about the terminology for this function. Some authors (Korenberg, 1988) use the nomenclature “second-order,” as does this book; others have used the term “third-order” (Marmarelis and Marmarelis, 1978) to describe the same relation. 2.4 MEAN-SQUARE PARAMETER ESTIMATION Given a model structure and a set of input–output measurements, it is often desirable to identify the parameter vector that generates the output that “best approximates” the measured output. Consider a model, M, with a parameter set, θ, whose response to the input u(t) will be written ŷ(θ, t) = M(θ, u(t)) A common definition for the “best approximation” is that which minimizes the mean- square error between the measured output, z(t), and the model output, ŷ(θ, t): MSE(M, θ, u(t)) = E z(t) − ŷ(θ, t) 2 (2.29) If the signals are ergodic, then this expectation can be evaluated using a time average over a single record. In discrete time, this results in the summation: VN (M, θ, u(t)) = 1 N N t=1 z(t) − ŷ(θ, t) 2 (2.30) which is often referred to as the “mean-square error,” even though it is computed using a time average rather than an ensemble average. Note that the MSE depends on the model structure, M, the parameter vector, θ, and the test input, u(t). For a particular structure, the goal is to find the parameter vector, θ̂, that minimizes (2.30). That is (Ljung, 1999): θ̂ = arg min θ VN (θ, u(t)) (2.31)
- 42. 26 BACKGROUND In general, there is no closed-form solution to this minimization problem, so the “param- eter space” must be searched using iterative methods as discussed in Chapter 8. 2.4.1 Linear Least-Squares Regression If the model output, ŷ(t), is a linear function of the parameters, the model may be formulated as a matrix equation (Beck and Arnold, 1977), ŷ(θ) = Uθ (2.32) where ŷ is an N element vector containing ŷ(t), and U is a matrix with N rows (one per data point) and M columns (one per model parameter). Equation (2.30) then becomes VN (θ) = 1 N (z − Uθ)T (z − Uθ) = 1 N zT z − 2θT UT z + θT UT Uθ which may be solved analytically as follows. First, differentiate with respect to θ to get the gradient ∂VN ∂θ = 2 N (UT Uθ − UT z) The minimum mean-square error solution, θ̂, is found by setting the gradient to zero and solving: UT Uθ̂ = UT z θ̂ = (UT U)−1 UT z (2.33) Thus, for any model structure where the output is linear in the parameters, as in equation (2.32), the optimal parameter vector may be determined directly by from equations (2.33), called the normal equations.∗ Many of the model structures examined in this text are linear in their parameters even though they describe nonlinear systems. Thus, solutions of the normal equations and their properties are fundamental to much of what will follow. 2.4.1.1 Example: Polynomial Fitting Consider, for example, the following prob- lem. Given N measurements of an input signal, u1, u2, . . . , uN and output z1, z2, . . . , zN , find the third-order polynomial that best describes the relation between uj and zj . To do so, assume that yj = c(0) + c(1) uj + c(2) u2 j + c(3) u3 j zj = yj + vj ∗In the MATLAB environment, the normal equation, (2.33), can be solved using the “left division” operator. θ̂ = Uz.
- 43. MEAN-SQUARE PARAMETER ESTIMATION 27 where v is a zero-mean, white Gaussian noise sequence. First, construct the regression matrix U = 1 u1 u2 1 u3 1 1 u2 u2 2 u3 2 . . . . . . . . . . . . 1 uN u2 N u3 N (2.34) Then, rewrite the expression for z as the matrix equation z = Uθ + v (2.35) where θ = c(0) c(1) c(2) c(3) T , and use equations (2.33) to solve for θ. 2.4.2 Properties of Estimates The solution of equations (2.33) gives the model parameters that provide the minimum mean-square error solution. It is important to know how close to the true parameters these estimated values are likely to be. Consider the following conditions: 1. The model structure is correct; that is, there is a parameter vector such that the system can be represented exactly as y = Uθ. 2. The output, z = y + v, contains additive noise, v(t), that is zero-mean and statisti- cally independent of the input, u(t). If conditions 1 and 2 hold, then the expected value of the estimated parameter vector is E[θ̂] = E[(UT U)−1 UT z] = θ + E[(UT U)−1 UT v] (2.36) However, since v is independent of u, it will be independent of all the columns of U and so E[θ̂] = θ (2.37) That is, the least-squares parameter estimate is unbiased (Beck and Arnold, 1977). The covariance matrix for these parameter estimates is Cθ̂ = E[(θ̂ − E[θ̂])(θ̂ − E[θ̂])T ] (2.38) = (UT U)−1 UT E[vvT ]U(UT U)−1 (2.39) Usually, equation (2.39) cannot be evaluated directly since E[vvT ], the covariance matrix of the measurement noise, is not known. However, if the measurement noise is white, then the noise covariance in equation (2.39) reduces to E[vvT ] = σ2 v IN, where σ2 v is the variance of v(t), and IN is an N ×N identity matrix. Furthermore, an unbiased estimate of the noise variance (Beck and Arnold, 1977) is given by σ̂2 v = 1 N − M N t=1 (z(t) − ŷ(t))2 (2.40)
- 44. 28 BACKGROUND where M is the number of model parameters. Thus the covariance matrix for the param- eter estimates, Cθ̂ , reduces to Cθ̂ = σ̂2 v (UT U)−1 (2.41) which can be evaluated directly. 2.4.2.1 Condition of the Hessian Instead of using probabilistic considerations, as in the previous section, it is also instructive to examine the numerical properties of the estimate. Thus, instead of taking an expected value over a hypothetical ensemble of records, consider the sensitivity of the computed estimate, θ̂, to changes in the single, finite-length, input–output record, [u(t) z(t)]. From the normal equations (2.33), it is evident that the error in the parameter esti- mate is θ̃ = θ − θ̂ = θ − (UT U)−1 UT z = −(UT U)−1 UT v (2.42) where v(t) is the noise in the measured output, z = Uθ + v. The error is the product of two terms. The first term, (UT U)−1, is the inverse of the Hessian, denoted H, an M × M matrix containing the second-order derivatives of the cost function with respect to the parameters. H(i, j) = ∂2VN (θ) ∂θi∂θj = UT U (2.43) Furthermore, if the measurement noise is white, the inverse of the Hessian is proportional to the parameter covariance matrix, Cθ̂ , given in equation (2.41). Let ν = UT v be the second term in equation (2.42). Substituting these two expres- sions gives θ̃ = −H−1 ν Now, consider the effect of a small change in ν on the parameter estimate. The Hessian is a non-negative definite matrix, so its singular value decomposition (SVD) (Golub and Van Loan, 1989) can be written as H = VSVT where V = [v1 v2 . . . vM] is an orthogonal matrix, VT V = I, and S is a diagonal matrix, S = diag[s1, s2 . . . , sM], where s1 ≥ s2 ≥ · · · ≥ sM ≥ 0. Using this, the Hessian can be expanded as H = M i=1 sivivi T
- 45. POLYNOMIALS 29 and the estimation error, θ̃, becomes θ̃ = − M i=1 1 si vivi T ν (2.44) Notice that if the noise term, ν, changes in the direction parallel to the kth singular vector, vk, then the change in θ̂ will be multiplied by 1/sk. Consequently, the ratio of the largest to smallest singular values will determine the relative sensitivity of the parameter estimates to noise. This ratio is referred to as the condition number (Golub and Van Loan, 1989) of the matrix and ideally should be close to 1. 2.5 POLYNOMIALS Polynomials provide a convenient means to model the instantaneous, or memoryless, relationship between two signals. Section 2.4.1.1 demonstrated that fitting a polynomial between two data sequences can be done by solving a linear least-squares problem. 2.5.1 Power Series The power series is a simple polynomial representation involving a sum of monomials in the input signal, m(u) = Q q=0 c(q) uq = Q q=0 c(q) M(q) (u) = c(0) + c(1) u + c(2) u2 + c(3) u3 + · · · where the notation M(q)(u) = uq is introduced to prepare for the discussion of orthogonal polynomials to follow. Uppercase script letters will be used throughout to represent polynomial systems. The polynomial coefficients, c(q), can be estimated by solving the linear least-squares problem defined by equations (2.34) and (2.35). This approach may give reasonable results provided that there is little noise and the polynomial is of low order. However, this problem often becomes badly conditioned, resulting in unreliable coefficient estimates. This is because in power-series formulations the Hessian will often have a large condition number; that is the ratio of the largest and smallest singular values will be large. As a result, the estimation problem is ill-conditioned, since as equation (2.44) shows, small singular values in the Hessian will amplify errors in the coefficient estimates. The large condition number arises for two reasons: 1. The columns of U will have widely different amplitudes, particularly for high-order polynomials, unless σu ≈ 1. As a result, the singular values of U, which are the square roots of the singular values of the Hessian, will differ widely.
- 46. 30 BACKGROUND 2. The columns of U will not be orthogonal. This is most easily seen by examining the Hessian, UT U, which will have the form H = N 1 E[u] E[u2] . . . E[uq] E[u] E[u2] E[u3] . . . E[uq+1] E[u2] E[u3] E[u4] . . . E[uq+2] . . . . . . . . . ... . . . E[uq] E[uq+1] E[uq+2] . . . E[u2q] Since H is not diagonal, the columns of U will not be orthogonal to each other. Note that the singular values of U can be viewed as the lengths of the semiaxes of a hyperellipsoid defined by the columns of U. Thus, nonorthogonal columns will stretch this ellipse in directions more nearly parallel to multiple columns and will shrink it in other directions, increasing the ratio of the axis lengths, and hence the condition number of the estimation problem (Golub and Van Loan, 1989). 2.5.2 Orthogonal Polynomials Ideally, the regressors should be mutually orthogonal, so the Hessian will be diagonal with elements of similar size. This can be achieved by replacing the power series basis functions with another polynomial function P(q)(u) m(u) = Q q=0 c(q) P(q) (u) (2.45) chosen to make the estimation problem well-conditioned. The objective is to find a polynomial function that makes the Hessian diagonal. That is, the (i, j)th element of the Hessian should be H(i, j) = N · E[P(i+1) (u)P(j+1) (u)] = δi,j which demonstrates that the terms of P(q)(u) must be orthonormal. The expected value of a function, g, of a random variable, u, with probability density f (u), is given by (Bendat and Piersol, 1986; Papoulis, 1984) E[g(u)] = ∞ −∞ f (u)g(u) du Consequently, the expected values of the elements of the Hessian will be E[P(i) (u)P(j) (u)] = ∞ −∞ f (u)P(i) (u)P(j) (u) du This demonstrates that a particular polynomial basis function, P(q)(u), will be orthogonal only for a particular input probability distribution. Thus each polynomial family will be orthogonal for a particular input distribution. Figure 2.5 shows the basis functions corresponding to three families of polynomials: the ordinary power series, as well as the Hermite and Tchebyshev families of orthogonal polynomials to be discussed next.
- 47. POLYNOMIALS 31 0 1 2 Power 0 1 2 Hermite −1 0 1 Tcheb 10 0 10 −5 0 5 −1 0 1 0 50 100 −10 0 10 −1 0 1 −1 k 0 1 k −20 0 20 −1 0 1 0 5 k 10 k −50 0 50 −1 0 1 −10 0 10 −100 k 0 100 k −3 0 3 −50 0 50 −1 0 1 −1 0 1 Order 0 Order 1 Order 2 Order 3 Order 4 Order 5 Figure 2.5 Power series, Hermite and Tchebyshev polynomials of orders 0 through 5. Left Col- umn: Power series polynomials over the arbitrarily chosen domain [−10 10]. Middle Column: Hermite polynomials over the domain [−3 3] corresponding to most of the range of the unit- variance, normal random variable. Right Column: Tchebyshev polynomials over their full domain [−1 1]. 2.5.3 Hermite Polynomials The Hermite polynomials H(q)(u) result when the orthogonalization is done for inputs with a zero-mean, unit-variance, Gaussian distribution. Thus, Hermite polynomials are constructed such that ∞ −∞ exp(−u2 )H(i) (u)H(j) (u) du = 0 for i = j (2.46)
- 48. 32 BACKGROUND Using equation (2.46) and the results for the expected value of products of Gaussian variables (2.3), the Hermite polynomials can be shown to be H(n) (u) = n! n/2 m=0 (−1)m m!2m(n − 2m)! u(n−2m) (2.47) The first four elements are H(0) (u) = 1 H(1) (u) = u H(2) (u) = u2 − 1 H(3) (u) = u3 − 3u The Hermite polynomials may also be generated using the recurrence relation, H(k+1) (u) = uH(k) (u) − kH(k−1) (u) (2.48) Note that these polynomials are only orthogonal for zero-mean, unit variance, Gaus- sian inputs. Consequently, input data are usually transformed to zero mean and unit variance before fitting Hermite polynomials. The transformation is retained as part of the polynomial representation and used to transform any other inputs that may be applied to the polynomial. 2.5.4 Tchebyshev Polynomials A Gaussian random variable can take on any value between −∞ and +∞ (although the probability of attaining the extreme values is negligible). Real data, on the other hand, always have a finite range since they are limited by the dynamic range of the recording apparatus, if nothing else. Thus, it is logical to develop a set of polynomials that are orthogonal over a finite range. The Tchebyshev polynomials, T , are orthogonalized over the range [−1 1] for the probability distribution (1 − u2)−1/2. 1 −1 T (i)(u)T (j)(u) √ 1 − u2 du = π 2 δi,j for i = 0, j = 0 π for i = j = 0 (2.49) This probability density tends to infinity at ±1, and thus does not correspond to the PDF of any realistic data set. Thus, the Tchebyshev polynomials will not be exactly orthogonal for any data. However, all Tchebyshev polynomial basis functions are bounded between −1 and +1 for inputs between −1 and +1 (see Figure 2.5). In addition, each goes through all of its local extrema (which are all ±1) in this range. Thus, although the regressors will not be exactly orthogonal, they will have similar variances, and so the estimation problem should remain well-scaled. Hence, Tchebyshev polynomials are used frequently since they usually lead to well-conditioned regressions. The general expression for the order-q Tchebyshev polynomial is T (q) (u) = q 2 q/2 m=0 (−1)m q − m q − m m (2u)q−2m (2.50)
- 49. POLYNOMIALS 33 The first four Tchebyshev polynomials are T (0) (u) = 1 T (1) (u) = u T (2) (u) = 2u2 − 1 T (3) (u) = 4u3 − 3u The Tchebyshev polynomials are also given by the recursive relationship T (q+1) (u) = 2uT (q) (u) − T (q−1) (u) (2.51) In practice, input data are transformed to [−1 1] prior to fitting the coefficients, and the scale factor is retained as part of the polynomial representation. 2.5.5 Multiple-Variable Polynomials Polynomials in two or more variables will also be needed to describe nonlinear systems. To establish the notation for this section, consider the types of terms involved in a multivariable power series. The order-zero term will be a constant, as in the single variable case. Similarly, the first-order terms will include only a single input, raised to the first power, and thus will have the same form as their single-input counterparts, M(1) (uk) = uk However, the second-order terms will involve products of two inputs, either two copies of the same signal or two distinct signals. Thus, the second-order terms will be of two types, M(2) (uk, uk) = u2 k M(2) (uj , uk) = uj uk For example, the second-order terms in a three-input, second-order polynomial will involve three single-input terms, u2 1, u2 2, and u2 3, and three two-input terms, u1u2, u1u3, and u2u3. To remain consistent with the notation for single-variable polynomials, let M(2) (uk) = M(2) (uk, uk) = u2 k Similarly, the order-q terms will involve from one to q inputs, raised to powers such that the sum of their exponents is q. For example, the third-order terms in a three-input polynomial are u3 1 u3 2 u3 3 u2 1u2 u2 1u3 u2 2u1 u2 2u3 u2 3u1 u2 3u2 u1u2u3 2.5.5.1 Orthogonal Multiple-Input Polynomials Next, consider the construction of orthogonal, multiple-input polynomials. Let u1, u2, . . . , un be mutually orthonormal, zero-mean Gaussian random variables. These may be measured signals that just happen to be orthonormal, but will more likely result from orthogonalizing and normalizing a set of more general (Gaussian) signals with a QR factorization or SVD.
- 50. 34 BACKGROUND For Gaussian inputs, it is reasonable to start with Hermite polynomials. The order- zero term will be a constant, since it is independent of the inputs. Similarly, since the first-order terms include only one input, they must be equivalent to their single-input counterparts: H(1) (uk) = uk (2.52) Next, consider the first type of second-order term, involving a second-degree function of a single input. These terms will have the form H(2) (uk, uk) = H(2) (uk) = u2 k − 1 (2.53) and will be orthogonal to order-zero terms since E[H(2) (uk, uk)H(0) (uj )] = E[(u2 k − 1) · 1] and E[u2 k − 1] = 0. They will also be orthogonal to all first-order terms, since E[H(2) (uk, uk)H(1) (uj )] = E[(u2 k − 1)uj ] = E[u2 kuj ] − E[uj ] Both terms in this expression are products of odd numbers of zero-mean Gaussian random variables, and so their expected values will be zero. Furthermore, any two second-order, single-input terms will be orthogonal. To see why, note that E[H(2) (uk, uk)H(2) (uj , uj )] = E[(u2 k − 1)(u2 j − 1)] = E[u2 ku2 j ] − E[u2 j ] − E[u2 k] + 1 The inputs are orthonormal, by assumption, and so E[u2 k] = E[u2 j ] = 1. Furthermore, from equation (2.2) we obtain E[u2 ku2 j ] = E[u2 k]E[u2 j ] + 2E[ukuj ]2 = 1 and so H(2)(uk, uk) and H(2)(uj , uj ) will be orthogonal (for j = k). Now, consider second-order terms involving different inputs. The corresponding Her- mite polynomial is simply the product of corresponding first-order terms, H(2) (uj , uk) = H(1) (uj )H(1) (uk) j = k = uj uk (2.54) Since E[ukuj ] = 0, this term will be orthogonal to the constant, order-zero term. It will also be orthogonal to any odd-order term because the resulting expectations will involve an odd number of Gaussian random variables. It remains to show that this term is orthogonal to the other second-order Hermite polynomial terms. First consider the two-input terms, E[H(2) (ui, uj )H(2) (uk, ul)] = E[uiuj ukul] = E[uiuj ]E[ukul] + E[uiuk]E[uj ul] + E[uiul]E[uj uk]
- 51. NOTES AND REFERENCES 35 Provided that we have neither i = k, j = l nor i = l, j = k (in which case the terms would be identical), each pair of expectations will contain at least one orthogonal pair of signals, and therefore equal zero. Thus, any distinct pair of two-input second-order terms will be orthogonal to each other. Next consider the orthogonality of the two-input term to a single-input, second-order Hermite polynomial. E[H(2) (ui, uj )H(2) (uk, uk)] = E[uiuj (u2 k − 1)] = E[uiuj u2 k] − E[uiuj ] = E[uiuj ]E[u2 k] − 2E[uiuk]E[uj uk] − E[uiuj ] Since ui and uj are orthogonal, the first and third terms will be zero. Similarly, uk is orthogonal to at least one of ui and uj , so the second term will also be zero. Thus, equations (2.53) and (2.54) define two-input polynomials orthogonalized for orthonormal, zero-mean Gaussian inputs. Similarly, there will be three types of third-order terms corresponding to the number of distinct inputs. The single-input terms are the same as the third-order, single-input Hermite polynomial term, H(3) (uk, uk, uk) = H(3) (uk) = u3 k − 3uk (2.55) Terms with distinct inputs are generated using products of lower-order single-input Her- mite polynomials, H(3) (uj , uj , uk) = H(2) (uj )H(1) (uk) j = k = (u2 j − 1)uk (2.56) H(3) (ui, uj , uk) = H(1) (ui)H(1) (uj )H(1) (uk) i = j = k = uiuj uk (2.57) Higher-order terms are constructed in a similar fashion resulting in an orthogonal series known as the Grad–Hermite (Barrett, 1963) polynomial series. The same approach may be used to construct multiple-input polynomials based on the Tchebyshev polynomials. 2.6 NOTES AND REFERENCES 1. Bendat and Piersol (1986) and Papoulis (1984) provide more detailed discussions of probability theory and first-order correlation functions. 2. More information concerning higher-order correlation functions can be found in Marmarelis and Marmarelis (1978). 3. There are many texts dealing with linear regression. In particular, Beck and Arnold (1977) and Seber (1977) are recommended. Discussions more relevant to system identification can be found in Ljung (1999) and Söderström and Stoica (1989).
- 52. 36 BACKGROUND 4. Beckmann (1973) contains a detailed discussion of several orthogonal polynomial systems, including both the Tchebyshev and Hermite polynomials. The classic reference for multiple-input, orthogonal polynomials is Barrett (1963). 2.7 PROBLEMS 1. Let x and y be zero-mean Gaussian random variables with variances σ2 x and σ2 y , and covariance E[xy] = σ2 xy. Evaluate E[(xy)2 ] E[x3 y] E[x3 y2 ] 2. Let x(t) and n(t) be mutually independent zero-mean Gaussian signals with variances σ2 x and σ2 n , respectively. Let y(t) = 3x(t − τ) + n(t). What is the cross-correlation between x(t) and y(t)? What is the cross-correlation between x(t) and z(t) = x3(t − τ) + n(t)? 3. Show that if x is a zero-mean unit variance Gaussian random variable, then the Hessian associated with fitting Hermite polynomials is given by H(i, j) = i!δi,j where δi,j is a Kronecker delta. 4. Prove the recurrence relation (2.51). 5. Prove the recurrence relation (2.48). 2.8 COMPUTER EXERCISES 1. Use the MATLAB NLID toolbox to create a RANDV object describing a zero-mean Gaussian random variable with a standard deviation of 0.5. Use it to generate a NLDAT object containing 1000 points with this distribution. Suppose that this signal is an unrectified EMG, measured in millivolts, and was sampled at 1000 Hz. Set the domain and range of your NLDAT object accordingly. Plot the NLDAT object (signal) as a function of time, and check that the axes are labeled properly. Use PDF to construct a histogram for this signal. Construct the theoretical probability density, using PDF, and your RANDV object. Vary the number of bins used to compute the histogram, and compare the result with the theoretical distribution. How does the histogram change if you use 10,000 data points, or 100? How many of the 1000 points in your original signal are greater than 1 (i.e., 2σ). How many are less than −1? 2. Use the RANDV object from question 1 to generate an NLDAT object containing white Gaussian noise. Use equation (2.3), and the variance of the original signal, to predict the expected value of the fourth power of this signal. Check the validity of this prediction by raising the signal to the fourth power and computing its mean. Try changing the length of the signal. How many points are necessary before equation (2.3)
- 53. COMPUTER EXERCISES 37 yields an accurate prediction of the mean of the fourth power of a Gaussian RV? Repeat this with the sixth and eighth power of the signal. 3. Generate an NLDAT object containing 10,000 samples of zero-mean, unit variance, white Gaussian noise. Assume that this signal was sampled at 200 Hz, and set the domain increment accordingly. Compute and plot the autocorrelation between lags of 0 and 1 s. What is the amplitude of the peak at 0 lag? Estimate the standard deviation of the remaining points. Compute this ratio for various record lengths. Filter your signal using a second-order Butterworth low-pass filter, with a cutoff frequency of 10 Hz, and plot the autocorrelation of the resulting signal. Compute the ratio of the peak, at zero lag, to the standard deviation of the points far distant from the peak. Where does the correlation first fall below the standard deviation of the “distant” points? 4. Generate a 10-s sample of a 20 Hz, 5 V peak to peak, square wave, sampled at 500 Hz. Compute its autocorrelation for lags of 0–0.5 s, and plot the result. Generate a 10-s sample of zero-mean white Gaussian noise, with a standard deviation of 10 V, sampled at 500 Hz. Add this to the square wave, and plot the result as a func- tion of time. Compute and plot the autocorrelation function of the noisy square wave. Filter the noise record with a fourth-order low-pass filter with a 20 Hz cutoff. Add the low-pass filtered noise to the square wave. Plot the resulting signal, and compute and plot its autocorrelation. 5. Generate an NLDAT object containing 10,000 samples of zero-mean, unit variance, white Gaussian noise, and compute and plot its autocorrelation. Next, compute and plot the second-order autocorrelation function for this signal. Square the Gaussian noise signal, and shift and rescale the result so that it has zero mean and unit variance. Compute and plot its first- and second-order autocorrelation functions. Repeat this procedure with the cube of the Gaussian noise signal. What, if anything, does the second-order autocorrelation tell you about the nonlinearity? Does it generalize to higher exponents? 6. Use POLYNOM to create an empty polynomial object. Set the type to “power,” and set the coefficients so that it represents y = 1 + 3u − 2u2 Generate an object containing 1000 samples of white Gaussian noise, with mean 1 and variance 2. Apply the polynomial to the signal. Set the mean, standard deviation, and range properties of the polynomial to reflect those of the input data. Transform the polynomial object into a Hermite polynomial (orthogonalized using the statis- tics of the input NLDAT object). Transform it into a Tchebyshev polynomial. Examine the coefficients in each case. Use the Tchebyshev and Hermite polynomials to trans- form the input. Compare the results to the output of the power series. Are there any differences? 7. Use POLYNOM to fit polynomials to the input–output data generated in the previous problem. Fit polynomials of type power, hermite, and tcheb to the data, and compare
- 54. 38 BACKGROUND your results. Add noise to the output, so that signal-to-noise ratio is 10 dB. Re-fit the polynomials using the noise corrupted output, and compare with the noise-free results. Generate another set of input–output data. This time, let the input have a mean 1 and a variance of 4. Use each of the polynomials that you identified to predict the output of this new data set. When does the prediction remain accurate? When does it break down.
- 55. Other documents randomly have different content
- 56. A. It appeared to be a scattered fire. As soon as they commenced firing, I started up on the hill. Some one called out they were firing blank cartridges, and I seen the dust flying around, and I threw myself down like everybody else. There was a man shot within the length of this room from me, and killed—a man named Ray, I think. I then started down hill, and when I was coming down I saw a man on the far side of Twenty-eighth street swing round a freight car, and throw into the company—he threw three or four stones or some missiles in among them, the last, when I was down almost to the track, and I thought every stone I seen throwed, I thought they would fire. Q. Was it before they had fired? A. After the firing, he swung around, and seemed to be inviting them, I thought, to do something. By Mr. Lindsey: Q. Inviting the soldiers? A. It looked as though he was. He was holding on to the iron rod on the car, and was swinging on in front of them. He was a large man, about six feet, very genteelly dressed—more so than the common run of them. By Senator Yutzy: Q. This man you saw swinging on round there, trying to make an effort to exasperate them? A. It looked as though he was inviting them. Q. It looked as though he was trying to exasperate them? A. It looked as though he was inviting them to fire. I crossed the track ten minutes after the firing was over. The soldiers seemed to
- 57. be laying huddled together. This stone throwing was right in among them. Q. Did you see any of the soldiers struck down by missiles before the firing took place? A. When the first advance was made, I thought I saw one of them stagger. I cannot tell whether they were hit. After they made the bayonet charge the parties took their hands and threw the guns up. Q. What was the appearance of the class of men that threw the stones? A. This I think was a half grown boy that threw the missiles from the back of the house. Q. A boy? A. It looked to me. Three or four have grown boys there. Q. Did you see any men there throwing stones? A. I don't recollect of seeing any stones throwed, except at this watch-box, until I saw this man, after the firing was all over. The track was perfectly clear when I crossed. Q. What was the character of the crowd immediately in front of the military, or near the military? A. Rough looking. I had seen the same crowd around for two or three days. I had been out and in on the railroad. I had seen them at … street and Twenty-eighth street, for two or three days. Q. Would you take them to be citizens of Harrisburg? A. Yes, sir; not as a general thing. I know some of the conductors of the trains remarked to me that everybody was going in and nobody was coming out—all the tramps come in town and none going out.
- 58. Q. These men—would you take them to be what is generally denominated tramps? A. Not all of them. Generally a pretty rough looking set. On the hill side there was plenty of women and children. Q. I mean in the immediate vicinity of where the troubles were? A. These were a rough looking set of men. I won't say they were all tramps. They were a rough looking set of men. I noticed them before the military came up. There was no disturbance at all until after the military came up. They were all quiet. Q. They resisted the military, when they came up? A. After they formed a line and made a charge. Q. They resisted the military before the firing? A. Yes; they stood right like a wall. The military marched up, and they didn't give the least bit. Then they stepped back a piece or two, and made a bayonet charge. I was not close enough to hear any orders given. C. H. Armstrong, sworn: By Mr. Lindsey: Q. Where do you reside? A. Thirty-second street. Q. What is your occupation? A. Coal business. Q. Where is your office?
- 59. A. Liberty street, between Twenty-eighth and Twenty-ninth. Q. Were you at your office on the 19th—Thursday, 19th of July last? A. Yes, sir. Q. How large a crowd of men was there gathered about there during Thursday? A. There was quite a large crowd there during Thursday morning. Towards the afternoon a great number had come up to see the strikers. There was very few railroad men among the crowd. Q. What class of men were there? A. Parties that lived around the railroad there, just come up to see the excitement. Q. Where they demonstrative? A. No, sir; they were not. They were all talking about double- headers, I do not know what they meant, and I asked them, and they told me about putting two engines on a long train. Q. How large a crowd was there at any one time during the day, Thursday? A. I guess three or four hundred—in the afternoon about four o'clock. Q. Did they remain there during the night, Thursday? A. Yes; I was up there about twelve o'clock, and there was a few men there—about thirty remained there during the night waiting for trains to go out. Q. How many on Friday? A. There was seven or eight hundred. They were expecting the soldiers in that evening. Were also expecting the Harrisburg men up
- 60. that evening. They did not come up. I went down Saturday morning and went down the railroad from our house. I saw the Harrisburg soldiers there on the side of the hill and also down by the railroad. Q. How large was it Saturday? A. I don't know how large it was; the streets were just jammed and the side of the hill on Twenty-eighth street. Q. Was you present when the firing took place? A. Yes; I was up on the side of the hill about seventy yards from where the troops were. Q. Did you see troops as they marched up? A. Yes; I saw them before they left the Union depot. Saw them get their cartridges before they left there. Q. Did you go up ahead of them? A. Yes, sir. Q. Did you have any talk with the mob? A. Yes, sir; I talked with some of the railroad men. Q. Tell them that the troops were coming? A. Told them that the Philadelphia troops were coming. Q. What did they say? A. They said that they were not afraid of that; as long as they didn't hurt them, they would not hurt them. Q. Was the sheriff's posse ahead of the line? A. Yes; I recognized a few of them, I believe the sheriff was ahead, and, I think, Deputy Sheriff Steward, and, I think, Mr. Pitcairn was also ahead of them. He was walking beside Sheriff Fife.
- 61. Q. Did you see any stones or missiles thrown by the crowd at the soldiers? A. Yes; about the time they were charging bayonets. Q. Was there any pistols fired by the crowd? A. There was one or two fired. A pistol about the center of Twenty- eighth street; held it over, and shot down the road. By that time there were stone throwing. There were two cannons, and there was some boys started to throw stones, and one of them hit a soldier against a car, and the moment he fell they started firing. He threw up his arm about the time they charged bayonets—the crowd was throwing the bayonets up. The crowd catched hold of the bayonets, and threw the guns up to save themselves. Q. Did you hear any command given by any of the officers to fire? A. No, sir; I did not. I heard them charge bayonets. I heard that command, and I heard them give their military manœuvres, but I heard no command to fire at all. Q. Did you remain there during the night—Saturday night? A. Yes; I was there until Sunday, at dinner time. Q. What time did the mob begin to fire the cars? A. At half past ten o'clock. Q. Do you know who set the first on fire? A. No, sir; I could not say, I heard them say it was small boys done it. It was right back of our office it first started. The time I saw it there was first one car on fire, and they started to run oil cars down against it. Q. Were you there during the time, Thursday or Friday, when the police force came out?
- 62. A. They were there. I did not see them come up, they were up there when I was there. Q. How many policemen? A. I do not know how many there was, only about ten or twelve, I think. I think there was only three or four on Thursday. Q. Did they make any efforts to disperse the mob? A. Not as I saw. Q. Did they assist in trying to start the train? A. I did not see them trying to do that at all. Q. Do you know who was in charge of the police? A. No, sir; I could not say. By Senator Yutzy: Q. You heard the command given by the officers to charge bayonets? A. Yes, sir. Q. Did you remain in the same position after you heard that command until after the firing commenced? A. No, sir; I did not, after the first volley was fired. Q. But from the time you heard the command given to charge bayonets up to the time of the firing, were you still in the same position? A. Yes; I was in the same position. Q. If there had been a command given by the officers to fire, you think you would have heard it?
- 63. A. I think so. I heard most of the other commands and the manœuvres they went through before they charged bayonets. Q. You heard that distinctly? A. I heard the order to carry arms, shoulder arms—— Q. Arms port? A. I do not know whether I heard arms port or not. Q. How long after the command was given to charge bayonets before the firing commenced? A. About two minutes. It was a different body of men that came up through the hollow-square. By Senator Reyburn: Q. How did they fire. Did they fire altogether, as if they were ordered to fire? A. The parties next to the cars. The men in their company did the first shooting, and they shot very low. At the same time those in front shot higher. Q. They commenced? A. Commenced right where these men fell. Q. On the road? A. No; that was, I run back against the car—a lot of flat cars filled with coal. Q. Was this the line that was formed parallel with the railroad tracks, on the right towards the hill side? A. No; towards the round-house. Q. There is where the first shot was fired?
- 64. A. Yes, sir. A man standing near the end of the cars fell, and just as he fell, they just put their guns up and shot. Q. Did you notice in what direction they fired? A. Towards the hill. Q. Over the heads of the other line? A. Yes, over the heads; I could see the dirt fly; the party in front of them shot. Q. Did they appear to fire in the direction of where the missiles and stones came from? A. The missiles came right in front of this other body of men that shot towards the side of the hill. The stones were right at the foot of the hill, and they shot up on the side of the hill. The boys that threw the stones, were down at the foot of the hill, right back of the tracks. There was two cannons there, and those boys were right among them throwing. By Senator Reyburn: Q. Did you say the troops came out of the round-house, Sunday? A. I didn't say they came out of the round-house; I say they passed Twenty-eighth street. I was on the corner of Twenty-eighth and Penn when they passed. Q. Was anybody shooting at them? A. I saw one man following them up as they came down Twenty- eighth street. Q. He followed them up? A. Followed them so far as I could see, about the middle of Twenty- eighth and Twenty-ninth. I was afraid they would shoot at me.
- 65. Q. What did he do while he was following up? A. Threatened to shoot several times—threatened to shoot on an alley in Twenty-eighth street. We put up our hands at him. He got up again and followed them at Penn street. I got back of a sign and I believe he shot after he got a piece further up. Q. You didn't see him fire? A. No. Q. He had a gun? A. A breech-loader. Q. Musket? A. It was one of those breech-loaders. I saw him throw it up and examine the cartridges. Q. Did you know the man? A. No; I knew he wore a linen coat and a white straw hat. Q. Was it a rifle or a shot gun? A. Yes; regular musket, called breech-loaders, something similar to what the militia have. At this point the committee adjourned, to meet at the arsenal, at half-past eight o'clock, this evening. Allegheny Arsenal, Friday Evening, February 22, 1878. The committee met pursuant to adjournment, at the United States Arsenal, at half-past eight o'clock. All members present.
- 66. Major A. R. Buffington, sworn: By Mr. Lindsey: Q. First state your rank and official position here under the United States Government? A. Major ordnance United States Army, commanding Allegheny arsenal—commandant Allegheny arsenal—which ever way you choose to put it. Q. If you will go on and get at the facts, probably it will be as easy as any other way to come at a statement of the facts that came within your knowledge? A. I presume what you want from me are simple facts. You want no opinions, nor anything else—my knowledge of the riot. I have here three or four little notes addressed to me: July 21, 5, P.M. Major Buffington, Commanding U.S. Arsenal: The troops of the first division, after having been fired at by the rioters, returned the fire, killing and wounding a number. It is said the rioters will take the arsenal, and take the arms and ammunition. It may be a rumor; I will give it to you for what it is worth. It would be well for you to be on your guard. (Signed) A. L. Pearson, Major General. Q. What time did you receive that note?
- 67. A. I received that note somewhere about six o'clock. It is dated five- thirty, P.M., July 21, about half an hour afterwards—about six o'clock, I should judge. Previous to this, I would state that three gentlemen came here from the city—came to my quarters—and informed me in substance the same thing, before this was received—perhaps an hour. Was received somewhere about that time. That was the first notice I had of it. Q. Who were those gentlemen? A. Their names I don't know. They were strangers to me, and they introduced themselves. I have forgotten their names. Q. Were they citizens of Pittsburgh? A. Yes, sir; and when they came they were very much afraid that the mob would see their vehicle out in front of the gate, and they immediately left and went away on that account, saying they were afraid their vehicle would be recognized, and they were afraid of the mob. This word I mention was brought to me by Doctor Speers, of Pittsburgh, in a buggy, and he also was very much afraid of the mob. Cautioned me saying anything about it; that they would spot me, or something to that purpose. I also received this one. Here is a copy of that one written in the handwriting of General Latta. This was handed through the picket fence, which is by my quarters, to a young lady visiting my house at that time, with a request to give it to me, which she did. She refused to take it, and told him to take it to the guard-house, and he expressed a fear about taking it to the guard-house, and insisted on her taking it. In addition, I received this one. It is headed, O.D. 7, 21—7th month 21st day. Commandant Arsenal: Mob has started to the arsenal for the purpose of taking arms. Serious trouble at Twenty-eighth street between them and military.
- 68. (Signed) J. M. or T. M. King, Superintendent. I believe he is superintendent of the Allegheny Valley railroad. Having received information from those gentlemen previous to those notes, I immediately took steps to receive the mob if they should come out there. Lieutenant Lyon was staying over there. He came to the office, and I told him to tell my sergeant to go out quietly and couple the six pounder guns, have one of them brought down to the inside of the gate, as you come in. I had but one box of ammunition. All my men had Springfield rifles, and I had a part of a box of ammunition for them—I had plenty of ammunition, but none of that kind. My sergeant got some for my men, and I gave him some general instructions to guard and close the gates, and lock them, and let no one in without my knowledge. These preparations were carried on, and we got in readiness, and matters remained so until night came on, and there was no signs of anybody coming, and at night I thought I would go out in the street and see what was going on. Lieutenant Lyon, I believe, went with me. We struck down street and consulted with various people. Very few people know me here. I went out in the street and talked with a few of them. In a field below here—about two squares—is a new livery stable, and over that building there was some sort of a meeting going on, and we went to the door. They had sentries at the door. There was quite a concourse of citizens around. We could not get in, and we waited there until they came out. They were cheering inside, and somebody making speeches. Presently they came out, about twenty armed with some muskets they had gotten out of an armory below here somewhere—half-grown boys they were, and a few men—and filed off down street cheered by the populace surrounding them, and one man along side of me fired a musket in the air, and that is all that I saw. I didn't see any disposition of any of them to come here at all,
- 69. and I returned, telling Lieutenant Lyon I had no doubt they would come out here, but I did not anticipate any trouble with them at all, and instructed the men to keep in their quarters with their clothing on—to lie down with their clothing on, ready at a moment's call. Between ten and eleven o'clock I heard drums beating down street, and I concluded the rioters were coming. I went out, and the men were turned out and placed up here behind that building, where they could not be seen, and by that time the mob had got at the gate. There was nobody there except a sentry and that six pounder gun there. I went out in citizen's dress. They were yelling and screaming about the gate. As I approached, one of them said, Here comes the commanding officer, we will talk to him. I walked up to the gate, too—the gate is armed with open pickets—they stopped their noise, and I said, Boys, what is the matter? They said, A party of Philadelphia troops have fired into a crowd down here and killed a lot of women and children, and we come to get arms; we want to fight them. I says, I cannot give you any arms. I said, I cannot help you, it is impossible for me to help you. He said women and children had been shot down, and I said, It was a sad thing, but it is impossible for me to help you. We don't want you, we want that gun. I ignored that request, and kept talking quiet to them. They seemed to be peaceable enough, except one man, and I imagined he was slightly intoxicated. I know there are only twenty men in the place, and if twenty-five will join me we scale the walls. He abused them for not following. Presently one of them said, He talks well, come on. Finally the better disposed of them called the others off, and they went up the street a short distance and returned again. This belligerent fellow staid near the gate and called for volunteers. There was nobody there except myself and the sentry. I kept the men out of sight, for I didn't wish to flourish a red handkerchief in the face of the bull. I was determined to exhaust all peaceable means. They came back again, and about the time they got opposite the gate, a cry of fire was raised, and an alarm struck on the bell, and they all raised the cry of fire, and they went off
- 70. down the street. This man who was calling for volunteers, says he, The guns are spiked, we will all go; and they all went off again. That is my knowledge of the mob. By Mr. Lindsey: Q. What stores of ammunition were here at that time that the mob could have got if they had entered? A. We have here many buildings full of ordinance stores. We had for years, and have yet, something like thirty-six or forty thousand stand of arms. Don't put these down as the exact figures. We had a great many thousand stand of arms, and two magazines full of powder and ammunition, prepared and partially prepared for service; that is, the powder in the shells, the powder in the cartridges, two large magazines full. We have there, in fact, two of them full, and another partially so in the upper park. Besides these arms I speak of, we have many thousand stands of arms, revolvers, carbines, muskets, and all sorts of things. We have many large warehouses here. There is one there, [indicating,] and here is one, [indicating,] and one on the other side of the street; above that are the magazines. We have got a great deal of property here, valuable property, too, but we had no small arm ammunition except some of the old style ammunition— a lot of the old style paper cartridges which I had broken up. The arms we had are mostly loaders, except fifty breech-loading muskets, and my men here are armed with caliber fifty. A year before the riot began I was impressed with the dangerous position of this place, and I drew the attention of the authorities in Washington to it. There is a map showing the arsenal, [indicating.] That is Butler street. There, you see, are four buildings called temporary magazines. Those are wooden buildings. There are a great mass of breech-loading ammunition in there, partially prepared for service. There is one magazine, and there is the other one. There is Penn avenue—it is called a pike there. A man might have thrown a lighted cigar over and set fire to this place. I drew the attention of
- 71. my chief to it, and called particular attention to this dangerous place. He saw the importance of it, and ordered me to break up the ammunition and otherwise get rid of it. Fortunately, all that was cleared out before the riot began. These magazines were all full, and the small arm ammunition I had broken up. Here the shops are below the work-shops, on a plateau just below this, and here is the road over which you came. Here is a sort of open space, and nothing but a low wall here with a picket. Right opposite, there is another gate leading into the upper park. My men were here, and this part is utterly defenseless, and in that place were a number of cannon. The mob would have cleaned me out here. There is not a man there, but a man in charge of the magazine, and twenty men, you see, would be a small force to defend it. It is not a fortified place, it is the same as houses surrounded by a wall with a wooden picket fence. The mob could push it over and come in, and there would be no trouble about it. Q. Not a very strong fortification? A. It is utterly defenseless; but, at the same time, I was not afraid the mob would do me any injury. Q. How many cannon had you that they could have taken and moved off? A. I don't know how many are in that shed. I have five or six pieces which I call in current service. Those pieces are mounted. Those are six-pounder guns, and there is plenty of ammunition which could be used for that purpose in those magazines. I had one of them on this side, [indicating,] and one on the other side. As mobs generally do, they always come where the danger is. Here was those six-pounder guns, with canister. The only hostile demonstration they made was to rush for the gate, but I merely raised my hands, and says, that won't do, and they stopped instantly. To show what the state of affairs was here, and my information of what was going on, sometime on Saturday night I received this communication from
- 72. General Latta, addressed to the commanding officer, United States arsenal, Pittsburgh, without date, or anything else. He says: Have you communicated with General Government about prospects of attack on your arsenal. (Signed) Gen. Latta. That was sometime late in the night. I don't know whether it was before the mob came or not. Here is the reply I sent to him: United States Arsenal, July 21. James W. Latta: In reply to your pencil note, without date, I have to say I have not communicated with General Government about prospects of attack on me, and shall not until such a course is necessary. I had no communication with Washington, and none with the State authorities, except just what I showed you. They didn't advise me about what was going on. I knew nothing but just what you see here, except to give them all the assistance I could, and, by a strange mistake, I gave them all the defense I had. Here is a communication: Head-Quarters Sixth Division, Pittsburgh, July 21, 1877, 11.30, P.M. Major E. R. Buffington: It is of the utmost importance that I should have two hundred rounds metallic ball cartridges. Please deliver them to Colonel Moore. In case I have none in store, I will deliver you the order of the Secretary of War to-morrow.
- 73. (Signed) A. L. Pearson, Major General. To explain, the State had here some ammunition which I had been giving to them as they wanted all along, and we had given it all out. General Pearson had been informed that there was no more here belonging to the State, but he said if he had none to give him some, and he would get authority from Washington. Down here is the note of the man in charge of the magazines. This is dated eleven-thirty, P.M. It did not come to me till long afterwards. I sent them with a guard to the upper park with instructions to tell the magazine man to give them if they had any of the State stores, and to tell this gentleman that I had none except a part of a box for my own men. It was just nine hundred rounds, and the guard carried the written order, and down to the bottom, pasted to it, is: July 22, '77.—4.20 A.M. You see how late it was in the morning. Deliver to General Pearson's messengers eight hundred and sixty center prime metallic ball cartridges United States property, there being no ball cartridges belonging to the State at this arsenal. (Signed) James Fitzsimmons. By mistake, the State troops got all the cartridges I had. My men got forty out of the nine hundred. Each man had two rounds simply.
- 74. By Senator Yutzy: Q. Do I understand you to say that you had no ammunition for any of your arms? A. I had not a round of ammunition suitable for any arm I had in here, for the simple reason that we are in profound peace, surrounded by friends. Since the Frankford arsenal got making metallic cartridges, we had a few rounds here for the use of my men, in case we wanted to shoot. We had a few blank cartridges. We had cannon ammunition, but all the small arm ammunition was broken up and powder taken out of it and balls thrown into the lead pile. We did not have any for arms we had here except, perhaps, a few cartridges for revolvers, which I issued afterwards to the citizens in the town to defend the city—two or three days afterwards. Q. You have some muzzle loaders? A. All the muskets are muzzle loaders except—— Q. And no ammunition for them? A. No; we had powder and ball. I had paper cartridges made for the committee since that, as the controller will tell you. They came to get muskets, and I had cartridges made. Q. You may state to what extent you supplied the citizens with ammunition? A. Well, to quite a large extent. This is a statement which I have made up for General Negley. He represents the committee of safety. Here are fifty Cosmopolitan carbines; three hundred and ninety-nine Springfield rifled muskets; fifty Remington revolvers; fifty cavalry sabers; forty-nine belt holsters; forty-seven pistol pouches; forty- seven cavalry saber belts; fifty carbine slings; forty-eight carbine slings swivels; two hundred bayonet scabbards; one hundred and ninety cap pouches; two hundred cartridge boxes; two hundred
- 75. cartridge-box belts; two hundred and fifteen waist belts; two hundred waist-belt plaits; fifty bridles—curb bridles; forty-nine holsters and straps; fifty saddles; fifty saddle bags; fifty saddle blankets; thirty-eight pounds of buckshot; four hundred and eighty- three musket percussion caps. These were to make buck and ball cartridges. Those were returned back to me since then. These are to be added: One hundred and twenty-six Springfield rifled muskets, caliber fifty-eight; twenty-six Remington revolvers, caliber forty-four; thirteen Whitney revolvers, caliber thirty-six; six cavalry sabers, thirty-nine pistol holsters, sixty cap pouches, eighteen waist belts, thirty-three waist belt plaits. That was to the committee of safety; besides that, to the mayor of Allegheny City I issued—I am not sure—I think it was three hundred muskets, and powder, and balls, and buckshot, and cartridge paper for making cartridges. Q. These cartridges were not paper, buck, and ball? A. Oh, no; just the material I had, which was the balls and the powder. In order to get the powder, I broke up cannon ammunition —one pound cartridges—broke it up. By Mr. Lindsey: Q. Did you witness any of the scenes of Sunday, the 22d. A. Yes; for a very short time. Q. Tell us what you saw on Sunday? A. By these papers I was nearly all night. I went to bed to get some rest, and was in bed when I heard firing down this street. That is what first wakened me was the sound of firing down the street. Otherwise, before that it was perfectly peaceable and quiet. That officer had his twenty men out on this side of that building, for Sunday morning inspection, and he had just dismissed them, and
- 76. hearing this firing down street, I jumped out of bed, and got into my pantaloons, and put my night shirt in my breeches, and got my coat on, and rushed out of the room, and before I got out I saw that, from the exclamations of those in the house, there were a number of men running through the grounds. Whoever they were, they were unauthorizedly coming in without any permission, and when I appeared on the grounds, there were a number of soldiers inside, how many, I do not know, and as I opened the door, the firing had ceased. I started towards the gate, and upon looking around in this direction, I saw Lieutenant Lyons coming towards my quarters, and an officer coming towards my house, and I turned to meet him. As he came up, I said to him, You must take your men right out, sir; there is no protection here for you. He answered, You have walls. Yes, I said, we have walls, raising my hand that way. He says, Have you any suggestion to make. I said, None, sir; except to organize your men and assault them. I supposed there was some fight going on, from what I saw and heard. That is all the words passed between me and the officer—who he was I do not know. He had a blouse coat, and looked like a second lieutenant. It does not matter who he was; under the circumstances there was no time, at that time, for wasting words, in my estimation. My orders were orders that were peremptory. I ordered him and his men out. As I told him these last words he started towards the gate, and I immediately turned and went towards the building, where a wounded man was brought in. Says I, I will take care of the wounded. I called my men, and ordered him taken to the hospital. There was a man lying inside of the gate, one of my men bathing his temples. I asked what was the matter. He did not know. I called my man and instructed him to take him to the hospital. I went to the gate, and I saw nothing there, except a few citizens—workingmen in their Sunday clothes—going to church. I did not see an armed man anywhere. Brinton and his command had gone up the street in the meantime, and left the arsenal. In a few moments I was joined by Lieutenant Lyon, and I authorized him to go immediately for a
- 77. surgeon for these wounded men, and he went down to Doctor Robinson's office, which is one square below, and brought Doctor Robinson in immediately, by a private entrance, to save time. I went to the hospital—there was no signs of any riot in the street, or anything of the kind—I went to the hospital, and there saw Doctor Robinson, who referred to Lieutenant Ash, and said his leg must come off, but I prefer to have some surgeon to consult with. I suggested to him Doctor Lemoyne, and he agreed to that, and I went to my quarters and wrote Doctor Lemoyne a note, telling him, in the letter, Doctor Robinson had been called in, and that he wished to consult with him, and I sent one of my own horses and a messenger into the city, for Doctor Lemoyne, and he came out. He lives, maybe, three miles from here. In the meantime, the wounded had the attention of Doctor Robinson and his partner, Doctor Evans. Doctor Lemoyne soon came, accompanied by Doctor Reed, and then the wounded had the attention of all four of them. That is all I saw. As to the condition of the Philadelphia men, Lieutenant Lyon can tell more than I can, because he saw the whole thing. The stragglers were inside here; were kept here, and fed, and taken care of until Tuesday evening, and they were clothed like my men. They were so demoralized that one of them, it was reported, could not eat, and in order to divert their minds, Lieutenant Lyon put them to work—they were so afraid they would be shot by somebody. Lieutenant Ash died here, and his brother and his wife were here at the time he died. Q. When did he die? A. Died on Tuesday about two o'clock. Q. What become of the other wounded man that was brought in? A. The other wounded man was playing possum. There was not anything the matter with him. When he went to the hospital he was lying on a bed, and I said, what is the matter with you? He did not answer. Says I, get up, we do not want anybody in the hospital except sick men. The other men wounded themselves getting over
- 78. the pickets. They came over the pickets, and I am satisfied they wounded themselves in getting over the pickets. One man had a wound in the center of the hand, which he said was caused by a piece of shell. I think he put his hand on a picket of the fence, and one man had his pantaloons torn. They had some little scratches on them. In a few minutes they got out, and went to the works with the other men. There was only one wounded man, that was Lieutenant Ash. One man was wounded down street here, and ran into the Catholic church—Corporal Ash—and a few days ago he came in here to see me. He was shot in the abdomen, and strange to say he got well. Those were the only wounded men down street here. Q. You did not know, at the time that this soldier approached you, that it was General Brinton? A. No; I do not know who it was. It did not make any difference who he was. As I wrote afterwards to Washington, they virtually forced my guard. It is a very different thing for a man outside and a man inside. I know what the place is, and it is presumed that I ought to know how to take care of it; and, in addition to that, I am responsible to the civil authority for every act of mine which comes in conflict with it. I am amenable, in other words, to the civil authorities, and it seems to me I ought to know beforehand what I am doing before I enter into a fight—to know what is going on. I shall certainly take care to do so. Suppose I had opened on some of those men; they would have had me up here for murder, sure, the next day. If it had once begun, it would have been a serious business. I questioned my guard. I said to him, when did you hear that firing—when that firing began, did you see anything in the streets? No, he said, there was a small boy in front, a newspaper boy, and I asked him to look down the street, and he said he saw no one. That is the fire that got me out. I said, were there any shots fired after you heard that? He said, there was not a shot fired while the men were here at all. I did not hear a shot fired after I left my quarters, until along sometimes afterwards, way down below
- 79. here somewhere. Some man—so it was reported, and I believe it was so—some man shot two of them with the same shot, from behind the cemetery wall, or somewhere near there. By Mr. Lindsey: Q. If that mob on Saturday night had made an attempt to enter your grounds here, would you have considered you were justifiable in resisting it with any amount of force? A. Unmistakably. I had it there ready to use, and the beck of my hand would have brought my men there. Q. Did you know this officer that approached you and asked permission to bring his men inside? A. He did not ask any permission—just as I have reported to you. The words were no more or less than just what I have said. The mere fact of his being there revealed to me that he had come there for that purpose. I did not wait for any request at all. The mere presence of himself and his men was sufficient for me. I took my action from what I saw. Q. How many men were inside of the grounds? A. That I do not know. Lieutenant Lyon can answer that better than I can. Q. The number of men that stayed here? A. The number of men that stayed here were eight besides Lieutenant Ash. These men I kept—afterwards, when I saw the command was gone, these men, I allowed them to stay in because I would not send them out in the streets. I told the men to join the command, but the command moved off, and these men were allowed to remain in, and were fed and kept. By Senator Yutzy:
- 80. Q. Was there any formal demand made by any officer of the militia to be admitted, or request to be admitted here? A. None, sir; except just what I told you. Lieutenant M. W. Lyon, sworn: By Mr. Lindsey: Q. Just state your rank? A. First lieutenant of ordinance. Q. Stationed at—— A. Allegheny arsenal. Q. State what came under your observation here on the morning of the 22d—Sunday morning? A. We have Sunday morning inspection about eight o'clock. I finished the inspection, and returned to my quarters and had hold of the door, when I heard the firing down street. I turned to look out to see what it was, when I heard a yell and a lot of men running over the wall—jumping over the wall. I ran up to the gate in that direction. I thought they were the mob. Soldiers were running. I thought it was our own guard. When I got as far as that large warehouse, I met this officer, and I took him to Major Buffington's quarters. Q. Did you know who the officer was? A. No, sir; there were several officers, and the only way I now know it was General Brinton, is the fact, that some of them say that he wore a blouse, and he was the only one that had a blouse. The others were in full dress uniform.
- 81. Q. Did he state to you what he wanted? A. No; he seemed to be commanding officer, and I took him to the major's quarters. He was in a great hurry. There were several officers with him. Q. What was the result of his interview with Major Buffington? A. As the major says, he made the remark to him, as he stated in his statement, and after that this officer, with the other officers, walked toward the entrance and went out, and I followed more leisurely. When we arrived there, they were bringing in the wounded, and the major told them all the wounded they had they might leave, and he ordered those that were bringing them in, to re-join the company— these eight men came in under the pretense that they were wounded, excepting one man, who had brought—I think he helped carry Lieutenant Ash in, and the major told him he would have to join his command, and he went to the gate and found the command had moved on, and he came to me and said he would willingly hide anywhere. He would hide in the coal-shed. He had never fired a gun off in his life, and only belonged to the militia three weeks. By Mr. Means: Q. Did the general commanding leave his command, in your opinion —the man that wore the blouse—had he left his command, and come in here for protection? A. I do not think he came in here for protection. Q. What brought him here? A. He came in to see if he could get admittance for his troops. Q. Did General Brinton then move on with his command? A. As far as I know. I went down with some of these men that were carrying the wounded, to show them the direction to the hospital;
- 82. then I returned to the gate to go for Doctor Robinson, and I do not think they stopped there more than a minute. Q. Did you see any mob following in the rear? A. There was none, I am quite positive. When I went to the gate, there was a man who keeps a beer saloon standing at the gate, and he said there was only one man following, and he gave the name of this one man. I went up to him and asked him, and he immediately stopped talking, and he said he did not know the man's name. Q. When you went for Doctor Robinson, did you see any of the mob? A. I saw no mob. I saw quite a number of people in the street that had come out of curiosity, hearing the firing, but they had no arms with them at all. Q. Did you have any conversation with these wounded men to ascertain how they were wounded? A. Oh! yes; I asked them all how they were wounded. One man said that they kept firing away from the middle of the street. They had two cannons, and loaded them up with glass and nails—little toy cannons. He said he got struck that way two or three times. Q. How long after Lieutenant Ash was brought into the hospital was it before Doctor Robinson arrived? A. I should think it was not more than five minutes, because I did not go down all the way to the hospital—the hospital is halfway between here and the guard-house, and I went immediately back to the gate, and went down to Thirty-seventh street, where Doctor Robinson lives, and he was sitting in his chair, reading the morning paper, and he came with me immediately, and I did not think it was more than five minutes, certainly not more than ten. Q. How long was it before Doctor Lemoyne arrived?
- 83. A. I do not think he came until about two hours afterwards. Q. Was there any amputation performed? A. No, sir; they tried to perform an amputation, but Lieutenant Ash was not strong enough. Q. Did you learn where he was wounded—where he was when he was shot? A. I never could learn. I did not ask him, because the doctors did not want him to have any conversation. Q. You do not know how far he had been carried? A. No, sir; it was my impression he was shot near Thirty-seventh street. Some of the men said he was shot near the round-house. Lieutenant Dermott, who was stationed at the university here as assistant professor in engineering, he was up here while these wounded men were in the arsenal, and together we went over to the commissary where their cartridge boxes were, and I found the cartridges they had in their boxes, and they all averaged twenty rounds a piece, and one man he had forty. Some had less than twenty. Q. Of the soldiers? A. Of those eight that were here. I asked him—he was an old man. In fact, he had been wounded in the hand at the battle of Gettysburg, he said. When they were passing them around, there were several extra cartridge boxes, and he took one. By Senator Yutzy: Q. Were you in the vicinity of the crossing of Twenty-eighth street and the railroad, the scene of the riot, on Saturday? A. Yes, sir.
- 84. Q. At the time the military arrived there? A. I was not there the time the military arrived there. I was there about an hour before they arrived. I was talking with Captain Breck. He had two six-pounder guns, and I told him they were not of much use. He ought to have Gatling guns. He said the Philadelphia troops did have a pair of them. I waited until my patience was exhausted, and I came home. Q. Did you see any of the movements of the military in that vicinity, or while you were there? A. They made no movement while I was there. They simply remained stationary where they were. There were some on the hill side with their arms all stacked. Q. None at the crossing of the railroad, were there? A. I am not positive about that. At any rate, I did not keep account of them. They could get across the track very readily, for I went across. I do not think there were any there. I think they were mostly on the hill, and those had their guns stacked, but they were down at the crossing and on Twenty-eighth street, talking with the people, about the same as though they were going to have a party. Q. That is, the soldiers were away from their command? A. Yes, sir. Q. Mingling with the crowd? A. Yes, sir. Q. In conversation with them? A. Yes, sir. Q. Was there any considerable number of them with their arms where they were stacked?
- 85. A. They were stacked there, and there were several sentinels along the line where the arms were stacked, but the men, as a rule, had their guns stacked. Q. They had broken ranks? A. They had broken ranks. Q. Did you see them make any effort to keep any portion of the track clear? A. Not while I was there; no, sir. Q. How long were you there. A. I was there three quarters of an hour, perhaps an hour. Q. Did you see any portion of the military in ranks? A. I saw no portion of them drawn up in line of battle, or anything like it; no, sir; or company front either. I think the only men I saw, were those that were without arms, walking up and down with the crowd, talking to them, and the sentinels on post over the stacked arms. Q. They appeared to be the only ones on duty? A. They appeared to be the only ones on duty at that time. Q. As the militia were passing here, did they throw away their arms or ammunition? A. Well, not that I saw, except that Major Buffington found a case filled with cartridges belonging to the Gatling guns. At this point the committee adjourned until to-morrow morning, at ten o'clock.
- 86. Pittsburgh, Saturday, February 23, 1878. The committee met, pursuant to adjournment, at half-past ten, in the orphans' court room, Mr. Lindsey in the chair. All the members present. O. Phillips, sworn: By Mr. Lindsey: Q. State your residence and your official position in July last, and then go on and give us the facts? A. My residence is 344 Ridge avenue, Allegheny. I was mayor of the city for the last three years, up to January, 1878. Q. Of the city of Allegheny? A. Yes; of the city of Allegheny. Q. Just commence and give us a statement in your own way, chronologically—give us the facts? A. On Thursday or Friday, the 19th or 20th of July last, I had been over in Pittsburgh during the day, and went back to my office in the afternoon, and there I found that the railroad officials of the Pennsylvania company had sent up the office for police assistance, stating that a crowd of men were interfering with the running of trains near the outer depot, and that Chief of Police Ross and ten or twelve policemen had gone down there. Q. The outer depot of the Fort Wayne road? A. Yes; I jumped in a horse car and went down there myself, deeming it my duty to go and see what was the trouble, and when I got to the outer depot I saw a number of men walking up and down
- 87. the track, and quite a large number of men at Strawberry lane. I noticed a locomotive pass me and go down. It was interrupted or stopped by some men climbing up on the engine, and gesticulating in a threatening way, but what they said I do not know, but the engine stopped, and returned to the round-house. I went down then to where this crowd of men was, and saw it was a very large assemblage—several hundreds—and the police force were an atom, a mere drop in the bucket. Some of the men wanted to talk to me about their troubles. I told them, as mayor of the city, I had nothing to do with that. I was simply there as a representative of peace and good order, and spoke to the men, cautioning prudence, asking them if they realized the seriousness of what they were doing. I noticed that a man by the name of Robert Ammon was recognized as their ring-leader. He came up to me and introduced himself as having known me at my factory, on the South Side, and said he would like to talk to me. I stepped aside to converse with him, and while we were talking, men would come up and say: What shall we do now, Bob? He would say: Stand aside, I do not want to be interrupted. He told me he had been an employé of the railroad company six weeks or two months before that, but had been discharged, and since that he had been around the country organizing Trainmen's Unions. He told me he had influence to stop these troubles; that if he had sent a telegram to Martinsburg the troubles would have been stopped. He said it was not worth while to go to the railroad men; he asked me to make a speech to the men; I told him that was not my style. The men gave me their assurance they would protect the railroad property, both day and night, and when they could not do anything further they would send to me for police. I then left my officers quietly mingling with these men, and then I went back to the mayor's office, which was on Thursday or Friday, I am not sure which, or Saturday. Word came to me that some of the supposed strikers had gone to one of the military organizations in Allegheny, and had taken thirty or forty arms, and had taken them down towards where the men were on a strike, and
- 88. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com