![Random Differential Equations in Scientific Computing
mands1 blend theory with application, and provide a solid foundation of the
principles of deterministic ordinary differential equations and their numerics.
dynamical systems for deterministic & randomly perturbed
(ordinary) differential equations
s
mathematics
i t
mathe
ith
all
topics
minary
talk)
seminary-part
lecture exercises workshop
teamwork
ematics
&
in
e
to
deal
wi
eir
own
sem
informatics
seminary-part
k
of
nformatics
udents
have
d
not
just
the
algorithms of
scientific computing
all
stu
(and
Figure 1. Design of the lecture “Dynamical Systems & Scientific Computing – Intro-
duction to the Theory & Simulation of Random Differential Equations” combining
the workshop with seminar contributions, exercises, and few lectures to increase
student-centred learning effects.
Concerning random differential equations there is just a limited amount of
literature discussing them on a graduate level, and despite their significance
in scientific and engineering applications there is only one book in German
by Helga Bunke [45] from 1972 and another one in Polish by Dobieslaw Bo-
browski [30] from 1987 totally dedicated to the theory of random differential
equations. T. T. Soong’s monograph [233] from 1973 dedicates about half of
his work to these equations. Arnold Kistner’s PhD thesis [157] from 1978
(in German, too) covers essential aspects of linear random (ordinary) differ-
ential equations. There are a couple of recent papers and the one or other
book chapter on the numerics of random differential equations. However, a
holistic approach is missing in particular taking into account recent results of
Ludwig Arnold on random dynamical systems, cf. [11].
This book is a holistic and self-contained treatment of the analysis and
numerics of random differential equations from a problem-centred point of
1
Note that the MATLAB examples presented throughout this book are mainly meant to
illustrate certain individual aspects in a compact manner; hence, these examples do not
represent a “nice” implementation from a software engineering point of view (mostly, we
skipped documenting comments for the sake of a compact representation, e.g.).
Preface iii](https://image.slidesharecdn.com/81403-250322050454-7be95d0a/85/Random-differential-equations-in-scientific-computing-1st-Edition-Tobias-Neckel-11-320.jpg)
![Random Differential Equations in Scientific Computing
elastic body motion. We discuss different types of meshes with an
emphasis on regular Cartesian grids. The three main spatial discretisa-
tions—finite differences, finite volumes, and finite elements—are briefly
explained before delving deeper into finite difference schemes (FD).
We derive the corresponding FD approximations for the fundamental
equations of elastic body motion and simulate steady-state scenarios
of buildings which are bent.
Chapter 3 motivates and mathematically rigorously discusses existence
and uniqueness of path-wise solutions of random (ordinary) differen-
tial equations. We start by modelling external and ground motion ex-
citations by means of stochastic processes which motivates the study of
random (ordinary) differential equations. Their solution, existence and
uniqueness concepts are then discussed together with the correspon-
dence between stochastic and random (ordinary) differential equations.
In particular, we study the conditions that lead to the existence of path-
wise unique solutions. Solutions in the extended sense are analysed as
well as the dependence of solutions on parameters and initial conditions.
As an excursion we finally give the equations of motion for single- and
multi-storey (wireframe) buildings. Our main source for the set-up and
discussion of random differential equations is Helga Bunke’s book [45].
Chapter 4 adds the notions of P- and mean-square solutions to our discus-
sion. The special nature of random (ordinary) differential equations of-
ten requires additional refined solution concepts going beyond that of a
path-wise solution. Taking, for instance, into account that a solution may
fulfill the given random differential equation with probability one, or that
the solution is a square integrable stochastic process leads to the notion
of a P-solution or a mean-square solution, respectively. Their properties
and interconnections, in particular with respect to path-wise solutions,
are studied here.
Chapter 5 widens the scope to additional categories of applications for ran-
dom differential equations. In particular, flow problems are discussed in
more detail. These problems represent an important class of applications
in computational science and engineering. The various possible random
effects in the model, the geometry, the boundary conditions, and the pa-
rameters may be generalised to other flow scenarios, involving coupled
scenarios such as fluid-structure interaction or biofilm growth.
Part II is a mini-course on the interplay between dynamical systems and
scientific computing in itself. Here, we cover the analytical and numerical
foundations of deterministic ordinary differential equations. A special em-
phasis is given to dynamical systems theory including essential phase space
Preface vii](https://image.slidesharecdn.com/81403-250322050454-7be95d0a/85/Random-differential-equations-in-scientific-computing-1st-Edition-Tobias-Neckel-15-320.jpg)
![Random Differential Equations in Scientific Computing
Chapter 1
Recap: Random Variables &
Stochastic Processes
This chapter provides a friendly review of the central concepts of probability
theory focusing on random variables and their properties that eventually lead
to the notion of a stochastic process. Our aim is to recall the basic definitions
and equip them with tailored illustrations and MATLAB commands rather than
emphasize the most general and abstract mathematical concepts.
1.1 Key Concepts
This chapter sums up the material on continuous random variables and
stochastic processes suitable for an undergraduate/ beginning graduate lec-
ture, see [62, 25, 61, 122, 14]. The blending of MATLAB commands into the
text is motivated by [79], [153], and [177].
As illustrative introductory examples, we motivate the concepts of (i) con-
vergence of random variables that will be essential to set-up stochastic sta-
bility as well as (ii) of ergodicity of stochastic processes.
Example 1.1 (Convergence of Random Variables, cf. [225], p. 148). Let
{Xi}n
i=1 be a sequence of normally distributed random variables with van-
ishing mean and variance i−1, i.e. Xi ∼ N(0, i−1). Figure 1.1 displays the (cu-
mulative) distribution functions of the first elements of this sequence. Based
on this figure, it seems as if limi→∞ Xi = X with the limiting random variable
X ∼ PointMass(0).
Though, P(Xi = X) = 0 for any i, since X ∼ PointMass(0) is a discrete ran-
dom variable with exactly one outcome and Xi ∼ N(0, i−1) is a continuous
random variable for any i ∈ N. In other words, a continuous random variable,
such as Xi, has vanishing probability of realising any single real number in
its support.
Thus, we need more sophisticated notions of convergence for sequences
of random variables.
Example 1.2 (Ergodic & Non-Ergodic Stochastic Processes). Figure 1.2 shows
some sample paths of a parameter-dependent stochastic process Xt, called
“geometric Brownian motion”, cf. [222]. In particular,
Xt = X0 exp
((
a − 1
2 b2
)
t + bWt
)
,
Chapter 1 3](https://image.slidesharecdn.com/81403-250322050454-7be95d0a/85/Random-differential-equations-in-scientific-computing-1st-Edition-Tobias-Neckel-33-320.jpg)
![Random Differential Equations in Scientific Computing
Time [t]
0.0 0.2 0.4 0.6 0.8 1.0
1.0
1.5
2.0
2.5
exp(0.8t)
a = 0.8, b = 0.4
a = 0.8, b = 0.25
a = 0.8, b = 0.1
68,2% conf. int. for b = 0.25
(a)
Time [t]
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.5
1.0
1.5
2.0
2.5
exp(−0.8t)
a = −0.8, b = 0.4
a = −0.8, b = 0.25
a = −0.8, b = 0.1
68.2% conf. int. for b = 0.25
(b)
Time [t]
0 1 2 3 4
0.0
0.5
1.0
1.5
2.0
2.5
3.0
exp(0.8t)
a = 0.8, b = 4.4
a = 0.8, b = 2.25
a = 0.8, b = 1.1
68.2% conf. int. for b = 2.25
(c)
Time [t]
0 1 2 3 4
0.0
0.5
1.0
1.5
2.0
2.5
3.0
exp(0.1t)
a = 0.1, b = 4.4
a = 0.1, b = 2.25
a = 0.1, b = 1.1
68.2% conf. int. for b = 2.25
(d)
Figure 1.2. Paths of the Geometric Brownian Motion for a = 0.8, X0 = 1 (a) a = −0.8,
X0 = 2 (b), a = 0.8, X0 = 1 (c), and a = 0.1, X0 = 1. In (a) and (b) the values of b are
the same (b = 0.4, 0.25, 0.1) and the 68.2% confidence interval for the b = 0.25 paths
is shown as well as the expectation value X0 exp(at). In (c) and (d) the values of b
are the same (b = 4.4, 2.25, 1.1) and the 68.2% confidence interval for the b = 2.25
paths is shown as well as the expectation value X0 exp(at).
6. Under what conditions are two stochastic processes indistinguishable?
7. What are the characteristics of Gaussian processes?
8. How can we utilize MATLAB to simulate random variables, stochastic pro-
cesses and their properties?
as well as the following key concepts
1. σ-algebras, probability measures and probability spaces,
2. Random variables as well as their density, distribution and moment func-
tions,
Section 1.1 5](https://image.slidesharecdn.com/81403-250322050454-7be95d0a/85/Random-differential-equations-in-scientific-computing-1st-Edition-Tobias-Neckel-35-320.jpg)
![Random Differential Equations in Scientific Computing
1.2.1 Continuity, Measures and Probability Spaces
Let us first recall the definition of Hölder- and Lipschitz-continuity as well as
that of Ck,α-functions, cf. for instance [3], p. 40:
Definition 1.3 (Hölder and Lipschitz Continuity/ Ck,α-functions). Let (X, ∥ ·
∥X), (Y, ∥ · ∥Y ) be normed spaces and 0 < α ≤ 1. A function f : X → Y is
called globally Hölder continuous of order α if there is a positive constant C
such that
∥f(x) − f(y)∥Y ≤ C∥x − y∥α
X ∀ x, y ∈ X . (1.1)
f is called locally Hölder continuous of order α if it satisfies the condition
(1.1) on every bounded subset of X. f is called globally (or locally) Lipschitz
continuous if it is globally (or locally) Hölder continuous of order α = 1. f is
called a Ck,α-function if it is k times continuously differentiable and the k-th
derivatives are locally Hölder continuous of order α for some k ∈ N.
The central problem in measure theory is to find a measure/ volume for
as many elements of the power set P(Rd) as possible, such that this mea-
sure/ volume is additive, translation invariant and normalized. As there is no
solution to define a measure/ volume for all elements of P(Rd), we have to
restrict ourselves to special sub-set systems:
Definition 1.4 (σ-Algebra). Let Ω be a nonempty set. A collection of sets A ⊂
P(Ω) is called σ-algebra, if
• A is a algebra, i.e.,
– Ω ∈ A,
– A ∈ A ⇒ Ac ∈ A and A, B ∈ A ⇒ A ∪ B ∈ A
• ∀n ∈ N : An ∈ A ⇒ ∪n∈NAn ∈ A
Trivial examples for σ-algebras are A = {∅, Ω} and A = P(Ω), moreover, for
any A ⊂ Ω the σ-algebra properties of A = {∅, A, Ac, Ω} are easily verified.
In particular, if E is a collection of subsets of Ω, then the smallest σ-algebra
generated by E, and denoted by σ(E), is defined as
σ(E) :=
∩
{A : E ⊂ A and A is a σ-algebra on Ω} .
For instance, the smallest σ-algebra containing all open subsets of Rd is called
the Borel σ-algebra, denoted by Bd or simply by B if the dimension d requires
no specific mentioning.
Let Ω be a nonempty set and E ⊂ P(Ω). The set-system E is called
intersection-stable, if
∀ E1, E2 ∈ E =⇒ E1 ∩ E2 ∈ E .
Section 1.2 7](https://image.slidesharecdn.com/81403-250322050454-7be95d0a/85/Random-differential-equations-in-scientific-computing-1st-Edition-Tobias-Neckel-37-320.jpg)
![Tobias Neckel & Florian Rupp
Obviously, every σ-algebra is intersection-stable1.
Let Ω be a nonempty set and A be a σ-algebra on Ω. The pair (Ω, A) is
called measurable space and the elements of A are called measurable sets.
Definition 1.5 (Measurable Function). Let (A, A) and (B, B) be measurable
spaces. A function f : A → B is called A-B-measurable, if f−1(B) ⊂ A.
For instance, every continuous function f : X → Y between two metric (or
topological) spaces X and Y is measurable.
Definition 1.6 (Measure and Probability Measure). Let Ω be a nonempty set
and A be a σ-algebra on Ω. Then a set-function µ on A is called a measure, if
• µ(A) ∈ [0, ∞] for all A ∈ A,
• µ(∅) = 0,
• µ is σ-additive, i.e., for any disjoint collection of sets A1, A2, · · · ∈ A with
∪n∈NAn ∈ A it holds that
µ
(
∪
n∈N
An
)
=
∞
∑
n=1
µ(An) .
Moreover, a measure µ is called a probability measure if it additionally satis-
fies
• µ(Ω) = 1.
A measure µ on a measurable space (Ω, F) is called σ-finite, if there exist
E1, E2, · · · ∈ F, pairwise disjoint, s. t. Ω = ∪n∈NEn and µ(En) < ∞ for all
n ∈ N. Moreover, for two measures µ, ν on a measurable space (Ω, F), the
measure ν is called absolutely continuous with respect to µ, if every µ-nullset
is a ν-nullset. The notation for this property is ν ≪ µ.
If µ is a measure on the σ-algebra A of a measurable space (Ω, A), then the
triplet (Ω, A, µ) is called measure-space. In particular:
Definition 1.7 (Probability Space). Let Ω be a nonempty set and A be a σ-
algebra on Ω. The triplet (Ω, A, P) is called probability space, if P is a proba-
bility measure on the measurable space (Ω, A).
Let (Ω, A, P) be a probability space: points ω ∈ Ω are usually addressed
as sample points and a set A ∈ A is called event, hereby P(A) denotes the
probability of the event A.
1
In an extended course on measure theory, the property of set-systems to be intersection-
stable motivates the discussion of Dynkin systems, see, e.g., [93], pp. 24.
8 Chapter 1](https://image.slidesharecdn.com/81403-250322050454-7be95d0a/85/Random-differential-equations-in-scientific-computing-1st-Edition-Tobias-Neckel-38-320.jpg)
![Random Differential Equations in Scientific Computing
0 1
p
X(ω) = x for an element ω of A
x
P(A) = p = P({ω : X(ω) = x})
R
A := {ω : X(ω) = x}
ω1
ω2
X(ω1)
X(ω2)
Ω
B := {ω : X(ω) = y}
y
Figure 1.3. Relation between a random variable X and its probability function, fol-
lowing [62], p. 53.
A property which is true except for an event of probability zero is said
to hold almost surely (abbreviated ”a.s.”) or almost everywhere (abbreviated
”a.e.”).
1.2.2 Random Variables and the σ-Algebras They Generate
Probabilities are a set-functions that assign a number between 0 and 1 to a
set of points of the sample space Ω, cf. [62], pp. 53. Their domain is the set
of events of a random experiment and their range is contained in the inter-
val [0, 1]. A random variable is also a function whose range is a set of real
numbers, but whose domain is the set of sample points ω ∈ Ω making up the
whole sample space Ω (not subsets of Ω), see Fig. 1.3.
Definition 1.8 (Random Variable). Let (Ω, A, P) be a probability space. Then
a function X : Ω → Rd is called random variable, if for each Borel-set B ∈
B ⊂ Rd
X−1
(B) = {ω ∈ Ω : X(ω) ∈ B} ∈ A .
I.e., a random variable is a Rd-valued A-measurable function on a probability
space (Ω, A, P).
We usually write X and not X(ω). This follows the custom within proba-
bility theory of mostly not displaying the dependence of random variables
on the sample point ω ∈ Ω. We also denote P(X−1(B)) as P(X ∈ B), the
probability that X is in B ∈ B.
Example 1.9 (Indicator and Simple Functions are Random Variables). Let A ∈
A. Then the indicator function of A,
IA(ω) :=
{
1 if ω ∈ A
0 if ω /
∈ A
Section 1.2 9](https://image.slidesharecdn.com/81403-250322050454-7be95d0a/85/Random-differential-equations-in-scientific-computing-1st-Edition-Tobias-Neckel-39-320.jpg)
![Tobias Neckel & Florian Rupp
is a random variable.
More generally, if A1, A2, . . . , An ∈ A are disjoint sets, such that Ω = ˙
∪
n
i=1Ai
and a1, a2, . . . , am ∈ R, then
X =
n
∑
i=1
aiIAi
is a random variable, called a simple or elementary function.
Example 1.10. Sums and products of random variables are themselves ran-
dom variables, too.
MATLAB can generate distributed pseudo-random numbers between 0 and
1 can be generated utilizing the rand command. For instance, the code below
generates a one row vector with five column entries the values of which are
uniformly distributed on the interval [1, 10]:
r = 1 + (10-1).*rand(1,5);
r =
7.8589 5.1082 1.1665 8.3927 5.0023
The MATLAB manual says: “MATLAB software initializes the random number
generator at startup. The generator creates a sequence of [pseudo-]random
numbers called the global stream. The rand function accesses the global
stream and draws a set of numbers to create the output. This means that
every time rand is called, the state of the global stream is changed and the
output is different.” Due to this algorithmic procedure, pseudo-random num-
bers are created that depend on the initial value/ seed of the generation al-
gorithm. This initial value/ seed can be controlled to allow for a repetition of
the generated sequence of pseudo-random numbers. The randn command
generates normally distributed pseudo-random numbers. For instance,
randn(’state’,100) % set the initial
r = 1 + 2.*randn(1,5); % value/ seed of randn
r =
2.8170 -3.4415 0.5219 1.1375 -3.0404
randn(’state’,100) % reset the initial
r = 1 + 2.*randn(1,5); % value/ seed of randn
r =
2.8170 -3.4415 0.5219 1.1375 -3.0404
generates two identical one row vectors with five column entries the values of
which are normally distributed with mean 1 and standard deviation 2. In both
cases the same initial value/ seed was applied to MATLAB’s random number
generator.
10 Chapter 1](https://image.slidesharecdn.com/81403-250322050454-7be95d0a/85/Random-differential-equations-in-scientific-computing-1st-Edition-Tobias-Neckel-40-320.jpg)
![Random Differential Equations in Scientific Computing
Lemma 1.11 (σ-Algebras Generated by Random Variables). Let X : Ω → Rd
be a random variable on the probability space (Ω, A, P). Then
A(X) := {X−1
(B) : B ∈ B}
is a σ-algebra, called the σ-algebra generated by X. This is the smallest sub-σ-
algebra of A with respect to which X is measurable.
Proof. It is easy to verify that {X−1(B) : B ∈ B} is a σ-algebra, and, more-
over, that it is indeed the smallest σ-algebra of A with respect to which X is
measurable.
It is essential to understand that, in probabilistic terms, the σ-algebra A(X)
can be interpreted as ”containing all relevant information” about the random
variable X.
In particular if a random variable Y is a function of the random variable
X, i.e., if Y = Ψ(X), for some reasonable function Ψ, then Y is A(X)-
measurable. Conversely, suppose Y : Ω → R is A(X)-measurable. Then,
there exists a function Ψ such that Y = Ψ(X). Hence, if Y is A(X)-
measurable, Y is in fact a function of X. Consequently, when we know the
value X(ω), we in principle know also Y (ω) = Ψ(X(ω)), although we may
have no practical way to construct Ψ.
1.2.3 Density and Distribution Functions
Our discussion of properties of random variables starts with the one-
dimensional setting: A scalar random variable X is associated to the (proba-
bility) density function X 7→ f(X) defined by the property
P(x1 ≤ X ≤ x2) =
∫ x2
x1
f(X)dX
where P(x1 ≤ X ≤ x2) denotes the probability of the event x1 ≤ X ≤ x2.
The (cumulative/ probability) distribution function F(X) is given by
x0 7→ FX(x0) := P(X ≤ x0) =
∫ x0
−∞
f(X)dX
I.e., the probability that x lies in the interval (x1; x2] is FX(x2) − FX(x1) with
x1 < x2. Of course F(∞) = 1, see Fig. 1.4.
The following theorem ensures the existence of a density:
Theorem 1.1 (Radon-Nikodym). Let (Ω, F) be a measurable space and µ a σ-
finite measure on F. Moreover, let ν ≪ µ be a measure. Then ν has a density
Section 1.2 11](https://image.slidesharecdn.com/81403-250322050454-7be95d0a/85/Random-differential-equations-in-scientific-computing-1st-Edition-Tobias-Neckel-41-320.jpg)
![Tobias Neckel & Florian Rupp
Probability Density Function Cummulative Density
Function
P(x1 < X < x2)
x1 x2
x1 x2
1
Figure 1.4. Sketch of a density and its corresponding distribution function. Shown
is the connection between the probability P(x1 < X < x2) and the graphs of these
two functions.
with respect to µ, i.e., there exists a measurable function f : (Ω, F) → ([0, ∞], B)
such that
for all E ∈ F it holds that ν(E) =
∫
E
fdµ .
Proof. see [93], pp. 277
In applications one often has to deal with data samples of random vari-
ables and tries to gather some information about the density and distribu-
tion of these random variables. The following MATLAB Example takes, for
convenience, 1000 sample points of a normally distributed random variable
with mean 3 and variance 25, i.e. of a N(3, 25)-random variable, and graphs
its distribution curve using histograms (see Fig. 1.5 for a visualisation of the
resulting plots).
MATLAB Example 1.1. plotting.m: Generating the images visualised in Fig. 1.5.
num = 1000; x = 3 + 5 . *randn (num, 1 ) ;
figure ( 1 ) ; plot ( [ 1 : length ( x ) ] , x )
[ count bins ] = h i s t ( x , sqrt (num) ) ;
figure ( 2 ) ; h i s t ( x , sqrt (num) )
figure ( 3 ) ; plot ( bins , count , ’−b ’ , ’ LineWidth ’ ,2)
count_sum = cumsum( count ) ;
figure ( 4 ) ; plot ( bins , count_sum , ’−b ’ , ’ LineWidth ’ ,2)
12 Chapter 1](https://image.slidesharecdn.com/81403-250322050454-7be95d0a/85/Random-differential-equations-in-scientific-computing-1st-Edition-Tobias-Neckel-42-320.jpg)
![Random Differential Equations in Scientific Computing
Thus, for a normally distributed random variable X, the moment generating
function is
MX(t) = E(exp(tX)) =
∫ ∞
−∞
1
√
2πσ
exp
(
−
(x − µ)2
2σ2
)
exp(tx)dx
= exp
(
µt + 1
2 σ2
t2
)
.
This leads to E(X) = µ, E(X2) = µ2 + σ2, E(X3) = µ3 + 3µσ2, . . . and E(X −
µ) = 0, E((X − µ)2) = E(X2 − 2µX + µ2) = σ2, E((X − µ)3) = 0, . . . .
Following [146], example 3.4, we apply the moment generating function to
derive the geometric distribution: First, via MATALB, we obtain a closed form
for the moment generating function by
ML = simplify( symsum(exp(t*k)*p^k*q, k, 0, inf) );
pretty(ML)
q
- ------------
exp(t) p - 1
The first and second moments are generated by differentiation and substi-
tution for t = 0 in the resultant expression. For the first moment this leads
to
MLP = limit(diff(ML), t, 0)
MLP =
1/(p-1)^2*q*p
We repeat the process for the second moment
MLPP = limit(diff(ML,2), t, 0)
MLPP =
-q*p*(p+1)/(p-1)^3
The variance is now computed and simplified by noting q = 1 − p and
substitution Var(X) = E(X2) − E(X)2:
VARL = subs(MLPP, q, 1-p) - subs(MLP, q, 1-p)^2
pretty( simplify(VARL) )
p
--------
2
(p - 1)
The first and second moments of a random vector x = (x1, . . . , xn)T are
defined by
µ := E(x) := (E(x1), . . . , E(xn))T
Section 1.3 17](https://image.slidesharecdn.com/81403-250322050454-7be95d0a/85/Random-differential-equations-in-scientific-computing-1st-Edition-Tobias-Neckel-47-320.jpg)
![Tobias Neckel & Florian Rupp
and by the covariance matrix (symmetric and positive definite)
P := E((x − µ)(x − µ)T )
:=
E((x1 − µ1)(x1 − µ1)) E((x1 − µ1)(x2 − µ2)) . . . E((x1 − µ1)(xn − µn))
E((x2 − µ2)(x1 − µ1)) E((x2 − µ2)(x2 − µ2)) . . . E((x2 − µ2)(xn − µn))
.
.
.
.
.
.
...
.
.
.
E((xn − µn)(x1 − µ1)) E((xn − µn)(x2 − µ2)) . . . E((xn − µn)(xn − µn))
.
The diagonal elements E((xi − µi)2) of P are the variances and the off-
diagonal element are the covariances of the vector components. The standard
square deviation is given by the trace of P:
tr(P) = E
( n
∑
i=1
(xi − µ)2
)
.
For instance, with the covariance matrix P, the density of a normally dis-
tributed n-vector is
f(x1, . . . , xn) =
1
√
(2π)n
√
det(P)
exp
(
−1
2 (x − µ)T
P−1
(x − µ)
)
.
To generate a realisation of a multivariate Gaussian random variable X ∼
N(µ, P), P ∈ Rd×d we can proceed as follows. cf. [153], pp. 487:
1. Perform a Cholesky decomposition of P to yield the non-singular d × d-
matrix G such that P = GGT .
2. Generate a realisation u ∈ Rd of a random vector U ∼ N(0, I) in Rd,
where I = diag(1, 1, . . . , 1) ∈ Rd×d denotes the d × d unit matrix.
3. Form the realisation of X as x = Gu + µ.
As an example, let us assume µ := 0 and
P :=
1 2/3 1/3
2/3 1 1/3
1/3 2/3 1
⇒ G =
1 0 0
0.6667 0.7454 0
0.3333 0.5963 0.7303
.
In Fig. 1.7, 100 realisations of X are plotted by use of the following MATLAB
commands:
18 Chapter 1](https://image.slidesharecdn.com/81403-250322050454-7be95d0a/85/Random-differential-equations-in-scientific-computing-1st-Edition-Tobias-Neckel-48-320.jpg)
![Random Differential Equations in Scientific Computing
−3
−2
−1
0
1
2
3
−4
−2
0
2
4
−3
−2
−1
0
1
2
3
4
Figure 1.7. Realisations of a 3-dimensional multivariate Gaussian random variable,
cf. [153], p. 488.
MATLAB Example 1.2. plotting3D.m: Generating Fig. 1.7.
P = [1 2/3 1/3;2/3 1 2/3;1/3 2/3 1 ] ;
G = chol ( P ) ’ ; % perform Cholesky decomposition
% MATLAB produces P=A ’ * A so G=A ’
M = 200;
for m = 1:M % generate r e a l i s a t i o n s of x
u = [ randn (1 ,1) randn (1 ,1) randn (1 ,1) ] ’ ;
x = G*u ;
scatter3 ( x ( 1 ) , x ( 2 ) , x ( 3 ) , ’ f i l l e d ’ )
hold on ;
end
Cross-expectations of products of random variables are generally difficult
to obtain, though the following proposition allows us to calculate an arbitrary
product of normally distributed random variables4.
Proposition 1.14 (Expectation of a Product of Normally Distributed Random
Variables). Suppose X = (x1, x2, . . . , xn)T ∼ N(0, P), where P is an n × n-
positive semi-definite matrix. For non-negative integers s1, s2, . . . , sn, we have
E
( n
∏
i=1
xsi
i
)
=
0 if s is odd ,
1
(s
2 )!
∑s1
ν1=0 · · ·
∑sn
νn=0 (−1)
∑n
i=1 νi
·
·
(
s1
ν1
)
. . .
(
sn
νn
)
(1
2 hT Ph
)s/2
if s is even ,
where s = s1 + s2 + · · · + sn and h =
(1
2 s1 − ν1, 1
2 s2 − ν2, . . . , 1
2 sn − νn
)T
.
4
See, [138], too. In the physics literature such closed form solutions for the expectation of
the product of normally distributed random variables are associated with Wick’s formula,
cf. [150], p.546.
Section 1.3 19](https://image.slidesharecdn.com/81403-250322050454-7be95d0a/85/Random-differential-equations-in-scientific-computing-1st-Edition-Tobias-Neckel-49-320.jpg)
Random differential equations in scientific computing 1st Edition Tobias Neckel
- 1. Instant Ebook Access, One Click Away – Begin at ebookgate.com Random differential equations in scientific computing 1st Edition Tobias Neckel https://ebookgate.com/product/random-differential-equations- in-scientific-computing-1st-edition-tobias-neckel/ OR CLICK BUTTON DOWLOAD EBOOK Get Instant Ebook Downloads – Browse at https://ebookgate.com Click here to visit ebookgate.com and download ebook now
- 2. Instant digital products (PDF, ePub, MOBI) available Download now and explore formats that suit you... Wavelet methods for solving partial differential equations and fractional differential equations 1st Edition Gupta https://ebookgate.com/product/wavelet-methods-for-solving-partial- differential-equations-and-fractional-differential-equations-1st- edition-gupta/ ebookgate.com Elementary Differential Equations 1st Edition Kenneth Kuttler https://ebookgate.com/product/elementary-differential-equations-1st- edition-kenneth-kuttler/ ebookgate.com Functional methods in differential equations 1st Edition Veli-Matti Hokkanen https://ebookgate.com/product/functional-methods-in-differential- equations-1st-edition-veli-matti-hokkanen/ ebookgate.com A Course in Ordinary Differential Equations 2nd Edition Swift https://ebookgate.com/product/a-course-in-ordinary-differential- equations-2nd-edition-swift/ ebookgate.com
- 3. Fractional Partial Differential Equations 1st Edition Yong Zhou https://ebookgate.com/product/fractional-partial-differential- equations-1st-edition-yong-zhou/ ebookgate.com Differential Equations For Dummies 1st Edition Steven Holzner https://ebookgate.com/product/differential-equations-for-dummies-1st- edition-steven-holzner/ ebookgate.com Handbook of Differential Equations Evolutionary Equations Volume 1 1st Edition Constantin M. Dafermos https://ebookgate.com/product/handbook-of-differential-equations- evolutionary-equations-volume-1-1st-edition-constantin-m-dafermos/ ebookgate.com Handbook of Differential Equations Evolutionary Equations Volume 1 1st Edition Constantin M. Dafermos https://ebookgate.com/product/handbook-of-differential-equations- evolutionary-equations-volume-1-1st-edition-constantin-m-dafermos-2/ ebookgate.com Elements of partial differential equations Pavel Drabek https://ebookgate.com/product/elements-of-partial-differential- equations-pavel-drabek/ ebookgate.com
- 5. 1 Tobias Neckel Florian Rupp Random Differential Equations in Scientific Computing
- 6. Versita Discipline: Mathematics Managing Editor: Aleksandra Nowacka-Leverton Language Editor: Nick Rogers
- 7. 3 Published by Versita, Versita Ltd, 78 York Street, London W1H 1DP, Great Britain. This work is licensed under the Creative Commons Attribution-NonCommercial- -NoDerivs 3.0 license, which means that the text may be used for non-commercial purposes, provided credit is given to the author. Copyright © 2013 Florian Rupp and Tobias Neckel ISBN (paperback): 978-83-7656-024-3 ISBN (hardcover): 978-83-7656-025-0 ISBN (for electronic copy): 978-83-7656-026-7 Managing Editor: Aleksandra Nowacka-Leverton Language Editor: Nick Rogers Cover illustration: © Florian Rupp and Tobias Neckel www.versita.com
- 9. To our beloved parents Rosemarie and Hans-Georg & Erika and Heinrich
- 10. Tobias Neckel & Florian Rupp Preface It is interesting thus to follow the intellectual truths of analysis in the phenomena of nature. This correspondence, of which the system of the world will offer us numerous examples, makes one of the greatest charms attached to mathe- matical speculations. P-S L (1749-1827) Mathematical discoveries, small or great are never born of spontaneous generation. They al- ways presuppose a soil seeded with preliminary knowledge and well prepared by labour, both conscious and subconscious. H P (1854-1912) This book on the theory and simulation of random differential equations came into being as the result of our lecture “Dynamical Systems & Scientific Computing – Introduction to the Theory & Simulation of Random Differential Equations” during the summer term 2012. This novel interdisciplinary way to cover dynamical systems and scientific computing brought Master students into contact with cutting edge research and was awarded the Ernst-Otto Fis- cher prize for innovative and trend-setting teaching paradigms by the depart- ment of Computer Science of the Technische Universität München. Figure 1 provides a short overview of the pieces of this lecture and their fit. We will discuss this course and in particular the workshop on random differential equations and their application to ground motion excited multi- storey buildings in chapters 17 and 18. These hands-on lecture notes serve as the theoretical foundation for our lecture and the workshop. MATLAB com- ii Preface
- 11. Random Differential Equations in Scientific Computing mands1 blend theory with application, and provide a solid foundation of the principles of deterministic ordinary differential equations and their numerics. dynamical systems for deterministic & randomly perturbed (ordinary) differential equations s mathematics i t mathe ith all topics minary talk) seminary-part lecture exercises workshop teamwork ematics & in e to deal wi eir own sem informatics seminary-part k of nformatics udents have d not just the algorithms of scientific computing all stu (and Figure 1. Design of the lecture “Dynamical Systems & Scientific Computing – Intro- duction to the Theory & Simulation of Random Differential Equations” combining the workshop with seminar contributions, exercises, and few lectures to increase student-centred learning effects. Concerning random differential equations there is just a limited amount of literature discussing them on a graduate level, and despite their significance in scientific and engineering applications there is only one book in German by Helga Bunke [45] from 1972 and another one in Polish by Dobieslaw Bo- browski [30] from 1987 totally dedicated to the theory of random differential equations. T. T. Soong’s monograph [233] from 1973 dedicates about half of his work to these equations. Arnold Kistner’s PhD thesis [157] from 1978 (in German, too) covers essential aspects of linear random (ordinary) differ- ential equations. There are a couple of recent papers and the one or other book chapter on the numerics of random differential equations. However, a holistic approach is missing in particular taking into account recent results of Ludwig Arnold on random dynamical systems, cf. [11]. This book is a holistic and self-contained treatment of the analysis and numerics of random differential equations from a problem-centred point of 1 Note that the MATLAB examples presented throughout this book are mainly meant to illustrate certain individual aspects in a compact manner; hence, these examples do not represent a “nice” implementation from a software engineering point of view (mostly, we skipped documenting comments for the sake of a compact representation, e.g.). Preface iii
- 12. Tobias Neckel & Florian Rupp view. We take an interdisciplinary approach by considering state-of-the-art concepts of both dynamical systems and scientific computing. Our intended audiences are those of beginning graduate/ master level courses on theory and numerics of stochastically perturbed differential equations. The areas covered here are of importance for interdisciplinary courses in informatics, engineering and mathematics. Increasing interest in “Uncertainty Quantifi- cation” during recent years warrants a textbook that is aimed at a new gen- eration of researchers in this field, and which is rooted in the principles of dynamical systems and scientific computing. This will foster a solid under- standing of both theory and simulation. From a methodological point of view, the red line pervading this book is the two-fold reduction of a random partial differential equation disturbed by some external force as present in many important applications in science and engineering. First, the random partial differential equation is reduced to a set of random ordinary differential equations in the spirit of the method of lines. These are then further reduced to a family of (deterministic) ordinary differential equations, see Fig. 2. In particular, this latter reduction step and the fields of mathematics and computer science which support it, form the basis of this exposition. t x Ω t x Ω Space-Discretization (Finite Differences) Finite-Dimensional System of R(O)DE Space-Time-Realization-Cube Partial Differential Equation with Stochastic Effects (RPDE or SPDE) t x Ω Path-Wise Solution Concept for R(O)DEs Finite-Dimensional System of an Infinite Family of ODEs Decrease Mesh-Size Compatibility Conditions 1) all solutions of the ODE family are defined on a common time interval 2) all solutions are stochastic processes Figure 2. Reduction from a given continuum mechanical random partial differential equation to a family of deterministic ordinary differential equations. Hereby, our main example is the motion of multi-storey buildings subject to stochastic ground motion excitations. The (simplified) buildings are either modeled as solids by standard assumptions of continuum mechanics (and their corresponding partial differential equation description) or wireframe- structures based on deterministic oscillators. The external forcing is due to a linear filtered white noise that describes the earth’s surface as a layer be- tween the ground surface and the nearest bedrock where the source of an earthquake is located and treats the wave propagation in this layer as be- ing one-dimensional and vertical. The corresponding stochastic models are known as the Kanai-Tajimi filter or the Clough-Penzien filter. iv Preface
- 13. Random Differential Equations in Scientific Computing From a didactical point of view, we put much effort into providing the de- terministic foundation of the theory and simulation of ordinary differential equations as well as that of random variables and stochastic processes in or- der to give a self-contained presentation. Every chapter begins with a list of key concepts and key questions the reader should keep in mind while study- ing the contents of the respective chapter. Moreover, quite uniquely for a mathematics text book, every sub-chapter ends with a set of quizzes in the type of oral exam questions, allowing the knowledge obtained to be consoli- dated quickly, and to enable a successful self-study of the materials covered. Outline of the Chapters Figure 3 sketches the rough outline of this book and focuses on randomly perturbed phenomena in science and engineering, their mathematical anal- ysis and effective as well as efficient numerical simulation. In contrast to the “classical” bottom-up textbook approach, we follow an application oriented top-down procedure, and proceed from discussions of complex applications to simpler known concepts for the following reason: This allows us to start with the complete picture and introduce the reader to applications, numerics and general theory quickly. Thus, in part I, we proceed from random par- tial differential equations (RPDEs), to random ordinary differential equations (RODEs) and then finally to ordinary differential equations (ODEs). During the lecture on which the book is based, we saw that the students struggled with the new concept of randomized ODEs at first and actually required knowledge on stochastic ordinary differential equations (SODEs) and other solution con- cepts to fully place RODEs into their body of knowledge and to fully appre- ciate the stochastic concepts. After the discussion of the “complete picture” we continue, in parts II and III, with a recap in the classical way, because of the interdisciplinary background of the intended readership, we believe that this is necessary in order to give a self contained representation. In particu- lar, the chapters are such that they may be skipped by those readers familiar with the corresponding concepts. The main part of the exposition proceeds in part IV with a discussion of those RODEs that can be treated more or less easily: the linear ones. Here, the existence and uniqueness results are based on the general theorems provided in part I. Finally, RODEs and simulations together with their evaluation are joined in the workshop part V. In particular, the single chapters of this book contain the following specific information: Part I serves as an introduction to the modelling of randomly perturbed phenomena in science and engineering by random partial differential equa- tions and their reduction to random ordinary differential equations. Here, we discuss the following aspects: Preface v
- 14. Tobias Neckel & Florian Rupp Problem Formulation/ Reduction & Motivation § 2 RPDEs § 3 & § 4 RODEs § 5 Additional Examples Background Materials & Review § 1 Stochastic Processes Part II: Path-Wise ODEs Part III: Fourier & Co. The Workshop Projects Theory & Simulation of Random (Ordinary) Differential Equations Holistic Theory: § 12 Linear RODEs I § 13 Linear RODEs II § 14 Simulation of RODEs § 15 Stability of RODEs § 6 ODE Theory § 7 ODE Numerics § 8 Dynamical Systems § 16 Random Dynamical Systems § 9 Fourier Transform § 10 Noise Spectra § 11 Space Filling Curves § 17 The Workshop Idea § 18 The Workshop Project Figure 3. Outline of the book from the point of view of theory and simulation of random (partial/ ordinary) differential equations. Chapter 1 provides a friendly review of the central concepts of probability theory focusing on random variables and their properties that eventu- ally lead to the notion of a stochastic process. Our aim is to recall the basic definitions and equip them with tailored illustrations and MATLAB commands rather than emphasize the most general and abstract mathe- matical concepts. Chapter 2 discusses how specific Random Partial Differential Equations are transformed to Random Ordinary Differential Equations by applying clas- sical spatial discretisations. Variants from a variety of applications, are discussed, leaving the time discretisation for Chap. 7. The derivation of the underlying system of (deterministic) partial differential equations and corresponding boundary conditions is shown for the example of vi Preface
- 15. Random Differential Equations in Scientific Computing elastic body motion. We discuss different types of meshes with an emphasis on regular Cartesian grids. The three main spatial discretisa- tions—finite differences, finite volumes, and finite elements—are briefly explained before delving deeper into finite difference schemes (FD). We derive the corresponding FD approximations for the fundamental equations of elastic body motion and simulate steady-state scenarios of buildings which are bent. Chapter 3 motivates and mathematically rigorously discusses existence and uniqueness of path-wise solutions of random (ordinary) differen- tial equations. We start by modelling external and ground motion ex- citations by means of stochastic processes which motivates the study of random (ordinary) differential equations. Their solution, existence and uniqueness concepts are then discussed together with the correspon- dence between stochastic and random (ordinary) differential equations. In particular, we study the conditions that lead to the existence of path- wise unique solutions. Solutions in the extended sense are analysed as well as the dependence of solutions on parameters and initial conditions. As an excursion we finally give the equations of motion for single- and multi-storey (wireframe) buildings. Our main source for the set-up and discussion of random differential equations is Helga Bunke’s book [45]. Chapter 4 adds the notions of P- and mean-square solutions to our discus- sion. The special nature of random (ordinary) differential equations of- ten requires additional refined solution concepts going beyond that of a path-wise solution. Taking, for instance, into account that a solution may fulfill the given random differential equation with probability one, or that the solution is a square integrable stochastic process leads to the notion of a P-solution or a mean-square solution, respectively. Their properties and interconnections, in particular with respect to path-wise solutions, are studied here. Chapter 5 widens the scope to additional categories of applications for ran- dom differential equations. In particular, flow problems are discussed in more detail. These problems represent an important class of applications in computational science and engineering. The various possible random effects in the model, the geometry, the boundary conditions, and the pa- rameters may be generalised to other flow scenarios, involving coupled scenarios such as fluid-structure interaction or biofilm growth. Part II is a mini-course on the interplay between dynamical systems and scientific computing in itself. Here, we cover the analytical and numerical foundations of deterministic ordinary differential equations. A special em- phasis is given to dynamical systems theory including essential phase space Preface vii
- 16. Tobias Neckel & Florian Rupp structures (equilibria, periodic orbits, invariant sets) as well as fundamental tools (Lyapunov exponents and Lyapunov functions). Chapter 6 serves as a holistic introduction to the theory of ordinary differ- ential equations (without singularities). After some preliminaries, inte- gral curves in vector fields are discussed, i.e., ordinary differential equa- tions ẋ = F(t, x). Hereby, we start with continuous right hand sides F and their ε-approximate solutions as well as the Peano-Cauchy existence theorem and its implications. We continue our discussion for Lipschitz- continuous functions F and the existence and uniqueness theorem of Picard-Lindelöf. In particular, we analyse maximal integral curves, give the three types of maximal integral curves that can occur in autonomous systems and show the transformation of a d-th order equation into a first order system. Next, we deal with the existence of solutions in the ex- tended sense where the right hand side function may be continuous except for a set of Lebesgue-measure zero. Caratheodory’s existence theorem and its implications are studied together with maximum and minimum solutions. Then, we study the broad class of linear ordinary differential equations by discussing the unique existence of their solu- tions and their explicit construction. Applications of the theory focus on first integrals and oscillations for the deterministic pendulum and the Volterra-Lotka system. Finally, we provide a first glance into the ex- istence, uniqueness and extension of solutions of ordinary differential equations on infinite-dimensional Banach spaces. Chapter 7 contains the relevant aspects of the numerical simulation of ordi- nary differential equations. Classical explicit one-step methods such as the explicit Euler or Runge-Kutta schemes are presented before motivat- ing implicit approaches for stiff ODEs. A variety of example implementa- tions show the behaviour of the different schemes applied to different initial value problems. The brief discussion of the Newmark family of schemes and of symplectic methods widens the scope of this chapter to approaches that are typically neglected but that provide useful features worth being on the radar in the context of RODE simulations. Chapter 8 provides a brief review on deterministic dynamical systems. Fun- damental notions and concepts are introduced, like that of (continuous) dynamical systems, long-time behavior, invariance and attraction. This paves the way to analyze stability in the sense of Lyapunov by utilizing Lyapunov-functions for proving (asymptotic) stability in non-linear sys- tems. Next, we analyze the correspondence between the stability prop- erties of non-linear systems and their linearisation. Here, we give the famous theorem of Hartman and Grobman, a classification of equilib- ria in planar systems with respect to their stability properties as well as viii Preface
- 17. Random Differential Equations in Scientific Computing the techniques for the determination of the position of Lyapunov expo- nents of a linear system, like the Routh-Hurwitz criterion or the Lozinskii- measure method. Part III covers important concepts and algorithms in Scientific Computing: the discrete Fourier transform and its variants, the frequency domain method for response analysis, as well as space-filling curves as paradigms for effec- tive and efficient data storage. Chapter 9 discusses the basic aspects of the continuous and the discrete Fourier transform, with the focus on the latter including various MAT- LAB examples. The famous Fast Fourier Transform is derived. We briefly present the trigonometric variants of the discrete Fourier trans- form related to symmetry properties of the underlying input data. These trigonometric transforms allow us to realise fast Poisson solvers on Cartesian grids which are needed in the workshop problem (cf. Chap. 18). Frequency domain aspects and the Fourier transform are essential to un- derstand key characteristics of stochastic processes (spectrum, power spectrum) and the propagation of excitations through mechanical struc- tures. Chapter 10 starts with the basic definitions and implications related to the spectral representation of stationary and periodic stochastic processes. Based on these, we study the notions of energy, power and spectral density. We give several examples for colored noise processes, the fre- quency domain method for response analysis, and linear filters. In partic- ular, we apply this method to our problem of multi-storey excitation due to seismic impacts and their propagation through wireframe structures. Chapter 11 introduces the fundamental concepts, definitions, and proper- ties of space-filling curves such as the Hilbert and Peano curves. We briefly present three different categories of possible applications moti- vating the usage of these special curves in the context of computational simulations. Two variants for the construction of (discrete iterations of) the curves are explained in detail such that the reader is in the position to use space-filling curves for a tangible tasks like ordering Cartesian mesh cells. Here, strong connections to spacial discretisation (cf. Chap. 2) and its efficient implementation are provided. Part IV is devoted to a more in depth study of the theory and simulation of random (ordinary) differential equations. It analyses the theory of linear random differential equations. Numerical schemes for (non-linear) random differential equations, like the the averaged Euler and Heun method are dis- cussed. Stability of the null-solution is considered and Lyapunov-type meth- Preface ix
- 18. Tobias Neckel & Florian Rupp ods are applied to the various concepts of stochastic stability. Finally, the recent theory of random dynamical systems and its impacts on the study of random (ordinary) differential equations is presented. Chapter 12 treats linear inhomogeneous ordinary random differential equa- tions of the type Ẋt = A(t)Xt + Zt where the randomness is located just in the inhomogeneous driving process Zt. These types of equations can be analysed in ways analogous to their deterministic counterparts already exhibiting a wealth of interesting phenomena. Of importance are the stochastic characteristics of the solutions process as well as pe- riodic and stationary solution types. In particular, we give first stability conditions with respect to which solutions converge towards periodic or stationary ones. Chapter 13 extends this body of knowledge on linear random ordinary dif- ferential equations by also allowing stochastic effects in the coefficients. We give the general solution formulas for these types of equations to- gether with equivalence result for path-wise and mean-square solutions. Moreover, on the one hand, we analyse the asymptotic properties of path-wise solutions focusing on (exponential) decay towards the null- solution as well as on upper bounds for path-wise solutions. On the other hand, we also study the properties of the moments of path-wise solutions with respect to the (exponential) decay as well as the existence of asymptotically θ-periodic solutions. As an excursion, the general so- lution formula of linear non-commutative path-wise continuous noise systems is constructed. Chapter 14 discusses all relevant aspects for simulation of path-wise RODE problems. We present lower-order explicit RODE schemes (Euler and Heun) as well as higher-order K-RODE Taylor schemes. Detailed infor- mation on the corresponding MATLAB implementation for the wireframe model are given and numerical results show the validity of the approach. Chapter 15 studies the various notions of stability of the null solution of a random (ordinary) differential equation with a focus on path-wise equi- stability, h-, P-, and W-stability. In particular, the relations/ implications and inter-connections between these concepts are discussed and the results of Chap. 13 on the path-wise stability of linear random differ- ential equations with stochastic coefficients are re-framed in the con- text of these concepts. Moreover, we extend the deterministic Lya- punov method to random differential equations. Based on suitable Lyapunov-functions, necessary conditions for h-stability and path-wise equi-stability are given. Finally, the stability of deterministic systems x Preface
- 19. Random Differential Equations in Scientific Computing subject to different classes of continuously acting random perturbations is analysed. Chapter 16 provides a glimpse into the very recent theory of random dynam- ical systems. We give the fundamental definitions of metric, measurable and random dynamical systems together with some illustrative exam- ples. Moreover, we study the notions of forward and backwards stability and their implications. Part V gives the problem set of the workshop associated to the course we gave in the summer term 2012 together with some key results and lessons learnt from this experiment in higher education. Chapter 17 focuses on the didactic aspects of the workshop. We discuss the integration of workshop as a central part of the complete course. Details on the design of the workshop are presented covering in particular the concept of a virtual software company, the choice of the environment, and the team role descriptions. Chapter 18 contains the project specification used in the workshop. We present a selection of example results which our students produced at the end of the project. Finally, we summarise the lessons learnt—both from the point of view of the participants and the supervisors—providing interesting hints for future or similar projects. Preface xi
- 20. Tobias Neckel & Florian Rupp Acknowledgments A number of chapters of this book have been read and criticized in manuscript. In alphabetical order, we would like to acknowledge the sup- port and comments of Michael Bader, Peter Gamnitzer, Miriam Mehl, Philipp Neumann, Horst Osberger, Alfredo Parra, Benjamin Peherstorfer Christoph Riesinger, Konrad Waldherr, and last but not least Jonathan Zinsl. We thank our student assistants Andreas Hauptmann, Veronika Ostler, and Alexander Wietek for their support in preparing the course. Furthermore, we thank Thorsten Knott, who thoroughly designed the tasks in the context of solving the Poisson equation via continuous Fourier transform in Sec. 9.6.1, as well as Michael Bader for providing basic course material w.r.t. the Fourier transform and, in particular, the space-filling curves easing the development of the corresponding chapters. Peter Gamnitzer contributed via fruitful dis- cussions and valuable hints in the context of structural dynamics in Chap. 2, a help that is gratefully acknowledged. Special thanks go to Alfredo Parra Hinojosa for various contributions, in particular concerning the simulation routines for random ordinary differential equations in Chap. 14. We are very grateful to Versita’s publishing team and our editors Aleksan- dra Nowacka-Leverton, Marcin Marciniak, and Grzegorz Pastuszak for realising this book in the open access format. Garching bei München, Tobias Neckel and Florian Rupp June 26, 2013 xii Preface
- 21. Random Differential Equations in Scientific Computing Contents Preface I Motivation and Decomposition of Multi-Storey Building Excitation Problems 1 Recap: Random Variables & Stochastic Processes 1.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Random Variables, Generated σ-Algebras and Density Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.1 Continuity, Measures and Probability Spaces . . . . . . . . . 7 1.2.2 Random Variables and the σ-Algebras They Generate . . 9 1.2.3 Density and Distribution Functions . . . . . . . . . . . . . . . . . . 11 1.3 Moments and Integrals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.3.1 (Central) Moments and Moment Generating Functions . 16 1.3.2 Integration with Respect to a Probability Measure . . . . . 20 1.4 Independence and Conditional Expectation . . . . . . . . . . . . . . . . 21 1.4.1 Conditional Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.4.2 Further Properties of the Conditional Expectation . . . . . 27 1.4.3 Convergence Concepts for Sequences of Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 1.5 A Primer on Stochastic Processes . . . . . . . . . . . . . . . . . . . . . . . . . . 33 1.5.1 Continuous Stochastic Processes . . . . . . . . . . . . . . . . . . . . 36 1.5.2 Filtrations, Martingales and Super-Martingales . . . . . . . . 43 1.5.3 Gaussian Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 1.6 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 2 Reduction of RPDEs to RODEs 2.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Contents xiii
- 22. Tobias Neckel & Florian Rupp 2.2 Elastic Materials & Material Laws . . . . . . . . . . . . . . . . . . . . . . . . . . 56 2.2.1 Basic Aspects of Continuum Mechanics . . . . . . . . . . . . . . 57 2.2.2 Stress & Strain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2.2.3 Fundamental Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 2.3 Spatial and Temporal Discretisation of PDEs . . . . . . . . . . . . . . . . 65 2.3.1 From Space-Time to Space & Time . . . . . . . . . . . . . . . . . . 66 2.3.2 Spatial Discretisation: Meshing . . . . . . . . . . . . . . . . . . . . . 68 2.3.3 Spatial Discretisation: Operators . . . . . . . . . . . . . . . . . . . . 72 2.4 Finite Difference Approximations . . . . . . . . . . . . . . . . . . . . . . . . . . 73 2.4.1 General Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 2.4.2 Quality of FD Approximations . . . . . . . . . . . . . . . . . . . . . . . 76 2.4.3 FD Approximations for Elastic Body Motion . . . . . . . . . . . 77 2.5 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 3 Path-Wise Solutions of RODEs 3.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 3.2 Stochastic Processes as Models for External and Ground Motion Excitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 3.2.1 Brown’s Experiment & White Noise . . . . . . . . . . . . . . . . . . 90 3.2.2 Stochastic Models for Earthquake Excitations . . . . . . . . . 94 3.3 Random Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 3.3.1 Counterexamples for Path-Wise Solutions . . . . . . . . . . . . 102 3.3.2 Connections between Random and Stochastic Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 3.4 Path-Wise Solutions of Random Differential Equations . . . . . . . 106 3.4.1 Path-Wise Solutions in the Extended Sense . . . . . . . . . . . 110 3.4.2 Dependence on Parameters and Initial Conditions . . . . . 117 3.5 Excursion: Deterministic Description of the Vibrations of Single & Multi-Storey Buildings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 3.5.1 Vibrations of a Single-Storey Building . . . . . . . . . . . . . . . . 123 3.5.2 Vibrations of a Multi-Storey Building . . . . . . . . . . . . . . . . . 125 3.6 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 4 Path-Wise, P- & Mean-Square Solutions of RODEs 4.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 xiv Contents
- 23. Random Differential Equations in Scientific Computing 4.2 P-Solutions of Random Differential Equations . . . . . . . . . . . . . . . 134 4.3 Review: Mean-Square Analysis of Second Order Processes . . . . 136 4.4 Mean-Square Solutions of Random Differential Equations . . . . 139 4.5 Excursion: A Primer on Itô’s Stochastic Calculus . . . . . . . . . . . . . 146 4.5.1 Integration with Respect to White Noise . . . . . . . . . . . . . 149 4.5.2 Introducing the 1D Itô & Stratonovich Stochastic Integral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 4.6 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 5 RDEs in Science & Engineering: Randomly Perturbed Flow Problems 5.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 5.2 Derivation of the Deterministic Navier-Stokes Equations . . . . . 169 5.3 Numerical Solution of the Navier-Stokes Equations . . . . . . . . . . 176 5.4 Random Perturbation of Incompressible Flow . . . . . . . . . . . . . . . 179 5.5 Extension to Other Flow Problems . . . . . . . . . . . . . . . . . . . . . . . . . 181 5.6 Chapter’s Summary and Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . 182 II The Path-Wise Deterministic Setting 6 Recap: Theory of Ordinary Differential Equations (ODEs) 6.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 6.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 6.2.1 Vector Fields and Their Representation . . . . . . . . . . . . . . 191 6.2.2 Technical Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 6.3 Integral Curves in Vector Fields: Ordinary Differential Equations - Part I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 6.3.1 Approximate Solutions & Prerequisites for the Proof of the Cauchy-Peano Existence Theorem . . . . . . . . . . . . . . . 199 6.3.2 The Cauchy-Peano Existence Theorem . . . . . . . . . . . . . . . 203 6.4 Integral Curves in Vector Fields: Ordinary Differential Equations - Part II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 6.4.1 Local Existence & Uniqueness of Solutions . . . . . . . . . . . 207 6.4.2 Interlude: Solving ODEs Symbolically with MATLAB . . . . 210 6.4.3 Maximal Integral Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 Contents xv
- 24. Tobias Neckel & Florian Rupp 6.4.4 Maximal Integral Curves in Time-Independent Vector Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 6.4.5 Systems of 1st Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 6.5 Solutions of ODEs in the Extended Sense . . . . . . . . . . . . . . . . . . . 222 6.5.1 The Theorem of Caratheodory . . . . . . . . . . . . . . . . . . . . . . 223 6.5.2 Maximum & Minimum Solutions . . . . . . . . . . . . . . . . . . . . 226 6.6 Linear Ordinary Differential Equations . . . . . . . . . . . . . . . . . . . . . 230 6.6.1 Existence & Uniqueness of Solutions . . . . . . . . . . . . . . . . 231 6.6.2 Construction of Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . 234 6.7 First Integrals & Oscillations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 6.7.1 Application 1: The General Oscillation Equation . . . . . . . 240 6.7.2 Application 2: The Deterministic Pendulum . . . . . . . . . . . 241 6.7.3 Application 3: The Volterra-Lotka System . . . . . . . . . . . . . 245 6.8 Ordinary Differential Equations on Banach Spaces . . . . . . . . . . . 247 6.8.1 Existence & Uniqueness of Solutions . . . . . . . . . . . . . . . . 248 6.8.2 Extension of Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 6.8.3 Linear Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 6.9 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 7 Recap: Simulation of Ordinary Differential Equations 7.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 7.2 General Aspects of Numerical Solution of ODEs . . . . . . . . . . . . . 266 7.3 Explicit One-Step Methods for ODEs . . . . . . . . . . . . . . . . . . . . . . . 269 7.3.1 Explicit Euler Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 7.3.2 Heun’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 7.3.3 Explicit Runge-Kutta Schemes . . . . . . . . . . . . . . . . . . . . . . 271 7.3.4 Consistency & Convergence . . . . . . . . . . . . . . . . . . . . . . . . 273 7.4 Implicit Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276 7.4.1 Stiff ODEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278 7.4.2 Implicit Euler Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280 7.4.3 Trapezoidal Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 7.4.4 Consistency & Convergence . . . . . . . . . . . . . . . . . . . . . . . . 282 7.5 Excursion: The Newmark Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . 285 7.6 Excursion: Symplectic Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 7.7 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 xvi Contents
- 25. Random Differential Equations in Scientific Computing 8 Deterministic Dynamical Systems and Stability of Solutions 8.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 8.2 Continuous Dynamical Systems from ODEs . . . . . . . . . . . . . . . . . 306 8.2.1 Long-time Behavior, Invariance and Attraction . . . . . . . . 308 8.3 Lyapunov Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310 8.3.1 The Method of Lyapunov-Functions . . . . . . . . . . . . . . . . . 311 8.3.2 La Salle’s Principle & its Implications . . . . . . . . . . . . . . . . 315 8.4 Structural Stability & Linearisation . . . . . . . . . . . . . . . . . . . . . . . . 317 8.4.1 The Principle of Linearized Stability & the Theorem of Hartman-Grobman . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 8.4.2 The Routh-Hurwitz Stability Criterion . . . . . . . . . . . . . . . . 321 8.4.3 The Lozinskii-Measure and Stability . . . . . . . . . . . . . . . . . 326 8.5 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330 III Efficient Data Structures & the Propagation of Random Excitations 9 Fourier-Transform 9.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 9.2 The Continuous Fourier Transform . . . . . . . . . . . . . . . . . . . . . . . . . 341 9.3 The Discrete Fourier Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 9.3.1 Definition of the DFT and IDFT . . . . . . . . . . . . . . . . . . . . . . 342 9.3.2 MATLAB Examples for DFT and IDFT . . . . . . . . . . . . . . . . . . 347 9.3.3 DFT in Higher Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . 348 9.4 The Fast Fourier Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350 9.4.1 FFT Idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350 9.4.2 MATLAB Examples: Recursive and Iterative FFT . . . . . . . . 352 9.4.3 Outlook: FFT Variants and Libraries . . . . . . . . . . . . . . . . . . 355 9.5 Variants of Fourier Transforms via Symmetry Properties . . . . . . 356 9.5.1 The Discrete Sine Transform . . . . . . . . . . . . . . . . . . . . . . . . 360 9.5.2 The Discrete Cosine Transform . . . . . . . . . . . . . . . . . . . . . . 363 9.6 Solving the Poisson Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 9.6.1 Fourier’s Method for Partial Differential Equations . . . . . 366 9.6.2 Fast Poisson Solver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369 9.7 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370 Contents xvii
- 26. Tobias Neckel & Florian Rupp Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 10 Noise Spectra and the Propagation of Oscillations 10.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373 10.2 Spectral Properties of Stationary & Periodic Processes . . . . . . . 374 10.2.1 Stochastic Integration & the Spectral Representation Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 10.2.2 Stationary & Periodic Processes . . . . . . . . . . . . . . . . . . . . . 376 10.3 Energy & Power Spectral Density, and Examples for Colored Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380 10.3.1 A More Realistic Model for Brown’s Observation: The Ornstein-Uhlenbeck Process . . . . . . . . . . . . . . . . . . . . . . . . 384 10.3.2 Power-Law Noise & its Simulation . . . . . . . . . . . . . . . . . . . 387 10.4 The Frequency Domain Method for Response Analysis . . . . . . . 387 10.4.1 Propagation of Excitations . . . . . . . . . . . . . . . . . . . . . . . . . 391 10.4.2 Linear Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 10.5 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 11 Space Filling Curves for Scientific Computing 11.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399 11.2 The Concept of Space-filling Curves . . . . . . . . . . . . . . . . . . . . . . . 400 11.3 Applications of Space-filling Curves . . . . . . . . . . . . . . . . . . . . . . . 403 11.4 Computational Construction of Space-filling Curves . . . . . . . . . . 408 11.4.1 Grammar-based Construction . . . . . . . . . . . . . . . . . . . . . . . 408 11.4.2 Arithmetisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 11.5 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418 IV Path-Wise Solutions of Random Differential Equations and Their Simulation 12 Linear RODEs with Stochastic Inhomogeneity 12.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 12.2 The General Solution Formula . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424 12.2.1 Stochastic Properties of Path-Wise Solution . . . . . . . . . . 426 12.2.2 The Special Case of a Gaussian Inhomogeneity . . . . . . . . 430 xviii Contents
- 27. Random Differential Equations in Scientific Computing 12.3 Periodic and Stationary Solutions . . . . . . . . . . . . . . . . . . . . . . . . . 431 12.3.1 Existence of Periodic and Stationary Solutions . . . . . . . . 432 12.3.2 Convergence Towards Periodic and Stationary Solutions 439 12.4 Higher-Order Linear Random Differential Equations . . . . . . . . . . 444 12.5 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 452 13 Linear RODEs with Stochastic Coefficients 13.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458 13.2 The General Solution Formula . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461 13.3 Asymptotic Properties of Path-Wise Solutions . . . . . . . . . . . . . . . 465 13.3.1 (Exponential) Decay of Path-Wise Solutions . . . . . . . . . . 465 13.3.2 Boundedness of Path-Wise Solutions . . . . . . . . . . . . . . . . 477 13.4 Asymptotic Properties of the Moments of Path-Wise Solutions 479 13.4.1 Exponential Decay of the Moments . . . . . . . . . . . . . . . . . . 479 13.4.2 Periodic & Stationary Solutions . . . . . . . . . . . . . . . . . . . . . 482 13.5 The Solution Formula for Linear Non-Commutative Colored Noise Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485 13.6 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498 14 Simulating Path-Wise Solutions 14.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503 14.2 Discretisation Error of Explicit One-Step Methods for RODEs . . 504 14.3 Lower-Order Schemes for Random Differential Equations . . . . . 505 14.3.1 The Euler & Heun Schemes for RODEs . . . . . . . . . . . . . . . 505 14.3.2 MATLAB Examples for Hybrid Deterministic & Averaged Euler & Heun Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508 14.3.3 Numerical Results for Euler & Heun Schemes . . . . . . . . . 513 14.4 Higher-Order Schemes through Implicit Taylor-like Expansions 515 14.4.1 The K-RODE Taylor Schemes for RODEs . . . . . . . . . . . . . . 515 14.4.2 MATLAB Examples for the K-RODE Taylor Scheme . . . . . 520 14.4.3 Numerical Results for K-RODE Taylor Schemes . . . . . . . . 520 14.5 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522 Contents xix
- 28. Tobias Neckel & Florian Rupp 15 Stability of Path-Wise Solutions 15.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525 15.2 Stability Notations for Path-Wise Solutions . . . . . . . . . . . . . . . . . 527 15.2.1 The Zoo of Stochastic Stability Concepts . . . . . . . . . . . . . 528 15.2.2 Relations Between the Different Stability Notions . . . . . 531 15.2.3 Stability of Path-Wise Solutions of Linear RODEs . . . . . . 535 15.3 Lyapunov-Functions and Stability of Solution of RODEs . . . . . . 536 15.3.1 Lyapunov-Functions and h-Stability . . . . . . . . . . . . . . . . . 536 15.3.2 Lyapunov-Functions and Path-Wise Equi-Stability . . . . . 544 15.4 Excursion: Stability Subject to Continuously Acting Perturbations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547 15.5 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554 16 Random Dynamical Systems 16.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556 16.2 Definition of a Random Dynamical System . . . . . . . . . . . . . . . . . . 557 16.3 Stability and Lyapunov-Functions . . . . . . . . . . . . . . . . . . . . . . . . . 562 16.3.1 Forward Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562 16.3.2 Backwards Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 564 16.4 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566 V The Workshop Project 17 The Workshop Idea 17.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571 17.2 Integration of the Workshop in the Course . . . . . . . . . . . . . . . . . . 571 17.3 Design of the Workshop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573 17.3.1 The Concept of a Virtual Software Company . . . . . . . . . . 573 17.3.2 Choice of the Workshop Environment . . . . . . . . . . . . . . . . 573 17.3.3 Team and Role Descriptions . . . . . . . . . . . . . . . . . . . . . . . . 574 17.4 Chapter’s Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576 18 The Workshop Project: Stochastic Excitations of Multi-Storey Buildings 18.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 577 xx Contents
- 29. Random Differential Equations in Scientific Computing 18.2 Project Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 578 18.3 Project Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592 18.4 Lessons Learnt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597 18.4.1 General Impressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597 18.4.2 Feedback of Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . 599 18.4.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 600 18.5 Outlook: Extension to Future Projects . . . . . . . . . . . . . . . . . . . . . . 600 Index Bibliography Contents xxi
- 30. Tobias Neckel & Florian Rupp xxii
- 31. Part I Motivation and Decomposition of Multi-Storey Building Excitation Problems 1
- 32. He who seeks for methods without having a definite problem in mind seeks for the most part in vein. D H (1864 - 1943)
- 33. Random Differential Equations in Scientific Computing Chapter 1 Recap: Random Variables & Stochastic Processes This chapter provides a friendly review of the central concepts of probability theory focusing on random variables and their properties that eventually lead to the notion of a stochastic process. Our aim is to recall the basic definitions and equip them with tailored illustrations and MATLAB commands rather than emphasize the most general and abstract mathematical concepts. 1.1 Key Concepts This chapter sums up the material on continuous random variables and stochastic processes suitable for an undergraduate/ beginning graduate lec- ture, see [62, 25, 61, 122, 14]. The blending of MATLAB commands into the text is motivated by [79], [153], and [177]. As illustrative introductory examples, we motivate the concepts of (i) con- vergence of random variables that will be essential to set-up stochastic sta- bility as well as (ii) of ergodicity of stochastic processes. Example 1.1 (Convergence of Random Variables, cf. [225], p. 148). Let {Xi}n i=1 be a sequence of normally distributed random variables with van- ishing mean and variance i−1, i.e. Xi ∼ N(0, i−1). Figure 1.1 displays the (cu- mulative) distribution functions of the first elements of this sequence. Based on this figure, it seems as if limi→∞ Xi = X with the limiting random variable X ∼ PointMass(0). Though, P(Xi = X) = 0 for any i, since X ∼ PointMass(0) is a discrete ran- dom variable with exactly one outcome and Xi ∼ N(0, i−1) is a continuous random variable for any i ∈ N. In other words, a continuous random variable, such as Xi, has vanishing probability of realising any single real number in its support. Thus, we need more sophisticated notions of convergence for sequences of random variables. Example 1.2 (Ergodic & Non-Ergodic Stochastic Processes). Figure 1.2 shows some sample paths of a parameter-dependent stochastic process Xt, called “geometric Brownian motion”, cf. [222]. In particular, Xt = X0 exp (( a − 1 2 b2 ) t + bWt ) , Chapter 1 3
- 34. Tobias Neckel & Florian Rupp −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 0 0.2 0.4 0.6 0.8 1 x−axis value of the cumulative distribution function σ 2 = 1 σ 2 = 0.1 σ2 = 0.01 σ 2 = 0.001 Figure 1.1. Cumulative distribution functions of several normally distributed random variables N(µ, σ2 ) with µ = 0 and σ2 = 1, 10−1 , 10−2 , 10−3 . with initial value X0 at time t = 0, a, b ∈ R, and a (standard) Wiener process Wt (we will discuss this fundamental stochastic process in Chap. 3.2.1). The geometric Brownian motion is, for instance, used to model stock prices in the famous Black-Scholes model and is the most widely used model of stock price behavior. With some deeper understanding of the properties of the geometric Brow- nian motion we see on the one hand that the expected value follows the de- terministic exponential function E(Xt) = exp(at). On the other hand, though, its path’s converge to zero for all a < 1 2 b2. I.e., for a ∈ (0, 1 2 b2) we are in the paradoxical situation that the expectation of the process diverges and all its samples converge to zero-solution. Processes for which statistical properties, like the expected value, can be derived from the sample paths are called “ergodic”. In Fig. 1.2 (a) and (b), sample averages will provide an excellent estimator for the expected value. This is not the case in the non-ergodic case displayed in Fig. 1.2 (c) and (d). When reading this chapter note the answers to the following questions 1. What is a (real valued) stochastic process over a probability space? 2. What does expectation and variance of a random variable or a stochastic process tell us? 3. What does conditional expectation of a random variable or a stochastic process mean? 4. Which concepts describe the convergence of one random variable to- wards another? 5. What does the Borel-Cantelli lemma state? 4 Chapter 1
- 35. Random Differential Equations in Scientific Computing Time [t] 0.0 0.2 0.4 0.6 0.8 1.0 1.0 1.5 2.0 2.5 exp(0.8t) a = 0.8, b = 0.4 a = 0.8, b = 0.25 a = 0.8, b = 0.1 68,2% conf. int. for b = 0.25 (a) Time [t] 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0 2.5 exp(−0.8t) a = −0.8, b = 0.4 a = −0.8, b = 0.25 a = −0.8, b = 0.1 68.2% conf. int. for b = 0.25 (b) Time [t] 0 1 2 3 4 0.0 0.5 1.0 1.5 2.0 2.5 3.0 exp(0.8t) a = 0.8, b = 4.4 a = 0.8, b = 2.25 a = 0.8, b = 1.1 68.2% conf. int. for b = 2.25 (c) Time [t] 0 1 2 3 4 0.0 0.5 1.0 1.5 2.0 2.5 3.0 exp(0.1t) a = 0.1, b = 4.4 a = 0.1, b = 2.25 a = 0.1, b = 1.1 68.2% conf. int. for b = 2.25 (d) Figure 1.2. Paths of the Geometric Brownian Motion for a = 0.8, X0 = 1 (a) a = −0.8, X0 = 2 (b), a = 0.8, X0 = 1 (c), and a = 0.1, X0 = 1. In (a) and (b) the values of b are the same (b = 0.4, 0.25, 0.1) and the 68.2% confidence interval for the b = 0.25 paths is shown as well as the expectation value X0 exp(at). In (c) and (d) the values of b are the same (b = 4.4, 2.25, 1.1) and the 68.2% confidence interval for the b = 2.25 paths is shown as well as the expectation value X0 exp(at). 6. Under what conditions are two stochastic processes indistinguishable? 7. What are the characteristics of Gaussian processes? 8. How can we utilize MATLAB to simulate random variables, stochastic pro- cesses and their properties? as well as the following key concepts 1. σ-algebras, probability measures and probability spaces, 2. Random variables as well as their density, distribution and moment func- tions, Section 1.1 5
- 36. Tobias Neckel & Florian Rupp 3. Independence and conditional expectation, 4. Convergence in distribution, in probability, in the r-th mean, almost sure convergence and sure convergence, 5. Stochastic processes, their continuity and indistinguishability, 6. Kolmogorow’s fundamental and continuity theory, 7. Martingales, super-martingales, and filtrations, and 8. Gaussian processes. This chapter is structured as follows: In Sec. 1.2, we start with the fun- damental concepts of random variables, generated σ-algebras and density functions. Next, Section 1.3, discusses moments of random variables like the expectation value and variance as well as integration with respect to proba- bility measures. In Sec. 1.4, the essential concepts of independence of ran- dom variables and conditional probabilities and conditional expectation are studied. In particular, it is here that we give the various definitions of conver- gence of random variables. Moreover, in Sec. 1.5, we give the basic defini- tions and concepts of continuous stochastic processes together with a brief discussion of Gaussian processes. Finally, Section 1.6 wraps up the contents of this chapter. Prerequisites: Some pre-knowledge on probability theory and stochastic processes are helpful. Teaching Remarks: Though labeled chapter 1, we certainly do not suggest to start a course for beginning graduate students with the basics presented in this chapter as its contents are heavily loaded with technical definitions that are not very motivating for the student interested in applications. In view of our top-down approach, we assume the concepts of this chapter as pre- requisites to be considered in a lecture after chapters 3 or 5 when required. For a lecture class, it seems to be most appropriate to give this chapter as a homework and discuss some relevant exercises together in the classroom. 1.2 Random Variables, Generated σ-Algebras and Density Functions With respect to the key elements and notations of probability theory we start with the introduction of random variables and especially the σ-algebras they generate. Next, density and distribution functions will be discussed followed by the definition of (central) moments and moment generating functions. Fi- nally, we define what we mean by integration with respect to a probability measure and give some useful inequalities. 6 Chapter 1
- 37. Random Differential Equations in Scientific Computing 1.2.1 Continuity, Measures and Probability Spaces Let us first recall the definition of Hölder- and Lipschitz-continuity as well as that of Ck,α-functions, cf. for instance [3], p. 40: Definition 1.3 (Hölder and Lipschitz Continuity/ Ck,α-functions). Let (X, ∥ · ∥X), (Y, ∥ · ∥Y ) be normed spaces and 0 < α ≤ 1. A function f : X → Y is called globally Hölder continuous of order α if there is a positive constant C such that ∥f(x) − f(y)∥Y ≤ C∥x − y∥α X ∀ x, y ∈ X . (1.1) f is called locally Hölder continuous of order α if it satisfies the condition (1.1) on every bounded subset of X. f is called globally (or locally) Lipschitz continuous if it is globally (or locally) Hölder continuous of order α = 1. f is called a Ck,α-function if it is k times continuously differentiable and the k-th derivatives are locally Hölder continuous of order α for some k ∈ N. The central problem in measure theory is to find a measure/ volume for as many elements of the power set P(Rd) as possible, such that this mea- sure/ volume is additive, translation invariant and normalized. As there is no solution to define a measure/ volume for all elements of P(Rd), we have to restrict ourselves to special sub-set systems: Definition 1.4 (σ-Algebra). Let Ω be a nonempty set. A collection of sets A ⊂ P(Ω) is called σ-algebra, if • A is a algebra, i.e., – Ω ∈ A, – A ∈ A ⇒ Ac ∈ A and A, B ∈ A ⇒ A ∪ B ∈ A • ∀n ∈ N : An ∈ A ⇒ ∪n∈NAn ∈ A Trivial examples for σ-algebras are A = {∅, Ω} and A = P(Ω), moreover, for any A ⊂ Ω the σ-algebra properties of A = {∅, A, Ac, Ω} are easily verified. In particular, if E is a collection of subsets of Ω, then the smallest σ-algebra generated by E, and denoted by σ(E), is defined as σ(E) := ∩ {A : E ⊂ A and A is a σ-algebra on Ω} . For instance, the smallest σ-algebra containing all open subsets of Rd is called the Borel σ-algebra, denoted by Bd or simply by B if the dimension d requires no specific mentioning. Let Ω be a nonempty set and E ⊂ P(Ω). The set-system E is called intersection-stable, if ∀ E1, E2 ∈ E =⇒ E1 ∩ E2 ∈ E . Section 1.2 7
- 38. Tobias Neckel & Florian Rupp Obviously, every σ-algebra is intersection-stable1. Let Ω be a nonempty set and A be a σ-algebra on Ω. The pair (Ω, A) is called measurable space and the elements of A are called measurable sets. Definition 1.5 (Measurable Function). Let (A, A) and (B, B) be measurable spaces. A function f : A → B is called A-B-measurable, if f−1(B) ⊂ A. For instance, every continuous function f : X → Y between two metric (or topological) spaces X and Y is measurable. Definition 1.6 (Measure and Probability Measure). Let Ω be a nonempty set and A be a σ-algebra on Ω. Then a set-function µ on A is called a measure, if • µ(A) ∈ [0, ∞] for all A ∈ A, • µ(∅) = 0, • µ is σ-additive, i.e., for any disjoint collection of sets A1, A2, · · · ∈ A with ∪n∈NAn ∈ A it holds that µ ( ∪ n∈N An ) = ∞ ∑ n=1 µ(An) . Moreover, a measure µ is called a probability measure if it additionally satis- fies • µ(Ω) = 1. A measure µ on a measurable space (Ω, F) is called σ-finite, if there exist E1, E2, · · · ∈ F, pairwise disjoint, s. t. Ω = ∪n∈NEn and µ(En) < ∞ for all n ∈ N. Moreover, for two measures µ, ν on a measurable space (Ω, F), the measure ν is called absolutely continuous with respect to µ, if every µ-nullset is a ν-nullset. The notation for this property is ν ≪ µ. If µ is a measure on the σ-algebra A of a measurable space (Ω, A), then the triplet (Ω, A, µ) is called measure-space. In particular: Definition 1.7 (Probability Space). Let Ω be a nonempty set and A be a σ- algebra on Ω. The triplet (Ω, A, P) is called probability space, if P is a proba- bility measure on the measurable space (Ω, A). Let (Ω, A, P) be a probability space: points ω ∈ Ω are usually addressed as sample points and a set A ∈ A is called event, hereby P(A) denotes the probability of the event A. 1 In an extended course on measure theory, the property of set-systems to be intersection- stable motivates the discussion of Dynkin systems, see, e.g., [93], pp. 24. 8 Chapter 1
- 39. Random Differential Equations in Scientific Computing 0 1 p X(ω) = x for an element ω of A x P(A) = p = P({ω : X(ω) = x}) R A := {ω : X(ω) = x} ω1 ω2 X(ω1) X(ω2) Ω B := {ω : X(ω) = y} y Figure 1.3. Relation between a random variable X and its probability function, fol- lowing [62], p. 53. A property which is true except for an event of probability zero is said to hold almost surely (abbreviated ”a.s.”) or almost everywhere (abbreviated ”a.e.”). 1.2.2 Random Variables and the σ-Algebras They Generate Probabilities are a set-functions that assign a number between 0 and 1 to a set of points of the sample space Ω, cf. [62], pp. 53. Their domain is the set of events of a random experiment and their range is contained in the inter- val [0, 1]. A random variable is also a function whose range is a set of real numbers, but whose domain is the set of sample points ω ∈ Ω making up the whole sample space Ω (not subsets of Ω), see Fig. 1.3. Definition 1.8 (Random Variable). Let (Ω, A, P) be a probability space. Then a function X : Ω → Rd is called random variable, if for each Borel-set B ∈ B ⊂ Rd X−1 (B) = {ω ∈ Ω : X(ω) ∈ B} ∈ A . I.e., a random variable is a Rd-valued A-measurable function on a probability space (Ω, A, P). We usually write X and not X(ω). This follows the custom within proba- bility theory of mostly not displaying the dependence of random variables on the sample point ω ∈ Ω. We also denote P(X−1(B)) as P(X ∈ B), the probability that X is in B ∈ B. Example 1.9 (Indicator and Simple Functions are Random Variables). Let A ∈ A. Then the indicator function of A, IA(ω) := { 1 if ω ∈ A 0 if ω / ∈ A Section 1.2 9
- 40. Tobias Neckel & Florian Rupp is a random variable. More generally, if A1, A2, . . . , An ∈ A are disjoint sets, such that Ω = ˙ ∪ n i=1Ai and a1, a2, . . . , am ∈ R, then X = n ∑ i=1 aiIAi is a random variable, called a simple or elementary function. Example 1.10. Sums and products of random variables are themselves ran- dom variables, too. MATLAB can generate distributed pseudo-random numbers between 0 and 1 can be generated utilizing the rand command. For instance, the code below generates a one row vector with five column entries the values of which are uniformly distributed on the interval [1, 10]: r = 1 + (10-1).*rand(1,5); r = 7.8589 5.1082 1.1665 8.3927 5.0023 The MATLAB manual says: “MATLAB software initializes the random number generator at startup. The generator creates a sequence of [pseudo-]random numbers called the global stream. The rand function accesses the global stream and draws a set of numbers to create the output. This means that every time rand is called, the state of the global stream is changed and the output is different.” Due to this algorithmic procedure, pseudo-random num- bers are created that depend on the initial value/ seed of the generation al- gorithm. This initial value/ seed can be controlled to allow for a repetition of the generated sequence of pseudo-random numbers. The randn command generates normally distributed pseudo-random numbers. For instance, randn(’state’,100) % set the initial r = 1 + 2.*randn(1,5); % value/ seed of randn r = 2.8170 -3.4415 0.5219 1.1375 -3.0404 randn(’state’,100) % reset the initial r = 1 + 2.*randn(1,5); % value/ seed of randn r = 2.8170 -3.4415 0.5219 1.1375 -3.0404 generates two identical one row vectors with five column entries the values of which are normally distributed with mean 1 and standard deviation 2. In both cases the same initial value/ seed was applied to MATLAB’s random number generator. 10 Chapter 1
- 41. Random Differential Equations in Scientific Computing Lemma 1.11 (σ-Algebras Generated by Random Variables). Let X : Ω → Rd be a random variable on the probability space (Ω, A, P). Then A(X) := {X−1 (B) : B ∈ B} is a σ-algebra, called the σ-algebra generated by X. This is the smallest sub-σ- algebra of A with respect to which X is measurable. Proof. It is easy to verify that {X−1(B) : B ∈ B} is a σ-algebra, and, more- over, that it is indeed the smallest σ-algebra of A with respect to which X is measurable. It is essential to understand that, in probabilistic terms, the σ-algebra A(X) can be interpreted as ”containing all relevant information” about the random variable X. In particular if a random variable Y is a function of the random variable X, i.e., if Y = Ψ(X), for some reasonable function Ψ, then Y is A(X)- measurable. Conversely, suppose Y : Ω → R is A(X)-measurable. Then, there exists a function Ψ such that Y = Ψ(X). Hence, if Y is A(X)- measurable, Y is in fact a function of X. Consequently, when we know the value X(ω), we in principle know also Y (ω) = Ψ(X(ω)), although we may have no practical way to construct Ψ. 1.2.3 Density and Distribution Functions Our discussion of properties of random variables starts with the one- dimensional setting: A scalar random variable X is associated to the (proba- bility) density function X 7→ f(X) defined by the property P(x1 ≤ X ≤ x2) = ∫ x2 x1 f(X)dX where P(x1 ≤ X ≤ x2) denotes the probability of the event x1 ≤ X ≤ x2. The (cumulative/ probability) distribution function F(X) is given by x0 7→ FX(x0) := P(X ≤ x0) = ∫ x0 −∞ f(X)dX I.e., the probability that x lies in the interval (x1; x2] is FX(x2) − FX(x1) with x1 < x2. Of course F(∞) = 1, see Fig. 1.4. The following theorem ensures the existence of a density: Theorem 1.1 (Radon-Nikodym). Let (Ω, F) be a measurable space and µ a σ- finite measure on F. Moreover, let ν ≪ µ be a measure. Then ν has a density Section 1.2 11
- 42. Tobias Neckel & Florian Rupp Probability Density Function Cummulative Density Function P(x1 < X < x2) x1 x2 x1 x2 1 Figure 1.4. Sketch of a density and its corresponding distribution function. Shown is the connection between the probability P(x1 < X < x2) and the graphs of these two functions. with respect to µ, i.e., there exists a measurable function f : (Ω, F) → ([0, ∞], B) such that for all E ∈ F it holds that ν(E) = ∫ E fdµ . Proof. see [93], pp. 277 In applications one often has to deal with data samples of random vari- ables and tries to gather some information about the density and distribu- tion of these random variables. The following MATLAB Example takes, for convenience, 1000 sample points of a normally distributed random variable with mean 3 and variance 25, i.e. of a N(3, 25)-random variable, and graphs its distribution curve using histograms (see Fig. 1.5 for a visualisation of the resulting plots). MATLAB Example 1.1. plotting.m: Generating the images visualised in Fig. 1.5. num = 1000; x = 3 + 5 . *randn (num, 1 ) ; figure ( 1 ) ; plot ( [ 1 : length ( x ) ] , x ) [ count bins ] = h i s t ( x , sqrt (num) ) ; figure ( 2 ) ; h i s t ( x , sqrt (num) ) figure ( 3 ) ; plot ( bins , count , ’−b ’ , ’ LineWidth ’ ,2) count_sum = cumsum( count ) ; figure ( 4 ) ; plot ( bins , count_sum , ’−b ’ , ’ LineWidth ’ ,2) 12 Chapter 1
- 43. Random Differential Equations in Scientific Computing 0 100 200 300 400 500 600 700 800 900 1000 −15 −10 −5 0 5 10 15 20 (a) −15 −10 −5 0 5 10 15 20 0 10 20 30 40 50 60 70 80 90 (b) −15 −10 −5 0 5 10 15 20 0 10 20 30 40 50 60 70 80 90 (c) −15 −10 −5 0 5 10 15 20 0 100 200 300 400 500 600 700 800 900 1000 (d) Figure 1.5. (a) 1000 sample points of a N(3, 25)-random variable, (b) histogram, (c) empirical density and (d) empirical distribution of these points. The density function of a random variable, the joint (probability) density function fX,Y (x, y) of two random variables X and Y is given analogously as P(x1 ≤ X ≤ x2, y1 ≤ Y ≤ y2) = ∫ x2 x1 ∫ y2 y1 fX,Y (x, y)dydx with fX,Y (x, y) = fY |X(y|x)fX(x) = fX|Y (x|y)fY (y) , where fY |X(y|x) and fX|Y (x|y) are the conditional densities of Y given X = x and of X given Y = y respectively, and fX(x) and fY (y) are the marginal densities for X and Y respectively2. In particular we have for the marginal densities: fX(x) = ∫ ∞ −∞ fX,Y (x, y)dy and fY (y) = ∫ ∞ −∞ fX,Y (x, y)dx . 2 E.g., the marginal density of X simply ignores all information of Y , and vice versa. Section 1.2 13
- 44. Tobias Neckel & Florian Rupp Two random variables are independent if the conditional probability distri- bution of either given the observed value of the other is the same as if the other’s value had not been observed, e.g. fY |X(y|x) = fY (y) In particular, two random variables are independent, if their joint density is the product of the marginal densities: fX,Y (x, y) = fX(x)fY (y) . Moreover, for independent random variables X, Y it holds, that E(X · Y ) = E(X) · E(Y ), Var(X + Y ) = Var(X) + Var(Y ) and Cov(X, Y ) = 0. For our two random variables X and Y , their joint distribution is the dis- tribution of the intersection of the events X and Y , that is, of both events X and Y occurring together. Consequently, the joint distribution function FX,Y (x, y) is given by FX,Y (x, y) = P(X ≤ x, Y ≤ y) = ∫ x −∞ ∫ y −∞ fX,Y (u, v)dvdu . In the case of only two random variables, this is called a bivariate distribution, but the concept (as well as that of the bivariate densities and independence) generalizes to any number of events or random variables. Example 1.12 (The Normal-Distribution). For a normally or Gaussian dis- tributed3 random variable X, the density f(x) is given by f(x) = 1 √ 2πσ exp ( − (x − µ)2 2σ2 ) , where µ and σ2 denote the mean value and the variance of X, respectively. To denote that a real-valued random variable X is normally distributed with mean µ and variance σ2, we write X ∼ N(µ, σ2). The relation between the joint density of two random variables, the marginal and conditional densities is sketched in Fig. 1.6. Note that in gen- eral, the conditional probability of X given Y is not the same as Y given X. The probability of both X and Y together is P(XY ), and if both P(X) and P(Y ) are non-zero this leads to a statement of Bayes Theorem: P(X|Y ) = P(Y |X) · P(X) P(Y ) , and P(Y |X) = P(X|Y ) · P(Y ) P(X) . Conditional probability is also the basis for statistical dependence and sta- tistical independence as we will see in Sec. 1.4. 3 The normal distribution was first introduced by Abraham de Moivre in an article in 1733, which was reprinted in the second edition of his ”The Doctrine of Chances”, 1738, in the context of approximating certain binomial distributions for large natural numbers. 14 Chapter 1
- 45. Random Differential Equations in Scientific Computing E(y) E(x) y x marginal density of y x = x0 conditional density of y, given x = x0 marginal density of x Figure 1.6. Relation between the joint, marginal and conditional densities. Before you continue, make sure to answer the following questions: Quiz: Section 1.2 Q1 Give the definitions of Hölder and Lipschitz continuity, and give an ex- ample of a Hölder-continuous function that is not Lipschitz-continuous. Q2 Give the definition of a σ-algebra, a probability measure, and a probabil- ity space. Give two examples for probability spaces. Q3 Give the definition of a random variable. Give two examples for random variables. Q4 How can σ-algebras be generated utilizing random variables? Q5 Give the definitions of the density and the distribution function of a ran- dom variable. Q6 What are conditional densities and marginal densities? Q7 What does the theorem of Radon-Nikodym state and why is it important? Section 1.2 15
- 46. Tobias Neckel & Florian Rupp 1.3 Moments and Integrals Particularly in (physical) experiments neither distributions nor densities are available easily throughout measurement processes; though the expectation and moments (as well as their properties) play an important role for those applications. 1.3.1 (Central) Moments and Moment Generating Functions Random variables can be described by their k-th moments which are defined bas: E(xk ) := ∫ ∞ −∞ xk f(x)dx and their k-th central moments, E((x − E(x))k ) := ∫ ∞ −∞ (x − E(x))k f(x)dx . The most important moments are the mean/ average value/ expected value/ 1st moment µ := E(x) = ∫ ∞ −∞ xf(x)dx and the variance/ 2nd central moment σ2 := Var(x) := E((x − µ)2 ) := ∫ ∞ −∞ (x − µ)2 f(x)dx , whereby the quantity σ is called the standard deviation. Example 1.13 (Moment Generating Function and First Moments of Normally Distributed Random Variables). For normally distributed random variables all higher moments (k > 2) can be expressed by the mean µ and the variance σ2. Given a real random variable X, the so-called moment generating function is defined as MX(t) := E(exp(tX)) Provided the moment generating function exists in an open interval around t = 0, the n-th moment is given as E(Xn ) = M (n) X (0) = dn MX(t) dtn t=0 . 16 Chapter 1
- 47. Random Differential Equations in Scientific Computing Thus, for a normally distributed random variable X, the moment generating function is MX(t) = E(exp(tX)) = ∫ ∞ −∞ 1 √ 2πσ exp ( − (x − µ)2 2σ2 ) exp(tx)dx = exp ( µt + 1 2 σ2 t2 ) . This leads to E(X) = µ, E(X2) = µ2 + σ2, E(X3) = µ3 + 3µσ2, . . . and E(X − µ) = 0, E((X − µ)2) = E(X2 − 2µX + µ2) = σ2, E((X − µ)3) = 0, . . . . Following [146], example 3.4, we apply the moment generating function to derive the geometric distribution: First, via MATALB, we obtain a closed form for the moment generating function by ML = simplify( symsum(exp(t*k)*p^k*q, k, 0, inf) ); pretty(ML) q - ------------ exp(t) p - 1 The first and second moments are generated by differentiation and substi- tution for t = 0 in the resultant expression. For the first moment this leads to MLP = limit(diff(ML), t, 0) MLP = 1/(p-1)^2*q*p We repeat the process for the second moment MLPP = limit(diff(ML,2), t, 0) MLPP = -q*p*(p+1)/(p-1)^3 The variance is now computed and simplified by noting q = 1 − p and substitution Var(X) = E(X2) − E(X)2: VARL = subs(MLPP, q, 1-p) - subs(MLP, q, 1-p)^2 pretty( simplify(VARL) ) p -------- 2 (p - 1) The first and second moments of a random vector x = (x1, . . . , xn)T are defined by µ := E(x) := (E(x1), . . . , E(xn))T Section 1.3 17
- 48. Tobias Neckel & Florian Rupp and by the covariance matrix (symmetric and positive definite) P := E((x − µ)(x − µ)T ) := E((x1 − µ1)(x1 − µ1)) E((x1 − µ1)(x2 − µ2)) . . . E((x1 − µ1)(xn − µn)) E((x2 − µ2)(x1 − µ1)) E((x2 − µ2)(x2 − µ2)) . . . E((x2 − µ2)(xn − µn)) . . . . . . ... . . . E((xn − µn)(x1 − µ1)) E((xn − µn)(x2 − µ2)) . . . E((xn − µn)(xn − µn)) . The diagonal elements E((xi − µi)2) of P are the variances and the off- diagonal element are the covariances of the vector components. The standard square deviation is given by the trace of P: tr(P) = E ( n ∑ i=1 (xi − µ)2 ) . For instance, with the covariance matrix P, the density of a normally dis- tributed n-vector is f(x1, . . . , xn) = 1 √ (2π)n √ det(P) exp ( −1 2 (x − µ)T P−1 (x − µ) ) . To generate a realisation of a multivariate Gaussian random variable X ∼ N(µ, P), P ∈ Rd×d we can proceed as follows. cf. [153], pp. 487: 1. Perform a Cholesky decomposition of P to yield the non-singular d × d- matrix G such that P = GGT . 2. Generate a realisation u ∈ Rd of a random vector U ∼ N(0, I) in Rd, where I = diag(1, 1, . . . , 1) ∈ Rd×d denotes the d × d unit matrix. 3. Form the realisation of X as x = Gu + µ. As an example, let us assume µ := 0 and P := 1 2/3 1/3 2/3 1 1/3 1/3 2/3 1 ⇒ G = 1 0 0 0.6667 0.7454 0 0.3333 0.5963 0.7303 . In Fig. 1.7, 100 realisations of X are plotted by use of the following MATLAB commands: 18 Chapter 1
- 49. Random Differential Equations in Scientific Computing −3 −2 −1 0 1 2 3 −4 −2 0 2 4 −3 −2 −1 0 1 2 3 4 Figure 1.7. Realisations of a 3-dimensional multivariate Gaussian random variable, cf. [153], p. 488. MATLAB Example 1.2. plotting3D.m: Generating Fig. 1.7. P = [1 2/3 1/3;2/3 1 2/3;1/3 2/3 1 ] ; G = chol ( P ) ’ ; % perform Cholesky decomposition % MATLAB produces P=A ’ * A so G=A ’ M = 200; for m = 1:M % generate r e a l i s a t i o n s of x u = [ randn (1 ,1) randn (1 ,1) randn (1 ,1) ] ’ ; x = G*u ; scatter3 ( x ( 1 ) , x ( 2 ) , x ( 3 ) , ’ f i l l e d ’ ) hold on ; end Cross-expectations of products of random variables are generally difficult to obtain, though the following proposition allows us to calculate an arbitrary product of normally distributed random variables4. Proposition 1.14 (Expectation of a Product of Normally Distributed Random Variables). Suppose X = (x1, x2, . . . , xn)T ∼ N(0, P), where P is an n × n- positive semi-definite matrix. For non-negative integers s1, s2, . . . , sn, we have E ( n ∏ i=1 xsi i ) = 0 if s is odd , 1 (s 2 )! ∑s1 ν1=0 · · · ∑sn νn=0 (−1) ∑n i=1 νi · · ( s1 ν1 ) . . . ( sn νn ) (1 2 hT Ph )s/2 if s is even , where s = s1 + s2 + · · · + sn and h = (1 2 s1 − ν1, 1 2 s2 − ν2, . . . , 1 2 sn − νn )T . 4 See, [138], too. In the physics literature such closed form solutions for the expectation of the product of normally distributed random variables are associated with Wick’s formula, cf. [150], p.546. Section 1.3 19
- 50. Another Random Scribd Document with Unrelated Content
- 51. Surprise and curiosity he plainly saw, but it was not so easy to discover the other. “Come, now, what have you been looking up my house for?” “On my honor, Joe, I’ve never set eyes on the building and don’t know whether it’s stone or brick, three story or two.” “Then what in the deuce—?” “Patience! Is your house in the market?” “Yes.” “Then perhaps it is one of a number given me by a real estate agent to look up for a friend of mine. I’ll preserve the slip,” taking it from Joe and folding it up. “It looks like a woman’s writing.” “Yes, all writing does after a man has fallen into the habit of looking for letters day by day—letters that are delayed—Come, you married men are very suspicious.” With that he dexterously whipped the subject around and began talking about something of decided interest, so that Joe, completely hoodwinked, speedily forgot about the singular little coincidence that had brought this address under the eyes of the owner of the house. He was not quite done with Joe yet. “You must own a good deal of property in and around the city, Joe?” “I do—property left to me by my mother.” “You have no need to work.” “Well, perhaps not. Some day when I take the notion I mean to figure up my income from this property, and if it’s a good sum, by Jove! I’ll fling business to the winds and take my little wife to Europe for a year—that is, if—”
- 52. Darrell did not let him finish. “Why, man alive, you talk as though you didn’t hardly know what property you owned, yourself.” “Neither do I—it’s all come to me since I married, and I’ve been so much taken up with my wife that I haven’t found time to attend to it as I should.” Darrell winked hard. He knew certain facts that would seem to indicate that Joe found time to spend an hour every afternoon with some one besides Lillian. If so then this was rank perjury. What was he to think of a hypocrite? “Jove! that’s a queer case. I don’t suppose your wife has any idea of where your property lies—never saw such places as this Twenty- seventh Street house, for instance?” “Heavens! no. That house is an eyesore to me. The neighborhood is not a good one and I will only let it to decent tenants. No, Lillian will never know I own a house there.” Darrell was satisfied. He had made his point. Soon after Joe bade him good evening, and hurried away. It was not far from five o’clock. Darrell snatched a disguise from a hook and changed his appearance in one minute. All he wanted was to effect such a change that Joe might not recognize him. Then he left the office and bolted down stairs after his friend.
- 53. Joe was discovered in the crowd, making his way toward the elevated station, and knowing his destination Darrell arrived there first. They got in the same car. At this time in the evening it was pretty crowded and both had to stand up. At Twenty-seventh Street a number left the train and those we follow with the rest. Darrell observed Joe eagerly consult his watch. “He’s late this evening and no doubt expects a scolding,” was his mental comment upon seeing the frown upon Joe’s usually good- natured face. The giant walked along so fast that Eric could hardly keep his place behind him. They approached the fatal number. Truly Joe acted like a guilty wretch—he glanced up and down the street as if to make sure no acquaintance was passing. Deception was a novelty to him—this was the first time Darrell had ever seen his friend acting in a mean role. When they reached the steps Joe ascended them, took a key out of his pocket and deliberately opened the front door. The detective was passing at the time, but his quick glance failed to reveal anything of interest. Evening was coming on, and the shadows of the approaching night had evidently gathered in the hall of the house—he could just see the glass globe of the hanging gas jet in the hall, but it was not lighted.
- 54. For that matter there was no light about the house at all, though the neighbors were beginning to illuminate their houses. Passing down the street a little distance, Eric Darrell crossed over, and came up the other side. He now noticed that there was a light in the second story front room, though almost ready to swear it had not been there previous to the entrance of the proprietor. The inside blinds were closed in such a way that Darrell could see nothing. He was deeply interested. Whatever this strange mystery attached to Joe’s daily visit here might mean, Darrell could not forget that the other was his friend. He would act as a surgeon might when one whom he regarded highly was brought before him for attention—his fingers would be very tender, but the cruel knife must do its duty. He was walking slowly along when he almost ran into a female who stood on the edge of the pavement opposite the house. Her black attire and the veil she wore attracted his attention immediately. Besides, she was looking upward toward the windows where the glimmer of light could be seen. A suspicion flashed into his mind. He touched the arm of the lady in black. “Lillian—Mrs. Leslie,” he said in a low voice. A cry came from under the veil. “Who speaks to me?” gasped the lady. “It is I—Eric Darrell. This is no place for a lady, especially at such an hour. You may be insulted here.”
- 55. “But he is here—Joe, my husband, and where he is his wife should not be afraid to go,” she said with some bitterness. “Theoretically true, madam, but there are lots of places in this wicked city where men daily pass and ladies dare not go. You promised to leave this to me and you must keep your word. Take my arm and let me see you to the elevated station.” She might have rebelled, but there was a touch of gentle but firm authority in his tone, and being a woman she yielded, knowing he was right. On the way to the elevated station she was silent, but finally, upon reaching the steps, she turned to her companion. “Mr. Darrell, does my husband know that I have sought your advice?” So intensely interested was she in the answer, that she even held her breath. “To my knowledge, Mrs. Leslie, Joe does not even suspect you of ever having seen me.” “Thank heaven,” she almost gasped, a world of relief showing itself upon her face, for, the better to look at her companion when expecting his answer, she had brushed her veil aside. “You need not borrow trouble on that score. Act naturally, as though you suspected nothing and had no reason to evade his eye.” She moved uneasily at his words. Darrell had spoken them with a purpose, just as the surgeon probes for the bullet before making any attempt to extract it. He believed he had met with a certain share of success too. “What did he want with you?” she asked, as if to cover her own confusion.
- 56. “Merely a matter of business.” “Did he mention me?” “He said I must come up and meet you sometime—whatever this may turn out, Mrs. Leslie, I know Joe fairly worships you—never doubt that fact. Some things seem hard to put together, but when the truth shines upon them they will be found very simple.” “Like Columbus and the egg, for example.” “Yes, indeed. Now, if at any time you and I should meet in Joe’s presence, don’t forget to treat me as a stranger.” “I will not.” “Then I shall say good evening, and as a last word, advise you to leave this to me.” “I shall, Mr. Darrell.” She flitted up the station stairs and Darrell, with a long sigh, turned down the street again. Somehow the pretty wife of his friend quite fascinated him, and he found himself wishing the sister would be like her. Walking down the street, he soon reached his old stamping ground. The light burned in the second story room and he believed Joe had not left the house. For perhaps ten minutes things went on this way. Then the light suddenly vanished. A minute later Joe Leslie came out. Darrell listened intently to see if he spoke to any one at the door but a wagon rattling by prevented his making sure.
- 57. Then Joe descended the steps and set briskly off for the elevated station. The detective did not follow him. He desired to do a little work around that region, and knew Joe was bound for home. The house seemed to be dark and deserted, but others were in the same condition, the shades being drawn and shutters closed. New York people, many of them, act as though their houses were meant to be dungeons, being hermetically sealed to shut out the light. Darrell surveyed the building a few minutes, crossed over, looked at it more closely, started up the steps, then shook his head negatively. “Not yet—I’ll wait a little,” he muttered. Glancing up and down the street he saw a small grocery store on the corner. People must eat, and these venders of daily provisions generally know more about those who live in the neighborhood than any other class. The gossip and small talk of the street passes current here, and the proprietor hears all. So Darrell made for the grocery. It was not a very extensive establishment—the owner and his clerk were not busy, and Darrell, picking out the former, asked: “Can you tell me who lives at No—?” The man looked at him with a smile. “A young woman named Mrs. Lester, whose husband I believe is in California—she was in here once or twice—quite a fine-looking lady,”
- 58. returned the groceryman. “Thanks,” replied the detective, turning and leaving the store as suddenly as he entered. “Jacob, what number did he ask about?” said the proprietor, turning to his clerk. The boy gave it, at which the other whistled. “That’s what they call a bull on me. I was five numbers out of the way. But let it pass. He didn’t want to buy nothing.” The blunder was destined to give Darrell trouble however.
- 59. CHAPTER V THE MAN DRESSED AS A BULL FIGHTER When Eric Darrell left the little grocery on the corner, it was with a bad feeling at his heart. It seemed as though a cold, clammy hand had suddenly come in contact with that member of his anatomy, and chilled it. Could this thing be? If Joe Leslie turned out to be that moral leper, a bigamist, Darrell believed he would never put any trust in human nature again. Did it not look like it? Nothing was lacking. Good heavens! even the names were almost alike—Leslie and Lester. He was horrified—dazed—dumfounded. Then his teeth came together with a snap, and he swore he would solve this mystery—the man might be living two lives—others had done it before—perhaps many in New York are doing it to-day. In his time Darrell had met with just such cases as this, and he believed his experience justified him in solving the puzzle. So her husband was in California. It was a likely story.
- 60. California must be very near by if he could drop in six times a week. He passed the house again and found that there were still no signs of light. Evidently those who lived there, perhaps enjoying the luxuries of the season, knew how to hide their light under a bushel. Darrell remembered what Joe had said—he had long since despaired of renting the house, and probably did not try very hard. Then again about his income—no wonder he did not know how he stood if he had to keep two separate establishments running. They might do that economically out in Salt Lake City among the Mormons but it is quite an expensive luxury in New York. So the detective made his way down to Twenty-third Street and entering a dairy kitchen where a thousand were being served to the music of an orchestra, had his dinner. He took his time over it, read the evening paper, and when he finally passed out it was well on to eight o’clock. Then he smoked a cigar and watched the passers by for half an hour more. Then he sauntered away. At nine o’clock he found himself one of a little crowd gathered at the door of a hall. A masquerade was to take place here, and as carriage after carriage drove up, depositing nymphs and devils, cavaliers and knights, upon the pavement, the crowd laughed in a good-natured way. Some of the rougher element might have indulged in jeers or remarks that would have brought on trouble, but for their fear of the law, which was represented by two stalwart policemen, armed with
- 61. their long night sticks which are a dread to the heathen of the slums. Darrell was interested too, and stood with the rest, looking on. While thus engaged, a gentleman and lady left a hack and walked toward the entrance. He represented a Spanish bull fighter, and with his splendid figure made a remarkably good matador, while his companion, as a lady of cards, caused a ripple of admiration among the lookers-on. Both were fully masked, and, having wraps over their costumes, only a portion of the latter were seen; but it was evident that the lady was possessed of a lovely figure, her arms were rounded and perfect, while her neck, glimpses of which could be seen, was dazzlingly white, and royally built. Darrell looked at her with interest. Then his eyes fell on her escort. He started. Surely that figure was owned by none other than Joe Leslie. What was he doing at the ball? Was this his wife? Of course it must be—the figure and beautiful neck corresponded with what Darrell remembered of Mrs. Leslie. Still, he could not help but think it odd, even at that brief moment, for Joe to bring his lovely wife here to this ball. True, it was a respectable affair, and many good people attended it, but none of the first families in New York would dream of being seen at the public masquerade—at least if they came they went away without unmasking.
- 62. As the couple passed him he could not resist saying aloud: “Hallo! Joe!” The man seemed to start, and muttered something to his companion, at which she laughed, but he did not look around to see who had spoken. Others were following them. Darrell stood a while longer, and then left the scene. Somehow or other he was troubled—he knew not exactly why. If that was Lillian with her husband, it was all well and good— although surprised at Joe taking his wife to such a carnival, so long as her husband was with her it was all right. But was it Lillian? This thought kept crowding into his brain. He could not expel it. After a little he became angry with himself for brooding over the matter so. “Hang it, I can settle the matter easily,” he muttered, as he found himself at the foot of the stairs leading to the elevated station. So up he ran. It was not a great while later when he found himself walking along the street on which the Leslies lived. He had never seen their house before, but having the number speedily found it. Of course it was one of a row. How neat and clean everything looked up in this region when compared with the neighborhood of the Twenty-seventh Street house.
- 63. His sympathies naturally ran in favor of Lillian—he seemed to believe she was the more innocent of Joe’s dupes—provided the case was really as bad as it seemed. Making sure he had the right number, as the houses were built pretty much alike, he ran up the steps and pulled the bell. A minute later a girl came to the door. “I wish to see Mr. Leslie.” “He is out, sir.” “Ah!” Darrell’s suspicions took firmer ground. The girl held the door open a crack, as though it were secured by a chain bolt. “Mrs. Leslie will do—can I see her?” He almost held his breath waiting for the answer—it seemed as though the fate of a seemingly happy household depended upon it— whether Joe Leslie were saint or sinner. “Mrs. Leslie is in—what name, please?” “You may say—stay, here is my card,” believing the girl would have no chance to read it on the way. He handed her a calling card which simply bore his name. In a minute she came back. “Mrs. Leslie will see you, sir.” The door opened. Eric Darrell found himself under the roof of Joe Leslie’s little “bird’s nest,” as the latter was fond of styling it. Everything around him showed evidences of good taste and plenty of money.
- 64. Poor bachelor Eric heaved a sigh as he noted the comfortable air of the cozy house. “What a fool,” he muttered, “but some men never know when they’re well off. With a wife and a home like his, Joe ought to be the happiest man in New York. Seems to me these things generally go to the ones least capable of appreciating them.” By this time the philosopher, in following the servant along the hall, came to the open library door, through which she motioned him to enter. He did so. Here his old bachelor soul was worse rattled than ever—such a dream of bliss may have come to him over his post-prandial cigar, but he had never believed it could be realized to a human being here below. The soft lights, the cases of books, the cheery fire in the large grate, and, chief of all, the pretty little lady seated at the table engaged in some delicate fancy work—it all took poor Eric’s breath away. He had sense enough to walk up and shake hands. “You see the plight I am in—you will forgive my not rising, Mr. Darrell,” she said, referring to her lap full of silk threads and such odds and ends. “Certainly, Mrs. Leslie, don’t move, I beg. I will find a seat near by,” he returned. She was looking at him eagerly. “Mr. Darrell, it is not accident that brings you up here to-night?” she said, and there was a question in her eyes as well as in her voice. He cannot get out of this. “I came on a little business.”
- 65. “You asked to see Mr. Leslie?” “In reality I expected to see you.” “Ah! you have already solved our terrible mystery—tell me the worst —does Joe visit that awful house to play cards?” It is hard work dealing with a woman—she is apt to ask so many questions and demand an answer—then, if important facts are told her she may in a fit of pique or anger disclose them to the very one who should not know. Darrell knows all this. He understands how to manage the gentler sex, and in the present instance does not mean to tell one whit more than is necessary. “I am sorry to say, Mrs. Leslie, that the case is not yet closed— indeed, the complications are growing more serious—but,” as he observes the look of pain on her sweet face, “I expect and hope to soon clear it all up.” “Heaven grant it,” she replied. Luckily Lillian had considerable reserve force in her nature, and now that this was brought into play, she gave promise of rising to meet the exigencies of the occasion. Darrell admired her courage. He found it harder to believe evil of her than he did of Joe, for he had great respect for the gentler sex, and believed all men had a good share of the old Adam in them—some fought the good fight and conquered—others lay down their arms and surrendered, while many ran to meet the evil half way, so misshapen were their souls. Alone, when speculating upon this strange double case, he might figure out this thing or that by force of logic; but when looking upon that truthful, lovely face, and into those calm eyes, he was ready to exclaim:
- 66. “Shame upon you, Eric Darrell, for ever even thinking this little woman and wrong could have anything in common. She’s an angel if ever there was one on earth, and I hope her sister is built upon the same pattern.” “Where is Joe?” he asked, suddenly. “You haven’t seen him then?” “I—no, indeed, not to speak to since he was in my office this afternoon.” “I—thought he had gone to you—he spoke your name in connection with the matter.” “What matter, may I ask?” “The sad affair that took him from me to-night.” Sad affair! As Darrell saw again in imagination the gay surroundings of the hall where the grand bal masque was being held, he ground his teeth in silent rage, but knowing that a pair of sharp eyes were upon him he did not allow his fury to find a vent. “Indeed! I am just as much in the dark as ever, Mrs. Leslie— enlighten me.” “I presume it’s the same sad business he went to see you about to- day.” Darrell thought not. “You know he has a young clerk and cashier in his employ, Georgie Kingsley, of whom Joe is very fond. Of late he has been led to believe the boy is getting a little wild—reports have been reaching Joe of little things, showing that Georgie is keeping bad company, and gambling. I know this has worried Joe of late.”
- 67. Darrell thought something else might be giving him a nervous spell too—no man can live a double life except at a great mental strain, for the risk of sudden exposure must be terrible. “So he’s gone to try and save poor Georgie to-night, has he? Noble- hearted old Joe.” She could not help but catch something of the sneer under his words, and trembled as she realized that the detective had grave doubts. “He said he would probably go to your room and get your company.” “He changed his mind, no doubt,” muttered the detective— indignation was apt to make him tell more than discretion warranted. “What do you mean—you know something that you do not want to tell me. I insist on your speaking. Have you seen my husband?” “I believe I have.” “Where was it?” “Entering the hall where a bal masque was being held—quite a large affair.” “Alone?” breathlessly. “No—with a lady. Good heavens! Mrs. Leslie, take it calmly, I beg of you!”
- 68. CHAPTER VI MARIAN He need not have been so alarmed. True, the blood seemed to leave Lillian’s face, and she gasped for breath, but a moment later she appeared so calm that even the detective was amazed. His admiration increased, for he saw this woman was no pretty doll, to faint at the first breath of adversity. “Do you know this as a fact, Mr. Darrell?” she asked in steady tones. “I do not, positively, and I think we ought to give Joe the benefit of the doubt.” “I shall do more than that. Until with his own lips he acknowledges such a thing to me, I will believe him innocent—I will trust him as I have always done, as the best and truest man on earth. And yet it cuts home to even have such suspicions aroused—oh, if Marian were only here!” “Your sister?” “Yes, the sister I love so dearly, and who would be such a comfort to me. She always believed in Joe. It would be a great shock to her.” Eric was struck by a sudden thought. They always came with a rush, and at times might fall under the name of an inspiration.
- 69. “Have you your sister’s photograph handy, Mrs. Leslie? Your husband spoke of her so much and said I must meet her some day. I am quite interested, and would like to see her picture.” “That is it on the mantel.” She did not evidently suspect the awful thought that came into his brain. He walked over and looked at the photograph. It attracted him very much. The face was very like Lillian’s, only the hair and eyes were dark. “I shall expect an invitation here when your sister comes on, Mrs. Leslie. She is in Chicago now, I believe.” “That is her home, but she is now traveling in California with a party of friends.” California! The mention of that far-away State sent a cold chill down his back. Was it not the grocery man who had said the beautiful Mrs. Lester’s husband was in California? Somehow he made the application, and the effect was a decided chill. It was growing blacker for Joe. “I shall take a run down and see if I can find Joe—he may be at my room waiting for me—who knows? Can I trust you to keep this matter from him, Mrs. Leslie—supposing this is all a mistake and that he is innocent, would you ever want him to believe that you harbored such suspicions?” “No, no, I would not,” she sobbed.
- 70. “Then do your part—you can act it I am sure. Appear natural—show no unusual coldness or warmth of affection—try not to meet his eye or your own may betray you. If he insists on finding out what ails you, retreat in the usual plea of a headache.” “I will not fail you, Mr. Darrell. You go about your work with the prayers of a faithful wife following you.” He believed it then—he would have staked his life on her truth—and yet in the near future such terrible doubts were to arise. “Surely that talisman ought to keep any man who is half a man, from evil—a loving mother and a faithful wife are the lodestones that have saved many a weak man from the pit of destruction. Good- night, Mrs. Leslie. Remember, should the worst come, you can depend upon Eric Darrell as your brother.” He had said more than he intended to, but he was not cold-blooded like a fish, and the evident distress of this angel on earth had wrought up all his feelings. Just then he felt as though he could have pommeled Joe Leslie with the greatest of pleasure. Any man was a brute who would give a woman like this sweet creature, pain. So Eric strode away angry with the wickedness of the world in general, and this friend of his in particular. If Joe Leslie turned out a rascal he could see no palliating circumstance connected with the case, and according to his ideas the man ought to be drawn and quartered. Hardly knowing where he was going, Darrell brought up at the hall where the bal masque was in progress. It was still early—not later than half past ten, and the affair had only started.
- 71. Any one could get in on payment of the regular price, two dollars, although none were allowed on the main floor but masks. Darrell went in. He had seen these things before, and hence had little interest in the ball itself. Most of the characters were old too, although here and there some genius had devised something new, and worth looking at. Eric had other ideas in view. Monks, flower girls, Indians, Chinese, knights, fortune tellers, dames and the endless chain of historical personages such an event gathers, passed before him without exciting more than a slight smile or a single glance of admiration. He was looking for the couple upon whom he meant to bestow his interest. Soon he sighted them. From that time on Eric seldom took his eyes off the pair. He imagined he detected certain little peculiarities in the man’s walk that marked him as Joe Leslie. As for the woman, Eric became quite interested trying to make her out—in figure she certainly resembled Lillian, and this only added to his eager pursuit. Another point he noticed—her hair was dark. Was she the one who had entered his mind? He noticed that when they danced it was always together—other couples might separate but the Spanish bull fighter and the Lady of Cards seemed inseparable.
- 72. Welcome to Our Bookstore - The Ultimate Destination for Book Lovers Are you passionate about books and eager to explore new worlds of knowledge? At our website, we offer a vast collection of books that cater to every interest and age group. From classic literature to specialized publications, self-help books, and children’s stories, we have it all! Each book is a gateway to new adventures, helping you expand your knowledge and nourish your soul Experience Convenient and Enjoyable Book Shopping Our website is more than just an online bookstore—it’s a bridge connecting readers to the timeless values of culture and wisdom. With a sleek and user-friendly interface and a smart search system, you can find your favorite books quickly and easily. Enjoy special promotions, fast home delivery, and a seamless shopping experience that saves you time and enhances your love for reading. Let us accompany you on the journey of exploring knowledge and personal growth! ebookgate.com