![CHAPTER 1
Mathematical Preliminaries
This chapter reviews some of the concepts and results from calculus that are frequently
used in this course. We recall important definitions and theorems, and outline proofs of
certain theorems. The readers are assumed to be familiar with a first course in calculus.
In Section 1.1, we introduce sequences of real numbers and discuss the concept of limit
and continuity in Section 1.2 with the intermediate value theorem. This theorem plays a
basic role in finding initial guesses in iterative methods for solving nonlinear equations.
In Section 1.3 we define the notion of derivative of a function, and prove Rolle’s theorem
and mean-value theorem for derivatives. The mean-value theorem for integration is
discussed in Section 1.4. These two theorems are crucially used in devising methods
for numerical integration and differentiation. Finally, Taylor’s theorem is discussed in
Section 1.5, which is essential for derivation and error analysis of almost all numerical
methods discussed in this course. In Section 1.6 we introduce tools useful in discussing
speed of convergence of sequences and rate at which a function f(x) approaches a point
f(x0) as x → x0.
1.1 Sequences of Real Numbers
Let a, b ∈ R be such that a < b. We use the notations [a, b] and (a, b) for the closed
and the open intervals, respectively, and are defined by
[a, b] = { x ∈ R : a ≤ x ≤ b } and (a, b) = { x ∈ R : a < x < b }.
S. Baskar and S. Sivaji Ganesh
1
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-13-320.jpg)
![Chapter 1. Mathematical Preliminaries
Definition 1.1.1 [Sequence].
A sequence of real numbers is an ordered list of real numbers
a1, a2, . . . , an, an+1, . . . .
In other words, a sequence is a function that associates the real number an for each
natural number n. The notation {an} is used to denote the sequence, i.e.
{an} := a1, a2, . . . , an, an+1, . . . .
The concept of convergence of a sequence plays an important role in the numerical
analysis. For instance, an iterative procedure is often constructed to approximate a
solution x of an equation. This procedure produces a sequence of approximations,
which is expected to convergence to the exact solution x.
Definition 1.1.2 [Convergence of a Sequence].
Let {an} be a sequence of real numbers and let a be a real number. The sequence
{an} is said to converge to a, written as
lim
n→∞
an = a (or an → a as n → ∞),
if for every ϵ > 0, there exists a natural number N such that
|an − a| < ϵ whenever n ≥ N.
The real number a is called the limit of the sequence {an}.
The following result is very useful in computing the limit of a sequence sandwiched
between two sequences having a common limit.
Theorem 1.1.3 [Sandwich Theorem].
Let {an}, {bn}, {cn} be sequences of real numbers such that
1. there exists an n0 ∈ N such that for every n ≥ n0, the sequences satisfy the
inequalities an ≤ bn ≤ cn and
2. lim
n→∞
an = lim
n→∞
cn = a, for some real number a.
Then the sequence {bn} also converges and lim
n→∞
bn = a.
S. Baskar and S. Sivaji Ganesh
2
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-14-320.jpg)
![Section 1.1 Sequences of Real Numbers
Definition 1.1.4 [Bounded Sequence].
A sequence {an} is said to be a bounded sequence if there exists a real number M
such that
|an| ≤ M for every n ∈ N.
Note that a bounded sequence, in general, need not converge, for instance {(−1)n
}.
However, it is always possible to find a convergent subsequence.
Theorem 1.1.5 [Bolzano-Weierstrass theorem].
Every bounded sequence {an} has a convergent subsequence {ank
}.
Definition 1.1.6 [Monotonic Sequences].
A sequence {an} of real numbers is said to be
1. an increasing sequence if an ≤ an+1, for every n ∈ N.
2. a strictly increasing sequence if an < an+1, for every n ∈ N.
3. a decreasing sequence if an ≥ an+1, for every n ∈ N.
4. a strictly decreasing sequence if an > an+1, for every n ∈ N.
A sequence {an} is said to be a (strictly) monotonic sequence if it is either
(strictly) increasing or (strictly) decreasing.
The following theorem shows the advantage of working with monotonic sequences.
Theorem 1.1.7.
Bounded monotonic sequences always converge.
The following result gives the algebra of limits of sequences.
Theorem 1.1.8.
Let {an} and {bn} be two sequences. Assume that lim
n→∞
an and lim
n→∞
bn exist. Then
1. lim
n→∞
(an + bn) = lim
n→∞
an + lim
n→∞
bn.
2. lim
n→∞
c an = c lim
n→∞
an, for any number c.
3. lim
n→∞
anbn = lim
n→∞
an lim
n→∞
bn .
4. lim
n→∞
1
an
=
1
lim
n→∞
an
, provided lim
n→∞
an ̸= 0.
S. Baskar and S. Sivaji Ganesh
3
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-15-320.jpg)
![Chapter 1. Mathematical Preliminaries
1.2 Limits and Continuity
In the previous section, we introduced the concept of limit for a sequences of real
numbers. We now define the “limit” in the context of functions.
Definition 1.2.1 [Limit of a Function].
1. Let f be a function defined on the left side (or both sides) of a, except possibly
at a itself. Then, we say “the left-hand limit of f(x) as x approaches a,
equals l” and denote
lim
x→a−
f(x) = l,
if we can make the values of f(x) arbitrarily close to l (as close to l as we like)
by taking x to be sufficiently close to a and x less than a.
2. Let f be a function defined on the right side (or both sides) of a, except possibly
at a itself. Then, we say “the right-hand limit of f(x) as x approaches a,
equals r” and denote
lim
x→a+
f(x) = r,
if we can make the values of f(x) arbitrarily close to r (as close to r as we like)
by taking x to be sufficiently close to a and x greater than a.
3. Let f be a function defined on both sides of a, except possibly at a itself. Then,
we say“the limit of f(x) as x approaches a, equals L” and denote
lim
x→a
f(x) = L,
if we can make the values of f(x) arbitrarily close to L (as close to L as we
like) by taking x to be sufficiently close to a (on either side of a) but not equal
to a.
Remark 1.2.2.
Note that in each of the above definitions the value of the function f at the point
a does not play any role. In fact, the function f need not be defined at a.
In the previous section, we have seen some limit laws in the context of sequences.
Similar limit laws also hold for limits of functions. We have the following result, often
referred to as “the limit laws” or as “algebra of limits”.
S. Baskar and S. Sivaji Ganesh
4
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-16-320.jpg)
![Section 1.2 Limits and Continuity
Theorem 1.2.3.
Let f, g be two functions defined on both sides of a, except possibly at a itself.
Assume that lim
x→a
f(x) and lim
x→a
g(x) exist. Then
1. lim
x→a
(f(x) + g(x)) = lim
x→a
f(x) + lim
x→a
g(x).
2. lim
x→a
c f(x) = c lim
x→a
f(x), for any number c.
3. lim
x→a
f(x)g(x) = lim
x→a
f(x) lim
x→a
g(x).
4. lim
x→a
1
g(x)
=
1
lim
x→a
g(x)
, provided lim
x→a
g(x) ̸= 0.
Remark 1.2.4.
Polynomials, rational functions, all trigonometric functions wherever they are de-
fined, have property called direct substitution property:
lim
x→a
f(x) = f(a).
The following theorem is often used to estimate limits of functions.
Theorem 1.2.5.
If f(x) ≤ g(x) when x is in an interval containing a (except possibly at a) and the
limits of f and g both exist as x approaches a, then
lim
x→a
f(x) ≤ lim
x→a
g(x).
The following theorem is often used to compute limits of functions.
Theorem 1.2.6 [Sandwich Theorem].
Let f, g, and h be given functions such that
1. f(x) ≤ g(x) ≤ h(x) when x is in an interval containing a (except possibly at
a) and
2. lim
x→a
f(x) = lim
x→a
h(x) = L,
then
lim
x→a
g(x) = L.
S. Baskar and S. Sivaji Ganesh
5
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-17-320.jpg)
![Chapter 1. Mathematical Preliminaries
We will now give a rigorous definition of the limit of a function. Similar definitions can
be written down for left-hand and right-hand limits of functions.
Definition 1.2.7.
Let f be a function defined on some open interval that contains a, except possibly
at a itself. Then we say that the limit of f(x) as x approaches a is L and we write
lim
x→a
f(x) = L.
if for every ϵ > 0 there is a number δ > 0 such that
|f(x) − L| < ϵ whenever 0 < |x − a| < δ.
Definition 1.2.8 [Continuity].
A function f is
1. continuous from the right at a if
lim
x→a+
f(x) = f(a).
2. continuous from the left at a if
lim
x→a−
f(x) = f(a).
3. continuous at a if
lim
x→a
f(x) = f(a).
A function f is said to be continuous on an open interval if f is continuous at
every point in the interval. If f is defined on a closed interval [a, b], then f is said
to be continuous at a if f is continuous from the right at a and similarly, f is said
to be continuous at b if f is continuous from left at b.
Remark 1.2.9.
Note that the definition for continuity of a function f at a, means the following
three conditions are satisfied:
1. The function f must be defined at a. i.e., a is in the domain of f,
2. lim
x→a
f(x) exists, and
3. lim
x→a
f(x) = f(a).
Equivalently, for any given ϵ > 0, there exists a δ > 0 such that
|f(x) − f(a)| < ϵ whenever |x − a| < δ.
S. Baskar and S. Sivaji Ganesh
6
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-18-320.jpg)
![Section 1.3 Differentiation
Theorem 1.2.10.
If f and g are continuous at a, then the functions f + g, f − g, cg (c is a constant),
fg, f/g (provided g(a) ̸= 0), f◦g (composition of f and g, whenever it makes sense)
are all continuous.
Thus polynomials, rational functions, trigonometric functions are all continuous on
their respective domains.
Theorem 1.2.11 [Intermediate Value Theorem].
Suppose that f is continuous on the closed interval [a, b] and let N be any number
between f(a) and f(b), where f(a) ̸= f(b). Then there exists a point c ∈ (a, b) such
that
f(c) = N.
1.3 Differentiation
We next give the basic definition of a differentiable function.
Definition 1.3.1 [Derivative].
The derivative of a function f at a point a, denoted by f′
(a), is defined as
f′
(a) = lim
h→0
f(a + h) − f(a)
h
, (1.1)
if this limit exists. We say f is differentiable at a. A function f is said to be
differentiable on (c, d) if f is differentiable at every point in (c, d).
There are alternate ways of defining the derivative of a function, which leads to different
difference formulae.
Remark 1.3.2.
The derivative of a function f at a point x = a can also be given by
f′
(a) = lim
h→0
f(a) − f(a − h)
h
, (1.2)
and
f′
(a) = lim
h→0
f(a + h) − f(a − h)
2h
, (1.3)
provided the limits exist.
S. Baskar and S. Sivaji Ganesh
7
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-19-320.jpg)
![Chapter 1. Mathematical Preliminaries
If we write x = a + h, then h = x − a and h → 0 if and only if x → a. Thus, formula
(1.1) can equivalently be written as
f′
(a) = lim
x→a
f(x) − f(a)
x − a
.
Remark 1.3.3 [Interpretation].
Take the graph of f, draw the line joining the points (a, f(a)) and (x, f(x)). Take
its slope and take the limit of these slopes as x → a. Then the point (x, f(x)) tends
to (a, f(a)). The limit is nothing but the slope of the tangent line at (a, f(a)) to
the curve y = f(x). This geometric interpretation will be very useful in describing
the Newton-Raphson method in the context of solving nonlinear equations.
Theorem 1.3.4.
If f is differentiable at a, then f is continuous at a.
Proof.
Using the identities
f(x) =
f(x) − f(a)
x − a
(x − a) + f(a)
and taking limit as x → a yields the desired result.
The converse of Theorem 1.3.4 is not true. For, the function f(x) = |x| is continuous
at x = 0 but is not differentiable there.
Theorem 1.3.5.
Suppose f is differentiable at a. Then there exists a function ϕ such that
f(x) = f(a) + (x − a)f′
(a) + (x − a)ϕ(x),
and limx→a ϕ(x) = 0.
Proof.
Define ϕ by
ϕ(x) =
f(x) − f(a)
x − a
− f′
(a).
Since f is differentiable at a, the result follows on taking limits on both sides of the
last equation as x → a.
S. Baskar and S. Sivaji Ganesh
8
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-20-320.jpg)
![Section 1.3 Differentiation
Theorem 1.3.6 [Rolle’s Theorem].
Let f be a function that satisfies the following three hypotheses:
1. f is continuous on the closed interval [a, b].
2. f is differentiable on the open interval (a, b).
3. f(a) = f(b).
Then there is a number c in the open interval (a, b) such that f′
(c) = 0.
Proof.
If f is a constant function i.e., f(x) = f(a) for every x ∈ [a, b], clearly such a c exists.
If f is not a constant, then at least one of the following holds.
Case 1: The graph of f goes above the line y = f(a) i.e., f(x) > f(a) for some
x ∈ (a, b).
Case 2: The graph of f goes below the line y = f(a) i.e., f(x) < f(a) for some
x ∈ (a, b).
In case (1), i.e., if the graph of f goes above the line y = f(a), then the global
maximum cannot be at a or b. Therefore, it must lie in the open interval (a, b).
Denote that point by c. That is, global maximum on [a, b] is actually a local maxi-
mum, and hence f′
(c) = 0. A similar argument can be given in case (2), to show the
existence of local minimum in (a, b). This completes the proof of Rolle’s theorem.
The mean value theorem plays an important role in obtaining error estimates for certain
numerical methods.
Theorem 1.3.7 [Mean Value Theorem].
Let f be a function that satisfies the following hypotheses:
1. f is continuous on the closed interval [a, b].
2. f is differentiable on the open interval (a, b).
Then there is a number c in the open interval (a, b) such that
f′
(c) =
f(b) − f(a)
b − a
.
or, equivalently,
f(b) − f(a) = f′
(c)(b − a).
S. Baskar and S. Sivaji Ganesh
9
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-21-320.jpg)
![Chapter 1. Mathematical Preliminaries
Proof.
Define ϕ on [a, b] by
ϕ(x) = f(x) − f(a) −
f(b) − f(a)
b − a
(x − a).
By Rolle’s theorem there exists a c ∈ (a, b) such that ϕ′
(c) = 0 and hence the proof
is complete.
1.4 Integration
In Theorem 1.3.7, we have discussed the mean value property for the derivative of a
function. We now discuss the mean value theorems for integration.
Theorem 1.4.1 [Mean Value Theorem for Integrals].
If f is continuous on [a, b], then there exists a number c in [a, b] such that
b∫
a
f(x) dx = f(c)(b − a).
Proof.
Let m and M be minimum and maximum values of f in the interval [a, b], respectively.
Then,
m(b − a) ≤
b∫
a
f(x) dx ≤ M(b − a).
Since f is continuous, the result follows from the intermediate value theorem.
Recall the average value of a function f on the interval [a, b] is defined by
1
b − a
b∫
a
f(x) dx.
Observe that the first mean value theorem for integrals asserts that the average of
an integrable function f on an interval [a, b] belongs to the range of the function
f.
S. Baskar and S. Sivaji Ganesh
10
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-22-320.jpg)
![Section 1.5 Taylor’s Theorem
Remark 1.4.2 [Interpretation].
Let f be a function on [a, b] with f > 0. Draw the graph of f and find the area
under the graph lying between the ordinates x = a and x = b. Also, look at a
rectangle with base as the interval [a, b] with height f(c) and compute its area.
Both values are the same.
The Theorem 1.4.1 is often referred to as the first mean value theorem for integrals.
We now state the second mean value theorem for integrals, which is a general form of
Theorem 1.4.1.
Theorem 1.4.3 [Second Mean Value Theorem for Integrals].
Let f and g be continuous on [a, b], and let g(x) ≥ 0 for all x ∈ R. Then there exists
a number c ∈ [a, b] such that
b∫
a
f(x)g(x) dx = f(c)
b∫
a
g(x) dx.
The proof of this theorem is left as an exercise.
1.5 Taylor’s Theorem
Let f be a real-valued function defined on an interval I. We say f ∈ Cn
(I) if f is
n-times continuously differentiable at every point in I. Also, we say f ∈ C∞
(I) if f is
continuously differentiable of any order at every point in I.
The most important result used very frequently in numerical analysis, especially in
error analysis of numerical methods, is the Taylor’s expansion of a C∞
function in a
neighborhood of a point a ∈ R. In this section, we define the Taylor’s polynomial
and prove an important theorem called the Taylor’s theorem. The idea of the proof of
this theorem is similar to the one used in proving the mean value theorem, where we
construct a function and apply Rolle’s theorem several times to it.
Definition 1.5.1 [Taylor’s Polynomial for a Function at a Point].
Let f be n-times continuously differentiable at a given point a. The Taylor’s
polynomial of degree n for the function f at the point a, denoted by Tn, is defined
by
Tn(x) =
n∑
k=0
f(k)
(a)
k!
(x − a)k
, x ∈ R. (1.4)
S. Baskar and S. Sivaji Ganesh
11
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-23-320.jpg)
![Chapter 1. Mathematical Preliminaries
Theorem 1.5.2 [Taylor’s Theorem].
Let f be (n + 1)-times continuously differentiable function on an open interval con-
taining the points a and x. Then there exists a number ξ between a and x such
that
f(x) = Tn(x) +
f(n+1)
(ξ)
(n + 1)!
(x − a)n+1
, (1.5)
where Tn is the Taylor’s polynomial of degree n for f at the point a given by (1.4)
and the second term on the right hand side is called the remainder term.
Proof.
Let us assume x > a and prove the theorem. The proof is similar if x < a.
Define g(t) by
g(t) = f(t) − Tn(t) − A(t − a)n+1
and choose A so that g(x) = 0, which gives
A =
f(x) − Tn(x)
(x − a)n+1
.
Note that
g(k)
(a) = 0 for k = 0, 1, . . . n.
Also, observe that the function g is continuous on [a, x] and differentiable in (a, x).
Apply Rolle’s theorem to g on [a, x] (after verifying all the hypotheses of Rolle’s
theorem) to get
a < c1 < x satisfying g′
(c1) = 0.
Again apply Rolle’s theorem to g′
on [a, c1] to get
a < c2 < c1 satisfying g′′
(c2) = 0.
In turn apply Rolle’s theorem to g(2)
, g(3)
, . . . , g(n)
on intervals [a, c2], [a, c3], . . . ,
[a, cn], respectively.
At the last step, we get
a < cn+1 < cn satisfying g(n+1)
(cn+1) = 0.
S. Baskar and S. Sivaji Ganesh
12
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-24-320.jpg)
![Chapter 1. Mathematical Preliminaries
Remark 1.5.4 [Estimate for Remainder Term in Taylor’s Formula].
Let f be an (n + 1)-times continuously differentiable function with the property
that there exists an Mn+1 such that
|f(n+1)
(ξ)| ≤ Mn+1, for all ξ ∈ I.
Then for fixed points a, x ∈ I, the remainder term in (1.5) satisfies the estimate
f(n+1)(ξ)
(n + 1)!
(x − a)n+1
≤
Mn+1
(n + 1)!
x − a
n+1
.
We can further get an estimate of the reminder term that is independent of x as
f(n+1)(ξ)
(n + 1)!
(x − a)n+1
≤
Mn+1
(n + 1)!
(b − a)n+1
,
which holds for all x ∈ I. Observe that the right hand side of the above estimate
is a fixed number. We refer to such estimates as remainder estimates.
Note
In most applications of Taylor’s theorem, one never knows ξ precisely. However
in view of remainder estimate given above, it does not matter as long as we know
that the remainder can be bounded by obtaining a bound Mn+1 which is valid
for all ξ between a and x.
We next introduce an important notion of error called the truncation error
Definition 1.5.5 [Truncation Error].
The remainder term involved in approximating f(x) by the Taylor’s polynomial Tn(x)
is also called the Truncation error.
Example 1.5.6.
A second degree polynomial approximation to
f(x) =
√
x + 1, x ∈ [−1, ∞)
using the Taylor’s formula about a = 0 is given by
f(x) ≈ 1 +
x
2
−
x2
8
,
where the remainder term is neglected and hence what we obtained here is only an
approximate representation of f.
S. Baskar and S. Sivaji Ganesh
14
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-26-320.jpg)
![Section 1.5 Taylor’s Theorem
The truncation error is obtained using the remainder term in the formula (1.5) with
n = 2 and is given by
x3
16(
√
1 + ξ )5
,
for some point ξ between 0 and x.
Note that we cannot obtain a remainder estimate in the present example as f′′′
is not
bounded in [−1, ∞). However, for any 0 < δ < 1, if we restrict the domain of f to
[−δ, ∞), then we can obtain the remainder estimate for a fixed x ∈ [−δ, ∞) as
x3
16(
√
1 − δ )5
.
Further, if we restrict the domain of f to [−δ, b] for some real number b > 0, then we
get the remainder estimate independent of x as
b3
16(
√
1 − δ )5
.
Definition 1.5.7 [Taylor’s Series].
Let f be C∞
in a neighborhood of a point a. The power series
∞∑
k=0
f(k)
(a)
k!
(x − a)k
is called the Taylor’s series of f about the point a.
The question now is when this series converges and what is the limit of this series.
These questions are answered in the following theorem whose proof is omitted in this
course.
Theorem 1.5.8.
Let f ∈ C∞
(I) and let a ∈ I. Assume that there exists an open neighborhood
(interval) Na ⊂ I of the point a and there exists a constant M (may depend on a)
such that
f(k)
(x) ≤ Mk
,
for all x ∈ Na and k = 0, 1, 2, · · · . Then for each x ∈ Ia, we have
f(x) =
∞∑
k=0
f(k)
(a)
k!
(x − a)k
.
S. Baskar and S. Sivaji Ganesh
15
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-27-320.jpg)
![Chapter 1. Mathematical Preliminaries
Definition 1.6.2 [Big Oh and Little oh].
Let {an} and {bn} be sequences of real numbers. Then
1. the sequence {an} is said to be Big Oh of {bn}, and write an = O(bn), if there
exists a real number C and a natural number N such that
|an| ≤ C |bn| for all n ≥ N.
2. the sequence {an} is said to be Little oh (sometimes said to be small oh) of
{bn}, and write an = o(bn), if for every ϵ > 0 there exists a natural number N
such that
|an| ≤ ϵ |bn| for all n ≥ N.
Remark 1.6.3.
1. If bn ̸= 0 for every n, then we have an = O(bn) if and only if the sequence{
an
bn
}
is bounded. That is, there exists a constant C such that
an
bn
≤ C
2. If bn ̸= 0 for every n, then we have an = o(bn) if and only if the sequence{
an
bn
}
converges to 0. That is,
lim
n→∞
an
bn
= 0.
3. For any pair of sequences {an} and {bn} such that an = o(bn), it follows that
an = O(bn).
The converse is not true. Consider the sequences an = n and bn = 2n + 3,
for which an = O(bn) holds but an = o(bn) does not hold.
4. Let {an} and {bn} be two sequences that converge to 0. Then an = O(bn)
means the sequence {an} tends to 0 at least as fast as the sequence {bn};
and an = o(bn) means the sequence {an} tends to 0 faster than the sequence
{bn}.
The Big Oh and Little oh notations can be adapted for functions as follows.
S. Baskar and S. Sivaji Ganesh
18
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-30-320.jpg)
![Section 1.6 Orders of Convergence
Definition 1.6.4 [Big Oh and Little oh for Functions].
Let x0 ∈ R. Let f and g be functions defined in an interval containing x0. Then
1. the function f is said to be Big Oh of g as x → x0, and write f(x) = O
(
g(x)
)
,
if there exists a real number C and a real number δ such that
|f(x)| ≤ C |g(x)| whenever |x − x0| ≤ δ.
2. the function f is said to be Little Oh (also, Small oh) of g as x → x0, and
write f(x) = o
(
g(x)
)
, if for every ϵ > 0 there exists a real number δ such that
|f(x)| ≤ ϵ |g(x)| whenever |x − x0| ≤ δ.
In case of functions also, a remark similar to the Remark 1.6.3 holds.
Example 1.6.5.
The Taylor’s formula for f(x) = cos(x) about the point a = 0 is
cos(x) =
n∑
k=0
(−1)k
(2k)!
x2k
+ (−1)n+1 cos(ξ)
(2(n + 1))!
x2(n+1)
where ξ lies between x and 0.
Let us denote the remainder term (truncation error) as (for a fixed n)
g(x) = (−1)n+1 cos(ξ)
(2(n + 1))!
x2(n+1)
.
Clearly, g(x) → 0 as x → 0. The question now is
‘How fast does g(x) → 0 as x → 0?’
The answer is
‘As fast as x2(n+1)
→ 0 as x → 0.’
That is,
g(x) = O
(
x2(n+1)
)
as x → 0.
In this case, we say that the sequence
{ n∑
k=0
(−1)k
(2k)!
x2k
}
converges to cos(x) with order 2(n + 1).
S. Baskar and S. Sivaji Ganesh
19
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-31-320.jpg)
![Chapter 1. Mathematical Preliminaries
1.6.2 Rates of Convergence
Let {an} be a sequence such that
lim
n→∞
an = a.
We would like to measure the speed at which the convergence takes place. For example,
consider
lim
n→∞
1
2n + 3
= 0
and
lim
n→∞
1
n2
= 0.
We feel that the first sequence goes to zero linearly and the second goes with a much
superior speed because of the presence of n2
in its denominator. We will define the
notion of order of convergence precisely.
Definition 1.6.6 [Rate of Convergence or Order of Convergence].
Let {an} be a sequence such that lim
n→∞
an = a.
1. We say that the rate of convergence is atleast linear if there exists a constant
c < 1 and a natural number N such that
|an+1 − a| ≤ c |an − a| for all n ≥ N.
2. We say that the rate of convergence is atleast superlinear if there exists a
sequence {ϵn} that converges to 0, and a natural number N such that
|an+1 − a| ≤ ϵn |an − a| for all n ≥ N.
3. We say that the rate of convergence is atleast quadratic if there exists a
constant C (not necessarily less than 1), and a natural number N such that
|an+1 − a| ≤ C |an − a|2
for all n ≥ N.
4. Let α ∈ R+. We say that the rate of convergence is atleast α if there exists a
constant C (not necessarily less than 1), and a natural number N such that
|an+1 − a| ≤ C |an − a|α
for all n ≥ N.
S. Baskar and S. Sivaji Ganesh
20
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-32-320.jpg)
![Section 1.7 Exercises
1.7 Exercises
Sequences of Real Numbers
1. Let L be a real number and let {an} be a sequence of real numbers. If there exists
a positive integer N such that
|an − L| ≤ µ|an−1 − L|,
for all n ≥ N and for some fixed µ ∈ (0, 1), then show that an → L as n → ∞.
2. Consider the sequences {an} and {bn}, where
an =
1
n
, bn =
1
n2
, n = 1, 2, · · · .
Clearly, both the sequences converge to zero. For the given ϵ = 10−2
, obtain the
smallest positive integers Na and Nb such that
|an| < ϵ whenever n ≥ Na, and |bn| < ϵ whenever n ≥ Nb.
For any ϵ > 0, show that Na > Nb.
3. Let {xn} and {yn} be two sequences such that xn, yn ∈ [a, b] and xn < yn for each
n = 1, 2, · · · . If xn → b as n → ∞, then show that the sequence {yn} converges.
Find the limit of the sequence {yn}.
4. Let In =
[
n − 2
2n
,
n + 2
2n
]
, n = 1, 2, · · · and {an} be a sequence with an is chosen
arbitrarily in In for each n = 1, 2, · · · . Show that an →
1
2
as n → ∞.
Limits and Continuity
5. Let f be a real-valued function such that f(x) ≥ sin(x) for all x ∈ R. If
lim
x→0
f(x) = L exists, then show that L ≥ 0.
6. Let P and Q be polynomials. Find
lim
x→∞
P(x)
Q(x)
and lim
x→0
P(x)
Q(x)
in each of the following cases.
i) The degree of P is less than the degree of Q.
ii) The degree of P is greater than the degree of Q.
iii) The agree of P is equal to the degree of Q.
S. Baskar and S. Sivaji Ganesh
21
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-33-320.jpg)
![Chapter 1. Mathematical Preliminaries
7. Show that the equation sin x+x2
= 1 has at least one solution in the interval [0, 1].
8. Let f(x) be continuous on [a, b], let x1, · · · , xn be points in [a, b], and let g1, · · · ,
gn be real numbers having same sign. Show that
n∑
i=1
f(xi)gi = f(ξ)
n∑
i=1
gi, for some ξ ∈ [a, b].
9. Let f : [0, 1] → [0, 1] be a continuous function. Prove that the equation f(x) = x
has at least one solution lying in the interval [0, 1] (Note: A solution of this equa-
tion is called a fixed point of the function f).
10. Show that the equation f(x) = x, where
f(x) = sin
(
πx + 1
2
)
, x ∈ [−1, 1]
has at least one solution in [−1, 1].
Differentiation
11. Let c ∈ (a, b) and f : (a, b) → R be differentiable at c. If c is a local extremum
(maximum or minimum) of f, then show that f′
(c) = 0.
12. Suppose f is differentiable in an open interval (a, b). Prove the following state-
ments
(a) If f′
(x) ≥ 0 for all x ∈ (a, b), then f is non-decreasing.
(b) If f′
(x) = 0 for all x ∈ (a, b), then f is constant.
(c) If f′
(x) ≤ 0 for all x ∈ (a, b), then f is non-increasing.
13. Let f : [a, b] → R be given by f(x) = x2
. Find a point c specified by the mean-
value theorem for derivatives. Verify that this point lies in the interval (a, b).
Integration
14. Let g : [0, 1] → R be a continuous function. Show that there exists a c ∈ (0, 1)
such that
1∫
0
x2
(1 − x)2
g(x)dx =
1
30
g(ξ).
S. Baskar and S. Sivaji Ganesh
22
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-34-320.jpg)
![Section 2.1 Floating-Point Representation
Although different computing devices have different ways of representing numbers, here
we introduce a mathematical form of this representation, which we will use throughout
this course.
Definition 2.1.2 [n-Digit Floating-point Number].
Let β ∈ N and β ≥ 2. An n-digit floating-point number in base β is of the form
(−1)s
× (.d1d2 · · · dn)β × βe
(2.3)
where
(.d1d2 · · · dn)β =
d1
β
+
d2
β2
+ · · · +
dn
βn
(2.4)
where di ∈ { 0, 1, · · · , β − 1 } with d1 ̸= 0 or d1 = d2 = d3 = · · · = 0, s = 0 or 1, and
an appropriate exponent e.
Remark 2.1.3.
When β = 2, the n-digit floating-point representation (2.3) is called the n-digit
binary floating-point representation and when β = 10, it is called the n-digit
decimal floating-point representation.
Example 2.1.4.
The following are examples of real numbers in the decimal floating point representa-
tion.
1. The real number x = 6.238 is represented in the decimal floating-point repre-
sentation as
6.238 = (−1)0
× 0.6238 × 101
,
in which case, we have s = 0, β = 10, e = 1, d1 = 6, d2 = 2, d3 = 3 and d4 = 8.
2. The real number x = −0.0014 is represented in the decimal floating-point rep-
resentation as
x = (−1)1
× 0.14 × 10−2
.
Here s = 1, β = 10, e = −2, d1 = 1 and d2 = 4.
S. Baskar and S. Sivaji Ganesh
27
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-39-320.jpg)
![Section 2.1 Floating-Point Representation
Example 2.1.7 [Overflow].
Run the following MATLAB code on a computer with 32-bit intel processor:
i=308.25471;
fprintf(’%f %fn’,i,10^i);
i=308.25472;
fprintf(’%f %fn’,i,10^i);
We see that the first print command shows a meaningful (but very large) number,
whereas the second print command simply prints inf. This is due to the overflow of
memory while representing a very large real number.
Also try running the following code on the MATLAB:
i=308.25471;
fprintf(’%f %fn’,i,10^i/10^i);
i=308.25472;
fprintf(’%f %fn’,i,10^i/10^i);
The output will be
308.254710 1.000000
308.254720 NaN
If your computer is not showing inf for i = 308.25472, try increasing the value of
i till you get inf.
Example 2.1.8 [Underflow].
Run the following MATLAB code on a computer with 32-bit intel processor:
j=-323.6;
if(10^j>0)
fprintf(’The given number is greater than zeron’);
elseif (10^j==0)
fprintf(’The given number is equal to zeron’);
else
fprintf(’The given number is less than zeron’);
end
The output will be
The given number is greater than zero
S. Baskar and S. Sivaji Ganesh
29
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-41-320.jpg)
![Chapter 2. Error Analysis
When the value of j is further reduced slightly as shown in the following program
j=-323.64;
if(10^j>0)
fprintf(’The given number is greater than zeron’);
elseif (10^j==0)
fprintf(’The given number is equal to zeron’);
else
fprintf(’The given number is less than zeron’);
end
the output shows
The given number is equal to zero
If your computer is not showing the above output, try decreasing the value of j till
you get the above output.
In this example, we see that the number 10−323.64
is recognized as zero by the com-
puter. This is due to the underflow of memory. Note that multiplying any large
number by this number will give zero as answer. If a computation involves such an
underflow of memory, then there is a danger of having a large difference between the
actual value and the computed value.
2.1.3 Chopping and Rounding a Number
The number of digits in the mantissa, as given in Definition 2.1.2, is called the pre-
cision or length of the floating-point number. In general, a real number can have
infinitely many digits, which a computing device cannot hold in its memory. Rather,
each computing device will have its own limitation on the length of the mantissa. If
a given real number has infinitely many (or sufficiently large number of) digits in the
mantissa of the floating-point form as in (2.1), then the computing device converts this
number into an n-digit floating-point form as in (2.3). Such an approximation is called
the floating-point approximation of a real number.
There are many ways to get floating-point approximation of a given real number. Here
we introduce two types of floating-point approximation.
Definition 2.1.9 [Chopped and Rounded Numbers].
Let x be a real number given in the floating-point representation (2.1) as
x = (−1)s
× (.d1d2 · · · dndn+1 · · · )β × βe
.
S. Baskar and S. Sivaji Ganesh
30
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-42-320.jpg)
![Section 2.2 Types of Errors
Finally, we have
fl(f(100000)) = fl(100000) × (0.1 × 10−2
)
= (0.1 × 106
) × (0.1 × 10−2
)
= 100.
Using six-digit chopping, the value of fl(f(100000)) is 200.
Definition 2.1.13 [Machine Epsilon].
The machine epsilon of a computer is the smallest positive floating-point number
δ such that
fl(1 + δ) > 1.
For any floating-point number ˆδ < δ, we have fl(1 + ˆδ) = 1, and 1 + ˆδ and 1 are
identical within the computer’s arithmetic.
Remark 2.1.14.
From Example 2.1.8, it is clear that the machine epsilon for a 32-bit intel processor
lies between the numbers 10−323.64 and 10−323.6. It is possible to get the exact
value of this number, but it is no way useful in our present course, and so we will
not attempt to do this here.
2.2 Types of Errors
The approximate representation of a real number obviously differs from the actual
number, whose difference is called an error.
Definition 2.2.1 [Errors].
1. The error in a computed quantity is defined as
Error = True Value - Approximate Value.
2. Absolute value of an error is called the absolute error.
3. The relative error is a measure of the error in relation to the size of the true
value as given by
Relative Error =
Error
True Value
.
Here, we assume that the true value is non-zero.
4. The percentage error is defined as
Percentage Error = 100 × Relative Error .
S. Baskar and S. Sivaji Ganesh
33
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-45-320.jpg)
![Section 2.3 Loss of Significance
Definition 2.3.1 [Significant β-Digits].
Let β be a radix and x ̸= 0. If xA is an approximation to x, then we say that xA
approximates x to r significant β-digits if r is the largest non-negative integer
such that
|x − xA|
|x|
≤
1
2
β−r+1
. (2.11)
We also say xA has r significant β-digits in x.
Note
When β = 10, we refer significant 10-digits by significant digits.
Example 2.3.2.
1. For x = 1/3, the approximate number xA = 0.333 has three significant digits,
since
|x − xA|
|x|
= 0.001 < 0.005 = 0.5 × 10−2
.
Thus, r = 3.
2. For x = 0.02138, the approximate number xA = 0.02144 has three significant
digits, since
|x − xA|
|x|
≈ 0.0028 < 0.005 = 0.5 × 10−2
.
Thus, r = 3.
3. For x = 0.02132, the approximate number xA = 0.02144 has two significant
digits, since
|x − xA|
|x|
≈ 0.0056 < 0.05 = 0.5 × 10−1
.
Thus, r = 2.
4. For x = 0.02138, the approximate number xA = 0.02149 has two significant
digits, since
|x − xA|
|x|
≈ 0.0051 < 0.05 = 0.5 × 10−1
.
Thus, r = 2.
5. For x = 0.02108, the approximate number xA = 0.0211 has three significant
digits, since
|x − xA|
|x|
≈ 0.0009 < 0.005 = 0.5 × 10−2
.
Thus, r = 3.
S. Baskar and S. Sivaji Ganesh
35
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-47-320.jpg)
![Chapter 2. Error Analysis
2.5 Propagation of Relative Error in Function Evaluation
For a given function f : R → R, consider evaluating f(x) at an approximate value
xA rather than at x. The question is how well does f(xA) approximate f(x)? To
answer this question, we compare Er(f(xA)) with Er(xA).
Assume that f is a C1
function. Using the mean-value theorem, we get
f(x) − f(xA) = f′
(ξ)(x − xA),
where ξ is an unknown point between x and xA. The relative error in f(xA) when
compared to f(x) is given by
Er(f(xA)) =
f′
(ξ)
f(x)
(x − xA).
Thus, we have
Er(f(xA)) =
(
f′
(ξ)
f(x)
x
)
Er(xA). (2.15)
Since xA and x are assumed to be very close to each other and ξ lies between x and
xA, we may make the approximation
f(x) − f(xA) ≈ f′
(x)(x − xA).
In view of (2.15), we have
Er(f(xA)) ≈
(
f′
(x)
f(x)
x
)
Er(xA). (2.16)
The expression inside the bracket on the right hand side of (2.16) is the amplifica-
tion factor for the relative error in f(xA) in terms of the relative error in xA. Thus,
this expression plays an important role in understanding the propagation of relative
error in evaluating the function value f(x) and hence motivates the following defini-
tion.
Definition 2.5.1 [Condition Number of a Function].
The condition number of a continuously differentiable function f at a point x = c
is given by
f′
(c)
f(c)
c . (2.17)
S. Baskar and S. Sivaji Ganesh
42
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-54-320.jpg)
![Section 2.5 Propagation of Relative Error in Function Evaluation
The condition number of a function at a point x = c can be used to decide whether the
evaluation of the function at x = c is well-conditioned or ill-conditioned depending on
whether this condition number is smaller or larger as we approach this point. It is not
possible to decide a priori how large the condition number should be to say that the
function evaluation is ill-conditioned and it depends on the circumstances in which we
are working.
Definition 2.5.2 [Well-Conditioned and Ill-Conditioned].
The process of evaluating a continuously differentiable function f at a point x = c
is said to be well-conditioned if the condition number
f′
(c)
f(c)
c
at c is ‘small’. The process of evaluating a function at x = c is said to be ill-
conditioned if it is not well-conditioned.
In the following example, we illustrate a well-conditioned computation.
Example 2.5.3.
Consider the function
f(x) =
√
x,
for all x ∈ [0, ∞). Then
f′
(x) =
1
2
√
x
, for all x ∈ [0, ∞).
The condition number of f is
f′
(x)
f(x)
x =
1
2
, for all x ∈ [0, ∞),
which shows that taking square roots is a well-conditioned process.
From (2.16), we have
|Er(f(xA))| ≈
1
2
|Er(xA)|.
Thus, Er(f(xA)) is more closer to zero than Er(xA).
We next illustrate an ill-conditioned computation.
S. Baskar and S. Sivaji Ganesh
43
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-55-320.jpg)
![Section 2.6 Exercises
2.6 Exercises
Floating-Point Approximation
1. Let X be a sufficiently large number which result in an overflow of memory on a
computing device. Let x be a sufficiently small number which result in underflow
of memory on the same computing device. Then give the output of the following
operations:
(i) x × X (ii) 3 × X (iii) 3 × x (iv) x/X (v) X/x.
2. In the following problems, show all the steps involved in the computation.
i) Using 5-digit rounding, compute 37654 + 25.874 − 37679.
ii) Let a = 0.00456, b = 0.123, c = −0.128. Using 3-digit rounding, compute
(a + b) + c, and a + (b + c). What is your conclusion?
iii) Let a = 2, b = −0.6, c = 0.602. Using 3-digit rounding, compute a × (b + c),
and (a × b) + (a × c). What is your conclusion?
3. To find the mid-point of an interval [a, b], the formula a+b
2
is often used. Compute
the mid-point of the interval [0.982, 0.987] using 3-digit chopping. On the number
line represent all the three points. What do you observe? Now use the more
geometric formula a + b−a
2
to compute the mid-point, once again using 3-digit
chopping. What do you observe this time? Why is the second formula more
geometric?
4. In a computing device that uses n-digit rounding binary floating-point arithmetic,
show that δ = 2−n
is the machine epsilon. What is the machine epsilon in a
computing device that uses n-digit rounding decimal floating-point arithmetic?
Justify your answer.
Types of Errors
5. If fl(x) is the approximation of a real number x in a computing device, and ϵ is
the corresponding relative error, then show that fl(x) = (1 − ϵ)x.
6. Let x, y and z be real numbers whose floating point approximations in a com-
puting device coincide with x, y and z respectively. Show that the relative er-
ror in computing x(y + z) equals ϵ1 + ϵ2 − ϵ1ϵ2, where ϵ1 = Er(fl(y + z)) and
ϵ2 = Er(fl(x × fl(y + z))).
7. Let ϵ = Er(fl(x)). Show that
i) |ϵ| ≤ 10−n+1
if the computing device uses n-digit (decimal) chopping.
ii) |ϵ| ≤ 1
2
10−n+1
if the computing device uses n-digit (decimal) rounding.
iii) Can the equality hold in the above inequalities?
S. Baskar and S. Sivaji Ganesh
47
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-59-320.jpg)
![Section 3.2 Direct Methods for Linear Systems
for x2. Substituting the values of x1 and x2 in the first equation, we can solve for x1.
This solution phase of the (Naive) Gaussian elimination method is called backward
substitution phase.
Note
The coefficient matrix of the system (3.9) is an upper triangular matrix given by
U =
a11 a12 a13
0 a
(2)
22 a
(2)
23
0 0 a
(3)
33
. (3.10)
Define a lower triangular matrix L by
L =
1 0 0
m21 1 0
m31 m32 1
. (3.11)
It is easy to verify that LU = A.
Remark 3.2.1 [Why Naive?].
We explain why the word “Naive” is used for the method described above.
1. First of all, we do not know if the method described here can be successfully
applied for all systems of linear equations which are uniquely solvable (that
is, the coefficient matrix is invertible).
2. Secondly, even when we apply the method successfully, it is not clear if the
computed solution is the exact solution. In fact, it is not even clear that the
computed solution is close to the exact solution.
We illustrate the above points in Example 3.2.2 and Example 3.2.3.
Example 3.2.2.
Consider the system of equations
(
0 1
1 1
) (
x1
x2
)
=
(
1
2
)
. (3.12)
The Step 1 cannot be started as a11 = 0. Thus the naive Gaussian elimination method
fails.
S. Baskar and S. Sivaji Ganesh
55
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-67-320.jpg)
![Section 3.2 Direct Methods for Linear Systems
In general, an invertible matrix A need not be one among the simple structures listed
above. However, in certain situations we can always find an invertible lower triangular
matrix L and an invertible upper triangular matrix U in such a way that
A = LU.
In this case, the system Ax = b becomes
L
(
Ux
)
= b.
To solve for x, we first solve the lower triangular system
Lz = b
for the vector z, which can be obtained easily using forward substitution. After ob-
taining z, we solve the upper triangular system
Ux = z
for the vector x, which is again obtained easily using backward substitution.
Remark 3.2.6.
In Gaussian elimination method discussed in Section 3.2.1, we have seen that a
given matrix A can be reduced to an upper triangular matrix U by an elimination
procedure and thereby can be solved using backward substitution. In the elim-
ination procedure, we also obtained a lower triangular matrix L in such a way
that
A = LU
as remarked in this section.
The above discussion motivates us to look for a LU factorization of a given ma-
trix.
Definition 3.2.7 [LU factorization].
A matrix A is said to have LU factorization (or LU decomposition) if there
exists a lower triangular matrix L and an upper triangular matrix U such that
A = LU.
S. Baskar and S. Sivaji Ganesh
65
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-77-320.jpg)
![Chapter 3. Numerical Linear Algebra
Remark 3.2.8.
Clearly if a matrix has an LU decomposition, the matrices L and U are not unique
as
A = LU = (LD)(D−1
U) = ˜L ˜U
for any invertible diagonal matrix D. Note that A = ˜L ˜U is also an LU decompo-
sition of A as ˜L is a lower triangular matrix and ˜U is an upper triangular matrix.
We discuss three different ways of constructing LU factorization of a given matrix.
Doolittle’s factorization
Definition 3.2.9 [Doolittle’s factorization].
A matrix A is said to have a Doolittle’s factorization if there exists a lower triangular
matrix L with all diagonal elements as 1, and an upper triangular matrix U such
that
A = LU.
We now state the sufficient condition under which the Doolittle’s factorization of a
given matrix exists. For this, we need the notion of leading principal minors of a given
matrix, which we define first and then state the required theorem.
Definition 3.2.10 [Principal Minors of a Matrix].
Let A be an n × n matrix.
1. A sub-matrix of order k (< n) of the matrix A is a k × k matrix obtained by
removing n − k rows and n − k columns from A.
The determinant of such a sub-matrix of order k of A is called a minor of
order k of the matrix A.
2. The principal sub-matrix of order k of the matrix A is obtained by removing
the last n − k rows and the last n − k columns from A.
The determinant of the leading principal sub-matrix of order k of A is called
the principal minor of order k of the matrix A.
3. A principal sub-matrix and the corresponding principal minor are called the
leading principal sub-matrix and the leading principal minor of order
k, respectively, if k < n.
S. Baskar and S. Sivaji Ganesh
66
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-78-320.jpg)
![Section 3.2 Direct Methods for Linear Systems
Example 3.2.11.
Consider the 3 × 3 matrix
A =
a11 a12 a13
a21 a22 a23
a31 a32 a33
.
There are two leading principal minors corresponding to the matrix A.
• The leading principal minor of order 1 of A is
|a11| = a11.
• The leading principal minor of order 2 of A is
a11 a12
a21 a22
= a11a22 − a12a21.
We now state the sufficient condition for the existence of a Doolittle factorization of a
given matrix.
Theorem 3.2.12.
Let n ≥ 2, and A be an n × n invertible matrix such that all of its first n − 1 leading
principal minors are non-zero. Then A has an LU-decomposition where L is a unit
lower triangular matrix (i.e. all its diagonal elements equal 1). That is, A has a
Doolittle’s factorization.
We omit the proof of this theorem but illustrate the construction procedure of a Doolit-
tle factorization in the case when n = 3.
Example 3.2.13 [Computing the Doolittle’s factorization for a 3 × 3 matrix].
Let us illustrate the direct computation of the factors L and U of a 3 × 3 matrix
A, whenever its leading principal minors of order 1, 2, and 3 are non-zero. Write
A = LU as
a11 a12 a13
a21 a22 a23
a31 a32 a33
=
1 0 0
l21 1 0
l31 l32 1
u11 u12 u13
0 u22 u23
0 0 u33
(3.30)
S. Baskar and S. Sivaji Ganesh
67
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-79-320.jpg)
![Section 3.2 Direct Methods for Linear Systems
Thus we obtain the Doolittle’s factorization of A as
A =
1 0 0
1 1 0
−2 3 1
1 1 −1
0 1 −1
0 0 2
.
Further, taking b = (1, 1, 1)T
, we now solve the system Ax = b using LU factorization,
with the matrix A given above. As discussed earlier, first we have to solve the lower
triangular system
1 0 0
1 1 0
−2 3 1
z1
z2
z3
=
1
1
1
.
Forward substitution yields z1 = 1, z2 = 0, z3 = 3. Keeping the vector z = (1, 0, 3)T
as the right hand side, we now solve the upper triangular system
1 1 −1
0 1 −1
0 0 2
x1
x2
x3
=
1
0
3
.
Backward substitution yields x1 = 1, x2 = 3/2, x3 = 3/2.
Crout’s factorization
In Doolittle’s factorization, the lower triangular matrix has special property. If we ask
the upper triangular matrix to have special property in the LU decomposition, it is
known as the Crout’s factorization.
Definition 3.2.15 [Crout’s factorization].
A matrix A is said to have a Crout’s factorization if there exists a lower triangular
matrix L, and an upper triangular matrix U with all diagonal elements as 1 such
that
A = LU.
The computation of the Crout’s factorization for a 3×3 matrix can be done in a similar
way as done for Doolittle’s factorization in Example 3.2.13. For invertible matrices, one
can easily obtain Doolittle’s factorization from Crout’s factorization and vice versa.
Cholesky’s factorization
Before we introduce Cholesky’s factorization, we recall the concept of a positive definite
matrix.
S. Baskar and S. Sivaji Ganesh
69
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-81-320.jpg)
![Chapter 3. Numerical Linear Algebra
Definition 3.2.16 [Positive Definite Matrix].
A symmetric matrix A is said to be positive definite if
xT
Ax > 0,
for every non-zero vector x.
We recall (from a course on Linear Algebra) a useful theorem concerning positive defi-
nite matrices.
Lemma 3.2.17.
The following statements concerning a symmetric n × n matrix A are equivalent.
1. The matrix A is positive definite.
2. All the principal minors of the matrix A are positive.
3. All the eigenvalues of the matrix A are positive.
The statements (1) and (2) are equivalent by definition. The proof of equivalence of
(3) with other two statements is out of the scope of this course.
We now define Cholesky’s factorization.
Definition 3.2.18 [Cholesky’s factorization].
A matrix A is said to have a Cholesky’s factorization if there exists a lower
triangular matrix L such that
A = LLT
.
Remark 3.2.19.
It is clear from the definition that if a matrix has a Cholesky’s factorization, then
the matrix have to be necessarily a symmetric matrix.
We now establish a sufficient condition for the existence of the Cholesky’s factoriza-
tion.
Theorem 3.2.20.
If A is an n × n symmetric and positive definite matrix, then A has a unique factor-
ization
A = LLT
,
where L is a lower triangular matrix with positive diagonal elements.
S. Baskar and S. Sivaji Ganesh
70
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-82-320.jpg)
![Section 3.3 Matrix Norms and Condition Number of a Matrix
Thus, if we want to solve a large set of linear systems where the coefficient matrix A
is fixed but the right hand side vector b varies, we can do the factorization once for
A and then use this factorization with different b vectors. This is obviously going to
reduce the computation cost drastically as we have seen in the operation counting of
Gaussian elimination method that the elimination part is the most expensive part of
the computation.
3.3 Matrix Norms and Condition Number of a Matrix
Given any two real numbers α and β, the distance between them is given by |α − β|.
We can also compare their magnitudes |α| and |β|. Vector norm may be thought of
as a generalization of the modulus | · | concept to deal with vectors and in general
matrices.
Definition 3.3.1 [Vector Norm].
A vector norm on Rn
is a function ∥ · ∥ : Rn
→ [0, ∞) having the following
properties:
1. ∥x∥ ≥ 0 for all x ∈ Rn
.
2. ∥x∥ = 0 if and only if x = 0.
3. ∥αx∥ = |α|∥x∥ for all x ∈ Rn
and for all α ∈ R.
4. ∥x + y∥ ≤ ∥x∥ + ∥y∥ for all x, y ∈ Rn
.
The condition 4 in the above definition is called the triangle inequality.
Note
We use the following notation for a vector x ∈ Rn in terms of its components:
x = (x1, x2, · · · , xn)T
.
Example 3.3.2.
There can be many vector norms on Rn
. We define three important vector norms on
Rn
, which are frequently used in matrix analysis.
1. The Euclidean norm (also called the l2-norm) on Rn
is denoted by ∥ · ∥2,
and is defined by
∥x∥2 =
n∑
i=1
|xi|2. (3.37)
S. Baskar and S. Sivaji Ganesh
73
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-85-320.jpg)
![Chapter 3. Numerical Linear Algebra
2. The l∞-norm, also called maximum norm, on Rn
is denoted by ∥ · ∥∞, and
is defined by
∥x∥∞ = max { |x1|, |x2|, · · · , |xn| } . (3.38)
3. l1-norm on Rn
is denoted by ∥ · ∥1, and is defined by
∥x∥1 = |x1| + |x2| + · · · + |xn|. (3.39)
All the three norms defined above are indeed norms; it is easy to verify that they
satisfy the defining conditions of a norm given in Definition 3.3.1.
Let us illustrate each of the above defined norms with some specific vectors.
Example 3.3.3.
Let us compute norms of some vectors now. Let x = (4, 4, −4, 4)T
, y = (0, 5, 5, 5)T
,
z = (6, 0, 0, 0)T
. Verify that ∥x∥1 = 16, ∥y∥1 = 15, ∥z∥1 = 6; ∥x∥2 = 8, ∥y∥2 = 8.66,
∥z∥2 = 6; ∥x∥∞ = 4, ∥y∥∞ = 5, ∥z∥∞ = 6.
From this example we see that asking which vector is big does not make sense. But
once the norm is fixed, this question makes sense as the answer depends on the norm
used. In this example each vector is big compared to other two but in different norms.
Remark 3.3.4.
In our computations, we employ any one of the norms depending on convenience.
It is a fact that “all vector norms on Rn are equivalent”; we will not elaborate
further on this.
We can also define matrix norms on the vector space Mn(R) of all n × n real matrices,
which helps us in finding distance between two matrices.
Definition 3.3.5 [Matrix Norm].
A matrix norm on the vector space of all n × n real matrices Mn(R) is a function
∥ · ∥ : Mn(R) → [0, ∞) having the following properties:
1. ∥A∥ ≥ 0 for all A ∈ Mn(R).
2. ∥A∥ = 0 if and only if A = 0.
3. ∥αA∥ = |α|∥A∥ for all A ∈ Mn(R) and for all α ∈ R.
4. ∥A + B∥ ≤ ∥A∥ + ∥B∥ for all A, B ∈ Mn(R).
S. Baskar and S. Sivaji Ganesh
74
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-86-320.jpg)
![Section 3.3 Matrix Norms and Condition Number of a Matrix
Note
As in the case of vector norms, the condition 4 in the above definition is called
the triangle inequality.
For a matrix A ∈ Mn(R), we use the notation
A = (aij)1≤i≤n, 1≤j≤n
where aij denotes the element in the ith row and jth column of A.
There can be many matrix norms on Mn(R). We will describe some of them now.
Example 3.3.6.
The following define norms on Mn(R).
1. ∥A∥ =
n∑
i=1
n∑
j=1
|aij|2 .
2. ∥A∥ = max { |aij| : 1 ≤ i ≤ n, 1 ≤ j ≤ n } .
3. ∥A∥ =
n∑
i=1
n∑
j=1
|aij|.
All the three norms defined above are indeed norms; it is easy to verify that they
satisfy the defining conditions of a matrix norm of Definition 3.3.5.
Among matrix norms, there are special ones that satisfy very useful and important
properties. They are called matrix norms subordinate to a vector norm. As the name
suggests, to define them we need to fix a vector norm. We will give a precise definition
now.
Definition 3.3.7 [Matrix Norm subordinate to a vector norm].
Let ∥ · ∥ be a vector norm on Rn
and let A ∈ Mn(R). The matrix norm of A
subordinate to the vector norm ∥ · ∥ is defined by
∥A∥ := sup {∥Ax∥ : x ∈ Rn
, ∥x∥ = 1} . (3.40)
The formula (3.40) indeed defines a matrix norm on Mn(R). The proof of this fact is
beyond the scope of our course. In this course, by matrix norm, we always mean a norm
subordinate to some vector norm. An equivalent and more useful formula for the matrix
norm subordinate to a vector norm is given in the following lemma.
S. Baskar and S. Sivaji Ganesh
75
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-87-320.jpg)
![Chapter 3. Numerical Linear Algebra
Lemma 3.3.8.
For any A ∈ Mn(R) and a given vector norm ∥ · ∥, we have
∥A∥ = max
z̸=0
∥Az∥
∥z∥
. (3.41)
Proof.
For any z ̸= 0, we have x = z/||z|| as a unit vector. Hence
max
||x||=1
||Ax|| = max
||z||̸=0
A
(
z
||z||
)
= max
||z||̸=0
||Az||
||z||
.
The matrix norm subordinate to a vector norm has additional properties as stated in
the following theorem whose proof is left as an exercise.
Theorem 3.3.9.
Let ∥ · ∥ be a matrix norm subordinate to a vector norm. Then
1. ∥Ax∥ ≤ ∥A∥∥x∥ for all x ∈ Rn
.
2. ∥I∥ = 1 where I is the identity matrix.
3. ∥AB∥ ≤ ∥A∥ ∥B∥ for all A, B ∈ Mn(R).
Note
We do not use different notations for a matrix norm and the subordinate vector
norm. These are to be understood depending on the argument.
We will now state a few results concerning matrix norms subordinate to some of the
vector norms described in Example 3.3.2. We omit their proofs.
Theorem 3.3.10 [Matrix norm subordinate to the maximum norm].
The matrix norm subordinate to the l∞-norm given in (3.38) on Rn
is denoted by
∥A∥∞ and is given by
∥A∥∞ = max
1≤i≤n
n∑
j=1
|aij|, (3.42)
which is called the maximum-of-row-sums norm.
S. Baskar and S. Sivaji Ganesh
76
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-88-320.jpg)
![Section 3.3 Matrix Norms and Condition Number of a Matrix
Theorem 3.3.11 [Matrix Norm Subordinate to the l1-norm].
The matrix norm subordinate to the l1-norm given in (3.39) on Rn
is denoted by
∥A∥1 and is given by
∥A∥1 = max
1≤j≤n
n∑
i=1
|aij|. (3.43)
The norm ∥A∥1 is called the maximum-of-column-sums norm of A.
Description and computation of the matrix norm subordinate to the Euclidean vector
norm on Rn
is more subtle.
Theorem 3.3.12 [Matrix norm subordinate to the Euclidean norm].
The matrix norm subordinate to the l2-norm (Euclidean norm) given by (3.37) on
Rn
is denoted by ∥A∥2 and is given by
∥A∥2 =
√
max
1≤i≤n
|λi|, (3.44)
where λ1, λ2, · · · , λn are eigenvalues of the matrix AT
A. The norm ∥A∥2 given by
the formula (3.44) is called the spectral norm of A.
Example 3.3.13.
Let us now compute ∥A∥∞ and ∥A∥2 for the matrix
1 1 −1
1 2 −2
−2 1 1
.
1. ∥A∥∞ = 5 since
3∑
j=1
|a1j| = |1| + |1| + | − 1| = 3,
3∑
j=1
|a2j| = |1| + |2| + | − 2| = 5,
3∑
j=1
|a3j| = | − 2| + |1| + |1| = 4.
2. ∥A∥2 ≈ 3.5934 as the eigenvalues of AT
A are λ1 ≈ 0.0616, λ2 ≈ 5.0256 and
λ3 ≈ 12.9128. Hence ∥A∥2 ≈
√
12.9128 ≈ 3.5934.
S. Baskar and S. Sivaji Ganesh
77
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-89-320.jpg)
![Section 3.3 Matrix Norms and Condition Number of a Matrix
Notice that the constant appearing on the right hand side of the inequality (3.45) (which
is ∥A∥ ∥A−1
∥) depends only on the matrix A (for a given vector norm). This number
is called the condition number of the matrix A. Notice that this condition number
depends very much on the vector norm being used on Rn
and the matrix norm that is
subordinate to the vector norm.
Definition 3.3.16 [Condition Number of a Matrix].
Let A be an n×n invertible matrix. Let a matrix norm be given that is subordinate
to a vector norm. Then the condition number of the matrix A (denoted by κ(A))
is defined as
κ(A) := ∥A∥ ∥A−1
∥. (3.47)
Remark 3.3.17.
From Theorem 3.3.14, it is clear that if the condition number is small, then the
relative error in the solution will also be small whenever the relative error in the
right hand side vector is small. On the other hand, if the condition number is very
large, then the relative error could be large even though the relative error in the
right hand side vector is small. We illustrate this in the following example.
Example 3.3.18.
The linear system
5x1 + 7x2 = 0.7
7x1 + 10x2 = 1
has the solution x1 = 0, x2 = 0.1. Let us denote this by x = (0, 0.1)T
, and the right
hand side vector by b = (0.7, 1)T
. The perturbed system
5x1 + 7x2 = 0.69
7x1 + 10x2 = 1.01
has the solution x1 = −0.17, x2 = 0.22, which we denote by ˜x = (−0.17, 0.22)T
,
and the right hand side vector by ˜b = (0.69, 1.01)T
. The relative error between the
solutions of the above systems in the l∞ vector norm is given by
∥x − ˜x∥∞
∥x∥∞
= 1.7,
S. Baskar and S. Sivaji Ganesh
79
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-91-320.jpg)
![Chapter 3. Numerical Linear Algebra
The above two examples shows that the Jacobi sequence need not converge always and
so we need to look for a condition on the system for which the Jacobi iterative sequence
converges to the exact solution.
Define the error in the kth
iterate x(k)
compared to the exact solution by
e(k)
= x − x(k)
.
It follows easily that e(k)
satisfies the system
e(k+1)
= Be(k)
,
where B is as defined in (3.51). Using any vector norm and the matrix norm subordinate
to it in the above equation, we get
∥e(k+1)
∥ = ∥Be(k)
∥
≤ ∥B∥∥e(k)
∥
≤ · · ·
≤ · · ·
≤ · · ·
≤ ∥B∥k+1
∥e(0)
∥.
Thus, when ∥B∥ < 1, the iteration method (3.51) always converges for any initial guess
x(0)
.
Again the question is
“what are all the matrices A for which the corresponding matrices B in (3.51) have
the property ∥B∥ < 1, for some matrix norm subordinate to some vector norm?”
One such class of matrices are the diagonally dominant matrices, which we define
now.
Definition 3.4.4 [Diagonally Dominant Matrices].
A matrix A is said to be diagonally dominant if it satisfies the inequality
n∑
j=1,j̸=i
|aij| < |aii|, i = 1, 2, · · · , n.
We now prove the sufficient condition for the convergence of the Jacobi method. This
theorem asserts that if A is a diagonally dominant matrix, then B in (3.51) of the
Jacobi method is such that ∥B∥∞ < 1.
S. Baskar and S. Sivaji Ganesh
86
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-98-320.jpg)
![Chapter 3. Numerical Linear Algebra
Then with i = l in (3.62),
∥e(k+1)
∥∞ ≤ αl∥e(k+1)
∥∞ + βl∥e(k)
∥∞. (3.63)
Since µ < 1, we have αl < 1 and therefore the above inequality gives
∥e(k+1)
∥∞ ≤
βl
1 − αl
∥e(k)
∥∞. (3.64)
Define
η = max
1≤i≤n
βi
1 − αi
. (3.65)
Then the above inequality yields
∥e(k+1)
∥∞ ≤ η∥e(k)
∥∞. (3.66)
Since for each i,
(αi + βi) −
βi
1 − αi
=
αi[1 − (αi + βi)]
1 − αi
≥
αi
1 − αi
[1 − µ] ≥ 0, (3.67)
we have
η ≤ µ < 1. (3.68)
Thus, when the coefficient matrix A is diagonally dominant, Gauss-Seidel method
converges.
Remark 3.4.10.
A careful observation of the proof of the above theorem reveals that the Gauss-
Seidel method converges faster than the Jacobi method by comparing (3.68) and
(3.59).
3.4.3 Mathematical Error
Let x∗
denote the computed solution using some method. The mathematical error
in the approximate solution when compared to the exact solution of a linear system
Ax = b is given by
e = x − x∗
. (3.69)
Recall from Chapter 2 that the mathematical error is due to the approximation made
in the numerical method where the computation is done without any floating-point
S. Baskar and S. Sivaji Ganesh
90
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-102-320.jpg)
![Chapter 3. Numerical Linear Algebra
Example 3.4.11 [Computation of residual at a higher precision].
Consider the system
0.729x1 + 0.81x2 + 0.9x3 = 0.6867
x1 + x2 + x3 = 0.8338
1.331x1 + 1.210x2 + 1.100x3 = 1.000
The exact solution of this system with 4-digit rounding is
x1 ≈ 0.2245, x2 ≈ 0.2814, x3 ≈ 0.3279.
The solution of the system by Gaussian elimination without pivoting using 4-digit
rounding leads to
x∗
1 ≈ 0.2251, x∗
2 ≈ 0.2790, x∗
3 ≈ 0.3295.
As we have used 4-digit rounding arithmetic in obtaining the approximate solution
vector x∗
, we use 8-digit rounding arithmetic in computing the residual, which is
obtained as
r = (0.00006210, 0.0002000, 0.0003519)T
.
Solving the linear system Ae = r using 8-digit rounding arithmetic, we obtain the
approximation as
e∗
= [−0.00044710, 0.00215000, −0.00150400]T
.
Compare this to the true error
e = x − x∗
= [−0.0007, 0.0024, −0.0016]T
Thus e∗
gives a good idea of the size of the error e in the computed solution x∗
.
Let us now propose an iterative procedure by first predicting the error by solving the
system (3.71) and then correcting the approximate solution x∗
by adding the predicted
error to the vector x∗
.
If we take x∗
= x(0)
, and define r(0)
= b − Ax(0)
, then the error e(0)
= x − x(0)
can be
obtained by solving the linear system
Ae(0)
= r(0)
.
Now, define
x(1)
= x(0)
+ e(0)
.
S. Baskar and S. Sivaji Ganesh
92
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-104-320.jpg)
![Section 3.5 Eigenvalue Problems
3.5 Eigenvalue Problems
Let A be an n × n matrix with real entries. Eigenvalues of A are defined as the roots
of the equation
det(λI − A) = 0. (3.74)
Note that det(λI −A) is a polynomial in λ of degree n, which is known as characteristic
polynomial of the matrix A. We know that even if A has real entries, the eigenvalues
need not be real numbers. It is also a known fact that for every matrix, there are n
eigenvalues λ1, λ2, · · · , λn (in this list, each eigenvalue is repeated as many times as its
algebraic multiplicity, i.e., multiplicity of the eigenvalue as a root of the characteristic
polynomial).
When n = 2 the characteristic polynomial is a quadratic polynomial for which there
is a nice formula for computing roots. When n = 3, there is a formula that many of
us do not remember. When n = 4, none has a formula. But computing eigenvalues
is important for applications. Therefore, numerically approximating the eigenvalues is
the only way out.
One obvious way of approximating an eigenvalue of a matrix is to first obtain the char-
acteristic polynomial (3.74) explicitly in λ and then use one of the numerical methods
discussed in the next chapter (on methods for solving nonlinear equations) to compute
a root of this polynomial. But this is not an efficient way of computing eigenvalues
because of two reasons. One reason is that obtaining explicit form of (3.74) is itself a
difficult task when the dimension of the matrix is very large. Secondly, if we make any
small error (like floating-point error) in obtaining the explicit form of the polynomial
(3.74), the resulting polynomial may have a root which is entirely different from any
of the eigenvalues that we are looking for. This is illustrated in the following example
by Wilkinson where we see that “the roots of polynomials are extremely sensitive to
perturbations in the coefficients”.
Example 3.5.1 [Wilkinson’s example].
Let f(x) and g(x) be two polynomials given by
f(x) = (x − 1)(x − 2) · · · (x − 10), g(x) = x10
.
The roots of the polynomial f(x) are 1, 2, · · · , 10, and all these roots are simple roots.
If we perturb this polynomial as F(x) = f(x) + 0.01g(x), then all the roots lie in the
interval [1, 3.5] (verified graphically). In fact, the largest root of the polynomial f(x)
is 10 and the largest root of the polynomial F(x) is approximately equal to 3.398067.
Thus, if the coefficient of x10
is perturbed by a small amount of 0.01, the root 10 of
f(x) could move as much a distance as approximately 6.6.
S. Baskar and S. Sivaji Ganesh
95
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-107-320.jpg)
![Chapter 3. Numerical Linear Algebra
Due to the two reasons discussed above, we look for an alternate method to compute
the eigenvalues of a given matrix. One such method is the power method that can be
used to obtain the eigenvalue which is the largest in magnitude among all the other
eigenvalues and the corresponding eigen vector. In Subsection 3.5.1, we present the
power method and discuss the condition under which this method can be applied. In
Subsection 3.5.2 we prove the Gerschgorin theorem which may be used as a tool to find
a class of matrices for which power method can be applied successfully.
3.5.1 Power Method
There are many variations of power method in the literature. We will present the most
elementary form of power method. We always deal with matrices with real entries, all
of whose eigenvalues are real numbers.
Power method is used to obtain a specific eigenvalue called dominant eigenvalue and
a corresponding eigenvector for a given n × n matrix A. The concept of a dominant
eigenvalue plays a very important role in many applications. The power method pro-
vides an approximation to it under some conditions on the matrix. We now define the
concept of a dominant eigenvalue.
Definition 3.5.2 [Dominant Eigenvalue of a Matrix].
An eigenvalue λ of an n × n matrix A is said to be a dominant eigenvalue of A if
|λ| = max{ |z| : z is an eigenvalue of A }.
Remark 3.5.3.
1. If a dominant eigenvalue of a matrix is equal to zero, then all its eigenvalues
are zero and an eigenvector can be found by solving the linear system Ax = 0.
2. If λ is a dominant eigenvalue of a matrix A, then there is no other eigenvalue
of A whose distance from zero is more than that of λ from zero.
3. Let µ1, µ2, · · · , µn be the eigenvalues of A (repeated according to their alge-
braic multiplicities) of an n×n matrix A. These eigenvalues can be renamed
(relabelled, reindexed) as λ1, λ2, · · · , λn such that they satisfy the condition:
|λ1| ≥ |λ2| ≥ · · · ≥ |λn|.
Note that λ1 is a dominant eigenvalue of A.
S. Baskar and S. Sivaji Ganesh
96
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-108-320.jpg)
![Chapter 3. Numerical Linear Algebra
Iteration No: 7
y7 = Ax6 = (1.498715, −0.000343, 2.998287)T
µ7 = 2.998287
x7 =
y7
µ7
= (0.499857, −0.000114, 1.000000)T
Iteration No: 8
y8 = Ax7 = (1.499571, −0.000114, 2.999429)T
µ8 = 2.999429
x8 =
y8
µ8
= (0.499952, −0.000038, 1.000000)T
Iteration No: 9
y9 = Ax8 = (1.499857, −0.000038, 2.999809)T
µ9 = 2.999809
x9 =
y9
µ9
= (0.499984, −0.000013, 1.000000)T
Iteration No: 10
y10 = Ax9 = (1.499952, −0.000013, 2.999936)T
µ10 = 2.999936
x10 =
y10
µ10
= (0.499995, −0.000004, 1.000000)T
These ten iterates suggest that the sequence {µk} converges to the eigenvalue λ1 = 3
and the sequence {x(k)
} converges to (0.5, 0, 1) =
1
2
v1.
Remark 3.5.8 [Disadvantages of power method].
1. The Power method requires at the beginning that the matrix has only one
dominant eigenvalue, and this information is generally unavailable.
2. Even when there is only one dominant eigenvalue, it is not clear how to
choose the initial guess x(0) such that it has a non-zero component (c1 in the
notation of the theorem) along the eigenvector v1.
S. Baskar and S. Sivaji Ganesh
102
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-114-320.jpg)
![Section 3.5 Eigenvalue Problems
Note that in the above example, all the hypothesis are satisfied. Now let us ask the
question
“What happens when any of the hypotheses of power method is violated?”
We discuss these situations through examples.
Example 3.5.9 [Dominant eigenvalue is not unique (Failure of H1)].
Consider the matrix
B =
1 3 4
0 2 1
0 0 −2
,
which has eigenvalues 1, −2, and 2. Clearly, the matrix B has two dominant eigen-
values, namely, −2 and 2. We start with an initial guess x(0)
= (1, 1, 1) and the first
five iterations generated using power method are given below:
Iteration No: 1
y1 = Ax0 = (8.000000, 3.000000, −2.000000)T
µ1 = 8.000000
x1 =
y1
µ1
= (1.000000, 0.375000, −0.250000)T
Iteration No: 2
y2 = Ax1 = (1.125000, 0.500000, 0.500000)T
µ2 = 1.125000
x2 =
y2
µ2
= (1.000000, 0.444444, 0.444444)T
Iteration No: 3
y3 = Ax2 = (4.111111, 1.333333, −0.888889)T
µ3 = 4.111111
x3 =
y3
µ3
= (1.000000, 0.324324, −0.216216)T
Iteration No: 4
y4 = Ax3 = (1.108108, 0.432432, 0.432432)T
µ4 = 1.108108
x4 =
y4
µ4
= (1.000000, 0.390244, 0.390244)T
S. Baskar and S. Sivaji Ganesh
103
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-115-320.jpg)
![Chapter 3. Numerical Linear Algebra
Iteration No: 5
y5 = Ax4 = (3.731707, 1.170732, −0.780488)T
µ5 = 3.731707
x5 =
y5
µ5
= (1.000000, 0.313725, −0.209150)T
It is observed that the sequence oscillates even till 1000 iterations as shown below:
Iteration No: 998
y998 = Ax997 = (1.103448, 0.413793, 0.413793)T
µ998 = 1.103448
x998 =
y998
µ998
= (1.000000, 0.375000, 0.375000)T
Iteration No: 999
y999 = Ax998 = (3.625000, 1.125000, −0.750000)T
µ999 = 3.625000
x999 =
y999
µ999
= (1.000000, 0.310345, −0.206897)T
Iteration No: 1000
y1000 = Ax999 = (1.103448, 0.413793, 0.413793)T
µ1000 = 1.103448
x1000 =
y1000
µ1000
= (1.000000, 0.375000, 0.375000)T
and so on. This is a clear indication that the power method is not converging in this
case.
Thus we conclude that the power method when applied to a matrix which has more
than one dominant eigenvalue may not converge.
Remark 3.5.10 [Dominant eigenvalue is not simple].
It is not necessary to have the dominant eigenvalue of algebraic multiplicity 1 in
order that the iterative sequences of power method converge. The important thing
is to have a unique dominant eigenvalue and it is allowed to have an algebraic
multiplicity r with r > 1. However it is necessary that the corresponding geometric
multiplicity should also be r to satisfy the hypothesis (H2) (also see later on, where
we discuss the situation where algebraic and geometric multiplicities do not match).
In such a case, power method computes only one eigenvector, as is usual.
S. Baskar and S. Sivaji Ganesh
104
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-116-320.jpg)
![Section 3.5 Eigenvalue Problems
Let us now illustrate the situation when the hypothesis (H3) is violated.
Example 3.5.11 [Failure of hypothesis (H3): Initial guess x(0) is such that c1 = 0].
Consider the matrix (same as in Example 3.5.7)
A =
3 0 0
−4 6 2
16 −15 −5
,
The eigenvalues of this matrix are
λ1 = 3, λ2 = 1 and λ3 = 0.
The corresponding eigenvectors are
v1 = (1, 0, 2)T
, v2 = (0, 2, −5)T
and v3 = (0, 1, −3)T
.
Thus, the hypothesis (H1) and (H2) are satisfied.
Here, we choose a different initial guess x0 = (0, 0.5, 0.25)T
. Note that the hypothesis
(H3) that c1 ̸= 0 is violated here. However, we can see that c2 ̸= 0. The first four
iterations of the power method are as follows:
Iteration No: 1
y1 = Ax0 = (0.000000, 3.500000, −8.750000)T
µ1 = −8.750000
x1 =
y1
µ1
= (−0.000000, −0.400000, 1.000000)T
Iteration No: 2
y2 = Ax1 = (0.000000, −0.400000, 1.000000)T
µ2 = 1.000000
x2 =
y2
µ2
= (0.000000, −0.400000, 1.000000)T
Iteration No: 3
y3 = Ax2 = (0.000000, −0.400000, 1.000000)T
µ3 = 1.000000
x3 =
y3
µ3
= (0.000000, −0.400000, 1.000000)T
S. Baskar and S. Sivaji Ganesh
105
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-117-320.jpg)
![Section 3.5 Eigenvalue Problems
Note that the matrix A satisfies the hypothesis (H1) since -5 is the unique dominant
eigenvalue and it is also a simple eigenvalue. The matrix A satisfies the hypothesis
(H2) as all eigenvalues are distinct and hence eigevectors form a basis for R3
. Thus
the fate of the power method iterates depends solely on the choice of the initial guess
x(0)
and whether it satisfies the hypothesis (H3)
• Let us take the initial guess x(0)
= (1, 0.5, 0.25)T
. Note that c1 ̸= 0 for this ini-
tial guess. Thus the initial guess satisfies the hypothesis (H3) and the iterative
sequences generated by power method converges to the dominant eigenvalue
λ1 = −5 and the corresponding eigenvector (with a scalar multiple) 1
5
v1.
• Let us take the initial guess x(0)
= (0, 0.5, 0.25)T
. Note that c1 ̸= 0 for this ini-
tial guess. Thus the initial guess satisfies the hypothesis (H3) and the iterative
sequences generated by power method converges to the dominant eigenvalue
λ1 = −5 and the corresponding eigenvector (with a scalar multiple) 1
5
v1. Com-
pare this with Example 3.5.11. In the present case the first coordinate of the
initial guess vector is zero, just as in Example 3.5.11. In Example 3.5.11 the
power method iterate converged to the second dominant eigenvalue and the
corresponding eigenvector, which does not happen in the present case. The
reason is that in the Example 3.5.11, c1 = 0 for the initial guess chosen, but in
the current example c1 ̸= 0.
3.5.2 Gerschgorin’s Theorem
An important tool in eigenvalue approximation is the ability to localize the eigenvalues,
and the most important tool in eigenvalue localization is the Gerschgorin’s theorem.
Gerschgorin’s Circle theorem helps us in localization of eigenvalues of a matrix. This
theorem explicitly constructs n disks in the complex plane with centers at the diagonal
elements of the matrix; and all the eigenvalues of the matrix lie in the union of these
disks.
Theorem 3.5.14 [Gerschgorin’s Circle Theorem].
Let A be an n × n matrix. For each k = 1, 2, · · · , n, define ρk by
ρk =
n∑
j=1
j̸=k
|akj|,
and Dk denotes the closed disk in the complex plane with centre akk and radius ρk,
i.e.,
Dk =
{
z ∈ C : |z − akk| ≤ ρk
}
.
S. Baskar and S. Sivaji Ganesh
107
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-119-320.jpg)
![Chapter 3. Numerical Linear Algebra
13. Let A be an n×n matrix which is positive definite. Let S be a non-empty susbset
of {1, 2, · · · , n}. Show that the submatrix AS = (aij)i,j∈S is positive definite.
14. Factor the matrix
A =
4 6 2
6 10 3
2 3 5
so that A = LLT
, where L is lower triangular.
15. Prove the uniqueness of the factorization A = LLT
, where L is a lower triangular
matrix all of whose diagonal entries are positive. (Hint: Assume that there
are lower triangular matrices L1 and L2 with positive diagonals. Prove that
L1L−1
2 = I.)
16. Use Cholesky factorization to solve the system of equations
x1 + 3x3 = 1,
2x1 − x2 = 3,
x1 + 2x3 = −1.
[Hint: Instead of solving for Ax = b, solve for AT
Ax = AT
b.]
Matrix Norms
17. The following inequalities show that the notion of convergent sequences of vectors
in Rn
is independent of the vector norm. Show that the following inequalities hold
for each x ∈ Rn
i) ∥x∥∞ ≤ ∥x∥2 ≤
√
n∥x∥∞,
ii) ∥x∥∞ ≤ ∥x∥1 ≤ n∥x∥∞,
iii) ∥x∥2 ≤ ∥x∥1 ≤
√
n∥x∥2.
18. Show that the norm defined on the set of all n × n matrices by
∥A∥ := max
1≤i≤n
1≤j≤n
|aij|
is not subordinate to any vector norm on Rn
.
19. Let A be an invertible matrix. Show that its condition number κ(A) satisfies
κ(A) ≥ 1.
S. Baskar and S. Sivaji Ganesh
114
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-126-320.jpg)
![Chapter 3. Numerical Linear Algebra
25. For each of the following systems, write down the formula for iterating sequences
of Jacobi and Gauss-Seidel methods. Compute three iterates by taking x0
=
(0, 0, 0)T
. Discuss if you can guarantee that these sequences converge to the exact
solution. In case you are not sure about convergence, suggest another iterating
sequence that converges to the exact solution if possible; and justify that the new
sequence converges to the exact solution.
i) 5x1 +2x2 +x3 = 0.12, 1.75x1 +7x2 +0.5x3 = 0.1, x1 +0.2x2 +4.5x3 = 0.5.
ii) x1 − 2x2 + 2x3 = 1, x1 + x2 − x3 = 1, 2x1 − 2x2 + x3 = 1.
iii) x1 + x2 + 10x3 = −1, 2x1 + 3x2 + 5x3 = −6, 3x1 + 2x2 − 3x3 = 4.
[Note: As it is only asked to discuss the guarantee of convergence, it is enough
to check the sufficient condition for convergence, ie. to check whether the given
system is diagonally dominant or not.]
26. Consider the following system of linear equations
0.8647x1 + 0.5766x2 = 0.2885, 0.4322x1 + 0.2882x2 = 0.1442.
i) Find the exact solution of the linear system.
ii) Using 4-digit chopping, do one iteration of residual corrector method to
obtain x(1)
. Use the initial guess to be (0, 0)T
.
iii) Compute the residual vector for the solution obtained in (ii) above.
Eigenvalue Problems
27. The matrix
A =
−3 0 0
2 1 0
1 0 2
has eigenvalues λ1 = −3, λ = 1 and λ3 = 2 and the corresponding eigenvectors
may be taken as v1 = (1, 0, 0)T
, v2 = (1, 2, 0)T
and v3 = (1, 0, 5)T
. Perform
3 iterations to find the eigenvalue and the corresponding eigen vector to which
the power method converges when we start the iteration with the initial guess
x(0)
= (0, 0.5, 0.75)T
. Without performing the iteration, find the eigenvalue and
the corresponding eigenvector to which the power method converges when we
start the iteration with the initial guess x(0)
= (0.001, 0.5, 0.75)T
. Justify your
answer.
S. Baskar and S. Sivaji Ganesh
116
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-128-320.jpg)
![Chapter 3. Numerical Linear Algebra
31. Use Gerschgorin’s circle theorem to determine the intervals in which the eigen-
values of the matrix
A =
0.5 0 0.2
0 3.15 −1
0.57 0 −7.43
.
lie, given that all eigenvalues of A are real. Show that power method can be
applied for this matrix to find the dominant eigenvalue without computing eigen-
values explicitly. Compute the first three iterates of Power method sequences.
32. Use the Gerschgorin Circle theorem to determine bounds for the eigenvalues for
the following matrices. Also find optimum bounds wherever possible. Also draw
the pictures of all the regions given by Greschgorin circles.
1 0 0
−1 0 1
−1 −1 2
,
4 −1 0
−1 4 −1
−1 −1 4
,
4.75 2.25 −0.25
2.25 4.75 1.25
−0.25 1.25 4.75
,
1 0 −1 1
2 2 −1 1
0 1 3 −2
1 0 1 4
.
33. Show that the imaginary parts of the eigenvalues of
3 1/3 2/3
1 −4 0
1/2 1/2 −1
all lie in the interval [−1, 1].
S. Baskar and S. Sivaji Ganesh
118
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-130-320.jpg)
![CHAPTER 4
Nonlinear Equations
One of the most frequently occurring problems in practical applications is to find the
roots of equations of the form
f(x) = 0, (4.1)
where f : [a, b] → R is a given nonlinear function. It is well-known that not all nonlinear
equations can be solved explicitly to obtain the exact value of the roots and hence, we
need to look for methods to compute approximate value of the roots. By approximate
root to (4.1), we mean a point x∗
∈ R for which the value f(x) is very near to zero, ie.,
f(x∗
) ≈ 0.
In this chapter, we introduce various iterative methods to obtain an approximation to
a real root of equations of the form (4.1) with f being a continuous nonlinear function.
The key idea in approximating the real roots of (4.1) consists of two steps:
1. Starting Step: Take one or more points (arbitrarily or following a procedure)
xi ∈ [a, b] (i = 0, 1, · · · , m, m ∈ N) around a root of the equation (4.1). Consider
xm as an approximation to the root of (4.1).
2. Improving Step: If xm is not ‘close’ to the required root, then devise a procedure
to obtain another point xm+1 that is ‘more close’ to the root than xm.
Repeat the improving step until we obtain a point xn (n ≥ m) which is ‘sufficiently
close’ to the required root.
This process of improving the approximation to the root is called the iterative process
(or iterative procedure), and such methods are called iterative methods. In an
iterative method, we obtain a sequence of numbers {xn} which is expected to converge
S. Baskar and S. Sivaji Ganesh
119
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-131-320.jpg)
![Chapter 4. Nonlinear Equations
to the root of (4.1) as n → ∞. We investigate conditions on the function f, its domain,
and co-domain under which the sequence of iterates converge to a root of the equation
(4.1).
We classify the iterative methods discussed in this chapter into two types, namely,
1. Closed Domain Methods: As the starting step, these methods need the knowl-
edge of an interval in which at least one root of the given nonlinear equation ex-
ists. Further iterations include the restriction of this interval to smaller intervals
in which root lies. These methods are also called bracketing methods.
2. Open Domain Methods: The xi’s mentioned in the starting step above are
chosen arbitrarily and the consecutive iterations are based on a formula.
In the case of closed domain methods, the difficult part is to locate an interval containing
a root. But, once this is done, the iterative sequence will surely converge as we will
see in Section 4.1. In this section, we discuss two closed domain methods, namely, the
bisection method and the regula-falsi method. In the case of open domain methods,
it is easy at the starting step as we can choose the xi’s arbitrarily. But, it is not
necessary that the sequence converges. We discuss the secant method, the Newton-
Raphson method and the fixed point method in Section 4.3, which are some of the open
domain methods.
4.1 Closed Domain Methods
The idea behind the closed domain methods is to start with an interval (denoted by
[a0, b0]) in which there exists at least one root of the given nonlinear equation and then
reduce the length of this interval iteratively with the condition that there is at least
one root of the equation at each iteration.
Note that the initial interval [a0, b0] can be obtained using the intermediate value theo-
rem (as we always assume that the nonlinear function f is continuous) by checking the
condition that
f(a0)f(b0) < 0.
That is, f(a0) and f(b0) are of opposite sign. The closed domain methods differ from
each other only by the way they go on reducing the length of this interval at each
iteration.
In the following subsections we discuss two closed domain methods, namely, (i) the
bisection method and (ii) the regula-falsi method.
S. Baskar and S. Sivaji Ganesh
120
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-132-320.jpg)
![Section 4.1 Closed Domain Methods
4.1.1 Bisection Method
The most simple way of reducing the length of the interval is to sub-divide the interval
into two equal parts and then take the sub-interval that contains a root of the equation
and discard the other part of the interval. This method is called the bisection method.
Let us explain the procedure of generating the first iteration of this method.
Step 1: Define x1 to be the mid-point of the interval [a0, b0]. That is,
x1 =
a0 + b0
2
.
Step 2: Now, exactly one of the following two statements hold.
1. x1 solves the nonlinear equation. That is, f(x1) = 0.
2. Either f(a0)f(x1) < 0 or f(b0)f(x1) < 0.
If case (1) above holds, then x1 is a required root of the given equation f(x) = 0 and
therefore we stop the iterative procedure. If f(x1) ̸= 0, then case (2) holds as f(a0)
and f(b0) are already of opposite signs. In this case, we define a subinterval [a1, b1] of
[a0, b0] as follows.
[a1, b1] =
{
[a0, x1], if f(a0)f(x1) < 0,
[x1, b0], if f(b0)f(x1) < 0.
The outcome of the first iteration of the bisection method is the interval [a1, b1] and the
first member of the corresponding iterative sequence is the real number x1. Observe
that
• the length of the interval [a1, b1] is exactly half of the length of [a0, b0] and
• [a1, b1] has at least one root of the nonlinear equation f(x) = 0.
Similarly, we can obtain x2 and [a2, b2] as the result of the second iteration and so on.
We now present the algorithm for the bisection method.
Algorithm 4.1.1 [Bisection Method].
Hypothesis: There exists an interval [a0, b0] such that the function f : [a0, b0] → R
is continuous, and the numbers f(a0) and f(b0) have opposite signs.
Algorithm:
Step 1: For n = 0, 1, 2, · · · , define the iterative sequence of the bisection method as
xn+1 =
an + bn
2
,
which is the midpoint of the interval [an, bn].
S. Baskar and S. Sivaji Ganesh
121
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-133-320.jpg)
![Chapter 4. Nonlinear Equations
Step 2: One of the following two cases hold.
1. xn+1 solves the nonlinear equation. That is, f(xn+1) = 0.
2. Either f(an)f(xn+1) < 0 or f(bn)f(xn+1) < 0.
Define the subinterval [an+1, bn+1] of [an, bn] as follows.
[an+1, bn+1] =
{
[an, xn+1], if f(an)f(xn+1) < 0,
[xn+1, bn], if f(bn)f(xn+1) < 0.
Step 3: Stop the iteration if one of the following happens:
• the case (1) in step 2 holds. Then declare the value of xn+1 as the required root.
• (bn+1 − an+1) is sufficiently small (less than a pre-assigned positive quantity).
Then declare the value of xn+2 as the required root up to the desired accuracy.
If any of the above stopping criteria does not hold, then repeat step 1 with n replaced
by n + 1. Continue this process till one of the above two stopping criteria is fulfilled.
Remark 4.1.2.
In practice, one may also use any of the stopping criteria listed in Section 4.2, either
single or multiple criteria.
Assuming that, for each n = 1, 2, · · · , the number xn is not a root of the nonlinear
equation f(x) = 0, we get a sequence of real numbers {xn}. The question is whether
this sequence converges to a root of the nonlinear equation f(x) = 0. We now discuss
the error estimate and convergence of the iterative sequence generated by the bisection
method.
Theorem 4.1.3 [Convergence and Error Estimate of Bisection Method].
Hypothesis: Let f : [a0, b0] → R be a continuous function such that the numbers
f(a0) and f(b0) have opposite signs.
Conclusion:
• There exists an r ∈ (a0, b0) such that f(r) = 0 and the iterative sequence {xn}
of the bisection method converges to r.
• For each n = 0, 1, 2, · · · , we have the following error estimate
|xn+1 − r| ≤
(
1
2
)n+1
(b0 − a0). (4.2)
S. Baskar and S. Sivaji Ganesh
122
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-134-320.jpg)
![Section 4.1 Closed Domain Methods
Proof.
It directly follows from the construction of the intervals [an, bn] that
bn − an =
1
2
(bn−1 − an−1) = · · · =
(
1
2
)n
(b0 − a0).
As a consequence, we get
lim
n→∞
(bn − an) = 0.
By algebra of limits of sequences, we get
lim
n→∞
an = lim
n→∞
bn.
Since for each n = 0, 1, 2, · · · , the number xn+1 is the mid-point of the interval [an, bn],
we also have
an < xn+1 < bn.
Now by sandwich theorem for sequences, we conclude that the sequence {xn} of mid-
points also converges to the same limit as the sequences of end-points. Thus we
have
lim
n→∞
an = lim
n→∞
bn = lim
n→∞
xn = r (say). (4.3)
Since for each n = 0, 1, 2, · · · , we have f(an)f(bn) < 0, applying limits on both sides
of the inequality and using the continuity of f, we get
f(r)f(r) ≤ 0, (4.4)
from which we conclude that f(r) = 0. That is, the sequence of mid-points {xn}
defined by the bisection method converges to a root of the nonlinear equation f(x) =
0.
Since the sequences {an} and {bn} are non-decreasing and non-increasing, respec-
tively, for each n = 0, 1, 2, · · · , we have r ∈ [an, bn]. Also, xn+1 is the mid-point of
the interval [an, bn]. Therefore, we have
|xn+1 − r| ≤
1
2
(bn − an) = · · · =
(
1
2
)n+1
(b0 − a0),
which is the required estimate.
S. Baskar and S. Sivaji Ganesh
123
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-135-320.jpg)
![Chapter 4. Nonlinear Equations
Corollary 4.1.4.
Let ϵ > 0 be given. Let f satisfy the hypothesis of bisection method with the interval
[a0, b0]. Let xn be as in the bisection method, and r be the root of the nonlinear
equation f(x) = 0 to which bisection method converges. Then |xn − r| ≤ ϵ whenever
n satisfies
n ≥
log(b0 − a0) − log ϵ
log 2
(4.5)
Proof.
By the error estimate of bisection method given by (4.2), we are sure that
|xn − r| ≤ ϵ,
whenever n is such that (
1
2
)n
(b0 − a0) ≤ ϵ.
By taking logarithm on both sides of the last inequality, we get the desired estimate
on n.
Remark 4.1.5.
The Corollary 4.1.4 tells us that if we want an approximation xn to the root r of
the given equation such that the absolute error is less than a pre-assigned positive
quantity, then we have to perform n iterations, where n is the least integer that
satisfies the inequality (4.5). It is interesting to observe that to obtain this n, we
don’t need to know the root r.
Example 4.1.6.
Let us find an approximate root to the nonlinear equation
sin x + x2
− 1 = 0
using bisection method so that the resultant absolute error is at most ϵ = 0.125.
To apply bisection method, we must choose an interval [a0, b0] such that the function
f(x) = sin x + x2
− 1
satisfies the hypothesis of bisection method. Note that f satisfies hypothesis of bi-
S. Baskar and S. Sivaji Ganesh
124
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-136-320.jpg)
![Section 4.1 Closed Domain Methods
section on the interval [0, 1]. In order to achieve the required accuracy, we should
first decide how many iterations are needed. The inequality (4.5), says that required
accuracy is achieved provided n satisfies
n ≥
log(1) − log(0.125)
log 2
= 3
Thus we have to compute x3. We will do it now.
Iteration 1: We have a0 = 0, b0 = 1. Thus x1 = 0.5. Since,
f(x1) = −0.27 < 0, f(0) < 0, and f(1) > 0,
we take [a1, b1] = [x1, b0] = [0.5, 1].
Iteration 2: The mid-point of [0.5, 1] is x2 = 0.75. Since
f(x2) = 0.24 > 0, f(0.5) < 0, and f(1) > 0,
we take [a2, b2] = [a1, x2] = [0.5, 0.75].
Iteration 3: The mid-point of [0.5, 0.75] is x3 = 0.625. Since
f(x3) = −0.024 < 0, f(0.5) < 0 and f(0.75) > 0,
we take [a3, b3] = [x3, b2] = [0.625, 0.75].
As suggested by the formula (4.5), we stop the iteration here and take the approximate
root of the given equation as the mid-point of the interval [a3, b3], which is x4 ≈ 0.6875.
Note that the true value is r ≈ 0.636733. The absolute error is 0.05 which is much
less than the required accuracy of 0.125.
Remark 4.1.7 [Comments on Bisection method].
1. Note that the mid-point of an interval [a, b] is precisely the x − coordinate
of the point of intersection of the line joining the points (a, sgn(f(a))) and
(b, sgn(f(b))) with the x-axis.
2. Let f, and the interval [a0, b0] satisfy the hypothesis of bisection method.
Even if the nonlinear equation f(x) = 0 has more than one root in the interval
[a, b], the bisection method chooses the root that it tries to approximate. In
other words, once we fix the initial interval [a0, b0], the bisection method
takes over and we cannot control it to find some specific root than what it
chooses to find.
S. Baskar and S. Sivaji Ganesh
125
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-137-320.jpg)
![Chapter 4. Nonlinear Equations
3. Given a function f, choosing an interval [a, b] such that f(a)f(b) < 0 is
crucial to bisection method. There is no general procedure to find such an
interval. This is one of the main drawbacks of bisection method.
4. Bisection method cannot be used to find zero of functions for which graph
touches x-axis but does not cross x-axis.
5. There is a misconception about bisection method’s order of convergence (see
Chapter 1 for the definition), which is claimed to be 1. However there is no
proof of |xn+1 − r| ≤ c|xn − r|. If the sequence {xn} converges linearly to r,
then we get
|xn+1 − r| ≤ cn+1
|x0 − r|.
In the case of bisection method, we have this inequality with c = 1/2, which is
only a necessary condition for the convergence to be linear but not a sufficient
condition.
4.1.2 Regula-falsi Method
The regula-falsi method is similar to the bisection method. Although the bisection
method (discussed in the previous subsection) always converges to the root, the con-
vergence is very slow, especially when the length of the initial interval [a0, b0] is very
large and the equation has the root very close to the one of the end points. This is
because, at every iteration, we are subdividing the interval [an, bn] into two equal parts
and taking the mid-point as xn+1 (the (n + 1)th
member of the iterative sequence).
Therefore, it takes several iterations to reduce the length of the interval to a very small
number, and as a consequence the distance between the root and xn+1.
The regula-falsi method differs from the bisection method only in the choice of xn+1
in the interval [an, bn] for each n = 0, 1, 2, · · · . Instead of taking the midpoint of the
interval, we now take the x-coordinate of the point of intersection of the line joining the
points (an, f(an)) and (bn, f(bn)) with the x-axis. Let us now explain the procedure of
generating the first iteration of this method.
Step 1: Assume the hypothesis of the bisection method and let [a0, b0] be the initial
interval. The line joining the points (a0, f(a0)) and (b0, f(b0)) is given by
y = f(a0) +
f(b0) − f(a0)
b0 − a0
(x − a0),
The first member x1 of the regula-falsi iterative sequence is the x-coordinate of the point
of intersection of the above line with the x-axis. Therefore, x1 satisfies the equation
f(a0) +
f(b0) − f(a0)
b0 − a0
(x1 − a0) = 0
S. Baskar and S. Sivaji Ganesh
126
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-138-320.jpg)
![Section 4.1 Closed Domain Methods
and is given by
x1 = a0 − f(a0)
b0 − a0
f(b0) − f(a0)
,
which can also be written as
x1 =
a0f(b0) − b0f(a0)
f(b0) − f(a0)
.
Step 2: Now, exactly one of the following two statements hold.
1. x1 solves the nonlinear equation. That is, f(x1) = 0.
2. Either f(a0)f(x1) < 0 or f(b0)f(x1) < 0.
If case (1) above holds, then x1 is a required root of the given equation f(x) = 0 and
therefore we stop the iterative procedure. If f(x1) ̸= 0, then case (2) holds as f(a0)
and f(b0) are already of opposite signs. We now define a subinterval [a1, b1] of [a0, b0]
as follows.
[a1, b1] =
{
[a0, x1], if f(a0)f(x1) < 0,
[x1, b0], if f(b0)f(x1) < 0.
The outcome of the first iteration of the regula-falsi method is the interval [a1, b1] and
the first member of the corresponding iterative sequence is the real number x1. Observe
that
• the length of the interval [a1, b1] may be (although not always) much less than
half of the length of [a0, b0] and
• [a1, b1] has at least one root of the nonlinear equation f(x) = 0.
We now summarize the regula-falsi method.
Algorithm 4.1.8 [Regula-falsi method].
Hypothesis: Same as bisection method.
Algorithm:
Step 1: For n = 0, 1, 2, · · · , define the iterative sequence of the regula-falsi method
as
xn+1 = an − f(an)
bn − an
f(bn) − f(an)
(4.6)
or
xn+1 =
anf(bn) − bnf(an)
f(bn) − f(an)
, (4.7)
which is the x-coordinate of the point of intersection of the line joining the points
S. Baskar and S. Sivaji Ganesh
127
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-139-320.jpg)
![Chapter 4. Nonlinear Equations
(an, f(an)) and (bn, f(bn)) (obtained at the nth
iteration) with the x-axis.
Step 2: One of the following two cases hold.
1. xn+1 solves the nonlinear equation. That is, f(xn+1) = 0.
2. Either f(an)f(xn+1) < 0 or f(bn)f(xn+1) < 0.
Define the subinterval [an+1, bn+1] of [an, bn] as follows.
[an+1, bn+1] =
{
[an, xn+1], if f(an)f(xn+1) < 0,
[xn+1, bn], if f(bn)f(xn+1) < 0.
Step 3: Stop the iteration if the case (1) in step 2 holds and declare the value of
xn+1 as the required root. Otherwise repeat step 1 with n replaced by n + 1.
Continue this process till a desired accuracy is achieved.
Remark 4.1.9 [Stopping criteria].
Unlike in the case of bisection method, there is no clear way of stopping the iteration
of regula-falsi method as the length of the interval [an, bn] obtained at the (n+1)th
iteration may not converge to zero as n → ∞. This situation occurs especially
when the function f is concave or convex in the interval [a0, b0] as illustrated in
Example 4.1.11.
Remark 4.1.10 [Important observations].
Assuming that, for each n = 1, 2, · · · , the number xn is not a root of the nonlinear
equation f(x) = 0, we get a sequence of real numbers {xn}. The question is
whether this sequence converges to a root of the nonlinear equation f(x) = 0.
Before addressing this question, let us consolidate the information that we have so
far.
1. The sequence of left end points of the intervals [an, bn] is a non-decreasing
sequence that is bounded above by b. That is,
a0 ≤ a1 ≤ · · · ≤ an · · · ≤ b0.
Hence the sequence {an} has a limit, i.e., limn→∞ an exists.
2. The sequence of right end points of the intervals [an, bn] is a non-increasing
S. Baskar and S. Sivaji Ganesh
128
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-140-320.jpg)
![Section 4.1 Closed Domain Methods
sequence that is bounded below by a. That is,
b0 ≥ b1 ≥ · · · ≥ bn · · · ≥ a0.
Hence the sequence {bn} has a limit, i.e., limn→∞ bn exists.
3. Since an < bn for all n = 0, 1, 2, · · · , we conclude that
lim
n→∞
an ≤ lim
n→∞
bn.
If the lengths of the intervals obtained in regula-falsi method tend to zero,
then we have
lim
n→∞
an = lim
n→∞
bn,
in which case, we also have by sandwich theorem
lim
n→∞
an = lim
n→∞
xn+1 = lim
n→∞
bn,
as an < xn+1 < bn for all n = 0, 1, 2, · · · . In this case, the common limit will
be a root of the nonlinear equation, as is the case with bisection method.
It is important to observe that, it may happen that the lengths of the subintervals
chosen by regula-falsi method do not go to zero. In other words, if an → α and
bn → β, then it may happen that α < β. Let us illustrate this case by the following
example.
Example 4.1.11.
Consider the nonlinear equation
ex
− 2 = 0.
Note that the function f : R → R defined by f(x) = ex
− 2 satisfies the hypothesis
of the regula-falsi method on the interval [0, 1]. Let us start the iteration with the
initial interval [a0, b0] = [0, 1].
Iteration 1: Using the formula (4.6), we get
x1 =
1
e − 1
.
Since f(x1) ≈ −0.21043, the subinterval chosen by regula-falsi method is the interval
[a1, b1] = [x1, 1].
S. Baskar and S. Sivaji Ganesh
129
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-141-320.jpg)
![Chapter 4. Nonlinear Equations
Iteration 2: We have x2 ≈ 0.67669. Since f(x2) ≈ −0.032645, the subinterval cho-
sen by regula falsi method is the interval [a2, b2] = [x2, 1].
By continuing these computations, we see that the subinterval at every iteration
chosen by regula-falsi method is the right half of the interval defined in the previous
iteration. In other words, for n = 1, 2, · · · , we have [an, bn] = [xn, 1]. This is clearly
seen geometrically as the function f is convex. A similar situation occurs when the
given function f is concave (at least in the initial interval [a0, b0]). We checked this
to be true for n = 1, 2 and a proof for a general n is left as an exercise.
Note in this example that β = 1 and α ≤ r ̸= 1, where r is the root of the given
equation f(x) = 0 to which the regula-falsi method is expected to converge.
The last example ruled out any hopes of proving that α = β. However, it is true that
the sequence {xn} converges to a root of the nonlinear equation f(x) = 0, and as can
be expected, the proof is by contradiction argument. Let us now state the theorem on
regula-falsi method.
Theorem 4.1.12 [Convergence of Regula-falsi method].
Hypothesis: Let f : [a0, b0] → R be a continuous function such that the numbers
f(a0) and f(b0) have opposite signs.
Conclusion: If {xn} is the iterative sequence defined by regula-falsi method. Then
the sequence {xn} converges to an r ∈ (a0, b0) such that f(r) = 0.
Proof is omitted.
Example 4.1.13.
Let us find an approximate root to the nonlinear equation
sin x + x2
− 1 = 0
using regula-falsi method. Exact root is approximately 0.637.
We choose the initial interval as [a0, b0] = [0, 1] as done in Example 4.1.6.
Iteration 1: We have a0 = 0, b0 = 1. Thus x1 = 0.54304. Since f(x1) = −0.18837 <
0, f(0) < 0, and f(1) > 0, we take [a1, b1] = [x1, b0] = [0.54304, 1].
Iteration 2: Using the regula-falsi formula (4.6), we get x2 = 0.62662. Since f(x2) =
−0.020937 < 0, f(0.54304) < 0, and f(1) > 0, we take [a2, b2] = [x2, 1] = [0.62662, 1].
S. Baskar and S. Sivaji Ganesh
130
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-142-320.jpg)
![Section 4.2 Stopping Criteria
Iteration 3: Using the regula-falsi formula (4.6), we get x3 = 0.63568. Since
f(x3) = −0.0021861 < 0, f(0.62662) < 0, and f(1) > 0, we take [a3, b3] = [x3, 1] =
[0.63658, 1].
Iteration 4: Using the regula-falsi formula (4.6), we get x4 = 0.63662.
and so on.
In this example also, we observe that the lengths of the intervals [an, bn] do not seem to
be tending to zero as n → ∞. However, we see that the sequence {xn} is approaching
the root of the given equation.
4.2 Stopping Criteria
The outcome of any iterative method for a given nonlinear equation is a sequence of real
numbers that is expected to converges to a root of the equation. When we implement
such a methods on a computer, we cannot go on computing the iterations indefinitely
and needs to stop the computation at some point. It is desirable to stop computing
the iterations when the xn’s are reasonably close to an exact root r for a sufficiently
large n. In other words, we want to stop the computation at the nth
iteration when the
computed value is such that
|xn − r| < ϵ
for some pre-assigned positive number ϵ.
In general, we don’t know the root r of the given nonlinear equation to which the
iterative sequence is converging. Therefore, we have no idea of when to stop the iteration
as we have seen in the case of regula-falsi method. In fact, this situation will be there
for any open domain methods discussed in the next section. An alternate way is to look
for some criteria that does not use the knowledge of the root r, but gives a rough idea
of how close we are to this root. Such a criteria is called a stopping criteria. We
now list some of the commonly used stopping criteria for iterative methods to nonlinear
equations.
Stopping Criterion 1: Fix a K ∈ N, and ask the iteration to stop after finding xK.
This criterion is borne out of fatigue, as it clearly has no mathematical reason why
the K fixed at the beginning of the iteration is more important than any other natural
number! If we stop the computation using this criterion, we will declare xK to be the
approximate root to the nonlinear equation f(x) = 0.
Stopping Criterion 2: Fix a real number ϵ > 0 and a natural number N. Ask the
iteration to stop after finding xk such that
|xk − xk−N | < ϵ.
S. Baskar and S. Sivaji Ganesh
131
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-143-320.jpg)
![Section 4.3 Open Domain Methods
using a computer program as we have to adopt a trial and error algorithm for this.
Therefore, it is highly desirable to device an iterative method that does not need this
information and gives freedom to choose any arbitrary starting point as done in the
iterative method for linear systems. This is the objective of an open domain method.
The open domain methods do not pre-suppose that a root is enclosed in an inter-
val [a0, b0]. These methods always start with a set of initial guesses x0, x1, · · · , xm
distributed anywhere on the real line (preferably sufficiently close to a root of the equa-
tion) and tell us how to compute the values xm+1, xm+2, · · · . in such a way that the
resulting iterative sequence {xn} converges to a root of the equation.
In this section we study three open domain iterative methods for approximating a root
of a given nonlinear equation.
4.3.1 Secant Method
The straightforward modification of the regula-falsi method is the well-known secant
method. The only difference in the secant method (when compared to the regula-falsi
method) is that here we do not demand the initial guesses a0 and b0 to be on the either
side of a root. Let us now present the algorithm of the secant method.
Algorithm 4.3.1 [Secant method].
Hypothesis: Given any initial values x0 and x1 (not necessarily on the either side
of a root) such that f(x0) ̸= f(x1).
Algorithm:
Step 1: For n = 1, 2, · · · , the iterative sequence for secant method is given by
xn+1 =
f(xn)xn−1 − f(xn−1)xn
f(xn) − f(xn−1)
. (4.8)
This expression can also be written as
xn+1 = xn − f(xn)
xn − xn−1
f(xn) − f(xn−1)
(4.9)
Step 2: Choose any one of the stopping criteria (or a combination of them) discussed
in Section 4.2. If this criterion is satisfied, stop the iteration. Otherwise, repeat the
step 1 by replacing n with n + 1 until the criterion is satisfied.
Recall that xn+1 for each n = 1, 2, · · · given by (4.8) (or (4.9)) is the x-coordinate of the
point of intersection of the secant line joining the points (xn−1, f(xn−1)) and (xn, f(xn))
S. Baskar and S. Sivaji Ganesh
133
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-145-320.jpg)
![Section 4.3 Open Domain Methods
Theorem 4.3.4 [Convergence of Secant Method].
Hypothesis:
1. Let f : R → R be a C2
(R) function.
2. Let r be a simple root of the nonlinear equation f(x) = 0, that is f′
(r) ̸= 0.
Conclusion: Then there exists a δ > 0 such that for every x0, x1 ∈ [r − δ, r + δ],
1. the secant method iterative sequence {xn} is well-defined.
2. the sequence {xn} belongs to the interval [r − δ, r + δ].
3. lim
n→∞
xn = r.
4. Further, we have
lim
n→∞
|xn+1 − r|
|xn − r|α
=
f′′
(r)
2f′(r)
α/(α+1)
, (4.10)
where α = (
√
5 + 1)/2 ≈ 1.62.
The proof for this is omitted for this course.
Remark 4.3.5.
The expression (4.10) implies that the order of convergence (see Chapter 1 for the
definition) of the iterative sequence of secant method is 1.62.
4.3.2 Newton-Raphson Method
In Theorem 4.3.4, we have seen that the secant method has more than linear order of
convergence. This method can further be modified to achieve quadratic convergence.
To do this, we first observe that when the iterative sequence of the secant method
converges, then as n increases, we see that xn−1 approaches xn. Thus, for a sufficiently
large value of n, we have
f′
(xn) ≈
f(xn) − f(xn−1)
xn − xn−1
,
provided f is a C1
function. Thus, if f(x) is differentiable, then on replacing in (4.9),
the slope of the secant by the slope of the tangent at xn, we get the iteration formula
xn+1 = xn −
f(xn)
f′(xn)
(4.11)
and is called the Newton-Raphson method.
S. Baskar and S. Sivaji Ganesh
135
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-147-320.jpg)
![Chapter 4. Nonlinear Equations
Remark 4.3.6 [Geometric Motivation].
If a function f is differentiable at a point x0, then the tangent (y = g(x)) to the
graph of f at the point (x0, f(x0)) is given by
g(x) = f′
(x0)(x − x0) + f(x0).
We may assume that for x ≈ x0, f(x) ≈ g(x). This can also be interpreted as
“If a function f is differentiable at a point, then the graph of f looks like a
straight line, for x ≈ x0 on zooming”.
Now, if we choose the initial guess x0 very close to the root r of f(x) = 0. That
is., if r ≈ x0, we have g(r) ≈ f(r) = 0. This gives (approximately)
0 ≈ f′
(x0)(r − x0) + f(x0).
Up on replacing r by x1 and using ‘=’ symbol instead of ‘≈’ symbol, we get the
first iteration of the Newton-Raphson’s iterative formula (4.11).
Recall in secant method, we need two initial guesses x0 and x1 to start the iteration.
In Newton-Raphson method, we need one initial guess x0 to start the iteration. The
consecutive iteration x1 is the x-coordinate of the point of intersection of the x-axis
and the tangent line at x0, and similarly for the other iterations. This geometrical
interpretation of the Newton-Raphson method is clearly observed in Figure 4.2.
We now derive the Newton-Raphson method under the assumption that f is a C2
function.
Let x0 be given. The Taylor’s polynomial of degree n = 1 with remainder is given by
f(x) = f(x0) + f′
(x0)(x − x0) +
(x − x0)2
2!
f′′
(ξ),
where ξ lies between x0 and x. When x0 is very close to x, the last term in the above
equation is smaller when compared to the other two terms on the right hand side. By
neglecting this term we have
f(x) ≈ f(x0) + f′
(x0)(x − x0). (4.12)
Notice that the graph of the function g(x) = f(x0) + f′
(x0)(x − x0) is precisely the
tangent line to the graph of f at the point (x0, f(x0)). We now define x1 to be the x-
coordinate of the point of intersection of this tangent line with the x-coordinate. That
is, the point x1 is such that g(x1) = 0, which gives
x1 = x0 −
f(x0)
f′(x0)
. (4.13)
S. Baskar and S. Sivaji Ganesh
136
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-148-320.jpg)
![Chapter 4. Nonlinear Equations
Remark 4.3.8.
It is evident that if the initial guess x0 is such that f′(xn) = 0, for some n ∈ N, then
the Newton-Raphson method fails. Geometrically, this means that the tangent line
to the graph of f at the point (xn, f(xn)) is parallel to the x-axis. Therefore, this
line never intersects x-axis and hence xn+1 never exists. See Remark 4.3.2 for the
failure of secant method and compare it with the present case.
Example 4.3.9.
Consider the equation
sin x + x2
− 1 = 0.
Let x0 = 1. Then the iterations from the Newton-Raphson method gives
n xn ϵ
1 0.668752 0.032019
2 0.637068 0.000335
3 0.636733 0.000000
Figure 4.2 shows the iterative points x2 and x3 in black bullet. Recall that the exact
root is x∗
≈ 0.636733. Obviously, the Newton-Raphson method is much faster than
both bisection and secant methods in this example.
Let us now discuss the convergence of the Newton-Raphson method.
Theorem 4.3.10 [Convergence of Newton-Raphson method].
Hypothesis:
1. Let f : R → R be a C2
(R) function.
2. Let r be a simple root of the nonlinear equation f(x) = 0, that is f′
(r) ̸= 0.
Conclusion: Then there exists a δ > 0 such that for every x0 ∈ [r − δ, r + δ],
1. each term of the Newton-Raphson iterative sequence {xn} is well-defined.
2. the sequence {xn} belongs to the interval [r − δ, r + δ].
3. limn→∞ xn = r.
4. Further, we have
lim
n→∞
|xn+1 − r|
|xn − r|2
=
f′′
(r)
2f′(r)
. (4.14)
The proofs of the conclusions (1)-(3) are omitted. Conclusion (4) implies that the
S. Baskar and S. Sivaji Ganesh
138
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-150-320.jpg)
![Section 4.3 Open Domain Methods
Newton-Raphson method converges quadratically. The proof is a direct application of
the Taylor’s theorem and is left as an exercise.
Theorem on Newton-Raphson method says that if we start near-by a root of the non-
linear equation, then Newton-Raphson iterative sequence is well-defined and converges.
For increasing convex functions, we need not be very careful in choosing the initial guess.
For such functions, the Newton-Raphson iterative sequence always converges, whatever
may be the initial guess. This is the content of the next theorem.
Theorem 4.3.11 [Convergence Result for Convex Functions].
Hypothesis: Let f : R → R be a twice continuously differentiable function such
that
1. f is convex, i.e., f′′
(x) > 0 for all x ∈ R.
2. f is strictly increasing, i.e., f′
(x) > 0 for all x ∈ R.
3. there exists an r ∈ R such that f(r) = 0.
Conclusion: Then
1. r is the unique root of f(x) = 0.
2. For every choice of x0, the Newton-Raphson iterative sequence converges to r.
Proof.
Proof of (1): Since f is strictly increasing, the function cannot take the same value
more than once. Thus f(x) = 0 has exactly one root.
Proof of (2): From
en+1 = −
f′′
(ξn)
2f′(xn)
e2
n. (4.15)
it follows that en+1 ≤ 0. This implies that r ≤ xn+1. Since f is a strictly increasing
function, we get f(r) ≤ f(xn+1). Thus f(xn+1) ≥ 0 for every n ∈ N. From (4.11),
we get xn+1 ≤ xn. That is, the sequence {xn} is a non-increasing sequence, and is
bounded below by r, and hence converges. Let us denote the limit by x∗
.
On the other hand, the sequence {en} is a non-decreasing sequence, bounded above
by 0. Let e∗
denote the limit of the sequence {en}.
Passing to the limit as n → ∞ in the equation
en+1 = en +
f(xn)
f′(xn)
,
we get the desired result.
S. Baskar and S. Sivaji Ganesh
139
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-151-320.jpg)
![Chapter 4. Nonlinear Equations
To following example illustrates the quadratic convergence of the Newton-Raphson
method.
Example 4.3.12.
Start with x0 = −2.4 and use Newton-Raphson iteration to find the root r = −2.0 of
the polynomial
f(x) = x3
− 3x + 2.
The iteration formula is
xn+1 =
2x3
n − 2
3x2
n − 3
.
It is easy to verify that |r − xn+1|/|r − xn|2
≈ 2/3, which shows the quadratic con-
vergence of Newton-Raphson’s method.
4.3.3 Fixed-Point Iteration Method
In fixed point iteration method, the problem of finding a root of a nonlinear equation
f(x) = 0 is equivalently viewed as the problem of finding a fixed point of a suitably
defined function g.
Let us first define the notion of fixed point of a function.
Definition 4.3.13 [Fixed point].
Let g be a continuous function. A point α ∈ R is said to be a fixed point of g if
α = g(α).
The idea behind the choice of g is to rewrite the given nonlinear equation f(x) = 0 in
the form x = g(x) for some function g. In general, there may be more than one choice
of g with this property as illustrated by the following example.
Example 4.3.14.
Note that α ∈ R is a root of the equation x2
− x − 2 = 0 if and only if α is a root of
each of the following equations.
1. x = x2
− 2
2. x =
√
x + 2
3. x = 1 +
2
x
.
In other words, obtaining a root of the equation f(x) = 0, where f(x) = x2
− x − 2
S. Baskar and S. Sivaji Ganesh
140
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-152-320.jpg)
![Chapter 4. Nonlinear Equations
Figure 4.3: Fixed-point Iteration Procedure.
x2, we need that x1 is in the domain of g once again. In fact the sequence is given by
x0, g(x0), g ◦ g(x0), g ◦ g ◦ g(x0), · · ·
Thus to have a well-defined iterative sequence, we require that
Range of the function g is contained in the domain of g.
A function with this property is called a self map. We make our first assumption on
the iterative function as
Assumption 1: a ≤ g(x) ≤ b for all a ≤ x ≤ b.
It follows that if a ≤ x0 ≤ b, then for all n, xn ∈ [a, b] and therefore xn+1 = g(xn) is
defined and belongs to [a, b].
S. Baskar and S. Sivaji Ganesh
142
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-154-320.jpg)
![Section 4.3 Open Domain Methods
Let us now discuss about the point 3. This is a natural expectation since the expression
x = g(x), which is the root of the required equation is precisely the definition of a fixed
point. To achieve this, we need g(x) to be a continuous function. For if xn → x∗
then
x∗
= lim
n→∞
xn = lim
n→∞
g(xn−1) = g( lim
n→∞
xn−1) = g(x∗
)
Therefore, we need
Assumption 2: The iterative function g is continuous.
It is easy to prove that a continuous self map on a bounded interval always has a fixed
point. However, the question is whether the sequence (4.16) generated by the iterative
function g converges, which is the requirement stated in point 2. This point is well
understood geometrically. Figure 4.3(a) and Figure 4.3(c) illustrate the convergence of
the fixed-point iterations whereas Figure 4.3(b) and Figure 4.3(d) illustrate the diverg-
ing iterations. In this geometrical observation, we see that when g′
(x) < 1, we have
convergence and otherwise, we have divergence. Therefore, we make the assumption
Assumption 3: The iteration function g(x) is differentiable on I = [a, b]. Further,
there exists a constant 0 < K < 1 such that
|g′
(x)| ≤ K, x ∈ I. (4.17)
Such a function is called the contraction map.
Let us now present the algorithm of the fixed-point iteration method.
Algorithm 4.3.16.
Hypothesis: Let g : [a, b] → [a, b] be an iteration function such that Assumptions 1,
2, and 3 stated above hold.
Algorithm:
Step 1: Choose an initial guess x0 ∈ [a, b].
Step 2: Define the iteration methods as
xn+1 = g(xn), n = 0, 1, · · ·
Step 3: For a pre-assigned positive quantity ϵ, check for one of the (fixed) stop-
ping criteria discussed in Section 4.2. If the criterion is satisfied, stop the iteration.
Otherwise, repeat the step 1 by replacing n with n + 1 until the criterion is satisfied.
S. Baskar and S. Sivaji Ganesh
143
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-155-320.jpg)
![Chapter 4. Nonlinear Equations
Theorem 4.3.17 [Convergence Result for Fixed-Point Iteration Method].
Hypothesis: Let the iterative function g be chosen so that
1. g is defined on the interval [a, b] and a ≤ g(x) ≤ b. That is, g is a self map on
[a, b]
2. g is continuously differentiable on [a, b]
3. g is a contraction map. That is,
λ = max
a≤x≤b
|g′
(x)| < 1. (4.18)
Conclusion: Then
1. x = g(x) has a unique root r in [a, b].
2. For any choice of x0 ∈ [a, b], with xn+1 = g(xn), n = 0, 1, · · · ,
lim
n→∞
xn = r.
3. We further have
|xn − r| ≤ λn
|x0 − r| ≤
λn
1 − λ
|x1 − x0| (4.19)
and
lim
n→∞
r − xn+1
r − xn
= g′
(r). (4.20)
Proof.
Proof for (1) is easy.
From mean-value theorem and (4.17), we have
|r − xn+1| = |g(r) − g(xn)| ≤ λ|r − xn|. (4.21)
By induction, we have
|r − xn+1| ≤ λn
|x0 − r|, n = 0, 1, · · · .
Since, as n → ∞, λn
→ 0, we have xn → r. Further, we have
|x0 − r| = |x0 − x1 + x1 − r|
≤ |x0 − x1| + |x1 − r|
≤ λ|x0 − r| + |x0 − x1|.
S. Baskar and S. Sivaji Ganesh
144
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-156-320.jpg)
![Section 4.3 Open Domain Methods
Then solving for |x0 − r|, we get (4.19).
Now we will prove the rate of convergence (4.20). From Mean-value theorem
r − xn+1 = g(r) − g(xn)
= g′
(ξn)(r − xn), n = 0, 1, · · · . (4.22)
with ξn an unknown point between r and xn. Since xn → r, we must have ξn → r
and therefore,
lim
n→∞
r − xn+1
r − xn
= lim
n→∞
g′
(ξn) = g′
(r).
This completes the proof.
Remark 4.3.18.
From the inequality (4.21), we see that the fixed point iteration method has linear
convergence. In other words, the order of convergence of this method is 1.
Example 4.3.19.
The nonlinear equation
x3
+ 4x2
− 10 = 0
has a unique root in [1, 2]. Note that the solution of each of the following fixed-point
problems is a root to the given nonlinear equation.
1. x = g1(x), where g1(x) = x − x3
− 4x2
+ 10
2. x = g2(x) where g2 =
√
10
x
− 4x
3. x = g3(x) where g3 =
1
2
√
10 − x3
4. x = g4(x) where g4 =
√
10
4 + x
5. x = g5(x) where g5 = x −
x3
+ 4x2
− 10
3x2 + 8x
.
We now show that among the five equivalent fixed-point formulations of the given
nonlinear equation, only some of them turn out to be good iterative functions. Let
us implement fixed-point iteration method with each of the five iterating functions,
and compare the results which are tabulated below.
S. Baskar and S. Sivaji Ganesh
145
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-157-320.jpg)
![Chapter 4. Nonlinear Equations
n x = g1(x) x = g2(x) x = g3(x) x = g4(x) x = g5(x)
0 1.5 1.5 1.5 1.5 1.5
1 -0.875 0.8165 1.286953768 1.348399725 1.373333333
2 6.732 2.9969 1.402540804 1.367376372 1.365262015
3 -469.7
√
−8.65 1.345458374 1.364957015 1.365230014
4 1.03 × 108
1.375170253 1.365264748 1.365230013
5 1.360094193 1.3652
6 1.367846968 1.365230576
7 1.363887004 1.365229942
8 1.365916734 1.365230022
9 1.364878217 1.365230012
10 1.365410062 1.365230014
15 1.365223680 1.365230013
20 1.365230236
25 1.365230006
30 1.365230013
From the above table, we conclude that the iterative functions g1 and g2 are very
bad, while that given by g5 is the best. However iterative functions g3 and g4 are also
good but requires more number of iterations compared to g5.
1. Let us start analyzing the iterative sequence generated by the iteration function
g1. In this case, the iterative sequence seems to be diverging. The main reason
behind this is the fact that g1 is not a contraction map for x ∈ [1, 2], which
can be seen easily. Thus the successive iterates using g1 have increasing moduli,
which also shows that g1 is not a self map.
2. Let us consider the iterative function g2. It is easy to check that g2 is not a self
map of [1, 2] to itself. In our computation above, we see that the entire iterative
sequence is not defined as one of the iterates becomes negative, when the initial
guess is taken as 1.5. The exact root is approximately equal to r = 1.365. There
is no interval containing r on which |g′
2(x)| < 1. In fact, g′
2(r) ≈ 3.4 and as a
consequence |g′
2(x)| > 3 on an interval containing r. Thus we don’t expect a
convergent iterative sequence even if the sequence is well-defined!
3. Regarding the iterative function g3, note that this iteration function is a de-
creasing function on [1, 2] as
g′
3(x) = −
3x2
4
√
10 − x3
< 0
S. Baskar and S. Sivaji Ganesh
146
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-158-320.jpg)
![Section 4.3 Open Domain Methods
on [1, 2]. Thus maximum of g3 is attained at x = 1, which is 1.5; and the
minimum is attained at x = 2 which is approximately equal to 0.707. Thus g3
is a self map of [1, 2]. But |g′
3(2)| ≈ 2.12. Thus the condition
|g′
3(x)| ≤ λ < 1
is violated on the interval [1, 2]. However by restricting to a smaller interval
[1, 1.5], we get that g3 is a self map of [1, 1.5] as g3 is still decreasing function
on [1, 1.5] and g3(1) = 1.5, g3(1.5) ≈ 1.28, and also
|g′
3(x)| ≤ |g′
3(1.5)| ≈ 0.66.
Thus g3 satisfies the hypothesis of theorem on fixed-point iteration, and as
expected the sequence of iterates converge.
4. Note that the iterative function g4 is given by
g4(x) =
√
10
4 + x
.
We have
|g′
4(x)| =
−5
√
10(4 + x)3/2
≤
5
√
10(5)3/2
< 0.15 for all x ∈ [1, 2].
The bound obtained on the derivative of g4 is considerably smaller when com-
pared to that of g3, which explains why the convergence is faster for the iterates
obtained using g4, when compared with those obtained by using g3.
5. Finally, let us consider the iterative sequence generated by g5. This sequence
converges much more faster compared to g3 and g4. Note that the fixed-point it-
erative sequence generated by g5 is nothing but the iterative sequence of Newton-
Raphson method for the root of the nonlinear equation f(x) = 0.
Example 4.3.20.
Consider the equation
sin x + x2
− 1 = 0.
Take the initial interval as [0, 1]. There are at least three possible choices for the
iteration function, namely,
1. g1(x) = sin−1
(1 − x2
),
S. Baskar and S. Sivaji Ganesh
147
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-159-320.jpg)
![Section 4.4 Comparison and Pitfalls of Iterative Methods
4.4 Comparison and Pitfalls of Iterative Methods
Closed domain methods: Bisection and Regula falsi methods
1. In both these methods, where we are trying to find a root of the nonlinear equa-
tion f(x) = 0, we are required to find an interval [a, b] such that f(a) and f(b)
have opposite signs. This calls for a complete study of the function f. In case
the function has no roots on the real line, this search for an interval will be fu-
tile. There is no way to realize this immediately, thus necessitating a full fledged
understanding of the funtion f.
2. Once it is known that we can start these methods, then surely the iterative se-
quences converge to a root of the nonlinear equation.
3. In bisection method, we can keep track of the error by means of an upper bound.
But such a thing is not available for regula falsi method. In general convergence
of bisection method iterates is slower compared to that of regula falsi method
iterates.
4. If the initial interval [a, b] is such that the equation f(x) = 0 has a unique root in
it, then both the methods converge to the root. If there are more than one roots
in [a, b], then usually both methods find different roots. The only way of finding
the desired root is to find an interval in which there is exactly one root to the
nonlinear equation.
Open domain methods: Secant, Newton-Raphson, and Fixed point methods
1. The main advantage of the open domain methods when compared to closed do-
main methods is that we don’t need to locate a root in an interval. Rather, we
can start the iteration with an arbitrarily chosen initial guess(es).
2. The disadvantage of the open domain methods is that the iterative sequence may
not be well-defined for all initial guesses. Even if the sequence is well-defined, it
may not converge. Even if it converges, it may not converge to a specific root of
interest.
3. In situations where both open and closed domain methods converge, open domain
methods are generally faster compared to closed domain methods. Especially,
Newton-Raphson’s method is faster than other methods as the order of conver-
gence of this method is 2. In fact, this is the fastest method known today.
4. In these methods, it may happen that we are trying to find a particular root of the
S. Baskar and S. Sivaji Ganesh
149
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-161-320.jpg)
![Section 4.5 Exercises
and x0 = 2, then
x1 = 4.0, x2 = 5.333333..., · · · , p15 = 19.72354..., · · · .
and the sequence diverges to +∞. This particular function has another suprising
problem. The value of f(x) goes to zero rapidly as x gets large, for example
f(x15) = 0.0000000536, and it is possible that p15 could be mistaken for a root as
per the residual error. Thus, using residual error for iterative methods nonlinear
equations is often not preferred.
6. The method can stuck in a cycle. For instance, let us compute the iterative
sequence generated by the Newton-Raphson’s method for the function f(x) =
x3
− x − 3 with the initial guess x0 = 0. The iterative sequence is
x1 = −3.00, x2 = −1.961538, x3 = −1.147176, x4 = −0.006579,
x5 = −3.000389, x6 = −1.961818, x7 = −1.147430, · · ·
and we are stuck in a cycle where xn+4 ≈ xk for k = 0, 1, · · · . But if we start
with a value x0 sufficiently close with the root r ≈ 1.6717, then the convergence
is obtained. The proof of this is left as an exercise.
7. If f(x) has no real root, then there is no indication by these methods and the iter-
ative sequence may simply oscillate. For example compute the Newton-Raphson
iteration for
f(x) = x2
− 4x + 5.
4.5 Exercises
Bisection Method and Regula-falsi Method
In the following problems on bisection method, the notation xn is used to denote the
mid-point of the interval [an−1, bn−1], and is termed as the bisection method’s nth
iterate
(or simply, the nth
iterate, as the context of bisection method is clear)
1. Let bisection method be used to solve the nonlinear equation (x is in radians)
x sin x − 1 = 0
starting with the initial interval [0, 2]. In order to approximate a solution of the
nonlinear equation with an absolute error less than or equal to 10−3
, what is the
minimum number of iterations required? Also find the corresponding approximate
solution.
S. Baskar and S. Sivaji Ganesh
151
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-163-320.jpg)
![Chapter 4. Nonlinear Equations
2. Consider the nonlinear equation
10x
+ x − 4 = 0
i) Find an interval [a0, b0] such that the function f(x) = 10x
+ x − 4 satisfies
the hypothesis of bisection method.
ii) Let r be the solution of the nonlinear equation to which bisection method
iterative sequence converges. Find an n such that xn (notation as in the
bisection method) approximates r to two significant digits. Also find xn.
3. If bisection method is used with the initial interval [a0, b0] = [2, 3], how many
iterations are required to assure that an approximate solution of the nonlinear
equation f(x) = 0 is obtained with an absolute error that is at most 10−5
?
4. In the notation of bisection method, determine (with justification) if the following
are possible.
i) a0 < a1 < a2 < · · · < an < · · ·
ii) b0 > b1 > b2 > · · · > bn > · · ·
iii) a0 = a1 < a2 = a3 < · · · < a2m = a2m+1 < · · · (Hint: First guess what
should be the solution found by bisection method in such a case, and then
find the simplest function having it as a root! Do not forget the connection
between bisection method and binary representation of a number, described
in the last problem)
5. Draw the graph of a function that satisfies the hypothesis of bisection method on
the interval [0, 1] such that the errors e1, e2, e3 satisfy e1 > e2, and e2 < e3. Give
formula for one such function.
6. Draw the graph of a function for which bisection method iterates satisfy x1 = 2,
x2 = 0, and x3 = 1 (in the usual notation of bisection method). Indicate in the
graph why x1 = 2, x2 = 0, and x3 = 1 hold. Also mention precisely the corre-
sponding intervals [a0, b0],[a1, b1], [a2, b2].
7. Draw the graph of a function (there is no need to give a formula for the func-
tion) for which a0, a1, a2, a3 (in the usual notation of bisection method) satisfy
a0 < a1 = a2 < a3. (Mark these points clearly on the graph.)
Secant Method and Newton-Raphson Method
8. Discuss some instances where the secant method fails. Note that failure of secant
method results from one of the following two situations: (i) the iterative sequence
is not well-defined, and (ii) the iterative sequence does not converge at all.
S. Baskar and S. Sivaji Ganesh
152
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-164-320.jpg)
![Section 4.5 Exercises
9. Let α be a positive real number. Find formula for an iterative sequence based on
Newton-Raphson method for finding
√
α and α1/3
. Apply the methods to α = 18
to obtain the results whcih are correct to two significant digits when compared
to their exact values.
10. Consider the nonlinear equation
1
3
x3
− x2
+ x + 1 = 0.
Show that there exists an initial guess x0 ∈ (0, 4) for which x2 of the Newton-
Raphson method iterative sequence is not defined.
11. Let a be a real number such that 0 < a ≤ 1. Let {xn}∞
n=1 be the iterative sequence
of the Newton-Raphson method to solve the nonlinear equation e−ax
= x. If x∗
denotes the exact root of this equation and x0 > 0, then show that
|x∗
− xn+1| ≤
1
2
(x∗
− xn)2
.
12. Newton-Raphson method is to be applied for approximating a root of the nonlin-
ear equation x4
− x − 10 = 0.
i) How many solutions of the nonlinear equation are there in [1, ∞)? Are they
simple?
ii) Find an interval [1, b] that contains the smallest positive solution of the
nonlinear equation.
iii) Compute five iterates of Newton-Raphson method, for each of the initial
guesses x0 = 1, x0 = 2, x0 = 100. What are your observations?
iv) A solution of the nonlinear equation is approximately equal to 1.85558 Find
a δ as in the proof of theorem on Newton-Raphson method, so that iterative
sequence of Newton-Raphson method always converges for every initial guess
x0 ∈ [1.85558 − δ, 1.85558 + δ].
v) Can we appeal to the theorem for convex functions in the context of Newton-
Raphson method? Justify.
13. Newton-Raphson method is to be applied for approximating a root of the equation
sin x = 0.
i) Find formula for the Newton-Raphson iterative sequence.
ii) Let α ∈ (−π/2, π/2) and α ̸= 0 be such that if x0 = α, then the iteration
becomes a cycle i.e.,
α = x0 = x2 = · · · = x2k = x2k+2 = · · · , x1 = x2 = · · · = x2k+1 = x2k+3 = · · ·
Find a non-linear equation g(y) = 0 whose solution is α.
S. Baskar and S. Sivaji Ganesh
153
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-165-320.jpg)
![Chapter 4. Nonlinear Equations
20. Show that g(x) = π + 1
2
sin(x/2) has a unique fixed point in [0, 2π]. Use fixed-
point iteration method with g as the iteration function and x0 = 0 to find an
approximate solution for the equaton 1
2
sin(x/2) − x + π = 0 with the stopping
criterion that the residual error is less than 10−4
.
21. Let α ∈ R and β ∈ R be the roots of x2
+ax+b = 0, and such that |α| > |β|. Let g
and h be two iterating functions satisfying the hypothesis of the theorem on fixed-
point method. Consider the iterative sequences {xn} and {yn} corresponding to
two the iterating functions g and h given by
xn+1 = −
axn + b
xn
, and yn+1 = −
b
yn + a
respectively. Show that the iterative sequences {xn} and {yn} converge to α and
β respectively.
22. Let {xn} ⊂ [a, b] be a sequence generated by a fixed point iteration method with
a continuously differentiable iteration function g(x). If this sequence converges to
x∗
, then show that
|xn+1 − x∗
| ≤
λ
1 − λ
|xn+1 − xn|,
where λ := max
x∈[a,b]
|g′
(x)|. (This estimate helps us to decide when to stop iterating
if we are using a stopping criterion based on the distance between successive it-
erates.)
23. Explain why the sequence of iterates xn+1 = 1 − 0.9x2
n, with initial guess x0 = 0,
does not converge to any solution of the quadratic equation 0.9x2
+ x − 1 = 0?
[Hint: Observe what happens after 25 iterations]
24. Let x∗
be the smallest positive root of the equation 20x3
− 20x2
− 25x + 4 = 0.
The following question is concerning the fixed-point formulation of the nonlinear
equation given by x = g(x), where g(x) = x3
− x2
− x
4
+ 1
5
.
i) Show that x∗
∈ [0, 1].
ii) Does the function g satisfy the hypothesis of theorem on fixed-point method?
If yes, we know that x∗
is the only fixed point of g lying in the interval
[0, 1]. In the notation of fixed-point iteration method, find an n such that
|x∗
− xn| < 10−3
, when the initial guess x0 is equal to 0.
25. Let f : [a, b] → R be a function such that f′
is continuous, f(a)f(b) < 0, and
there exists an α > 0 such that f′
(x) ≥ α > 0.
S. Baskar and S. Sivaji Ganesh
156
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-168-320.jpg)
![Section 4.5 Exercises
i) Show that f(x) = 0 has exactly one solution in the interval [a, b].
ii) Show that with a suitable choice of the parameter λ, the solution of the
nonlinear equation f(x) = 0 can be obtained by applying the fixed-point
iteration method applied to the function F(x) = x + λf(x).
26. Let p > 1 be a real number. Show that the following expression has a meaning
and find its value.
x =
√
p +
√
p +
√
p + · · ·
Note that the last equation is interpreted as x = limn→∞ xn, where x1 =
√
p,
x2 =
√
p +
√
p, · · · . (Hint: Note that xn+1 =
√
p + xn, and show that the se-
quence {xn} converges using the theorem on fixed-point method.)
27. Draw the graph of a function having the following properties: (i) The function
has exactly TWO fixed points. (ii) Give two choices of the initial guess x0 and
y0 such that the corresponding sequences {xn} and {yn} have the properties that
{xn} converges to one of the fixed points and the sequence {yn} goes away and
diverges. Point out the first three terms of both the sequences on the graph.
S. Baskar and S. Sivaji Ganesh
157
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-169-320.jpg)
![Chapter 5. Interpolation
In certain circumstances, the function f may be known explicitly, but still too dif-
ficult to perform certain operations like differentiation and integration. Thus, it is
often preferred to restrict the class of interpolating functions to polynomials, where the
differentiation and the integration can be done more easily.
In Section 5.1, we introduce the basic problem of polynomial interpolation and prove
the existence and uniqueness of polynomial interpolating the given data. There are at
least two ways to obtain the unique polynomial interpolating a given data, one is the
Lagrange and another one is the Newton. In Section 5.1.2, we introduce Lagrange form
of interpolating polynomial, whereas Section 5.1.3 introduces the notion of divided
differences and Newton form of interpolating polynomial. The error analysis of the
polynomial interpolation is studied in Section 5.3. In certain cases, the interpolating
polynomial can differ significantly from the exact function. This is illustrated by Carl
Runge and is called the Runge Phenomenon. In Section 5.3.4, we present the example
due to Runge and state a few results on convergence of the interpolating polynomials.
The concept of piecewise polynomial interpolation and Spline interpolation are discussed
in Section 5.5.
5.1 Polynomial Interpolation
Polynomial interpolation is a concept of fitting a polynomial to a given data. Thus, to
construct an interpolating polynomial, we first need a set of points at which the data
values are known.
Definition 5.1.1.
Any collection of distinct real numbers x0, x1, · · · , xn (not necessarily in increasing
order) is called nodes.
Definition 5.1.2 [Interpolating Polynomial].
Let x0, x1, · · · , xn be the given nodes and y0, y1, · · · , yn be real numbers. A polyno-
mial pn(x) of degree less than or equal to n is said to be a polynomial interpolating
the given data or an interpolating polynomial for the given data if
pn(xi) = yi, i = 0, 1, · · · n. (5.1)
The condition (5.1) is called the interpolation condition.
S. Baskar and S. Sivaji Ganesh
160
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-172-320.jpg)
![Section 5.1 Polynomial Interpolation
Remark 5.1.3.
Let x0, x1, · · · , xn be given nodes, and y0, y1, · · · , yn be real numbers. Let pn(x) be
a polynomial interpolating the given data. Then the graph of pn(x) passes through
the set of (n + 1) distinct points in the xy-plane given by the table
x x0 x1 x2 x3 · · · xn
y y0 y1 y2 y3 · · · yn
.
We call the set {(xi, yi), i = 0, 1, · · · , n} as data and quite often we represent this
set in the above form of a table.
5.1.1 Existence and Uniqueness of Interpolating Polynomial
The following result asserts that an interpolating polynomial exists and is unique.
Theorem 5.1.4 [Existence and Uniqueness of Interpolating polynomial].
Let x0, x1, · · · , xn be given nodes, and y0, y1, · · · , yn be real numbers.
1. Then there exists a polynomial pn(x) of degree less than or equal to n such
that
pn(xi) = yi, i = 0, 1, · · · , n.
That is, there exists an interpolating polynomial pn(x) for the given data
{(xi, yi), i = 0, 1, · · · , n}.
2. Such a polynomial is unique.
Proof.
Proof of uniqueness: Assume that pn(x) and qn(x) are interpolating polynomials
of degree less than or equal to n that satisfies the interpolation condition (5.1). Let
rn(x) = pn(x) − qn(x).
Then, rn(x) is also a polynomial of degree less than or equal to n, and by the inter-
polation condition, we have
rn(xi) = 0,
for every i = 0, 1, · · · , n. Thus, rn(x) is a polynomial of degree less than or equal to n
with n + 1 distinct roots. By the fundamental theorem of algebra, we conclude that
rn(x) is the zero polynomial. That is, the interpolating polynomial is unique.
S. Baskar and S. Sivaji Ganesh
161
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-173-320.jpg)
![Section 5.1 Polynomial Interpolation
Example 5.1.6.
Let the following data represent the values of f:
x 0 0.5 1
f(x) 1.0000 0.5242 −0.9037
The questions are the following:
1. What is the exact expression for the function f?
2. What is the value of f(0.75)?
We cannot get the exact expression for the function f just from the given data,
because there are infinitely many functions having same value at the given set of
points. Due to this, we cannot expect an exact value for f(0.75), in fact, it can be
any real number. On the other hand, if we look for f in the class of polynomials of
degree less than or equal to 2, then Theorem 5.1.4 tells us that there is exactly one
such polynomial and hence we can obtain a unique value for f(0.75).
The interpolating polynomial happens to be
p2(x) = −1.9042x2
+ 0.0005x + 1
and we have
p2(0.75) = −0.0707380.
The function used to generate the above table of data is
f(x) = sin
(π
2
ex
)
.
With this expression of f, we have (using 7-digit rounding)
f(0.75) ≈ −0.1827495.
The relative error is given by
Er(p2(0.75)) =
f(0.75) − p2(0.75)
f(0.75)
≈ 0.6129237.
That is, at the point x = 0.75 the polynomial approximation to the given function f
has more than 61% error. The graph of the function f (blue solid line) and p2 (green
dash line) are depicted in Figure 5.1. The blue dots denote the given data, magenta
‘+’ symbol indicates the value of p2(0.75) and the red ‘O’ symbol represents the value
of f(0.75). It is also observed from the graph that if we approximate the function f
for x ∈ [0, 0.5], then we obtain a better accuracy than approximating f in the interval
(0.5, 1).
S. Baskar and S. Sivaji Ganesh
163
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-175-320.jpg)
![Chapter 5. Interpolation
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
x
y
Figure 5.1: The function f(x) = sin
(π
2
ex
)
(blue solid line) and p2(x) (green dash line).
Blue dots represent the given data, magenta ‘+’ symbol indicates the value of p2(0.75)
and the red ‘O’ symbol represents the value of f(0.75).
5.1.2 Lagrange’s Form of Interpolating Polynomial
Definition 5.1.7 [Lagrange’s Polynomial].
Let x0, x1, · · · , xn be the given nodes. For each k = 0, 1, · · · , n, the polynomial lk(x)
defined by
lk(x) =
n∏
i=0
i̸=k
(x − xi)
(xk − xi)
(5.3)
is called the kth
Lagrange polynomial or the kth
Lagrange cardinal function.
Remark 5.1.8.
Note that the kth Lagrange polynomial depends on all the n+1 nodes x0, x1, · · · , xn.
S. Baskar and S. Sivaji Ganesh
164
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-176-320.jpg)
![Section 5.1 Polynomial Interpolation
Theorem 5.1.9 [Lagrange’s form of Interpolating Polynomial].
Hypothesis:
1. Let x0, x1, · · · , xn be given nodes.
2. Let the values of a function f be given at these nodes.
3. For each k = 0, 1, · · · , n, let lk(x) be the kth
Lagrange polynomial.
4. Let pn(x) (of degree ≤ n) be the polynomial interpolating the function f at
the nodes x0, x1, · · · , xn.
Conclusion: Then, pn(x) can be written as
pn(x) =
n∑
i=0
f(xi)li(x). (5.4)
This form of the interpolating polynomial is called the Lagrange’s form of In-
terpolating Polynomial.
Proof.
Firstly, we will prove that q(x) :=
∑n
i=0 f(xi)li(x) is an interpolating polynomial for
the function f at the nodes x0, x1, · · · , xn. Since
li(xj) =
{
1 if i = j
0 if i ̸= j
,
we get q(xj) = f(xj) for each j = 0, 1, · · · , n. Thus, q(x) is an interpolating polyno-
mial. Since interpolating polynomial is unique by Theorem 5.1.4, the polynomial q(x)
must be the same as pn(x). This completes the proof of the theorem.
Example 5.1.10.
Consider the case n = 1 in which we have two distinct points x0 and x1. Thus, we
have
l0(x) =
x − x1
x0 − x1
, l1(x) =
x − x0
x1 − x0
and therefore,
p1(x) = f(x0)l0(x) + f(x1)l1(x)
= f(x0)
x − x1
x0 − x1
+ f(x1)
x − x0
x1 − x0
=
f(x0)(x − x1) − f(x1)(x − x0)
x0 − x1
S. Baskar and S. Sivaji Ganesh
165
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-177-320.jpg)
![Section 5.1 Polynomial Interpolation
Remark 5.1.12.
The above example gives us a feeling that if we increase the number of nodes,
and thereby increasing the degree of the interpolating polynomial, the polynomial
approximates the orginal function more accurately. But this is not true in general,
and we will discuss this further in Section 5.3.4.
Remark 5.1.13.
Let x0, x1, · · · , xn be nodes, and f be a function. Recall that computing an in-
terpolating polynomial in Lagrange’s form requires us to compute for each k =
0, 1, · · · , n, the kth Lagrange’s polynomial lk(x) which depends on the given nodes
x0, x1, · · · , xn. Suppose that we have found the corresponding interpolating poly-
nomial pn(x) of f in the Lagrange’s form for the given data. Now if we add one
more node xn+1, the computation of the interpolating polynomial pn+1(x) in the
Lagrange’s form requires us to compute a new set of Lagrange’s polynomials cor-
responding to the set of (n + 1) nodes, and no advantage can be taken of the fact
that pn is already available.
An alternative form of the interpolating polynomial, namely, Newton’s form of
interpolating polynomial, avoids this problem, and will be discussed in the next
section.
5.1.3 Newton’s Form of Interpolating Polynomial
We saw in the last section that it is easy to write the Lagrange form of the interpolating
polynomial once the Lagrange polynomials associated to a given set of nodes have been
written. However we observed in Remark 5.1.13 that the knowledge of pn (in Lagrange
form) cannot be utilized to construct pn+1 in the Lagrange form. In this section we
describe Newton’s form of interpolating polynomial, which uses the knowledge of pn in
constructing pn+1.
Theorem 5.1.14 [Newton’s form of Interpolating Polynomial].
Hypothesis:
1. Let x0, x1, · · · , xn be given nodes.
2. Let the values of a function f be given at these nodes.
3. Let pn(x) (of degree ≤ n) be the polynomial interpolating the function f at
the nodes x0, x1, · · · , xn.
S. Baskar and S. Sivaji Ganesh
167
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-179-320.jpg)
![Section 5.2 Newton’s Divided Difference Formulas
5.2 Newton’s Divided Difference Formulas
Definition 5.2.1 [Divided Differences].
Let x0, x1, · · · , xn be distinct nodes. Let pn(x) be the polynomial interpolating a
function f at the nodes x0, x1, · · · , xn. The coefficient of xn
in the polynomial pn(x)
is denoted by f[x0, x1, · · · , xn], and is called an nth
divided difference of f.
Remark 5.2.2.
1. Since the interpolating polynomial for the function f at the nodes x0, x1, · · · , xn
is unique, there is one and only one coefficient of xn; even though interpo-
lation polynomial may have many forms like Lagrange’s form and Newton’s
form. Thus the quantity f[x0, x1, · · · , xn] is well-defined.
2. More generally, the divided difference f[xi, xi+1, · · · , xi+k] is the coefficient
of xk in the polynomial interpolating f at the nodes xi, xi+1, · · · , xi+k.
3. The Newton’s form of interpolating polynomial may be written, using divided
differences, as
pn(x) = f[x0] +
n∑
k=1
f[x0, x1, · · · , xk]
k−1∏
i=0
(x − xi) (5.9)
Example 5.2.3.
As a continuation of Example 5.1.10, let us construct the linear interpolating polyno-
mial of a function f in the Newton’s form. In this case, the interpolating polynomial
is given by
p1(x) = f[x0] + f[x0, x1](x − x0),
where
f[x0] = f(x0), f[x0, x1] =
f(x0) − f(x1)
x0 − x1
(5.10)
are zeroth and first order divided differences, respectively. Observe that this polyno-
mial is exactly the same as the interpolating polynomial obtained using Lagrange’s
form in Example 5.1.10.
The following result is concerning the symmetry properties of divided differences.
S. Baskar and S. Sivaji Ganesh
169
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-181-320.jpg)
![Chapter 5. Interpolation
Theorem 5.2.4 [Symmetry].
The divided difference is a symmetric function of its arguments. That is, if z0, z1, · · · , zn
is a permutation of x0, x1, · · · , xn, then
f[x0, x1, · · · , xn] = f[z0, z1, · · · , zn] (5.11)
Proof.
Since z0, z1, · · · , zn is a permutation of x0, x1, · · · , xn, which means that the nodes
x0, x1, · · · , xn have only been re-labelled as z0, z1, · · · , zn, and hence the polynomial
interpolating the function f at both these sets of nodes is the same. By definition
f[x0, x1, · · · , xn] is the coefficient of xn
in the polynomial interpolating the func-
tion f at the nodes x0, x1, · · · , xn, and f[z0, z1, · · · , zn] is the coefficient of xn
in the
polynomial interpolating the function f at the nodes z0, z1, · · · , zn. Since both the
interpolating polynomials are equal, so are the coefficients of xn
in them. Thus, we
get
f[x0, x1, · · · , xn] = f[z0, z1, · · · , zn].
This completes the proof.
The following result helps us in computing recursively the divided differences of higher
order.
Theorem 5.2.5 [Higher-order divided differences].
Divided differences satisfy the equation
f[x0, x1, · · · , xn] =
f[x1, x2, · · · , xn] − f[x0, x1, · · · , xn−1]
xn − x0
(5.12)
Proof.
Let us start the proof by setting up the following notations.
• Let pn(x) be the polynomial interpolating f at the nodes x0, x1, · · · , xn.
• Let pn−1(x) be the polynomial interpolating f at the nodes x0, x1, · · · , xn−1.
• Let q(x) be the polynomial interpolating f at the nodes x1, x2, · · · , xn.
Claim: We will prove the following relation between pn−1, pn, and q:
pn(x) = pn−1(x) +
x − x0
xn − x0
(
q(x) − pn−1(x)
)
(5.13)
S. Baskar and S. Sivaji Ganesh
170
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-182-320.jpg)
![Section 5.2 Newton’s Divided Difference Formulas
Since both sides of the equality in (5.13) are polynomials of degree less than or equal
to n, and pn(x) is the polynomial interpolating f at the nodes x0, x1, · · · , xn, the
equality in (5.13) holds for all x if and only if it holds for x ∈ { x0, x1, · · · , xn } and
both sides of the equality reduce to f(x) for x ∈ { x0, x1, · · · , xn }. Let us now verify
the equation (5.13) for x ∈ { x0, x1, · · · , xn }.
1. When x = x0,
pn−1(x0) +
x0 − x0
xn − x0
(
q(x0) − pn−1(x0)
)
= pn−1(x0) = f(x0) = pn(x0).
2. When x = xk for 1 ≤ k ≤ n − 1, q(xk) = pn−1(xk) and thus we have
pn−1(xk) +
xk − x0
xn − x0
(
q(xk) − pn−1(xk)
)
= pn−1(xk) = f(xk) = pn(xk).
3. When x = xn, we have
pn−1(xn)+
xn − x0
xn − x0
(
q(xn)−pn−1(xn)
)
= pn−1(xn)+
(
f(xn)−pn−1(xn)
)
= f(xn) = pn(xn).
This finishes the proof of the Claim.
The coefficient of xn
in the polynomial pn(x) is f[x0, x1, · · · , xn]. The coefficient of
xn
using the right hand side of the equation (5.13) is given by
(
coefficient of xn
in pn−1(x)
)
+
1
xn − x0
(
coefficient of xn
in (x−x0)
(
q(x)−pn−1(x)
))
.
On noting that the coefficient of xn−1
in the polynomial pn−1 is f[x0, x1, · · · , xn−1],
the coefficient of xn−1
in the polynomial q is f[x1, x2, · · · , xn], and the coefficient of
xn
in the polynomial pn−1 is zero, we get that the coefficient of xn
using the right
hand side of the equation (5.13) becomes
f[x1, x2, · · · , xn] − f[x0, x1, · · · , xn−1]
xn − x0
.
Comparing the coefficients of xn
in the left and right hand sides of the equation (5.13)
yields
f[x0, x1, · · · , xn] =
f[x1, x2, · · · , xn] − f[x0, x1, · · · , xn−1]
xn − x0
.
S. Baskar and S. Sivaji Ganesh
171
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-183-320.jpg)
![Chapter 5. Interpolation
Remark 5.2.6.
Let i, j ∈ N. Applying Theorem 5.2.5 to a set of nodes xi, xi+1, · · · , xi+j, we
conclude
f[xi, xi+1, · · · , xi+j] =
f[xi+1, xi+2, · · · , xi+j] − f[xi, xi+1, · · · , xi+j−1]
xi+j − xi
(5.14)
Note that the divided differences f[x0, x1, · · · , xn] are defined only for distinct nodes
x0, x1, · · · , xn.
5.2.1 Divided Differences Table
Given a collection of (n + 1) nodes x0, x1, · · · , xn and the values of the function f at
these nodes, we can construct the Newton’s form of interpolating polynomial pn(x) using
divided differences. As observed earlier, the Newton’s form of interpolation polynomial
has the formula
pn(x) = f[x0] +
n∑
k=1
f[x0, x1, · · · , xk]
k−1∏
i=0
(x − xi) (5.15)
One can explicitly write the formula (5.15) for n = 1, 2, 3, 4, 5, · · · . For instance, when
n = 5, the formula (5.15) reads
p5(x) = f[x0]
+ f[x0, x1](x − x0)
+ f[x0, x1, x2](x − x0)(x − x1)
+ f[x0, x1, x2, x3](x − x0)(x − x1)(x − x2)
+ f[x0, x1, x2, x3, x4](x − x0)(x − x1)(x − x2)(x − x3)
+ f[x0, x1, x2, x3, x4, x5](x − x0)(x − x1)(x − x2)(x − x3)(x − x4)
.
(5.16)
For easy computation of the divided differences in view of the formula (5.12), it is
convenient to write the divided differences in a table form. For n = 5, the divided
difference table is given by
S. Baskar and S. Sivaji Ganesh
172
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-184-320.jpg)
![Section 5.2 Newton’s Divided Difference Formulas
x0 f(x0)
f[x0, x1]
x1 f(x1) f[x0, x1, x2]
f[x1, x2] f[x0, x1, x2, x3]
x2 f(x2) f[x1, x2, x3] f[x0, x1, x2, x3, x4]
f[x2, x3] f[x1, x2, x3, x4] f[x0, x1, x2, x3, x4, x5]
x3 f(x3) f[x2, x3, x4] f[x1, x2, x3, x4, x5]
f[x3, x4] f[x2, x3, x4, x5]
x4 f(x4) f[x3, x4, x5]
f[x4, x5]
x5 f(x5)
Comparing the above divided differences table and the interpolating polynomial p5
given by (5.16), we see that the leading members of each column (denoted in bold font)
are the required divided differences used in p5(x).
5.2.2 Divided Difference Formula for Repeated Nodes
It follows from the symmetric property (Theorem 5.2.4) that the nth
order divided
difference formula (5.12) is not well-defined if at least two nodes coincide. But such
situations do occur quite often in numerical analysis, for instance, in the error analysis
of quadrature formulas. So, we need to interpret the define of the divided difference
in such a way that it is still well defined if two or more nodes coincide. For this, the
following theorem is useful.
Theorem 5.2.7 [Hermite-Genocchi Formula].
Hypothesis:
1. Let x0, x1, · · · , xn ∈ [a, b] be distinct nodes.
2. Let f be n-times continuously differentiable function on the interval [a, b].
Conclusion: Then
f[x0, x1, · · · , xn] =
∫
· · ·
∫
τn
f(n)
(t0x0 + · · · + tnxn) dt1 · · · dtn, (5.17)
where
τn =
{
(t1, t2, · · · , tn) / ti ≥ 0, i = 1, 2, · · · , n;
n∑
i=1
ti ≤ 1
}
, t0 = 1 −
n∑
i=1
ti.
(5.18)
S. Baskar and S. Sivaji Ganesh
173
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-185-320.jpg)
![Chapter 5. Interpolation
Proof.
We prove the theorem by induction.
Step 1 : First, we prove (5.17) for n = 1 . From (5.18), we see that τ1 = [0, 1]. Using
the expression for t0 in (5.18), we have
1∫
0
f′
(t0x0 + t1x1)dt1 =
1∫
0
f′
(x0 + t1(x1 − x0)) dt1 =
1
x1 − x0
f(x0 + t1(x1 − x0))
t1=1
t1=0
=
f(x1) − f(x0)
x1 − x0
= f[x0, x1].
Thus, we have proved the result (5.17) for n = 1.
Step 2 : Now assume that the formula (5.17) holds for n = k ≥ 1 and prove the
formula for n = k + 1. We have
∫
· · ·
∫
τk+1
f(k+1)
(t0x0 + t1x1 + · · · + tk+1xk+1) dt1 · · · dtk+1
=
∫
· · ·
∫
τk
1−(t1+···+tk)∫
0
f(k+1)
(x0 + t1(x1 − x0) + · · · + tk+1(xk+1 − x0)) dtk+1
dt1 · · · dtk
=
∫
· · ·
∫
τk
1
xk+1 − x0
[
f(k)
(x0 + t1(x1 − x0) + · · · + tk+1(xk+1 − x0))
]tk+1=1−(t1+···+tk)
tk+1=0
dt1 · · · dtk
=
1
xk+1 − x0
∫
· · ·
∫
τk
f(k)
(xk+1 + t1(x1 − xk+1) + · · · + tk(xk − xk+1)) dt1 · · · dtk
−
∫
· · ·
∫
τk
f(k)
(x0 + t1(x1 − x0) + · · · + tk(xk − x0)) dt1 · · · dtk
=
f[x1, · · · , xk, xk+1] − f[x0, x1, · · · , xk]
xk+1 − x0
= f[x0, x1, · · · , xk+1].
By the principle of mathematical induction, the result is true for all n ∈ N.
Remark 5.2.8.
Since f is n-times continuously differentiable, the right hand side of (5.17) is mean-
ingful for any set of points x0, x1, · · · , xn, which are not necessarily distinct. This
gives a meaning, which can be used to define the nth order divided difference when
x0, x1, · · · , xn are not distinct (i.e., one or more points are repeated).
S. Baskar and S. Sivaji Ganesh
174
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-186-320.jpg)
![Section 5.2 Newton’s Divided Difference Formulas
Following results are direct consequences of the above theorem.
Corollary 5.2.9.
Hypothesis:
1. Let x0, x1, · · · , xn ∈ [a, b] be a set of points, not necessarily distinct.
2. Let f be n-times continuously differentiable function on the interval [a, b].
Conclusion:
1. The function (x0, x1, · · · , xn) −→ f[x0, x1, · · · , xn] is continuous on Rn+1
.
2. For any x ∈ [a, b], the limit
lim
(x0,x1,··· ,xn)→(x,x,··· ,x)
f[x0, x1, · · · , xn]
exists and this limit is the nth
-order divided difference f[x, x, · · · , x] (x repeated
(n + 1)-times). Further, we have
f[x, x, · · · , x] =
f(n)
(x)
n!
. (5.19)
Proof.
1. The proof is out side the scope of this course and hence omitted.
2. The proof follows from the fact that
∫
· · ·
∫
τn
1 dt1 · · · dtn =
1
n!
and the Hermite-Genocchi formula (5.17).
The following theorem gives an expression for the divided difference of an (n+2)-times
differentiable function when two nodes are repeated.
Theorem 5.2.10.
Hypothesis:
1. Let x0, x1, · · · , xn be given (distinct) nodes in an interval [a, b].
2. Let x ∈ [a, b], x /∈ {x0, x1, · · · , xn}.
Conclusion: The (n + 2)nd
-order divided difference f[x0, x1, · · · , xn, x, x] of an (n +
2)-times continuously differentiable function f is given by
f[x0, x1, · · · , xn, x, x] =
d
dx
f[x0, x1, · · · , xn, x]. (5.20)
S. Baskar and S. Sivaji Ganesh
175
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-187-320.jpg)
![Chapter 5. Interpolation
Proof.
It follows from (5.17) that the function F(x) = f[x0, x1, · · · , xn, x] for all x ∈ [a, b] is
well-defined and continuous. Therefore, using the symmetry property of the divided
differences, we get
f[x0, x1, · · · , xn, x, x] = lim
h→0
f[x0, x1, · · · , xn, x, x + h]
= lim
h→0
f[x, x0, x1, · · · , xn, x + h].
As all the points used on the right hand side are distinct, we can use the formula
(5.12) to get
f[x0, x1, · · · , xn, x, x] = lim
h→0
f[x0, x1, · · · , xn, x + h] − f[x, x0, x1, · · · , xn]
h
= lim
h→0
f[x0, x1, · · · , xn, x + h] − f[x0, x1, · · · , xn, x]
h
=
d
dx
f[x0, x1, · · · , xn, x].
This completes the proof.
5.3 Error in Polynomial Interpolation
Let f be a function defined on an interval I = [a, b]. Let pn(x) be a polynomial of
degree less than or equal to n that interpolates the function f at n + 1 nodes x0, x1,
· · · , xn in I. How well does pn(x) approximate f on the interval I? This question leads
to the analysis of interpolation error.
As we discussed at the beginning of Chapter 2 (Error Analysis), the error given by
MEn(x) = f(x) − pn(x). (5.21)
is the mathematical error involved in the interpolating polynomial. But, when we per-
form the calculations using finite precision floating-point arithmetic, then the polyno-
mial obtained, denoted by ˜pn(x), need not be the same as the interpolating polynomial
pn(x). The error
AEn(x) = pn(x) − ˜pn(x). (5.22)
is the arithmetic error involved in the interpolating polynomial.
The total error, denoted by TEn(x), involved in the polynomial interpolation is therefore
given by
TEn(x) = f(x) − ˜pn(x) = (f(x) − pn(x)) + (pn(x) − ˜pn(x)) = MEn(x) + AEn(x).
S. Baskar and S. Sivaji Ganesh
176
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-188-320.jpg)
![Section 5.3 Error in Polynomial Interpolation
In Subsection 5.3.1, we derive the mathematical error involved in polynomial interpola-
tion. We analyze the effect of arithmetic error in Subsection 5.3.2.
5.3.1 Mathematical Error
The following theorem provides a formula for the interpolation error when we as-
sume that the necessary data are given exactly without any floating-point approxi-
mation.
Theorem 5.3.1 [Mathematical Error in Interpolation].
Let pn(x) be the polynomial interpolating a function f ∈ Cn+1
[a, b] at the nodes
x0, x1, · · · , xn lying in I = [a, b]. Then for each x ∈ I, there exists a ξx ∈ (a, b) such
that
MEn(x) =
f(n+1)
(ξx)
(n + 1)!
n∏
i=0
(x − xi) (5.23)
Proof.
If x is one of the nodes, then (5.23) holds trivially for any choice of ξx ∈ (a, b).
Therefore, it is enough to prove the theorem when x is not a node. The idea is to
obtain a function having at least n + 2 distinct zeros; and then apply Rolle’s theorem
n + 1 times to get the desired conclusion.
For a given x ∈ I with x ̸= xi (i = 0, 1, · · · , n), define a new function ψ on the
interval I by
ψ(t) = f(t) − pn(t) − λ
n∏
i=0
(t − xi), t ∈ I, (5.24)
where λ is chosen so that ψ(x) = 0. This gives
λ =
f(x) − pn(x)
n∏
i=0
(x − xi)
.
Note that ψ(xi) = 0 for each i = 0, 1, · · · , n. Thus, ψ has at least n + 2 distinct
zeros. By Rolle’s theorem, the function ψ′
has at least n + 1 distinct zeros in (a, b).
A repeated application of Rolle’s theorem n + 1 times to the function ψ gives that
ψ(n+1)
has at least one zero in (a, b); call it ξx. Differentiating (5.24) (n + 1)-times
S. Baskar and S. Sivaji Ganesh
177
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-189-320.jpg)
![Chapter 5. Interpolation
and noting that
p(n+1)
n (t) = 0,
( n∏
i=0
(t − xi)
)(n+1)
= (n + 1)!, ψ(n+1)
(ξx) = 0,
we see that ξx satisfies
0 = ψ(n+1)
(ξx) = f(n+1)
(ξx) −
f(x) − pn(x)
n∏
i=0
(x − xi)
(n + 1) !.
Thus,
f(n+1)
(ξx) −
MEn(x)
n∏
i=0
(x − xi)
(n + 1) ! = 0.
The last equation yields (5.23).
The following theorem plays an important role in the error analysis of numerical inte-
gration. The idea behind the proof of this theorem is similar to the idea used in the
above theorem and is left as an exercise.
Theorem 5.3.2.
If f ∈ Cn+1
[a, b] and if x0, x1, · · · , xn are distinct nodes in [a, b], then for any x ̸= xi,
i = 0, 1, · · · , n, there exists a point ξx ∈ (a, b) such that
f[x0, x1, · · · , xn, x] =
f(n+1)
(ξx)
(n + 1) !
. (5.25)
Remark 5.3.3.
It is interesting to note that when all the nodes coincide, then (5.25) reduces to
(5.19).
Definition 5.3.4 [Infinity Norm].
If f is continuous on a closed interval I = [a, b], then the infinity norm of f,
denoted by ∥f∥∞,I, is defined as
∥f∥∞,I = max
x∈I
|f(x)|. (5.26)
S. Baskar and S. Sivaji Ganesh
178
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-190-320.jpg)
,
where f[x0, x1] is given by (5.10). For each x ∈ I := [x0, x1], using the formula (5.23),
the error ME1(x) is given by
ME1(x) =
(x − x0)(x − x1)
2
f′′
(ξx),
where ξx ∈ (x0, x1) depends on x. Therefore,
|ME1(x)| ≤ |(x − x0)(x − x1)|
∥f′′
∥∞,I
2
.
Note that the maximum value of |(x − x0)(x − x1)| as x varies in the interval [x0, x1],
occurs at x = (x0 + x1)/2. Therefore, we have
|(x − x0)(x − x1)| ≤
(x1 − x0)2
4
.
Using this inequality, we get an upper bound
|ME1(x)| ≤ (x1 − x0)2 ∥f′′
∥∞,I
8
, for all x ∈ [x0, x1].
Note that the above inequality is true for all x ∈ [x0, x1]. In particular, this inequality
is true for an x at with the function |ME1| attains its maximum. Thus, we have
∥ME1∥∞,I ≤ (x1 − x0)2 ∥f′′
∥∞,I
8
.
The right hand side quantity, which is a real number, is an upper bound for the
mathematical error in linear interpolation with respect to the infinity norm.
Example 5.3.6.
Let the function
f(x) = sin x
be approximated by an interpolating polynomial p9(x) of degree less than or equal to
S. Baskar and S. Sivaji Ganesh
179
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-191-320.jpg)
![Chapter 5. Interpolation
9 for f at the nodes x0, x1, · · · , x9 in the interval I := [0, 1]. Let us obtain an upper
bound for ∥ME9∥∞,I, where (from (5.23))
ME9(x) =
f(10)
(ξx)
10!
9∏
i=0
(x − xi).
Since |f(10)
(ξx)| ≤ 1 and
∏9
i=0 |x − xi| ≤ 1, we have
| sin x − p9(x)| = |ME9(x)| ≤
1
10!
< 2.8 × 10−7
, for all x ∈ [0, 1].
Since this holds for all x ∈ [0, 1], we have
∥ME9∥∞,I < 2.8 × 10−7
.
The right hand side number is the upper bound for the mathematical error ME9 with
respect to the infinity norm on the interval I.
5.3.2 Arithmetic Error
Quite often, the polynomial interpolation that we compute is based on the function
data subjected to floating-point approximation. In this subsection, we analyze the
arithmetic error arising due to the floating-point approximation fl(f(xi)) of f(xi) for
each node point xi, i = 0, 1, · · · , n in the interval I = [a, b].
The Lagrange form of interpolating polynomial that uses the values fl(f(xi)) instead of
f(xi), i = 0, 1, · · · , n is given by
˜pn(x) =
n∑
i=0
fl(f(xi)) li(x).
We now analyze the arithmetic error. Denoting
ϵi := f(xi) − fl(f(xi); ||ϵ||∞ = max{|ϵi| : i = 0, 1, · · · , n},
we have
|AEn(x)| = |pn(x) − ˜pn(x)| =
n∑
i=0
(
f(xi) − fl(f(xi))
)
li(x) ≤ ||ϵ||∞
n∑
i=0
||li||∞,I,
for all x ∈ I. Therefore,
∥AEn∥∞,I = ∥pn − ˜pn∥∞,I ≤ ||ϵ||∞
n∑
i=0
||li||∞,I. (5.27)
S. Baskar and S. Sivaji Ganesh
180
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-192-320.jpg)
![Section 5.3 Error in Polynomial Interpolation
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
1
1.05
1.1
1.15
1.2
1.25
x
l(x)
(a)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
1
2
3
4
5
6
7
8
9
10
11
x
l(x)
(b)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
500
1000
1500
2000
2500
3000
3500
x
l(x)
(c)
0 2 4 6 8 10 12 14 16 18 20
0
2000
4000
6000
8000
10000
12000
14000
n
Mn
(d)
Figure 5.2: (a) to (c) depicts the graph of function l given by (5.28) for x ∈ [0, 1] when
n = 2, 8, and 18. (d) depicts the function n in the x-axis and Mn given by (5.29) in the
y-axis.
The upper bound in (5.27) might grow quite large as n increases, especially when the
nodes are equally spaced as we will study now.
Assume that the nodes are equally spaced in the interval [a, b], with x0 = a and xn = b,
and xi+1 − xi = h for all i = 0, 1, · · · , n − 1. Note that h = (b − a)/n. We write
xi = a + ih, i = 0, 1, · · · , n.
Any x ∈ I can be written as
x = a + ηh,
where 0 ≤ η ≤ n. Here η is not necessarily an integer. Therefore, for any x ∈ I, we
have
lk(x) =
n∏
i=0
i̸=k
(x − xi)
(xk − xi)
=
n∏
i=0
i̸=k
(η − i)
(k − i)
, k = 0, · · · , n.
Clearly, the Lagrange polynomials are not dependent on the choice of a, b, and h. They
depend entirely on n and η (which depends on x). The Figure 5.2 (a), (b) and (c) shows
S. Baskar and S. Sivaji Ganesh
181
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-193-320.jpg)
![Section 5.3 Error in Polynomial Interpolation
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
x
pn(x)
(a)
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
−5
0
5
10
15
20
25
30
x
pn(x)
(b)
Figure 5.3: Runge Phenomenon is illustrated. Figure (a) depicts the graph of the
function f given by (5.31) (blue solid line) along with the interpolating polynomial
of degree 2 (red dash line) and 8 (magenta dash dot line) with equally spaced nodes.
Figure (b) shows the graph of f (blue solid line) along with the interpolating polynomial
of degree 18 (red dash line) with equally spaced nodes.
This gives us a feeling that if the calculation is done with infinite precision (that is,
without any finite digit floating point arithmetic) and the function f is smooth, then we
always have a better approximation for a larger value of n. In other words, we expect
lim
n→∞
∥f − pn∥∞,I = 0.
But this is not true in the case of equally spaced nodes. This was first shown by Carl
Runge, where he discovered that there are certain functions for which, as we go on
increasing the degree of interpolating polynomial, the total error increases drastically
and the corresponding interpolating polynomial oscillates near the boundary of the
interval in which the interpolation is done. Such a phenomenon is called the Runge
Phenomenon. This phenomenon is well understood by the following example given by
Carl Runge.
Example 5.3.7 [Runge’s Function].
Consider the Runge’s function defined on the interval [−1, 1] given by
f(x) =
1
1 + 25x2
. (5.31)
The interpolating polynomials with n = 2, n = 8 and n = 18 are depicted in Figure
5.3. This figure clearly shows that as we increase the degree of the polynomial, the
interpolating polynomial differs significantly from the actual function especially, near
the end points of the interval.
S. Baskar and S. Sivaji Ganesh
183
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-195-320.jpg)
![Chapter 5. Interpolation
In the light of the discussion made in Subsection 5.3.2, we may think that the Runge
phenomenon is due to the amplification of the arithmetic error. But, even if the cal-
culation is done with infinite precision (that is, without any finite digit floating point
arithmetic), we may still have the Runge phenomenon due to the amplification in
mathematical error. This can be observed by taking infinity norm on both sides of the
formula (5.23). This gives the upper bound of the infinity norm of MEn(x) as
∥MEn∥∞,I ≤
(b − a)n+1
(n + 1)!
∥f(n+1)
∥∞,I.
Although the first part, (b−a)n+1
/(n+1)! in the upper bound tends to zero as n → ∞,
if the second part, ∥f(n+1)
∥∞,I increases significantly as n increases, then the upper
bound can still increase and makes it possible for the mathematical error to be quite
high.
A more deeper analysis is required to understand the Runge phenomenon more rigor-
ously. But this is outside the scope of this course and therefore is omitted.
5.3.5 Convergence of Sequence of Interpolating Polynomials
We end this section by stating without proof a negative result and a positive result
concerning the convergence of sequence of interpolating polynomials.
Theorem 5.3.8 [Faber].
For n ∈ N, let the sequence of nodes
a ≤ x
(n)
0 < x
(n)
1 < · · · < x(n)
n ≤ b
be given. Then there exists a continuous function f defined on the interval [a, b] such
that the polynomials pn(x) that interpolate the function f at these nodes have the
property that ∥pn − f∥∞,[a,b] does not tend to zero as n → ∞.
Example 5.3.9.
In fact, the interpolating polynomial pn(x) for the Runge’s function goes worser and
worser as shown in Figure 5.3 for increasing values of n with equally spaced nodes.
That is, ∥f − pn∥∞,[−1,1] → ∞ as n → ∞ for any sequence of equally spaced nodes.
Let us now state a positive result concerning the convergence of sequence of interpolat-
ing polynomials to a given continuous function.
S. Baskar and S. Sivaji Ganesh
184
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-196-320.jpg)
![Section 5.4 Piecewise Polynomial Interpolation
Theorem 5.3.10.
Let f be a continuous function on the interval [a, b]. Then there exists a sequence of
nodes
a ≤ x
(n)
0 < x
(n)
1 < · · · < x(n)
n ≤ b, for n ∈ N,
such that the polynomials pn(x) that interpolate the function f at these nodes satisfy
∥pn − f∥∞,[a,b] → 0 as n → ∞.
The Theorem 5.3.10 is very interesting because it implies that for the Runge’s function,
we can find a sequence of nodes for which the corresponding interpolating polynomial
yields a good approximation even for a large value of n.
Example 5.3.11.
For instance, define a sequence of nodes
x
(n)
i = cos
(
(2i + 1)π
2(n + 1)
)
, i = 0, 1, · · · , n (5.32)
for each n = 0, 1, 2, · · · . The nodes x
(n)
i defined by (5.32) are called Chebyshev
nodes.
In particular, if n = 4, the nodes are
x
(4)
0 = cos(π/10), x
(4)
1 = cos(3π/10),
x
(4)
2 = cos(5π/10), x
(4)
3 = cos(7π/10), x4
4 = cos(9π/10).
Figure 5.4 depicts pn(x) for n = 4, 18, 32, and 64 along with the Runge’s function.
From these figures, we observe that the interpolating polynomial pn(x) agrees well
with the Runge’s function.
5.4 Piecewise Polynomial Interpolation
Quite often polynomial interpolation will be unsatisfactory as an approximation tool.
This is true if we insist on letting the order of the polynomial get larger and larger.
However, if we keep the order of the polynomial fixed, and use different polynomials
over different intervals, with the length of the intervals getting smaller and smaller, then
the resulting interpolating function approximates the given function more accurately.
S. Baskar and S. Sivaji Ganesh
185
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-197-320.jpg)
![Section 5.4 Piecewise Polynomial Interpolation
Let us start with linear interpolation over an interval I = [a, b] which leads to
p1(x) = f(a) + f[a, b](x − a) = f(a) +
f(b) − f(a)
b − a
(x − a) =
x − b
a − b
f(a) +
x − a
b − a
f(b),
where the nodes are x0 = a, x2 = b. In addition to these two nodes, we now choose one
more point x1 such that x0 < x1 < x2. With these three nodes, can obtain a quadratic
interpolation polynomial. Instead, we can interpolate the function f(x) in [x0, x1] by
a linear polynomial with nodes x0 and x1, and in [x1, x2] by another linear polynomial
with nodes x1 and x2. Such polynomials are given by
p1,1(x) :=
x − x1
x0 − x1
f(x0) +
x − x0
x1 − x0
f(x1), p1,2(x) :=
x − x2
x1 − x2
f(x1) +
x − x1
x2 − x1
f(x2)
and the interpolating function is given by
s(x) =
{
p1,1(x) , x ∈ [x0, x1]
p1,2(x) , x ∈ [x1, x2]
.
Note that s(x) is a continuous function on [x0, x2], which interpolates f(x) and is
linear in [x0, x1] and [x1, x2]. Such an interpolating function is called piecewise linear
interpolating function.
In a similar way as done above, we can also obtain piecewise quadratic, cubic interpo-
lating functions and so on.
Example 5.4.1.
Consider the Example 5.1, where we have obtained the quadratic interpolating poly-
nomial for the function
f(x) = sin
(π
2
ex
)
.
The piecewise linear polynomial interpolating function for the data
x 0 0.5 1
f(x) 1.0000 0.5242 −0.9037
is given by
s(x) =
{
1 − 0.9516 x , x ∈ [0, 0.5]
1.9521 − 2.8558 x , x ∈ [0.5, 1].
The following table shows the value of the function f at x = 0.25 and x = 0.75 along
with the values of p2(x) and s(x) with relative errors.
S. Baskar and S. Sivaji Ganesh
187
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-199-320.jpg)
![Chapter 5. Interpolation
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
x
y
Figure 5.5: The function f(x) = sin
(π
2
ex
)
(blue line), p2(x) (green bash line) and
the piecewise linear interpolation s(x) (red dash and dot line) are shown. Blue dots
represent the given data, blue ‘x’ symbol indicates the value of f(0.25) and f(0.75),
green ‘+’ symbol indicates the value of p2(0.25) and p2(0.75), and the red ‘O’ symbol
represents the value of s(0.25) and s(0.75).
x f(x) p2(x) s(x) Er(p2(x)) Er(s(x))
0.25 0.902117 0.881117 0.762105 0.023278 0.155203
0.75 -0.182750 -0.070720 -0.189732 0.613022 0.038204
Figure 5.5 depicts the graph of f, p2(x) and s(x). In this figure, we observe that the
quadratic polynomial p2(x) agrees well with f(x) than s(x) for x ∈ [0, 0.5], whereas
s(x) agrees well with f(x) than p2(x) for x ∈ [0.5, 1].
5.5 Spline Interpolation
Although a piecewise interpolating function introduced in Section 5.4 is continuous, it
may not be differentiable at the nodes. We wish to find an interpolating function that
is sufficiently smooth and does a better approximation to f(x). This can be achieved
by spline interpolation.
S. Baskar and S. Sivaji Ganesh
188
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-200-320.jpg)
![Section 5.5 Spline Interpolation
Definition 5.5.1 [Spline Interpolating Function].
A spline interpolating function of degree d with nodes xi, i = 0, 1, · · · , n
(x0 < x1 < · · · < xn) is a function s(x) with the following properties
1. On each subinterval [xi−1, xi], i = 1, 2, · · · , n, s(x) is a polynomial of degree
less than or equal to d.
2. s(x) and its first (d − 1) derivatives are continous on [x0, xn].
3. The interpolation conditions s(xi) = f(xi), i = 0, 1, · · · , n are satisfied.
We shall now study how we can obtain the interpolation of a function f as spline
interpolating functions instead of polynomials. For the sake of simplicity, we restrict
only to d = 3, called the cubic spline interpolating function.
Remark 5.5.2.
Note that in each subinterval [xi−1, xi], i = 1, 2, · · · , n, we only know f(xi−1) and
f(xi). But we look for a cubic polynomial in this subinterval. Therefore, we cannot
follow the Lagrange’s or Newton’s interpolating formula, as these formulas demand
the function values at four distinct nodes in the subinterval. We need to adopt a
different method for the construction of the polynomial in each subinterval in order
to obtain the spline interpolation.
Construction of a Cubic Spline
The construction of a cubic spline interpolating function s(x) of a function f(x) is as
follows:
Step 1: Let us denote by M1, · · · , Mn,
Mi = s′′
(xi), i = 0, 1, · · · , n
and first obtain s(x) in terms of Mi’s which are unknowns.
Step 2: Since s(x) is cubic on each [xi−1, xi], the function s′′
(x) is linear on the interval
such that
s′′
(xi−1) = Mi−1, s′′
(xi) = Mi.
Therefore, it is given by
s′′
(x) =
(xi − x)Mi−1 + (x − xi−1)Mi
xi − xi−1
, xi−1 ≤ x ≤ xi (5.33)
Integrating (5.33) two times with respect to x, we get
s(x) =
(xi − x)3
Mi−1
6(xi − xi−1)
+
(x − xi−1)3
Mi
6(xi − xi−1)
+ K1x + K2,
S. Baskar and S. Sivaji Ganesh
189
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-201-320.jpg)
![Chapter 5. Interpolation
where K1 and K2 are integrating constants to be determined by using the interpolation
conditions s(xi−1) = f(xi−1) and s(xi) = f(xi). We have
K1 =
f(xi) − f(xi−1)
xi − xi−1
−
(Mi − Mi−1)(xi − xi−1)
6
K2 =
xif(xi−1) − xi−1f(xi)
xi − xi−1
−
(Mi−1xi − Mixi−1)(xi − xi−1)
6
Substituting these values in the above equation, we get
s(x) =
(xi − x)3
Mi−1 + (x − xi−1)3
Mi
6(xi − xi−1)
+
(xi − x)f(xi−1) + (x − xi−1)f(xi)
xi − xi−1
−
1
6
(xi − xi−1)[(xi − x)Mi−1 + (x − xi−1)Mi], xi−1 ≤ x ≤ xi (5.34)
Formula (5.34) applies to each of the intervals [x1, x2], · · · , [xn−1, xn]. The formulas
for adjacent intervals [xi−1, xi] and [xi, xi+1] will agree at their common point x =
xi because of the interpolating condition s(xi) = f(xi). This implies that s(x) is
continuous over the entire interval [a, b]. Similarly, formula (5.33) for s′′
(x) implies that
it is continuous on [a, b].
Step 3: All that remains is to find the values of Mi for all i = 0, 1, · · · , n. This
is obtained by ensuring the continuity of s′
(x) over [a, b], ie., the formula for s′
(x) on
[xi−1, xi] and [xi, xi+1] are required to give the same value at their common point x = xi,
for i = 1, 2, · · · , n − 1. After simplification, we get the system of linear equations for
i = 1, 2, · · · n − 1
xi − xi−1
6
Mi−1 +
xi+1 − xi−1
3
Mi +
xi+1 − xi
6
Mi+1
=
f(xi+1) − f(xi)
xi+1 − xi
−
f(xi) − f(xi−1)
xi − xi−1
. (5.35)
These n − 1 equations together with the assumption that
M0 = Mn = 0 (5.36)
leads to the values of M0, M1, · · · , Mn and hence to the interpolation function s(x).
A spline constructed above is called a natural spline.
Example 5.5.3.
Calculate the natural cubic spline interpolating the data
{
(1, 1),
(
2,
1
2
)
,
(
3,
1
3
)
,
(
4,
1
4
)}
.
S. Baskar and S. Sivaji Ganesh
190
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-202-320.jpg)
![Section 5.5 Spline Interpolation
1 1.5 2 2.5 3 3.5 4
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
s(x)
Figure 5.6: The function f(x) = 1/x (blue line) and the corresponding cubic natural
spline interpolating function s(x) (red dash line) are shown. The data are represented
by blue dots.
The number of points is n = 3 and xi − xi−1 = 1, for x = 1, 2, 3.
Step 1: Here, we have
M0 = s′′
(1), M1 = s′′
(2), M2 = s′′
(3), M3 = s′′
(4).
Step 2: The function s′′
(x) is given by
s′′
(x) =
(2 − x)M0 + (x − 1)M1 , x ∈ [1, 2]
(3 − x)M1 + (x − 2)M2 , x ∈ [2, 3]
(4 − x)M2 + (x − 3)M3 , x ∈ [3, 4]
.
Integrating s′′
(x) two times with respect to x, we get
s(x) =
(2 − x)3
M0
6
+
(x − 1)3
M1
6
+ K1x + K2 , x ∈ [1, 2]
(3 − x)3
M1
6
+
(x − 2)3
M2
6
+ K1x + K2 , x ∈ [2, 3]
(4 − x)3
M2
6
+
(x − 3)3
M3
6
+ K1x + K2 , x ∈ [3, 4]
.
S. Baskar and S. Sivaji Ganesh
191
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-203-320.jpg)
![Chapter 5. Interpolation
where K1 and K2 are integrating constants to be determined by using the conditions
s(xi−1) = f(xi−1) and s(xi) = f(xi). We have
K1 =
−
1
2
−
(M1 − M0)
6
, x ∈ [1, 2]
−
1
6
−
(M2 − M1)
6
, x ∈ [2, 3]
−
1
12
−
(M3 − M2)
6
, x ∈ [3, 4]
.
K2 =
3
2
−
2M0 − M1
6
, x ∈ [1, 2]
5
6
−
3M1 − 2M2
6
, x ∈ [2, 3]
7
12
−
4M2 − 3M3
6
, x ∈ [3, 4]
.
Substituting these expressions in the expression of s(x), we get the required cubic
spline as given in (5.34).
Step 3: Since we are constructing the natural spline, we take M0 = M3 = 0. The
system (5.35) gives
2
3
M1 +
1
6
M2 =
1
3
,
1
6
M1 +
2
3
M2 =
1
12
.
Solving this system ,we get M1 = 1
2
, M2 = 0. Substituting these values into (5.34),
we obtain
s(x) =
1
12
x3
−
1
4
x2
−
1
3
x +
3
2
, x ∈ [1, 2]
−
1
12
x3
+
3
4
x2
−
7
3
x +
17
6
, x ∈ [2, 3]
−
1
12
x +
7
12
, x ∈ [3, 4]
which is the required natural cubic spline approximation to the given data. A com-
parison result is depicted in Figure 5.6.
S. Baskar and S. Sivaji Ganesh
192
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-204-320.jpg)
![Chapter 5. Interpolation
7. Find the Lagrange form of interpolating polynomial p2(x) that interpolates the
function f(x) = e−x2
at the nodes x0 = −1, x1 = 0 and x2 = 1. Further, find the
value of p2(−0.9) (use 6-digit rounding). Compare the value with the true value
f(−0.9) (use 6-digit rounding). Find the percentage error in this calculation.
Newton’s Divided Difference Formula
8. For the particular function f(x) = xm
(m ∈ N), show that
f[x0, x1, · · · , xn] =
{
1 if n = m
0 if n > m
9. Let x0, x1, · · · , xn be nodes, and f be a given function. Define w(x) =
∏n
i=0(x −
xi). Prove that
f[x0, x1, · · · , xn] =
n∑
i=0
f(xi)
w′(xi)
.
10. The following data correspond to a polynomial P(x) of unknown degree
x 0 1 2
P(x) 2 −1 4
.
Determine the coefficient of x in the polynomial P(x) if all the third order divided
differences are 1.
Error in Polynomial Interpolation
11. Let pn(x) be a polynomial of degree less than or equal to n that interpolates a
function f at a set of distinct nodes x0, x1, · · · , xn. If x /∈ { x0, x1, · · · , xn }, then
show that the error is given by
f(x) − pn(x) = f[x0, x1, · · · , xn, x]
n∏
i=0
(x − xi).
12. If f ∈ Cn+1
[a, b] and if x0, x1, · · · , xn are distinct nodes in [a, b], then show that
there exists a point ξx ∈ (a, b) such that
f[x0, x1, · · · , xn, x] =
f(n+1)
(ξx)
(n + 1) !
13. Let N be a natural number. Let p1(x) denote the linear interpolating polynomial
on the interval [N, N +1] interpolating the function f(x) = x2
at the nodes N and
N + 1. Find an upper bound for the mathematical error ME1 using the infinity
norm on the interval [N, N + 1] (i.e., ∥ME1∥∞, [N,N+1]).
S. Baskar and S. Sivaji Ganesh
194
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-206-320.jpg)
![Section 5.6 Exercises
14. Let p3(x) denote a polynomial of degree less than or equal to 3 that interpolates
the function f(x) = ln x at the nodes x0 = 1, x1 = 4
3
, x2 = 5
3
, x3 = 2. Find a
lower bound on the absolute value of mathematical error |ME3(x)| at the point
x = 3
2
, using the formula for mathematical error in interpolation.
15. Let f : [0, π
6
] → R be a given function. The following is the meaning for the
Cubic interpolation in a table of function values
x x0 x1 · · · xN
f(x) f(x0) f(x1) · · · f(xN )
The values of f(x) are tabulated for a set of equally spaced points in [a, b], say
xi for i = 0, 1, · · · , N with x0 = 0, xN = π
6
, and h = xi+1 − xi > 0 for every
i = 0, 1, · · · , N − 1. For an ¯x ∈ [0, π
6
] at which the function value f(¯x) is not
tabulated, the value of f(¯x) is taken to be the value of p3(¯x), where p3(x) is the
polynomial of degree less than or equal to 3 that interpolates f at the nodes
xi, xi+1, xi+2, xi+3 where i is the least index such that x ∈ [xi, xi+3].
Take f(x) = sin x for x ∈ [0, π
6
]; and answer the following questions.
i) When ¯x and p3 are as described above, then show that |f(¯x) − p3(¯x)| ≤ h4
48
.
ii) If h = 0.005, then show that cubic interpolation in the table of function
values yields the value of f(¯x) with at least 10 decimal-place accuracy.
16. Let x0, x1, · · · , xn be n + 1 distinct nodes, and f be a function. For each
i = 0, 1, · · · , n, let fl (f(xi)) denote the floating point approximation of f(xi) ob-
tained by rounding to 5 decimal places (note this is different from using 5-digit
rounding). Assume that 0.1 ≤ f(xi) < 1 for all i = 0, 1, · · · , n. Let pn(x) de-
note the Lagrange form of interpolating polynomial corresponding to the data
{(xi, f(xi)) : i = 0, 1, · · · , n}. Let ˜pn(x) denote the Lagrange form of interpolat-
ing polynomial corresponding to the data {(xi, fl (f(xi))) : i = 0, 1, · · · , n}. Show
that the arithmetic error at a point ˜x satisfies the inequality
|pn(˜x) − ˜pn(˜x)| ≤
1
2
10−5
n∑
k=0
|lk(˜x)|.
Spline Interpolation
17. Find a natural cubic spline interpolating function for the data
x -1 0 1
y 5 7 9
.
18. Determine whether the natural cubic spline function that interpolates the table
x -1 0 1
y −3 −1 −1
S. Baskar and S. Sivaji Ganesh
195
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-207-320.jpg)
![Chapter 5. Interpolation
is or is not the function
S(x) =
{
x3
+ x − 1 x ∈ [−1, 0],
x3
− x − 1 x ∈ [0, 1].
19. Determine whether the coefficients a, b, c, d exist so that the function
S(x) =
1 − 2x x ∈ [−4, −3]
a + bx + cx2
+ dx3
x ∈ [−3, 4]
157 − 32x x ∈ [4, 5]
is a natural cubic spline interpolating function on the interval [−4, 5] for the data
x -4 -3 4 5
y 9 7 29 -3
.
20. Does there exist real numbers a and b so that the function
S(x) =
(x − 2)3
+ a(x − 1)2
x ∈ [−1, 2]
(x − 2)3
− (x − 3)2
x ∈ [2, 3]
(x − 3)3
+ b(x − 2)2
x ∈ [3, 5]
is a natural cubic spline interpolating function on the interval [−1, 5] for the data
x -1 2 3 5
y -31 -1 1 17
?
21. Show that the natural cubic spline interpolation function for a given data is
unique.
S. Baskar and S. Sivaji Ganesh
196
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-208-320.jpg)
![CHAPTER 6
Numerical Integration and Differentiation
There are two reasons for approximating derivatives and integrals of a function f(x).
One is when the function is very difficult to differentiate or integrate, or only the
tabular values are available for the function. Another reason is to obtain solution of a
differential or integral equation. In this chapter we introduce some basic methods to
approximate integral and derivative of a function given either explicitly or by tabulated
values.
In Section 6.1, we obtain numerical methods for evaluating the integral of a given
integrable function f defined on the interval [a, b]. Section 6.2 introduces various ways
to obtain numerical formulas for approximating derivatives of a given differentiable
function.
6.1 Numerical Integration
In this section we derive and analyze numerical methods for evaluating definite integrals.
The problem is to evaluate the number
I(f) =
b∫
a
f(x)dx. (6.1)
Most such integrals cannot be evaluated explicitly, and with many others, it is faster to
integrate numerically than explicitly. The process of approximating the value of I(f)
is usually referred to as numerical integration or quadrature rule.
S. Baskar and S. Sivaji Ganesh
197
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-209-320.jpg)
![Section 6.1 Numerical Integration
0000000000
0000000000
0000000000
00000000000000000000
0000000000
0000000000
0000000000
1111111111
1111111111
1111111111
11111111111111111111
1111111111
1111111111
1111111111
x
y
a b
y=f(x)
Figure 6.1: Geometric interpretation of the rectangle Rule.
We now obtain the mathematical error in rectangle rule, given by,
MER(f) := I(f) − I(p0).
Theorem 6.1.1 [Error in Rectangle Rule].
Let f ∈ C1
[a, b]. The mathematical error MER(f) of the rectangle rule takes the
form
MER(f) =
f′
(η)(b − a)2
2
, (6.5)
for some η ∈ (a, b).
Proof.
For each x ∈ (a, b], the linear interpolating polynomial for f at the nodes a and x is
given by
f(x) = p0(x) + f[a, x](x − a).
Therefore, the mathematical error in the rectangle rule is given by
MER(f) = I(f) − IR(f) =
b∫
a
f[a, x](x − a) dx.
S. Baskar and S. Sivaji Ganesh
199
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-211-320.jpg)
![Chapter 6. Numerical Integration and Differentiation
000000000000000000000000000000000
000000000000000000000000000000000
0000000000000000000000
0000000000000000000000
0000000000000000000000
111111111111111111111111111111111
111111111111111111111111111111111
1111111111111111111111
1111111111111111111111
1111111111111111111111
x
y
a b
y=f(x)
Figure 6.2: Geometric interpretation of the trapezoidal Rule.
Using mean-value theorem for integrals, we get
MER(f) = f[a, ξ]
b∫
a
(x − a) dx,
for some ξ ∈ (a, b). By mean value theorem for derivatives, f[a, ξ] = f′
(η) for some
η ∈ (a, ξ). Thus, we get
MER(f) =
f′
(η)(b − a)2
2
,
for some η ∈ (a, b).
6.1.2 Trapezoidal Rule
We now consider the case when n = 1. Then
p1(x) = f(x0) + f[x0, x1](x − x0),
and therefore
I(f) ≈ IT (f) :=
b∫
a
(f(x0) + f[x0, x1](x − x0)) dx.
Taking x0 = a and x1 = b, we get
IT (f) = (b − a)
(
f(a) + f(b)
2
)
(6.6)
and is called the trapezoidal Rule. The Trapezoidal rule is illustrated in Figure 6.2.
S. Baskar and S. Sivaji Ganesh
200
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-212-320.jpg)
![Section 6.1 Numerical Integration
We now obtain the mathematical error in trapezoidal rule, given by
MET(f) := I(f) − I(p1).
Theorem 6.1.2 [Error in Trapezoidal Rule].
Let f ∈ C2
[a, b]. The mathematical error MET(f) of the trapezoidal rule takes the
form
MET(f) = −
f′′
(η)(b − a)3
12
, (6.7)
for some η ∈ (a, b).
Proof.
We have
f(x) = f(a) + f[a, b](x − a) + f[a, b, x](x − a)(x − b).
Integrating over the interval [a, b], we get
I(f) = IT (f) +
b∫
a
f[a, b, x](x − a)(x − b)dx.
Therefore the mathematical error is given by
MET(f) = I(f) − IT (f) =
b∫
a
f[a, b, x](x − a)(x − b)dx. (6.8)
From Corollary 5.2.9 (Conclusion 1), we see that the function x −→ f[a, b, x] is con-
tinuous. Therefore, from the mean value theorem for integrals (after noting that
(x − a)(x − b) is negative for all x ∈ [a, b]), the expression (6.8) for the mathematical
error takes the form
MET(f) = f[a, b, η]
b∫
a
(x − a)(x − b)dx,
for some η ∈ (a, b). The formula (6.7) now follows from (5.25) and a direct evaluation
of the above integral.
S. Baskar and S. Sivaji Ganesh
201
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-213-320.jpg)
![Chapter 6. Numerical Integration and Differentiation
Example 6.1.3.
For the function f(x) = 1/(x + 1), we approximate the integral
I =
1∫
0
f(x)dx,
using trapezoidal rule to get
IT (f) =
1
2
(
1 +
1
2
)
=
3
4
= 0.75.
The true value is I(f) = log(2) ≈ 0.693147. Therefore, the error is MET(f) ≈
−0.0569. Using the formula (6.7), we get the bounds for MET(f) as
−
1
6
< MET(f) < −
1
48
which clearly holds in the present case.
Composite Trapezoidal Rule
We can improve the approximation of trapezoidal rule by breaking the interval [a, b]
into smaller subintervals and apply the trapezoidal rule (6.6) on each subinterval. We
will derive a general formula for this.
Let us subdivide the interval [a, b] into n equal subintervals of length
h =
b − a
n
with endpoints of the subintervals as
xj = a + jh, j = 0, 1, · · · , n.
Then, we get
I(f) =
b∫
a
f(x)dx =
xn∫
x0
f(x)dx =
n−1∑
j=0
xj+1∫
xj
f(x)dx.
Using trapezoidal rule (6.6) on the subinterval [xj, xj+1], we get
xj+1∫
xj
f(x)dx ≈ h
(
f(xj) + f(xj+1)
2
)
, j = 0, 1, · · · , n − 1.
S. Baskar and S. Sivaji Ganesh
202
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-214-320.jpg)
![Section 6.1 Numerical Integration
Substituting this in the above equation, we get
I(f) ≈ h
[
f(x0) + f(x1)
2
]
+ h
[
f(x1) + f(x2)
2
]
+ · · · + h
[
f(xn−1) + f(xn)
2
]
.
The terms on the right hand side can be combined to give the simpler formula
In
T (f) := h
[
1
2
f(x0) + f(x1) + f(x2) + · · · + f(xn−1) +
1
2
f(xn)
]
. (6.9)
This rule is called the composite trapezoidal rule.
Example 6.1.4.
Using composite trapezoidal rule with n = 2, let us approximate the integral
I =
1∫
0
f(x)dx,
where
f(x) =
1
1 + x
.
As we have seen in Example 6.1.3, the true value is I(f) = log(2) ≈ 0.693147. Now,
the composite trapezoidal rule with x0 = 0, x1 = 1/2 and x2 = 1 gives
I2
T (f) ≈ 0.70833.
Thus the error is -0.0152. Recall from Example 6.1.3 that with n = 1, the trapezoidal
rule gave an error of -0.0569.
6.1.3 Simpson’s Rule
We now calculate I(p2(x)) to obtain the formula for the case when n = 2. Let us choose
x0 = a, x1 = (a + b)/2 and x2 = b. The Lagrange form of interpolating polynomial is
p2(x) = f(x0)l0(x) + f(x1)l1(x) + f(x2)l2(x).
Then
b∫
a
p2(x)dx = f(x0)
b∫
a
l0(x) dx + f(x1)
b∫
a
l1(x) dx + f(x2)
b∫
a
l2(x) dx.
Using the change of variable
x =
b − a
2
t +
b + a
2
,
S. Baskar and S. Sivaji Ganesh
203
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-215-320.jpg)
![Section 6.1 Numerical Integration
Theorem 6.1.5 [Error in Simpson’s Rule].
Let f ∈ C4
[a, b]. The mathematical error MES(f) of the Simpson’s rule takes the
form
MES(f) = −
f(4)
(η)(b − a)5
2880
, (6.11)
for some η ∈ (a, b).
Proof.
Consider the collection of nodes a, (a + b)/2, b, x in the interval [a, b]. Let x0 ∈ [a, b]
be a given point different from a, (a + b)/2, b, and x. From the fourth order divided
difference formula, we have
f[a, (a + b)/2, b, x] = f[x0, a, (a + b)/2, b] + f[x0, a, (a + b)/2, b, x](x − x0).
The mathematical error in Simpson’s method can therefore be written as
MES(f) =
b∫
a
f[x0, a, (a + b)/2, b]ϕ(x)dx +
b∫
a
f[x0, a, (a + b)/2, b, x](x − x0)ϕ(x)dx,
where
ϕ(x) = (x − a)(x − (a + b)/2)(x − b).
A direct integration shows
b∫
a
ϕ(x)dx =
b∫
a
(x − a)
(
x −
a + b
2
)
(x − b)dx = 0.
Thus, we have
MES(f) =
b∫
a
f[x0, a, (a + b)/2, b, x](x − x0)ϕ(x)dx,
Recall in the proof of Theorem 6.1.2, we used mean value theorem at this stage
to arrive at the conclusion. But, in the present case, we cannot use this theorem
because the function (x − x0)ϕ(x) need not be of one sign on [a, b]. However, the
choice x0 = (a + b)/2 makes this function one signed (negative). Now using Corollary
5.2.9 (Conclusion 1) and following the idea of the proof of Theorem 6.1.2, we arrive
at the formula (6.11) (this is left as an exercise).
S. Baskar and S. Sivaji Ganesh
205
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-217-320.jpg)
![Chapter 6. Numerical Integration and Differentiation
Example 6.1.6.
We now use the Simpson’s rule to approximate the integral
I(f) =
1∫
0
f(x) dx,
where
f(x) =
1
1 + x
.
The true value is I(f) = log(2) ≈ 0.693147. Using the Simpson’s rule (6.10), we get
IS(f) =
1
6
(
1 +
8
3
+
1
2
)
=
25
36
≈ 0.694444.
Therefore, the error is MES(f) ≈ 0.001297.
Composite Simpson’s Rule
Let us now derive the composite Simpson’s rule. First let us subdivide the interval
[a, b] into n equal parts as we discussed in composite trapezoidal rule (Subsection 6.1.2).
Taking a = xi−1, b = xi+1, (note that xi = (xi−1 +xi+1)/2) and xi −xi−1 = h in Simpson
rule, we get
∫ xi+1
xi−1
f(x)dx ≈
h
3
{f(xi−1) + 4f(xi) + f(xi+1)} .
Summing for i = 1, · · · , N/2, (N even) we get
∫ b
a
f(x)dx =
N/2
∑
i=1
∫ x2i
x2i−2
f(x)dx
≈
h
3
N/2
∑
i=1
{f(x2i−2) + 4f(x2i−1) + f(x2i)} .
Therefore, the composite Simpson’s rule takes the form
I
N/2
S (f) =
h
3
f(x0) + f(xN ) + 2
N/2
∑
i=2
f(x2i−2) + 4
N/2
∑
i=1
f(x2i−1)
. (6.12)
S. Baskar and S. Sivaji Ganesh
206
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-218-320.jpg)
![Chapter 6. Numerical Integration and Differentiation
yields
b3
− a3
3
=
b∫
a
x2
dx = a2
w0 +
(
a + b
2
)2
w1 + b2
w2
Thus, we have a linear system of three equations satisfied by w0, w1, and w2. By
solving this system, we get
w0 =
1
6
(b − a), w1 =
2
3
(b − a), w2 =
1
6
(b − a),
which gives us the familiar Simpson’s rule.
This motivates the following definition.
Definition 6.1.8 [Degree of Precision].
The degree of precision (also called order of exactness) of a quadrature for-
mula is the largest positive integer n such that the formula is exact for all polynomials
of degree less than or equal to n.
Example 6.1.9.
Let us determine the degree of precision of Simpson’s rule. It will suffice to apply
the rule over the interval [0, 2] (in fact any interval is good enough and we chose this
interval for the sake of having easy computation).
2∫
0
dx = 2 =
2
6
(1 + 4 + 1),
2∫
0
xdx = 2 =
2
6
(0 + 4 + 2),
2∫
0
x2
dx =
8
3
=
2
6
(0 + 4 + 4),
2∫
0
x3
dx = 4 =
2
6
(0 + 4 + 8),
2∫
0
x4
dx =
32
5
̸=
2
6
(0 + 4 + 16) =
20
3
.
Therefore, the degree of precision of Simpson’s rule is 3.
S. Baskar and S. Sivaji Ganesh
208
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-220-320.jpg)
![Chapter 6. Numerical Integration and Differentiation
These conditions give w0 = 2 and x0 = 0. Thus, we have the formula
1∫
−1
f(x)dx ≈ 2f(0) =: IG0 (f), (6.16)
which is the required Gaussian rule for n = 0.
Case 2: (n = 1). In this case, the quadrature formula (6.14) takes the form
1∫
−1
f(x)dx ≈ w0f(x0) + w1f(x1).
The condition (6.15) gives
w0 + w1 = 2,
w0x0 + w1x1 = 0,
w0x2
0 + w1x2
1 =
2
3
,
w0x3
0 + w1x3
1 = 0.
A solution of this nonlinear system is w0 = w1 = 1, x0 = −1/
√
3 and x1 = 1/
√
3. This
lead to the formula
1∫
−1
f(x)dx ≈ f
(
−
1
√
3
)
+ f
(
1
√
3
)
=: IG1(f), (6.17)
which is the required Gaussian rule for n = 1.
Case 3: (General). In general, the quadrature formula is given by (6.14), where there
are 2(n + 1) free parameters xi and wi for i = 0, 1, · · · , n. The condition (6.15) leads
to the nonlinear system
n∑
j=0
wjxi
j =
0 , i = 1, 3, · · · , 2n + 1
2
i + 1
, i = 0, 2, · · · , 2n
.
These are nonlinear equations and their solvability is not at all obvious and therefore
the discussion is outside the scope of this course.
So far, we derived Gaussian rule for integrals over [−1, 1]. But this is not a limitation
as any integral on the interval [a, b] can easily be transformed to an integral on [−1, 1]
S. Baskar and S. Sivaji Ganesh
210
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-222-320.jpg)
![Chapter 6. Numerical Integration and Differentiation
Theorem 6.1.12.
Let f(x) be continous for a ≤ x ≤ b, and let n ≥ 1. Then the absolute error |En(f)|
in using Gaussian numerical integration rule to obtain I(f) satisfies
|En(f)| ≤ 2(b − a)ρ2n+1(f) (6.19)
where
ρ2n+1(f) = inf
deg q≤2n+1
∥f − q∥∞
is the minmax error of order 2n + 1 for f(x) on [a, b].
Proof.
En(p) = 0 for any polynomial p(x) of degree ≤ 2n + 1. Also, the error function En
satisfies En(F + G) = En(F) + En(G) for all F, G ∈ C[a, b]. Let p(x) = q∗
2n+1(x), the
minimax approximation of degree ≤ 2n + 1 to f(x) on [a, b]. Then
En(f) = En(f) − En(q∗
2n+1)
= En(f − q∗
2n+1)
=
∫ b
a
(f(x) − q∗
2n+1(x))dx −
n∑
j=0
wj(f(xj) − q∗
2n+1(xj)).
Therefore, we have
|En(f)| ≤ ∥f − q∗
2n+1∥∞[(b − a) +
n∑
j=0
|wj|].
But we have
n∑
j=0
|wj| = b − a and therefore, we get the desired result.
6.2 Numerical Differentiation
The aim of this section is to obtain formulas to approximate the values of derivatives of
a given function at a given point. Such a formula for the first derivative of a function
can be obtained directly using the definition of the derivative, namely, the difference
quotients of the function. This will be discussed in Subsection 6.2.1. But this idea
cannot be adopted for higher order derivatives. Approximating formulas for derivatives
can be obtained in at least two ways, namely,
S. Baskar and S. Sivaji Ganesh
212
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-224-320.jpg)
![Section 6.2 Numerical Differentiation
1. Methods based on Interpolation
2. Methods based on Undetermined Coefficients
These methods are discussed in Subsection 6.2.2 and 6.2.3, respectively.
6.2.1 Approximations of First Derivative
Forward Difference Formula
The most simple way to obtain a numerical method for approximating the derivative
of a C1
function f is to use the definition of derivative
f′
(x) = lim
h→0
f(x + h) − f(x)
h
.
The approximating formula can therefore be taken as
f′
(x) ≈
f(x + h) − f(x)
h
=: D+
h f(x) (6.20)
for a sufficiently small value of h > 0. The formula D+
h f(x) is called the forward
difference formula for the derivative of f at the point x.
Theorem 6.2.1.
Let f ∈ C2
[a, b]. The mathematical error in the forward difference formula is given
by
f′
(x) − D+
h f(x) = −
h
2
f′′
(η) (6.21)
for some η ∈ (x, x + h).
Proof.
By Taylor’s theorem, we have
f(x + h) = f(x) + hf′
(x) +
h2
2
f′′
(η) (6.22)
for some η ∈ (x, x + h). Using (6.20) and (6.22), we obtain
D+
h f(x) =
1
h
{[
f(x) + hf′
(x) +
h2
2
f′′
(η)
]
− f(x)
}
= f′
(x) +
h
2
f′′
(η).
This completes the proof.
S. Baskar and S. Sivaji Ganesh
213
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-225-320.jpg)
![Chapter 6. Numerical Integration and Differentiation
Remark 6.2.2.
If we consider the left hand side of (6.21) as a function of h, i.e., if
g(h) = f′
(x) − Dhf(x),
then we see that
g(h)
h
=
1
2
|f′′
(η)|.
Let M > 0 be such that |f′′(x)| ≤ M for all x ∈ [a, b]. Then we see that
g(h)
h
≤
M
2
.
That is, g = O(h) as h → 0. We say that the forward difference formula D+
h f(x)
is of order 1 (order of accuracy).
Backward Difference Formula
The derivative of a function f is also given by
f′
(x) = lim
h→0
f(x) − f(x − h)
h
.
Therefore, the approximating formula for the first derivative of f can also be taken as
f′
(x) ≈
f(x) − f(x − h)
h
=: D−
h f(x) (6.23)
The formula D−
h f(x) is called the backward difference formula for the derivative of f
at the point x.
Deriving the mathematical error for backward difference formula is similar to that of
the forward difference formula. It can be shown that the backward difference formula
is of order 1.
Central Difference Formula
The derivative of a function f is also given by
f′
(x) = lim
h→0
f(x + h) − f(x − h)
2h
.
Therefore, the approximating formula for the first derivative of f can also be taken as
f′
(x) ≈
f(x + h) − f(x − h)
2h
=: D0
hf(x), (6.24)
S. Baskar and S. Sivaji Ganesh
214
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-226-320.jpg)
![Section 6.2 Numerical Differentiation
. .. xx−h x+h
Forward
Backward
Central
x
y
y=f(x)
f’
Figure 6.4: Geometrical interpretation of difference formulae.
for a sufficiently small value of h > 0. The formula D0
hf(x) is called the central difference
formula for the derivative of f at the point x.
The central difference formula is of order 2 as shown in the following theorem.
Theorem 6.2.3.
Let f ∈ C3
[a, b]. The mathematical error in the central difference formula is given
by
f′
(x) − D0
hf(x) = −
h2
6
f′′′
(η), (6.25)
where η ∈ (x − h, x + h).
Proof.
Using Taylor’s theorem, we have
f(x + h) = f(x) + hf′
(x) +
h2
2!
f′′
(x) +
h3
3!
f′′′
(η1)
and
f(x − h) = f(x) − hf′
(x) +
h2
2!
f′′
(x) −
h3
3!
f′′′
(η2),
S. Baskar and S. Sivaji Ganesh
215
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-227-320.jpg)
.
Hence, we have the formula
f′
(x) ≈ p′
1(x) = f[x0, x1]. (6.26)
In particular,
• if we take x0 = x and x1 = x + h for a small value h > 0, we obtain the forward
difference formula D+
h f(x).
• if we take x0 = x−h and x1 = x for small value of h > 0, we obtain the backward
difference formula D−
h f(x).
• if we take x0 = x − h and x1 = x + h for small value of h > 0, we get the central
difference formula D0
hf(x).
We next prove the formula for mathematical error in approximating the first derivative
using interpolating polynomial.
Theorem 6.2.5 [Mathematical Error].
Hypothesis:
1. Let f be an (n + 2)-times continuously differentiable function on the interval
[a, b].
2. Let x0, x1, · · · , xn be n + 1 distinct nodes in [a, b].
3. Let pn(x) denote the polynomial that interpolates f at the nodes x0, x1, · · · , xn.
4. Let x be any point in [a, b] such that x /∈ {x0, x1 · · · , xn}.
Conclusion: Then
f′
(x) − p′
n(x) = wn(x)
f(n+2)
(ηx)
(n + 2)!
+ w′
n(x)
f(n+1)
(ξx)
(n + 1)!
(6.27)
with wn(x) =
n∏
i=0
(x − xi) and ξx and ηx are points in between the maximum and
minimum of x0, x1 · · · , xn and x, that depend on x.
Proof.
For any x ∈ [a, b] with x /∈ {x0, x1 · · · , xn}, by Newton’s form of interpolating
polynomial, we have
f(x) = pn(x) + f[x0, · · · , xn, x]wn(x),
S. Baskar and S. Sivaji Ganesh
217
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-229-320.jpg)
![Chapter 6. Numerical Integration and Differentiation
where pn(x) is the polynomial (of degree ≤ n) that interpolates f at x0, · · · , xn.
Taking derivative on both sides, we get
f′
(x) = p′
n(x) + wn(x)
d
dx
f[x0, · · · , xn, x] + w′
n(x)f[x0, · · · , xn, x].
But we know that (Theorem 5.2.10)
d
dx
f[x0, · · · , xn, x] = f[x0, · · · , xn, x, x].
Therefore, we have
f′
(x) = p′
n(x) + wn(x)f[x0, · · · , xn, x, x] + w′
n(x)f[x0, · · · , xn, x].
Further, from Theorem 5.3.2, we see that there exists an ξx ∈ (a, b) such that
f[x0, · · · , xn, x] =
f(n+1)
(ξx)
(n + 1)!
.
Using Theorem 5.3.2 along with the Hermite-Genocchi formula, we see that there
exists an ηx ∈ (a, b) such that
f[x0, · · · , xn, x, x] =
f(n+2)
(ηx)
(n + 2)!
.
Therefore, we get
f′
(x) − p′
n(x) = wn(x)
f(n+2)
(ηx)
(n + 2)!
+ w′
n(x)
f(n+1)
(ξx)
(n + 1)!
which is what we wish to show.
Difference formulas for higher order derivatives and their mathematical error can be
obtained similarly. The derivation of the mathematical error for the formulas of higher
order derivatives are omitted for this course.
Example 6.2.6.
Let x0, x1, and x2 be the given nodes. Then, the Newton’s form of interpolating
polynomial for f is given by
p2(x) = f(x0) + f[x0, x1](x − x0) + f[x0, x1, x2](x2
− (x0 + x1)x + x0x1).
Therefore, we take the first derivative of f as
f′
(x) ≈ p′
2(x) = f[x0, x1] + f[x0, x1, x2](2x − x0 − x1).
S. Baskar and S. Sivaji Ganesh
218
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-230-320.jpg)
![Section 6.2 Numerical Differentiation
• If we take x0 = x − h, x1 = x, and x2 = x + h for any given x ∈ [a, b], we obtain
the central difference formula D0
h(f) and the corresponding error obtained from
(6.27) is precisely the error given in (6.25).
• If we take x0 = x, x1 = x+h and x2 = x+2h for any given x ∈ [a, b], we obtain
the difference formula
f′
(x) ≈
−3f(x) + 4f(x + h) − f(x + 2h)
2h
with mathematical error obtained using (6.27) as
f′
(x) − p′
2(x) =
h2
3
f′′′
(ξ),
for some ξ ∈ (x, x + 2h).
6.2.3 Methods based on Undetermined Coefficients
Another method to derive formulas for numerical differentiation is called the method of
undetermined coefficients. The idea behind this method is similar to the one discussed
in deriving quadrature formulas.
Suppose we seek a formula for f(k)
(x) that involves the nodes x0, x1, · · · , xn. Then,
write the formula in the form
f(k)
(x) ≈ w0f(x0) + w1f(x1) + · · · + wnf(xn) (6.28)
where wi, i = 0, 1, · · · , n are free variables that are obtained by imposing the con-
dition that this formula is exact for polynomials of degree less than or equal to n.
Example 6.2.7.
We will illustrate the method by deriving the formula for f′′
(x) at nodes x0 = x − h,
x1 = x and x2 = x + h for a small value of h > 0.
For a small value of h > 0, let
f′′
(x) ≈ D
(2)
h f(x) := w0f(x − h) + w1f(x) + w2f(x + h) (6.29)
where w0, w1 and w2 are to be obtained so that this formula is exact when f(x) is
a polynomial of degree less than or equal to 2. This condition is equivalent to the
exactness for the three polynomials 1, x and x2
.
Step 1: When f(x) = 1 for all x. Then the formula of the form (6.29) is assumed to
S. Baskar and S. Sivaji Ganesh
219
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-231-320.jpg)
![Section 6.2 Numerical Differentiation
After simplification, we get
D
(2)
h f(x) = f′′
(x) +
h2
24
[(f(4)
(x) + · · · ) + (f(4)
(x) − · · · )].
Now treating the fourth order terms on the right hand side as remainders in Taylor’s
series, we get
D
(2)
h f(x) = f′′
(x) +
h2
24
[f(4)
(ξ1) + f(4)
(ξ2)],
for some ξ1, ξ2 ∈ (x − h, x + h). Using intermediate value theorem for the function
f(4)
, we get the mathematical error as
f′′
(x) − D
(2)
h f(x) = −
h2
12
f(4)
(ξ) (6.34)
for some ξ ∈ (x − h, x + h), which is the required mathematical error.
6.2.4 Arithmetic Error in Numerical Differentiation
Difference formulas are useful when deriving methods for solving differential equations.
But they can lead to serious errors when applied to function values that are subjected
to floating-point approximations. Let
f(xi) = fi + ϵi, i = 0, 1, 2.
To illustrate the effect of such errors, we choose the approximation D
(2)
h f(x) given by
(6.33) for the second derivative of f with x0 = x − h, x1 = x and x2 = x + h. Instead of
using the exact values f(xi), we use the appoximate values fi in the difference formula
(6.33). That is,
¯D
(2)
h f(x1) =
f2 − 2f1 + f0
h2
.
The total error committed is
f′′
(x1) − ¯D
(2)
h f(x1) = f′′
(x1) −
f(x2) − 2f(x1) + f(x0)
h2
+
ϵ2 − 2ϵ1 + ϵ0
h2
= −
h2
12
f(4)
(ξ) +
ϵ2 − 2ϵ1 + ϵ0
h2
.
Using the notation ϵ∞ := max{|ϵ0|, |ϵ1|, |ϵ3|}, we have
|f′′
(x1) − ¯D
(2)
h f(x1)| ≤
h2
12
|f(4)
(ξ)| +
4ϵ∞
h2
. (6.35)
The error bound in (6.35) clearly shows that, although the first term (bound of math-
ematical error) tends to zero as h → 0, the second term (bound of arithmetic error)
S. Baskar and S. Sivaji Ganesh
221
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-233-320.jpg)
![Chapter 6. Numerical Integration and Differentiation
2. Write down the errors in the approximation of
1∫
0
x4
dx and
1∫
0
x5
dx
by the Trapezoidal rule and Simpson’s rule. Find the value of the constant C for
which the Trapezoidal rule gives the exact result for the calculation of the integral
1∫
0
(x5
− Cx4
)dx.
3. A function f has the values shown below:
x 1 1.25 1.5 1.75 2
f(x) 10 8 7 6 5
.
i) Use trapezoidal rule to approximate
∫ 2
1
f(x) dx.
ii) Use Simpson’s rule to approximate
∫ 2
1
f(x) dx.
iii) Use composite Simpson’s rule to approximate
∫ 2
1
f(x) dx.
4. Obtain expressions for the arithmetic error in approximating the integral
∫ b
a
f(x) dx
using the trapezoidal and the Simpson’s rules. Also obtain upper bounds.
5. Let a = x0 < x1 < · · · < xn = b be equally spaced nodes (i.e., xk = x0 + kh for
k = 1, 2, · · · , n) in the interval [a, b]. Note that h = (b − a)/n. Let f be a twice
continuously differentiable function on [a, b].
i) Show that the expression for the mathematical error in approximating the
integral
∫ b
a
f(x) dx using the composite trapezoidal rule, denoted by En
T (f),
is given by
En
T (f) = −
(b − a)h2
12
f′′
(ξ),
for some ξ ∈ (a, b).
ii) Show that the mathematical error En
T (f) tends to zero as n → ∞ (one uses
the terminology composite trapezoidal rule is convergent in such a case).
6. Determine the minimum number of subintervals and the corresponding step size
h so that the error for the composite trapezoidal rule is less than 5 × 10−9
for
approximating the integral
∫ 7
2
dx/x.
7. Let a = x0 < x1 < · · · < xn = b be equally spaced nodes (i.e., xk = x0 + kh for
k = 1, 2, · · · , n) in the interval [a, b], and n is an even natural number. Note that
h = (b − a)/n. Let f be a four times continuously differentiable function on [a, b].
i) Show that the expression for the mathematical error in approximating the
integral
∫ b
a
f(x) dx using the composite Simpson rule, denoted by En
S(f), is
given by
En
S(f) = −
(b − a)h4
180
f(4)
(ξ),
S. Baskar and S. Sivaji Ganesh
224
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-236-320.jpg)
![CHAPTER 7
Numerical Ordinary Differential Equations
Consider a first order ordinary differential equation (ODE) of the form
y′
= f(x, y),
where y = y(x) is an unknown variable, x ∈ R is an independent variable and the
function f : R × R → R is given. Here, we used the notation y′
:= dy/dx. The
objective is to find the solution y for all x ∈ [a, b] subject to an initial condition
y(x0) = y0, x0 ∈ [a, b].
We call the problem of solving the above ODE along with this initial condition, the
initial value problem.
It is well-known that there are many ODEs of physical interest that cannot be solved
exactly although we know that such problems have unique solutions. If one wants
the solution of such problems, then the only way is to obtain it approximately. One
common way of obtaining an approximate solution to a given initial value problem is to
numerically compute the solution using a numerical method (or numerical scheme). In
this chapter, we introduce some basic numerical methods for approximating the solution
of a given initial value problem.
In Section 7.1, we review the exact solvability of a given initial value problem and
motivate the need of numerical methods. In Section 7.3, we introduce a basic numerical
method called Euler’s method for obtaining approximate solution of an initial value
problem and discussed the error involved in this method. We also show in this section
that the Euler method is of order 1. Modified forms of Euler method can be obtained
S. Baskar and S. Sivaji Ganesh
229
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-241-320.jpg)
![Chapter 7. Numerical Ordinary Differential Equations
Example 7.1.3 [Peano].
Let us consider the initial value problem
y′
= 3 y2/3
, y(0) = 0. (7.4)
Note that y(x) = 0 for all x ∈ R is clearly a solution for this initial value problem.
Also, y(x) = x3
for all x ∈ R. This initial value problem has infinite family of
solutions parametrized by c ≥ 0, given by
yc(x) =
{
0 if x ≤ c,
(x − c)3
if x > c,
defined for all x ∈ R. Thus, we see that a solution to an initial value problem need
not be unique.
However by placing additional conditions on f, we can achieve uniqueness as stated in
the following theorem. The proof of this theorem is omitted for this course.
Theorem 7.1.4 [Cauchy-Lipschitz-Picard’s Existence and Uniqueness Theorem].
Let D ⊆ R2
be a domain and I ⊂ R be an interval. Let f : D → R be a continuous
function. Let (x0, y0) ∈ D be a point such that the rectangle R defined by
R = {x : |x − x0| ≤ a} × {y : |y − y0| ≤ b} (7.5)
is contained in D. Let f be Lipschitz continuous with respect to the variable y on
R, i.e., there exists a K > 0 such that
|f(x, y1) − f(x, y1)| ≤ K|y1 − y2| ∀(x, y1), (x, y2) ∈ R.
Then the initial value problem (7.1) has at least one solution on the interval |x−x0| ≤
δ where δ = min{a, b
M
}. Moreover, the initial value problem (7.1) has exactly one
solution on this interval.
Remark 7.1.5.
We state without proof that the function f(x, y) = y2/3 in Example 7.1.3 is not a
Lipschitz function on any rectangle containing the point (0, 0) in it.
While verifying that a given function f is Lipschitz continuous is a little difficult, one
can easily give a set of alternative conditions that guarantee Lipschitz continuity, and
also these conditions are easy to verify.
S. Baskar and S. Sivaji Ganesh
232
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-244-320.jpg)
![Section 7.1 Review of Theory
Lemma 7.1.6.
Let D ⊆ R2
be an open set and f : D → R be a continuous function such that
its partial derivative with respect to the variable y is also continuous on D, i.e.,
∂f
∂y
: D → R is a continuous function. Let the rectangle R defined by
R = {x : |x − x0| ≤ a} × {y : |y − y0| ≤ b}
be contained in D. Then f is Lipschitz continuous with respect to y in R.
Proof.
Let (x, y1), (x, y2) ∈ R. Applying mean value theorem with respect to the y variable,
we get
f(x, y1) − f(x, y1) =
∂f
∂y
(x, ξ)(y1 − y2), (7.6)
for some ξ between y1 and y2. Since we are applying mean value theorem by fixing
x, this ξ will also depend on x. However since ∂f
∂y
: D → R is a continuous function,
it will be bounded on R. That is, there exists a number L > 0 such that
∂f
∂y
(x, y) ≤ L for all (x, y) ∈ R. (7.7)
Taking modulus in the equation (7.6), and using the last inequality we get
|f(x, y1) − f(x, y1)| =
∂f
∂y
(x, ξ) |(y1 − y2)| ≤ L |(y1 − y2)|.
This finishes the proof of lemma.
The following theorem can be used as a tool to check the existence and uniqueness of
solution of a given initial value problem (7.1).
Corollary 7.1.7 [Existence and Uniqueness Theorem].
Let D ⊆ R2
be a domain and I ⊆ R be an interval. Let f : D → R be a continuous
function. Let (x0, y0) ∈ D be a point such that the rectangle R defined by
R = {x : |x − x0| ≤ a} × {y : |y − y0| ≤ b} (7.8)
is contained in D.
If the partial derivative ∂f/∂y is also continuous in D, then there exists a unique
solution y = y(x) of the initial value problem (7.1) defined on the interval |x−x0| ≤ δ
where δ = min{a, b
M
}.
S. Baskar and S. Sivaji Ganesh
233
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-245-320.jpg)
![Chapter 7. Numerical Ordinary Differential Equations
We state an example (without details) of an initial value problem that has a unique
solution but the right hand side function in the differential equation is not a Lipschitz
function. Thus this example illustrates the fact that condition of Lipschitz continuity is
only a sufficient condition for uniqueness and by no means necessary.
Example 7.1.8.
The IVP
y′
=
{
y sin 1
y
if y ̸= 0
0 if y = 0,
y(0) = 0.
has unique solution, despite the RHS not being Lipschitz continuous with respect to
the variable y on any rectangle containing (0, 0).
Remark 7.1.9 [On Existence and Uniqueness Theorem].
1. Though we have stated existence theorems without proof, we can confirm that
their proofs do not give an explicit soluton of the initial value problem being
considered. In fact, the proofs give a sequence of functions that converge to
a solution of the initial value problem.
2. Even when the RHS of the ordinary differential equation is a function of x
only, we do not have explicit solutions. For example,
y′
= e−x2
, y(0) = 0.
The only alternative is to approximate its solution by a numerical procedure.
7.2 Discretization Notations
A numerical method gives approximate value of the solution of the initial value problem
(7.1) at only a discrete set of point. Thus, if we are interested in obtaining solution for
(7.1a) in an interval [a, b], then we first discretize the interval as
a = x0 < x1 < · · · < xn = b, (7.9)
where each point xi, i = 0, 1, · · · , n is called a node. Unless otherwise stated, we always
assume that the nodes are equally spaced. That is,
xj = x0 + jh, j = 0, 1, · · · n (7.10)
for a sufficiently small positive real number h. We use the notation for the approximate
solution as
yj = yh(xj) ≈ y(xj), j = 0, 1, · · · , n. (7.11)
S. Baskar and S. Sivaji Ganesh
234
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-246-320.jpg)
![Chapter 7. Numerical Ordinary Differential Equations
Figure 7.1: Geometrical interpretation of Euler’s method
Remark 7.3.1 [Geometric Interpretation].
A geometric insight into Euler’s method is shown in Figure 7.1 The tangent line to
the graph of y(x) at x = xj has slope f(xj, yj). The Euler’s method approximates
the value of y(xj+1) by the corresponding value of tangent line at the point x =
xj+1.
Example 7.3.2.
Consider the initial value problem
y′
= y, y(0) = 1.
The Euler method (7.12) for this equation takes the form
yj+1 = yj + hyj = (1 + h)yj.
Note that the exact solution for the given initial value problem is y(x) = ex
.
On applying Euler’s method with h = 0.01 and using 7-digit rounding, we get
y(0.01) ≈ y1 = 1 + 0.01 = 1.01
y(0.02) ≈ y2 = 1.01 + 0.01(1.01) = 1.0201
y(0.03) ≈ y3 = 1.0201 + 0.01(1.0201) = 1.030301
y(0.04) ≈ y4 = 1.030301 + 0.01(1.030301) = 1.040604
S. Baskar and S. Sivaji Ganesh
236
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-248-320.jpg)
![Chapter 7. Numerical Ordinary Differential Equations
similar way.
Using Taylor’s theorem, write
y(xj+1) = y(xj) + hy′
(xj) +
h2
2
y′′
(ξj)
for some xj < ξj < xj+1. Since y(x) satisfies the ODE y′
= f(x, y(x)), we get
y(xj+1) = y(xj) + hf(xj, y(xj)) +
h2
2
y′′
(ξj).
Thus, the local truncation error in forward Euler’s method is
Tj+1 =
h2
2
y′′
(ξj), (7.14)
which is the error involved in obtaining the value y(xj+1) using the exact value y(xj).
The forward Euler’s method uses the approximate value yj in the formula and therefore
the finally computed value yj+1 not only involves the truncation error but also the
propagated error involved in computing yj. Thus, the local mathematical error in the
forward Euler’s method is given by
ME(yj+1) := y(xj+1) − yj+1 = y(xj) − yj + h(f(xj, y(xj)) − f(xj, yj)) +
h2
2
y′′
(ξj).
Here, y(xj) − yj + h(f(xj, y(xj)) − f(xj, yj)) is the propagated error.
The propagated error can be simplified by applying the mean value theorem to f(x, z)
considering it as a functin of z:
f(xj, y(xj)) − f(xj, yj) =
∂f(xj, ηj)
∂z
[y(xj) − yj],
for some ηj lying between y(xj) and yj. Using this, we get the mathematical error
ME(yj+1) =
[
1 + h
∂f(xj, ηj)
∂z
]
ME(yj) +
h2
2
y′′
(ξj) (7.15)
for some xj < ξj < xj+1, and ηj lying between y(xj) and yj.
We now assume that over the interval of interest,
∂f(xj, y(xj))
∂z
< L, |y′′
(x)| < Y ,
where L and Y are fixed positive constants. On taking absolute values in (7.15), we
obtain
|ME(yj+1)| ≤ (1 + hL)|ME(yj)| +
h2
2
Y . (7.16)
S. Baskar and S. Sivaji Ganesh
238
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-250-320.jpg)
![Section 7.3 Euler’s Method
Applying the above estimate recursively, we get
|ME(yj+1)| ≤ (1 + hL)2
|ME(yj−1)| + (1 + (1 + hL))
h2
2
Y
≤ · · ·
≤ · · ·
≤ (1 + hL)j+1
|ME(y0)| +
(
1 + (1 + hL) + (1 + hL)2
+ · · · + (1 + hL)j
)h2
2
Y .
Using the formulas
1. For any α ̸= 1,
1 + α + α2
+ · · · + αj
=
αj+1
− 1
α − 1
2. For any x ≥ −1,
(1 + x)N
≤ eNx
,
in the above inequality, we have proved the following theorem.
Theorem 7.3.3.
Let y ∈ C2
[a, b] be a solution of the IVP (7.1) with
∂f(x, y)
∂y
< L, |y′′
(x)| < Y ,
for all x and y, and some constants L > 0 and Y > 0. The mathematical error in
the forward Euler’s method at a point xj = x0 + jh satisfies
|ME(yj)| ≤
hY
2L
(
e(xn−x0)L
− 1
)
+ e(xn−x0)L
|y(x0) − y0| (7.17)
Example 7.3.4.
Consider the initial value problem
y′
= y, y(0) = 1, x ∈ [0, 1].
Let us now find the upper bound for the mathematical error of forward Euler’s method
in solving this problem.
Here f(x, y) = y. Therefore, ∂f/∂y = 1 and hence we can take L = 1.
Since y = ex
, y′′
= ex
and |y′′
(x)| ≤ e for 0 ≤ x ≤ 1. Therefore, we take Y = e.
S. Baskar and S. Sivaji Ganesh
239
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-251-320.jpg)
![Chapter 7. Numerical Ordinary Differential Equations
We now use the estimate (7.17) with x0 = 0 and xn = 1 to obtain
|ME(yj)| ≤
he
2
(e − 1) ≈ 2.3354h.
Here, we assume that there is no approximation in the initial condition and therefore
the second term in (7.17) is zero.
To validate the upper bound obtained above, we shall compute the approximate
solution of the given IVP using forward Euler’s method. The method for the given
IVP reads
yj+1 = yj + hf(xj, yj) = (1 + h)yj.
The solution of this difference equation satisfing y(0) = 1 is
yj = (1 + h)j
.
Now, if h = 0.1, n = 10, we have yj = (1.1)10
. Therefore, the forward Euler’s method
gives y(1) ≈ y10 ≈ 2.5937. But the exact value is y(1) = e ≈ 2.71828. The error is
0.12466, whereas the bound obtained from (7.17) was 0.2354.
Remark 7.3.5.
The error bound (7.17) is valid for a large family of the initial value problems. But,
it usually produces a very poor estimate due to the presence of the exponential
terms. For instance, in the above example, if we take xn to be very large, then the
corresponding bound will also be very large.
The above error analysis assumes that the numbers used are of infinite precision and
no floating point approximation is assumed. When we include the floating point ap-
proximation that yn = ˜yn + ϵn, then the bound for total error is given in the following
theorem. The proof of this theorem is left as an exercise.
Theorem 7.3.6.
Let y ∈ C2
[a, b] be a solution of the IVP (7.1) with
∂f(x, y)
∂y
< L, |y′′
(x)| < Y ,
for all x and y, and some constants L > 0 and Y > 0. Let yj be the approximate
solution of (7.1) computed using the forward Euler’s method (7.12) with infinite
S. Baskar and S. Sivaji Ganesh
240
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-252-320.jpg)
![Section 7.4 Modified Euler’s Methods
precision and let ˜yj be the corresponding computed value using finite digit floating-
point arithmetic. If
yj = ˜yj + ϵj,
then the total error TE(yj) := y(xj) − ˜yj in forward Euler’s method at a point
xj = x0 + jh satisfies
|TE(yj)| ≤
1
L
(
hY
2
+
ϵ
h
)
(
e(xn−x0)L
− 1
)
+ e(xn−x0)L
|ϵ0|, (7.18)
where ϵ := max{|ϵi|/i = 0, 1, · · · , n}.
7.4 Modified Euler’s Methods
The Euler’s method derived in the previous section can also be derived using the equiv-
alent integral form of the IVP (7.1) as discussed in Lemma 7.1.1. Using this integral
form in the interval [xj, xj+1], we get
y(xj+1) = y(xj) +
∫ xj+1
xj
f(s, y)ds. (7.19)
The Euler’s method can be obtained by replacing the integral on the right hand side
by the rectangle rule.
Using the integral form in the interval [xj−1, xj+1] and using the mid-point quadrature
formula given by ∫ xj+1
xj−1
f(s, y)ds ≈ f(xj, yj)(xj+1 − xj−1),
we get the Euler’s mid-point method
yj+1 = yj−1 + 2hf(xj, yj). (7.20)
To compute the value of yj+1, we need to know the value of yj−1 and yj. Note that
the above formula cannot be used to compute the value of y1. Hence, we need another
method to obtain y1 and then yj, for j = 2, 3, · · · , n can be obtained using (7.20). This
method belongs to the class of 2-step methods.
Example 7.4.1.
Consider the initial-value problem
y′
= y, y(0) = 1.
To obtain the approximate value of y(0.04) with h = 0.01:
S. Baskar and S. Sivaji Ganesh
241
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-253-320.jpg)
![Chapter 7. Numerical Ordinary Differential Equations
quadratic term is given by
y(x + h) = y(x) + hy′
(x) +
h2
2
y′′
(x) + O(h3
). (7.22)
Since y satisfies the given ODE y′
= f(x, y), by differentiating this ODE with respect
to x both sides gives
y′′
(x) =
∂f
∂x
(x, y(x)) + y′
(x)
∂f
∂y
(x, y(x)) (7.23)
Substituting the values of y′
, y′′
in (7.22), we get
y(x + h) = y(x) + hf(x, y(x)) +
h2
2
[
∂f
∂x
(x, y(x)) + f(x, y(x))
∂f
∂y
(x, y(x))
]
+ O(h3
).
The last equation is re-written as
y(x + h) = y(x) +
h
2
f(x, y(x))
+
h
2
[
f(x, y(x)) + h
∂f
∂x
(x, y(x)) + hf(x, y(x))
∂f
∂y
(x, y(x))
]
+ O(h3
) (7.24)
Taking x = xj for j = 0, 1, · · · , n − 1 with xj+1 = xj + h in (7.24), we get
y(xj+1) = y(xj) +
h
2
f(xj, y(xj))
+
h
2
[
f(xj, y(xj)) + h
∂f
∂x
(xj, y(xj)) + hf(xj, y(xj))
∂f
∂y
(xj, y(xj))
]
+ O(h3
). (7.25)
Let us now expand the function f = f(s, t), which is a function of two variables, into
its Taylor series at the point (ξ, τ) and truncate the series after the linear term. It is
given by
f(s, t) = f(ξ, τ) + (s − ξ)
∂f
∂s
(ξ, τ) + (t − τ)
∂f
∂t
(ξ, τ) + O
(
(s − ξ)2
)
+ O
(
(t − τ)2
)
.
Taking (ξ, τ) = (xj, y(xj)) and comparing the above equation with the term in the
square brackets in the equation (7.25), we get
y(xj+1) = y(xj) +
h
2
f(xj, y(xj)) +
h
2
[f(xj+1, y(xj) + hf(xj, y(xj)))] + O(h3
). (7.26)
S. Baskar and S. Sivaji Ganesh
244
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-256-320.jpg)
![Section 7.5 Runge-Kutta Methods
Truncating the higher order terms and denoting the approximate value of y(xj+1) as
yj+1, we get
yj+1 = yj +
h
2
f(xj, yj) +
h
2
[f(xj+1, yj + hf(xj, yj))] . (7.27)
Although the terms dropped from (7.26) to get (7.27) are of order 3 (namely, O(h3
)),
the resultant approximation to y′
is of order 2 as is evident from the Taylor’s formula
(7.22). The equation (7.27) is therefore known as Runge-Kutta method of order
2. To facilitate easy memorizing, the formula (7.27) may be written as
yj+1 = yj +
1
2
(k1 + k2),
where
k1 = h f(xj, yj)
k2 = h f(xj+1, yj + k1).
The truncation error of Runge-Kutta method of order 2 is of order O(h3
) whereas
the Euler’s method is of order O(h2
). Therefore, for a fixed h > 0 we expect to get
more accurate result from Runge-Kutta method order 2 when compared to Euler’s
method.
Example 7.5.1.
Consider the initial-value problem
y′
= y, y(0) = 1.
Using Runge-Kutta method of order 2, we obtain
x y k1 k2
0.000000 1.000000 0.010000 0.010100
0.010000 1.010050 0.010000 0.010100
0.020000 1.020201 0.010100 0.010202
0.030000 1.030454 0.010202 0.010304
0.040000 1.040810 0.010305 0.010408
Recall the exact solution is y(x) = ex
and y(0.04) ≈ 1.040811. Therefore, the error
involved is 0.000001 which is much less than the error (0.000199) obtained in Euler’s
method for h = 0.01.
S. Baskar and S. Sivaji Ganesh
245
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-257-320.jpg)
![Chapter 7. Numerical Ordinary Differential Equations
7.5.2 Order Four
We state without derivation, the formula for the Runge-Kutta method of order 4.
yj+1 = yj +
1
6
(k1 + 2k2 + 2k3 + k4)
where
k1 = hf(xj, yj),
k2 = hf
(
xj +
h
2
, yj +
k1
2
)
,
k3 = hf
(
xj +
h
2
, yj +
k2
2
)
,
k4 = hf(xj + h, yj + k3)
The local truncation error of the 4th
order Runge-Kutta Method is of O(h5
).
Example 7.5.2.
Consider the initial-value problem
y′
= y, y(0) = 1.
Using Runge-Kutta method of order 4, we obtain
xj yj Exact Solution Error Relative Error
0.00 1.000000 1.000000 0.000000 0.000000
0.01 1.010050 1.010050 0.000000 0.000000
0.02 1.020201 1.020201 0.000000 0.000000
0.03 1.030455 1.030455 0.000000 0.000000
0.04 1.040811 1.040811 0.000000 0.000000
Note that the exact solution is y(x) = ex
.
7.6 Exercises
1. Let h > 0 and let xj = x0 + jh (j = 1, 2, · · · , n) be given nodes. Consider the
initial value problem y′
(x) = f(x, y), y(x0) = y0, with
∂f(x, y)
∂y
≤ 0,
for all x ∈ [x0, xn] and for all y.
S. Baskar and S. Sivaji Ganesh
246
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-258-320.jpg)
![Section 7.6 Exercises
i) Using error analysis of the Euler’s method, show that there exists an h > 0
such that
|en| ≤ |en−1| +
h2
2
f′′
(ξ) for some ξ ∈ (xn−1, xn),
where en = y(xn) − yn with yn obtained using Euler method.
ii) Applying the conclusion of (i) above recursively, prove that
|en| ≤ |e0| + n h2
Y where Y =
1
2
max
x0≤x≤xn
|y′′
(x)|. (∗∗)
2. The solution of the initial value problem
y′
(x) = λy(x) + cos x − λ sin x, y(0) = 0
is y(x) = sin x. For λ = −20, find the approximate value of y(3) using the Euler’s
method with h = 0.5. Compute the error bound given in (∗∗), and Show that the
actual absolute error exceeds the computed error bound given in (∗∗). Explain
why it does not contradict the validity of (∗∗).
3. Derive the backward Euler’s method for finding the approximate value of y(xn) for
some xn < 0, where y satisfies the initial value problem y′
(x) = f(x, y), y(0) = y0.
4. Consider the initial value problem y′
= −2y, 0 ≤ x ≤ 1, y(0) = 1.
i) Find an upper bound on the error in approximating the value of y(1) com-
puted using the Euler’s method (at x = 1) in terms of the step size h.
ii) For each h, solve the difference equation which results from the Euler’s
method, and obtain an approximate value of y(1).
iii) Find the error involved in the approximate value of y(1) obtained in (ii)
above by comparing with the exact solution.
iv) Compare the error bound obtained in (i) with the actual error obtained in
(iii) for h = 0.1, and for h = 0.01.
v) If we want the absolute value of the error obtained in (iii) to be at most
0.5 × 10−6
, then how small the step size h should be?
5. Consider the initial value problem y′
= xy, y(0) = 1. Estimate the error involved
in the approximate value of y(1) computed using the Euler’s method (with infinite
precision) with step size h = 0.01.
6. Find an upper bound for the propagated error in Euler method (with infinite
precision) with h = 0.1 for solving the initial value problem y′
= y, y(0) = 1, in
the interval
i) [0, 1] and
S. Baskar and S. Sivaji Ganesh
247
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-259-320.jpg)
![Chapter 7. Numerical Ordinary Differential Equations
ii) [0, 5].
7. For each n ∈ N, write down the Euler’s method for the solution of the initial value
problem y′
= y, y(0) = 1 on the interval [0, 1] with step size h = 1/n. Let the
resulting approximation to y(1) be denoted by αn. Show using limiting argument
(without using the error bound) that αn → y(1) as n → ∞.
8. In each of the following initial value problems, use the Euler’s method, Runge-
Kutta method of order 2 and 4 to find the solution at the specified point with
specified step size h:
i) y′
= x + y; y(0) = 1. Find y(0.2) (For the Euler’s method take h = 0.1 and
for other methods, take h = 0.2). The exact solution is y(x) = −1−x+2ex
.
ii) y′
= 2 cos x − y, y(0) = 1. Find y(0.6) (For the Euler’s method take h = 0.1
and for other methods, take h = 0.2) The exact solution is y(x) = sin x +
cos x.
9. Use the Euler’s, Runge-Kutta methods of order 2 and 4 to solve the IVP y′
=
0.5(x − y) for all x ∈ [0, 3] with initial condition y(0) = 1. Compare the solutions
for h = 1, 0.5, 0.25, 0.125 along with the exact solution y(x) = 3e−x/2
+ x − 2.
10. Show that the Euler’s and Runge-Kutta methods fail to determine an approxima-
tion to the non-trivial solution of the initial value problem y′
= yα
, 0 < α < 1,
y(0) = 0. Note that the ODE is in separable form and hence a non-trivial solution
to initial value problem can be found analytically.
11. Write the formula using the Euler’s method for approximating the solution to the
initial value problem
y′
= x2
+ y2
, y(x0) = y0
at the point x = x1 with h = x0 − x1 > 0. Find the approximate value y1 of the
solution to this initial value problem at the point x1 when x0 = 0, y0 = 1, and
h = 0.25. Find a bound for the truncation error in obtaining y1.
12. Using the Gaussian rule determine a formula for solving the initial value problem
y′
= e−x2
, y(x0) = y0.
in the form
yj+1 = yj−1 + h
(
e
−(xj− h√
3
)2
+ e
−(xj+ h√
3
)2
)
when the nodes are equally spaced with spacing h = xj+1 − xj, j ∈ Z. (h > 0).
Let x0 = 0, and y0 = 1. Using the method derived above, obtain approximate
values of y(−0.1) and y(0.1).
S. Baskar and S. Sivaji Ganesh
248
Spring 2017-18](https://image.slidesharecdn.com/full-ssg-ma214-napostmidsem-201718-180411131203/85/Numerical-Analysis-260-320.jpg)
Numerical Analysis
- 1. Introduction to Numerical Analysis Study Material for MA 214 S. Baskar and S. Sivaji Ganesh Department of Mathematics Indian Institute of Technology Bombay Powai, Mumbai 400 076.
- 3. Introduction to Numerical Analysis Study Material for MA 214 S. Baskar and S. Sivaji Ganesh Instructors: S. Sivaji Ganesh and Amlan Barua Spring 2017-18, Mathematics Department, Indian Institute of Technology Dharwad. Edition-3, Spring 2017-18
- 5. Contents 1 Mathematical Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Sequences of Real Numbers 1 1.2 Limits and Continuity 4 1.3 Differentiation 7 1.4 Integration 10 1.5 Taylor’s Theorem 11 1.6 Orders of Convergence 17 1.6.1 Big Oh and Little oh Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.6.2 Rates of Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 1.7 Exercises 21 2 Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.1 Floating-Point Representation 26 2.1.1 Floating-Point Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.1.2 Underflow and Overflow of Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.1.3 Chopping and Rounding a Number . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.1.4 Arithmetic Using n-Digit Rounding and Chopping . . . . . . . . . . . . . . . . . . . 32 2.2 Types of Errors 33 2.3 Loss of Significance 34 2.4 Propagation of Relative Error in Arithmetic Operations 38 2.4.1 Addition and Subtraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2.4.2 Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.4.3 Division . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.4.4 Total Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 5
- 6. Contents 2.5 Propagation of Relative Error in Function Evaluation 42 2.5.1 Stable and Unstable Computations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 2.6 Exercises 47 3 Numerical Linear Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.1 System of Linear Equations 52 3.2 Direct Methods for Linear Systems 53 3.2.1 Naive Gaussian Elimination Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.2.2 Modified Gaussian Elimination Method with Partial Pivoting . . . . . . . . . . 57 3.2.3 Operations Count in Naive Gaussian Elimination Method . . . . . . . . . . . . . 60 3.2.4 Thomas Method for Tri-diagonal System . . . . . . . . . . . . . . . . . . . . . . . . . 61 3.2.5 LU Factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.3 Matrix Norms and Condition Number of a Matrix 73 3.4 Iterative Methods for Linear Systems 82 3.4.1 Jacobi Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 3.4.2 Gauss-Seidel Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 3.4.3 Mathematical Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 3.4.4 Residual Corrector Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 3.4.5 Stopping Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 3.5 Eigenvalue Problems 95 3.5.1 Power Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 3.5.2 Gerschgorin’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 3.6 Exercises 111 4 Nonlinear Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 4.1 Closed Domain Methods 120 4.1.1 Bisection Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 4.1.2 Regula-falsi Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 4.2 Stopping Criteria 131 4.3 Open Domain Methods 132 4.3.1 Secant Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 4.3.2 Newton-Raphson Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 4.3.3 Fixed-Point Iteration Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 4.4 Comparison and Pitfalls of Iterative Methods 149 6
- 7. Section Contents 4.5 Exercises 151 5 Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 5.1 Polynomial Interpolation 160 5.1.1 Existence and Uniqueness of Interpolating Polynomial . . . . . . . . . . . . . . . 161 5.1.2 Lagrange’s Form of Interpolating Polynomial . . . . . . . . . . . . . . . . . . . . . 164 5.1.3 Newton’s Form of Interpolating Polynomial . . . . . . . . . . . . . . . . . . . . . . . 167 5.2 Newton’s Divided Difference Formulas 169 5.2.1 Divided Differences Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 5.2.2 Divided Difference Formula for Repeated Nodes . . . . . . . . . . . . . . . . . . . 173 5.3 Error in Polynomial Interpolation 176 5.3.1 Mathematical Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 5.3.2 Arithmetic Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 5.3.3 Total Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 5.3.4 Runge Phenomenon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 5.3.5 Convergence of Sequence of Interpolating Polynomials . . . . . . . . . . . . . . 184 5.4 Piecewise Polynomial Interpolation 185 5.5 Spline Interpolation 188 5.6 Exercises 193 6 Numerical Integration and Differentiation . . . . . . . . . . . . . . . . . . . . 197 6.1 Numerical Integration 197 6.1.1 Rectangle Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 6.1.2 Trapezoidal Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 6.1.3 Simpson’s Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 6.1.4 Method of Undetermined Coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 6.1.5 Gaussian Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 6.2 Numerical Differentiation 212 6.2.1 Approximations of First Derivative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 6.2.2 Methods based on Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 6.2.3 Methods based on Undetermined Coefficients . . . . . . . . . . . . . . . . . . . . . 219 6.2.4 Arithmetic Error in Numerical Differentiation . . . . . . . . . . . . . . . . . . . . . 221 6.3 Exercises 223 7
- 8. 7 Numerical Ordinary Differential Equations . . . . . . . . . . . . . . . . . . . 229 7.1 Review of Theory 230 7.2 Discretization Notations 234 7.3 Euler’s Method 235 7.3.1 Error in Euler’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 7.4 Modified Euler’s Methods 241 7.5 Runge-Kutta Methods 243 7.5.1 Order Two . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 7.5.2 Order Four . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 7.6 Exercises 246 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
- 11. Preface These lecture notes are prepared to serve as a supplementary study material for the students of the course MA 214 on Introduction to Numerical Analysis at the department of Mathematics, IIT Goa/Dharwad in Spring 2017-18. These notes are prepared mainly from the books quoted in the syllabus. However, other books were also used especially exercise problems are taken from various other books.
- 13. CHAPTER 1 Mathematical Preliminaries This chapter reviews some of the concepts and results from calculus that are frequently used in this course. We recall important definitions and theorems, and outline proofs of certain theorems. The readers are assumed to be familiar with a first course in calculus. In Section 1.1, we introduce sequences of real numbers and discuss the concept of limit and continuity in Section 1.2 with the intermediate value theorem. This theorem plays a basic role in finding initial guesses in iterative methods for solving nonlinear equations. In Section 1.3 we define the notion of derivative of a function, and prove Rolle’s theorem and mean-value theorem for derivatives. The mean-value theorem for integration is discussed in Section 1.4. These two theorems are crucially used in devising methods for numerical integration and differentiation. Finally, Taylor’s theorem is discussed in Section 1.5, which is essential for derivation and error analysis of almost all numerical methods discussed in this course. In Section 1.6 we introduce tools useful in discussing speed of convergence of sequences and rate at which a function f(x) approaches a point f(x0) as x → x0. 1.1 Sequences of Real Numbers Let a, b ∈ R be such that a < b. We use the notations [a, b] and (a, b) for the closed and the open intervals, respectively, and are defined by [a, b] = { x ∈ R : a ≤ x ≤ b } and (a, b) = { x ∈ R : a < x < b }. S. Baskar and S. Sivaji Ganesh 1 Spring 2017-18
- 14. Chapter 1. Mathematical Preliminaries Definition 1.1.1 [Sequence]. A sequence of real numbers is an ordered list of real numbers a1, a2, . . . , an, an+1, . . . . In other words, a sequence is a function that associates the real number an for each natural number n. The notation {an} is used to denote the sequence, i.e. {an} := a1, a2, . . . , an, an+1, . . . . The concept of convergence of a sequence plays an important role in the numerical analysis. For instance, an iterative procedure is often constructed to approximate a solution x of an equation. This procedure produces a sequence of approximations, which is expected to convergence to the exact solution x. Definition 1.1.2 [Convergence of a Sequence]. Let {an} be a sequence of real numbers and let a be a real number. The sequence {an} is said to converge to a, written as lim n→∞ an = a (or an → a as n → ∞), if for every ϵ > 0, there exists a natural number N such that |an − a| < ϵ whenever n ≥ N. The real number a is called the limit of the sequence {an}. The following result is very useful in computing the limit of a sequence sandwiched between two sequences having a common limit. Theorem 1.1.3 [Sandwich Theorem]. Let {an}, {bn}, {cn} be sequences of real numbers such that 1. there exists an n0 ∈ N such that for every n ≥ n0, the sequences satisfy the inequalities an ≤ bn ≤ cn and 2. lim n→∞ an = lim n→∞ cn = a, for some real number a. Then the sequence {bn} also converges and lim n→∞ bn = a. S. Baskar and S. Sivaji Ganesh 2 Spring 2017-18
- 15. Section 1.1 Sequences of Real Numbers Definition 1.1.4 [Bounded Sequence]. A sequence {an} is said to be a bounded sequence if there exists a real number M such that |an| ≤ M for every n ∈ N. Note that a bounded sequence, in general, need not converge, for instance {(−1)n }. However, it is always possible to find a convergent subsequence. Theorem 1.1.5 [Bolzano-Weierstrass theorem]. Every bounded sequence {an} has a convergent subsequence {ank }. Definition 1.1.6 [Monotonic Sequences]. A sequence {an} of real numbers is said to be 1. an increasing sequence if an ≤ an+1, for every n ∈ N. 2. a strictly increasing sequence if an < an+1, for every n ∈ N. 3. a decreasing sequence if an ≥ an+1, for every n ∈ N. 4. a strictly decreasing sequence if an > an+1, for every n ∈ N. A sequence {an} is said to be a (strictly) monotonic sequence if it is either (strictly) increasing or (strictly) decreasing. The following theorem shows the advantage of working with monotonic sequences. Theorem 1.1.7. Bounded monotonic sequences always converge. The following result gives the algebra of limits of sequences. Theorem 1.1.8. Let {an} and {bn} be two sequences. Assume that lim n→∞ an and lim n→∞ bn exist. Then 1. lim n→∞ (an + bn) = lim n→∞ an + lim n→∞ bn. 2. lim n→∞ c an = c lim n→∞ an, for any number c. 3. lim n→∞ anbn = lim n→∞ an lim n→∞ bn . 4. lim n→∞ 1 an = 1 lim n→∞ an , provided lim n→∞ an ̸= 0. S. Baskar and S. Sivaji Ganesh 3 Spring 2017-18
- 16. Chapter 1. Mathematical Preliminaries 1.2 Limits and Continuity In the previous section, we introduced the concept of limit for a sequences of real numbers. We now define the “limit” in the context of functions. Definition 1.2.1 [Limit of a Function]. 1. Let f be a function defined on the left side (or both sides) of a, except possibly at a itself. Then, we say “the left-hand limit of f(x) as x approaches a, equals l” and denote lim x→a− f(x) = l, if we can make the values of f(x) arbitrarily close to l (as close to l as we like) by taking x to be sufficiently close to a and x less than a. 2. Let f be a function defined on the right side (or both sides) of a, except possibly at a itself. Then, we say “the right-hand limit of f(x) as x approaches a, equals r” and denote lim x→a+ f(x) = r, if we can make the values of f(x) arbitrarily close to r (as close to r as we like) by taking x to be sufficiently close to a and x greater than a. 3. Let f be a function defined on both sides of a, except possibly at a itself. Then, we say“the limit of f(x) as x approaches a, equals L” and denote lim x→a f(x) = L, if we can make the values of f(x) arbitrarily close to L (as close to L as we like) by taking x to be sufficiently close to a (on either side of a) but not equal to a. Remark 1.2.2. Note that in each of the above definitions the value of the function f at the point a does not play any role. In fact, the function f need not be defined at a. In the previous section, we have seen some limit laws in the context of sequences. Similar limit laws also hold for limits of functions. We have the following result, often referred to as “the limit laws” or as “algebra of limits”. S. Baskar and S. Sivaji Ganesh 4 Spring 2017-18
- 17. Section 1.2 Limits and Continuity Theorem 1.2.3. Let f, g be two functions defined on both sides of a, except possibly at a itself. Assume that lim x→a f(x) and lim x→a g(x) exist. Then 1. lim x→a (f(x) + g(x)) = lim x→a f(x) + lim x→a g(x). 2. lim x→a c f(x) = c lim x→a f(x), for any number c. 3. lim x→a f(x)g(x) = lim x→a f(x) lim x→a g(x). 4. lim x→a 1 g(x) = 1 lim x→a g(x) , provided lim x→a g(x) ̸= 0. Remark 1.2.4. Polynomials, rational functions, all trigonometric functions wherever they are de- fined, have property called direct substitution property: lim x→a f(x) = f(a). The following theorem is often used to estimate limits of functions. Theorem 1.2.5. If f(x) ≤ g(x) when x is in an interval containing a (except possibly at a) and the limits of f and g both exist as x approaches a, then lim x→a f(x) ≤ lim x→a g(x). The following theorem is often used to compute limits of functions. Theorem 1.2.6 [Sandwich Theorem]. Let f, g, and h be given functions such that 1. f(x) ≤ g(x) ≤ h(x) when x is in an interval containing a (except possibly at a) and 2. lim x→a f(x) = lim x→a h(x) = L, then lim x→a g(x) = L. S. Baskar and S. Sivaji Ganesh 5 Spring 2017-18
- 18. Chapter 1. Mathematical Preliminaries We will now give a rigorous definition of the limit of a function. Similar definitions can be written down for left-hand and right-hand limits of functions. Definition 1.2.7. Let f be a function defined on some open interval that contains a, except possibly at a itself. Then we say that the limit of f(x) as x approaches a is L and we write lim x→a f(x) = L. if for every ϵ > 0 there is a number δ > 0 such that |f(x) − L| < ϵ whenever 0 < |x − a| < δ. Definition 1.2.8 [Continuity]. A function f is 1. continuous from the right at a if lim x→a+ f(x) = f(a). 2. continuous from the left at a if lim x→a− f(x) = f(a). 3. continuous at a if lim x→a f(x) = f(a). A function f is said to be continuous on an open interval if f is continuous at every point in the interval. If f is defined on a closed interval [a, b], then f is said to be continuous at a if f is continuous from the right at a and similarly, f is said to be continuous at b if f is continuous from left at b. Remark 1.2.9. Note that the definition for continuity of a function f at a, means the following three conditions are satisfied: 1. The function f must be defined at a. i.e., a is in the domain of f, 2. lim x→a f(x) exists, and 3. lim x→a f(x) = f(a). Equivalently, for any given ϵ > 0, there exists a δ > 0 such that |f(x) − f(a)| < ϵ whenever |x − a| < δ. S. Baskar and S. Sivaji Ganesh 6 Spring 2017-18
- 19. Section 1.3 Differentiation Theorem 1.2.10. If f and g are continuous at a, then the functions f + g, f − g, cg (c is a constant), fg, f/g (provided g(a) ̸= 0), f◦g (composition of f and g, whenever it makes sense) are all continuous. Thus polynomials, rational functions, trigonometric functions are all continuous on their respective domains. Theorem 1.2.11 [Intermediate Value Theorem]. Suppose that f is continuous on the closed interval [a, b] and let N be any number between f(a) and f(b), where f(a) ̸= f(b). Then there exists a point c ∈ (a, b) such that f(c) = N. 1.3 Differentiation We next give the basic definition of a differentiable function. Definition 1.3.1 [Derivative]. The derivative of a function f at a point a, denoted by f′ (a), is defined as f′ (a) = lim h→0 f(a + h) − f(a) h , (1.1) if this limit exists. We say f is differentiable at a. A function f is said to be differentiable on (c, d) if f is differentiable at every point in (c, d). There are alternate ways of defining the derivative of a function, which leads to different difference formulae. Remark 1.3.2. The derivative of a function f at a point x = a can also be given by f′ (a) = lim h→0 f(a) − f(a − h) h , (1.2) and f′ (a) = lim h→0 f(a + h) − f(a − h) 2h , (1.3) provided the limits exist. S. Baskar and S. Sivaji Ganesh 7 Spring 2017-18
- 20. Chapter 1. Mathematical Preliminaries If we write x = a + h, then h = x − a and h → 0 if and only if x → a. Thus, formula (1.1) can equivalently be written as f′ (a) = lim x→a f(x) − f(a) x − a . Remark 1.3.3 [Interpretation]. Take the graph of f, draw the line joining the points (a, f(a)) and (x, f(x)). Take its slope and take the limit of these slopes as x → a. Then the point (x, f(x)) tends to (a, f(a)). The limit is nothing but the slope of the tangent line at (a, f(a)) to the curve y = f(x). This geometric interpretation will be very useful in describing the Newton-Raphson method in the context of solving nonlinear equations. Theorem 1.3.4. If f is differentiable at a, then f is continuous at a. Proof. Using the identities f(x) = f(x) − f(a) x − a (x − a) + f(a) and taking limit as x → a yields the desired result. The converse of Theorem 1.3.4 is not true. For, the function f(x) = |x| is continuous at x = 0 but is not differentiable there. Theorem 1.3.5. Suppose f is differentiable at a. Then there exists a function ϕ such that f(x) = f(a) + (x − a)f′ (a) + (x − a)ϕ(x), and limx→a ϕ(x) = 0. Proof. Define ϕ by ϕ(x) = f(x) − f(a) x − a − f′ (a). Since f is differentiable at a, the result follows on taking limits on both sides of the last equation as x → a. S. Baskar and S. Sivaji Ganesh 8 Spring 2017-18
- 21. Section 1.3 Differentiation Theorem 1.3.6 [Rolle’s Theorem]. Let f be a function that satisfies the following three hypotheses: 1. f is continuous on the closed interval [a, b]. 2. f is differentiable on the open interval (a, b). 3. f(a) = f(b). Then there is a number c in the open interval (a, b) such that f′ (c) = 0. Proof. If f is a constant function i.e., f(x) = f(a) for every x ∈ [a, b], clearly such a c exists. If f is not a constant, then at least one of the following holds. Case 1: The graph of f goes above the line y = f(a) i.e., f(x) > f(a) for some x ∈ (a, b). Case 2: The graph of f goes below the line y = f(a) i.e., f(x) < f(a) for some x ∈ (a, b). In case (1), i.e., if the graph of f goes above the line y = f(a), then the global maximum cannot be at a or b. Therefore, it must lie in the open interval (a, b). Denote that point by c. That is, global maximum on [a, b] is actually a local maxi- mum, and hence f′ (c) = 0. A similar argument can be given in case (2), to show the existence of local minimum in (a, b). This completes the proof of Rolle’s theorem. The mean value theorem plays an important role in obtaining error estimates for certain numerical methods. Theorem 1.3.7 [Mean Value Theorem]. Let f be a function that satisfies the following hypotheses: 1. f is continuous on the closed interval [a, b]. 2. f is differentiable on the open interval (a, b). Then there is a number c in the open interval (a, b) such that f′ (c) = f(b) − f(a) b − a . or, equivalently, f(b) − f(a) = f′ (c)(b − a). S. Baskar and S. Sivaji Ganesh 9 Spring 2017-18
- 22. Chapter 1. Mathematical Preliminaries Proof. Define ϕ on [a, b] by ϕ(x) = f(x) − f(a) − f(b) − f(a) b − a (x − a). By Rolle’s theorem there exists a c ∈ (a, b) such that ϕ′ (c) = 0 and hence the proof is complete. 1.4 Integration In Theorem 1.3.7, we have discussed the mean value property for the derivative of a function. We now discuss the mean value theorems for integration. Theorem 1.4.1 [Mean Value Theorem for Integrals]. If f is continuous on [a, b], then there exists a number c in [a, b] such that b∫ a f(x) dx = f(c)(b − a). Proof. Let m and M be minimum and maximum values of f in the interval [a, b], respectively. Then, m(b − a) ≤ b∫ a f(x) dx ≤ M(b − a). Since f is continuous, the result follows from the intermediate value theorem. Recall the average value of a function f on the interval [a, b] is defined by 1 b − a b∫ a f(x) dx. Observe that the first mean value theorem for integrals asserts that the average of an integrable function f on an interval [a, b] belongs to the range of the function f. S. Baskar and S. Sivaji Ganesh 10 Spring 2017-18
- 23. Section 1.5 Taylor’s Theorem Remark 1.4.2 [Interpretation]. Let f be a function on [a, b] with f > 0. Draw the graph of f and find the area under the graph lying between the ordinates x = a and x = b. Also, look at a rectangle with base as the interval [a, b] with height f(c) and compute its area. Both values are the same. The Theorem 1.4.1 is often referred to as the first mean value theorem for integrals. We now state the second mean value theorem for integrals, which is a general form of Theorem 1.4.1. Theorem 1.4.3 [Second Mean Value Theorem for Integrals]. Let f and g be continuous on [a, b], and let g(x) ≥ 0 for all x ∈ R. Then there exists a number c ∈ [a, b] such that b∫ a f(x)g(x) dx = f(c) b∫ a g(x) dx. The proof of this theorem is left as an exercise. 1.5 Taylor’s Theorem Let f be a real-valued function defined on an interval I. We say f ∈ Cn (I) if f is n-times continuously differentiable at every point in I. Also, we say f ∈ C∞ (I) if f is continuously differentiable of any order at every point in I. The most important result used very frequently in numerical analysis, especially in error analysis of numerical methods, is the Taylor’s expansion of a C∞ function in a neighborhood of a point a ∈ R. In this section, we define the Taylor’s polynomial and prove an important theorem called the Taylor’s theorem. The idea of the proof of this theorem is similar to the one used in proving the mean value theorem, where we construct a function and apply Rolle’s theorem several times to it. Definition 1.5.1 [Taylor’s Polynomial for a Function at a Point]. Let f be n-times continuously differentiable at a given point a. The Taylor’s polynomial of degree n for the function f at the point a, denoted by Tn, is defined by Tn(x) = n∑ k=0 f(k) (a) k! (x − a)k , x ∈ R. (1.4) S. Baskar and S. Sivaji Ganesh 11 Spring 2017-18
- 24. Chapter 1. Mathematical Preliminaries Theorem 1.5.2 [Taylor’s Theorem]. Let f be (n + 1)-times continuously differentiable function on an open interval con- taining the points a and x. Then there exists a number ξ between a and x such that f(x) = Tn(x) + f(n+1) (ξ) (n + 1)! (x − a)n+1 , (1.5) where Tn is the Taylor’s polynomial of degree n for f at the point a given by (1.4) and the second term on the right hand side is called the remainder term. Proof. Let us assume x > a and prove the theorem. The proof is similar if x < a. Define g(t) by g(t) = f(t) − Tn(t) − A(t − a)n+1 and choose A so that g(x) = 0, which gives A = f(x) − Tn(x) (x − a)n+1 . Note that g(k) (a) = 0 for k = 0, 1, . . . n. Also, observe that the function g is continuous on [a, x] and differentiable in (a, x). Apply Rolle’s theorem to g on [a, x] (after verifying all the hypotheses of Rolle’s theorem) to get a < c1 < x satisfying g′ (c1) = 0. Again apply Rolle’s theorem to g′ on [a, c1] to get a < c2 < c1 satisfying g′′ (c2) = 0. In turn apply Rolle’s theorem to g(2) , g(3) , . . . , g(n) on intervals [a, c2], [a, c3], . . . , [a, cn], respectively. At the last step, we get a < cn+1 < cn satisfying g(n+1) (cn+1) = 0. S. Baskar and S. Sivaji Ganesh 12 Spring 2017-18
- 25. Section 1.5 Taylor’s Theorem But g(n+1) (cn+1) = f(n+1) (cn+1) − A(n + 1)!, which gives A = f(n+1) (cn+1) (n + 1)! . Equating both values of A, we get f(x) = Tn(x) + f(n+1) (cn+1) (n + 1)! (x − a)n+1 . This completes the proof. Observe that the mean value theorem (Theorem 1.3.7) is a particular case of the Taylor’s theorem. Remark 1.5.3. The representation (1.5) is called the Taylor’s formula for the function f about the point a. The Taylor’s theorem helps us to obtain an approximate value of a sufficiently smooth function in a small neighborhood of a given point a when the value of f and all its derivatives up to a sufficient order is known at the point a. For instance, if we know f(a), f′(a), · · · , f(n)(a), and we seek an approximate value of f(a + h) for some real number h, then the Taylor’s theorem can be used to get f(a + h) ≈ f(a) + f′ (a)h + f′′(a) 2! h2 + · · · + f(n)(a) n! hn . Note here that we have not added the remainder term and therefore used the approximation symbol ≈. Observe that the remainder term f(n+1)(ξ) (n + 1)! hn+1 is not known since it involves the evaluation of f(n+1) at some unknown value ξ lying between a and a + h. Also, observe that as h → 0, the remainder term approaches to zero, provided f(n+1) is bounded. This means that for smaller values of h, the Taylor’s polynomial gives a good approximation of f(a + h). As remarked above, the remainder term involves an unknown parameter and often this term cannot be calculated explicitly. However, an estimate of the error involved in the approximation of the function by its Taylor’s polynomial can be obtained by finding a bound for the remainder term. S. Baskar and S. Sivaji Ganesh 13 Spring 2017-18
- 26. Chapter 1. Mathematical Preliminaries Remark 1.5.4 [Estimate for Remainder Term in Taylor’s Formula]. Let f be an (n + 1)-times continuously differentiable function with the property that there exists an Mn+1 such that |f(n+1) (ξ)| ≤ Mn+1, for all ξ ∈ I. Then for fixed points a, x ∈ I, the remainder term in (1.5) satisfies the estimate f(n+1)(ξ) (n + 1)! (x − a)n+1 ≤ Mn+1 (n + 1)! x − a n+1 . We can further get an estimate of the reminder term that is independent of x as f(n+1)(ξ) (n + 1)! (x − a)n+1 ≤ Mn+1 (n + 1)! (b − a)n+1 , which holds for all x ∈ I. Observe that the right hand side of the above estimate is a fixed number. We refer to such estimates as remainder estimates. Note In most applications of Taylor’s theorem, one never knows ξ precisely. However in view of remainder estimate given above, it does not matter as long as we know that the remainder can be bounded by obtaining a bound Mn+1 which is valid for all ξ between a and x. We next introduce an important notion of error called the truncation error Definition 1.5.5 [Truncation Error]. The remainder term involved in approximating f(x) by the Taylor’s polynomial Tn(x) is also called the Truncation error. Example 1.5.6. A second degree polynomial approximation to f(x) = √ x + 1, x ∈ [−1, ∞) using the Taylor’s formula about a = 0 is given by f(x) ≈ 1 + x 2 − x2 8 , where the remainder term is neglected and hence what we obtained here is only an approximate representation of f. S. Baskar and S. Sivaji Ganesh 14 Spring 2017-18
- 27. Section 1.5 Taylor’s Theorem The truncation error is obtained using the remainder term in the formula (1.5) with n = 2 and is given by x3 16( √ 1 + ξ )5 , for some point ξ between 0 and x. Note that we cannot obtain a remainder estimate in the present example as f′′′ is not bounded in [−1, ∞). However, for any 0 < δ < 1, if we restrict the domain of f to [−δ, ∞), then we can obtain the remainder estimate for a fixed x ∈ [−δ, ∞) as x3 16( √ 1 − δ )5 . Further, if we restrict the domain of f to [−δ, b] for some real number b > 0, then we get the remainder estimate independent of x as b3 16( √ 1 − δ )5 . Definition 1.5.7 [Taylor’s Series]. Let f be C∞ in a neighborhood of a point a. The power series ∞∑ k=0 f(k) (a) k! (x − a)k is called the Taylor’s series of f about the point a. The question now is when this series converges and what is the limit of this series. These questions are answered in the following theorem whose proof is omitted in this course. Theorem 1.5.8. Let f ∈ C∞ (I) and let a ∈ I. Assume that there exists an open neighborhood (interval) Na ⊂ I of the point a and there exists a constant M (may depend on a) such that f(k) (x) ≤ Mk , for all x ∈ Na and k = 0, 1, 2, · · · . Then for each x ∈ Ia, we have f(x) = ∞∑ k=0 f(k) (a) k! (x − a)k . S. Baskar and S. Sivaji Ganesh 15 Spring 2017-18
- 28. Chapter 1. Mathematical Preliminaries −6 −4 −2 0 2 4 6 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 x y f(x)=cos(x) Taylor polynomial of degree 2 (n=1) −6 −4 −2 0 2 4 6 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 x y f(x)=cos(x) Taylor polynomial of degree 10 (n=5) Figure 1.1: Comparison between the graph of f(x) = cos(x) and the Taylor polynomial of degree 2 and 10 about the point a = 0. The following example illustrates the procedure for approximating a given function by its Taylor’s polynomial and obtaining the corresponding truncation error. Example 1.5.9. As another example, let us approximate the function f(x) = cos(x) by a polynomial using Taylor’s theorem about the point a = 0. First, let us take the Taylor’s series expansion f(x) = cos(0) − sin(0)x − cos(0) 2! x2 + sin(0) 3! x3 + · · · = ∞∑ k=0 (−1)k (2k)! x2k . Now, we truncate this infinite series to get an approximate representation of f(x) in a sufficiently small neighborhood of a = 0 as f(x) ≈ n∑ k=0 (−1)k (2k)! x2k , which is the Taylor polynomial of degree 2n for the function f(x) = cos(x) about the point a = 0. The remainder term is given by (−1)n+1 cos(ξ) (2(n + 1))! x2(n+1) , S. Baskar and S. Sivaji Ganesh 16 Spring 2017-18
- 29. Section 1.6 Orders of Convergence where ξ lies between 0 and x. It is important to observe in this particular example that for a given n, we get the Taylor polynomial of degree 2n. Figure 1.1 shows the comparison between the Taylor polynomial (red dot and dash line) of degree 2 (n = 1) and degree 10 (n = 5) for f(x) = cos(x) about a = 0 and the graph of cos(x) (blue solid line). We observe that for n = 1, Taylor polynomial gives a good approximation in a small neighborhood of a = 0. But sufficiently away from 0, this polynomial deviates significantly from the actual graph of f(x) = cos(x). Whereas, for n = 5, we get a good approximation in a relatively larger neighborhood of a = 0. 1.6 Orders of Convergence In Section 1.1, we defined convergent sequences and discussed some conditions under which a given sequence of real numbers converges. The definition and the discussed conditions never tell us how fast the sequence converges to the limit. Even if we know that a sequence of approximations converge to the exact one (limit), it is very important in numerical analysis to know how fast the sequence of approximate values converge to the exact value. In this section, we introduce two very important notations called big Oh and little oh, which are basic tools for the study of speed of convergence. We end this section by defining the rate of convergence, which is also called order of convergence. 1.6.1 Big Oh and Little oh Notations The notions of big Oh and little oh are well understood through the following exam- ple. Example 1.6.1. Consider the two sequences {n} and {n2 } both of which are unbounded and tend to infinity as n → ∞. However we feel that the sequence {n} grows ‘slowly’ compared to the sequence {n2 }. Consider also the sequences {1/n} and {1/n2 } both of which decrease to zero as n → ∞. However we feel that the sequence {1/n2 } decreases more rapidly compared to the sequence {1/n}. The above examples motivate us to develop tools that compare two sequences {an} and {bn}. Landau has introduced the concepts of Big Oh and Little oh for comparing two sequences that we will define below. S. Baskar and S. Sivaji Ganesh 17 Spring 2017-18
- 30. Chapter 1. Mathematical Preliminaries Definition 1.6.2 [Big Oh and Little oh]. Let {an} and {bn} be sequences of real numbers. Then 1. the sequence {an} is said to be Big Oh of {bn}, and write an = O(bn), if there exists a real number C and a natural number N such that |an| ≤ C |bn| for all n ≥ N. 2. the sequence {an} is said to be Little oh (sometimes said to be small oh) of {bn}, and write an = o(bn), if for every ϵ > 0 there exists a natural number N such that |an| ≤ ϵ |bn| for all n ≥ N. Remark 1.6.3. 1. If bn ̸= 0 for every n, then we have an = O(bn) if and only if the sequence{ an bn } is bounded. That is, there exists a constant C such that an bn ≤ C 2. If bn ̸= 0 for every n, then we have an = o(bn) if and only if the sequence{ an bn } converges to 0. That is, lim n→∞ an bn = 0. 3. For any pair of sequences {an} and {bn} such that an = o(bn), it follows that an = O(bn). The converse is not true. Consider the sequences an = n and bn = 2n + 3, for which an = O(bn) holds but an = o(bn) does not hold. 4. Let {an} and {bn} be two sequences that converge to 0. Then an = O(bn) means the sequence {an} tends to 0 at least as fast as the sequence {bn}; and an = o(bn) means the sequence {an} tends to 0 faster than the sequence {bn}. The Big Oh and Little oh notations can be adapted for functions as follows. S. Baskar and S. Sivaji Ganesh 18 Spring 2017-18
- 31. Section 1.6 Orders of Convergence Definition 1.6.4 [Big Oh and Little oh for Functions]. Let x0 ∈ R. Let f and g be functions defined in an interval containing x0. Then 1. the function f is said to be Big Oh of g as x → x0, and write f(x) = O ( g(x) ) , if there exists a real number C and a real number δ such that |f(x)| ≤ C |g(x)| whenever |x − x0| ≤ δ. 2. the function f is said to be Little Oh (also, Small oh) of g as x → x0, and write f(x) = o ( g(x) ) , if for every ϵ > 0 there exists a real number δ such that |f(x)| ≤ ϵ |g(x)| whenever |x − x0| ≤ δ. In case of functions also, a remark similar to the Remark 1.6.3 holds. Example 1.6.5. The Taylor’s formula for f(x) = cos(x) about the point a = 0 is cos(x) = n∑ k=0 (−1)k (2k)! x2k + (−1)n+1 cos(ξ) (2(n + 1))! x2(n+1) where ξ lies between x and 0. Let us denote the remainder term (truncation error) as (for a fixed n) g(x) = (−1)n+1 cos(ξ) (2(n + 1))! x2(n+1) . Clearly, g(x) → 0 as x → 0. The question now is ‘How fast does g(x) → 0 as x → 0?’ The answer is ‘As fast as x2(n+1) → 0 as x → 0.’ That is, g(x) = O ( x2(n+1) ) as x → 0. In this case, we say that the sequence { n∑ k=0 (−1)k (2k)! x2k } converges to cos(x) with order 2(n + 1). S. Baskar and S. Sivaji Ganesh 19 Spring 2017-18
- 32. Chapter 1. Mathematical Preliminaries 1.6.2 Rates of Convergence Let {an} be a sequence such that lim n→∞ an = a. We would like to measure the speed at which the convergence takes place. For example, consider lim n→∞ 1 2n + 3 = 0 and lim n→∞ 1 n2 = 0. We feel that the first sequence goes to zero linearly and the second goes with a much superior speed because of the presence of n2 in its denominator. We will define the notion of order of convergence precisely. Definition 1.6.6 [Rate of Convergence or Order of Convergence]. Let {an} be a sequence such that lim n→∞ an = a. 1. We say that the rate of convergence is atleast linear if there exists a constant c < 1 and a natural number N such that |an+1 − a| ≤ c |an − a| for all n ≥ N. 2. We say that the rate of convergence is atleast superlinear if there exists a sequence {ϵn} that converges to 0, and a natural number N such that |an+1 − a| ≤ ϵn |an − a| for all n ≥ N. 3. We say that the rate of convergence is atleast quadratic if there exists a constant C (not necessarily less than 1), and a natural number N such that |an+1 − a| ≤ C |an − a|2 for all n ≥ N. 4. Let α ∈ R+. We say that the rate of convergence is atleast α if there exists a constant C (not necessarily less than 1), and a natural number N such that |an+1 − a| ≤ C |an − a|α for all n ≥ N. S. Baskar and S. Sivaji Ganesh 20 Spring 2017-18
- 33. Section 1.7 Exercises 1.7 Exercises Sequences of Real Numbers 1. Let L be a real number and let {an} be a sequence of real numbers. If there exists a positive integer N such that |an − L| ≤ µ|an−1 − L|, for all n ≥ N and for some fixed µ ∈ (0, 1), then show that an → L as n → ∞. 2. Consider the sequences {an} and {bn}, where an = 1 n , bn = 1 n2 , n = 1, 2, · · · . Clearly, both the sequences converge to zero. For the given ϵ = 10−2 , obtain the smallest positive integers Na and Nb such that |an| < ϵ whenever n ≥ Na, and |bn| < ϵ whenever n ≥ Nb. For any ϵ > 0, show that Na > Nb. 3. Let {xn} and {yn} be two sequences such that xn, yn ∈ [a, b] and xn < yn for each n = 1, 2, · · · . If xn → b as n → ∞, then show that the sequence {yn} converges. Find the limit of the sequence {yn}. 4. Let In = [ n − 2 2n , n + 2 2n ] , n = 1, 2, · · · and {an} be a sequence with an is chosen arbitrarily in In for each n = 1, 2, · · · . Show that an → 1 2 as n → ∞. Limits and Continuity 5. Let f be a real-valued function such that f(x) ≥ sin(x) for all x ∈ R. If lim x→0 f(x) = L exists, then show that L ≥ 0. 6. Let P and Q be polynomials. Find lim x→∞ P(x) Q(x) and lim x→0 P(x) Q(x) in each of the following cases. i) The degree of P is less than the degree of Q. ii) The degree of P is greater than the degree of Q. iii) The agree of P is equal to the degree of Q. S. Baskar and S. Sivaji Ganesh 21 Spring 2017-18
- 34. Chapter 1. Mathematical Preliminaries 7. Show that the equation sin x+x2 = 1 has at least one solution in the interval [0, 1]. 8. Let f(x) be continuous on [a, b], let x1, · · · , xn be points in [a, b], and let g1, · · · , gn be real numbers having same sign. Show that n∑ i=1 f(xi)gi = f(ξ) n∑ i=1 gi, for some ξ ∈ [a, b]. 9. Let f : [0, 1] → [0, 1] be a continuous function. Prove that the equation f(x) = x has at least one solution lying in the interval [0, 1] (Note: A solution of this equa- tion is called a fixed point of the function f). 10. Show that the equation f(x) = x, where f(x) = sin ( πx + 1 2 ) , x ∈ [−1, 1] has at least one solution in [−1, 1]. Differentiation 11. Let c ∈ (a, b) and f : (a, b) → R be differentiable at c. If c is a local extremum (maximum or minimum) of f, then show that f′ (c) = 0. 12. Suppose f is differentiable in an open interval (a, b). Prove the following state- ments (a) If f′ (x) ≥ 0 for all x ∈ (a, b), then f is non-decreasing. (b) If f′ (x) = 0 for all x ∈ (a, b), then f is constant. (c) If f′ (x) ≤ 0 for all x ∈ (a, b), then f is non-increasing. 13. Let f : [a, b] → R be given by f(x) = x2 . Find a point c specified by the mean- value theorem for derivatives. Verify that this point lies in the interval (a, b). Integration 14. Let g : [0, 1] → R be a continuous function. Show that there exists a c ∈ (0, 1) such that 1∫ 0 x2 (1 − x)2 g(x)dx = 1 30 g(ξ). S. Baskar and S. Sivaji Ganesh 22 Spring 2017-18
- 35. Section 1.7 Exercises 15. If n is a positive integer, show that √ (n+1)π ∫ √ nπ sin(t2 ) dt = (−1)n c , where √ nπ ≤ c ≤ √ (n + 1)π. Taylor’s Theorem 16. Find the Taylor’s polynomial of degree 2 for the function f(x) = √ x + 1 about the point a = 1. Also find the remainder. 17. Use Taylor’s formula about a = 0 to evaluate approximately the value of the func- tion f(x) = ex at x = 0.5 using three terms (i.e., n = 2) in the formula. Obtain the remainder R2(0.5) in terms of the unknown c. Compute approximately the possible values of c and show that these values lie in the interval (0, 0.5). Big Oh, Little Oh, and Orders of Convergence 18. Prove or disprove: i) 2n2 + 3n + 4 = o(n) as n → ∞. ii) n+1 n2 = o(1 n ) as n → ∞. iii) n+1 n2 = O(1 n ) as n → ∞. iv) n+1√ n = o(1) as n → ∞. v) 1 ln n = o(1 n ) as n → ∞. vi) 1 n ln n = o(1 n ) as n → ∞. vii) en n5 = O(1 n ) as n → ∞. 19. Prove or disprove: i) ex − 1 = O(x2 ) as x → 0. ii) x−2 = O(cot x) as x → 0. iii) cot x = o(x−1 ) as x → 0. iv) For r > 0, xr = O(ex ) as x → ∞. v) For r > 0, ln x = O(xr ) as x → ∞. S. Baskar and S. Sivaji Ganesh 23 Spring 2017-18
- 37. CHAPTER 2 Error Analysis Numerical analysis deals with developing methods, called numerical methods, to ap- proximate a solution of a given Mathematical problem (whenever a solution exists). The approximate solution obtained by a method involves an error, which we call the mathematical error, and is precisely the difference between the exact solution and the approximate solution. Thus, we have Exact Solution = Approximate Solution + Mathematical Error. The study of numerical methods is incomplete if we don’t develop algorithms and implement the algorithms as computer codes. The outcome of the computer code is a set of numerical values to the approximate solution obtained using a numerical method. Such a set of numerical values is called the numerical solution to the given Mathematical problem. During the process of computation, the computer introduces a new error, called the arithmetic error , and we have Approximate Solution = Numerical Solution + Arithmetic Error. The error involved in the numerical solution when compared to the exact solution can be worser than the mathematical error and is now given by Exact Solution = Numerical Solution + Mathematical Error + Arithmetic Error. The Total Error is defined as Total Error = Mathematical Error + Arithmetic Error. S. Baskar and S. Sivaji Ganesh 25 Spring 2017-18
- 38. Chapter 2. Error Analysis A digital calculating device can hold only a finite number of digits because of memory restrictions. Therefore, a number cannot be stored exactly. Certain approximation needs to be done, and only an approximate value of the given number will finally be stored in the device. For further calculations, this approximate value is used instead of the exact value of the number. This is the source of arithmetic error. In this chapter, we introduce the floating-point representation of a real number and illustrate a few ways to obtain floating-point approximation of a given real number. We further introduce different types of errors that we come across in numerical analysis and their effects in the computation. At the end of this chapter, we will be familiar with the arithmetic errors, their effect on computed results and some ways to minimize this error in the computation. 2.1 Floating-Point Representation Let β ∈ N and β ≥ 2. Any real number can be represented exactly in base β as (−1)s × (.d1d2 · · · dndn+1 · · · )β × βe , (2.1) where di ∈ { 0, 1, · · · , β − 1 } with d1 ̸= 0 or d1 = d2 = d3 = · · · = 0, s = 0 or 1, and an appropriate integer e called the exponent. Here (.d1d2 · · · dndn+1 · · · )β = d1 β + d2 β2 + · · · + dn βn + dn+1 βn+1 + · · · (2.2) is a β-fraction called the mantissa, s is called the sign and the number β is called the radix. The representation (2.1) of a real number is called the floating-point representation. Remark 2.1.1. When β = 2, the floating-point representation (2.1) is called the binary floating- point representation and when β = 10, it is called the decimal floating-point representation. Note Throughout this course, we always take β = 10. 2.1.1 Floating-Point Approximation Due to memory restrictions, a computing device can store only a finite number of digits in the mantissa. In this section, we introduce the floating-point approximation and discuss how a given real number can be approximated. S. Baskar and S. Sivaji Ganesh 26 Spring 2017-18
- 39. Section 2.1 Floating-Point Representation Although different computing devices have different ways of representing numbers, here we introduce a mathematical form of this representation, which we will use throughout this course. Definition 2.1.2 [n-Digit Floating-point Number]. Let β ∈ N and β ≥ 2. An n-digit floating-point number in base β is of the form (−1)s × (.d1d2 · · · dn)β × βe (2.3) where (.d1d2 · · · dn)β = d1 β + d2 β2 + · · · + dn βn (2.4) where di ∈ { 0, 1, · · · , β − 1 } with d1 ̸= 0 or d1 = d2 = d3 = · · · = 0, s = 0 or 1, and an appropriate exponent e. Remark 2.1.3. When β = 2, the n-digit floating-point representation (2.3) is called the n-digit binary floating-point representation and when β = 10, it is called the n-digit decimal floating-point representation. Example 2.1.4. The following are examples of real numbers in the decimal floating point representa- tion. 1. The real number x = 6.238 is represented in the decimal floating-point repre- sentation as 6.238 = (−1)0 × 0.6238 × 101 , in which case, we have s = 0, β = 10, e = 1, d1 = 6, d2 = 2, d3 = 3 and d4 = 8. 2. The real number x = −0.0014 is represented in the decimal floating-point rep- resentation as x = (−1)1 × 0.14 × 10−2 . Here s = 1, β = 10, e = −2, d1 = 1 and d2 = 4. S. Baskar and S. Sivaji Ganesh 27 Spring 2017-18
- 40. Chapter 2. Error Analysis Remark 2.1.5. The floating-point representation of the number 1/3 is 1 3 = 0.33333 · · · = (−1)0 × (0.33333 · · · )10 × 100 . An n-digit decimal floating-point representation of this number has to contain only n digits in its mantissa. Therefore, the representation (2.3) is (in general) only an approximation to a real number. Any computing device has its own memory limitations in storing a real number. In terms of the floating-point representation, these limitations lead to the restrictions in the number of digits in the mantissa (n) and the range of the exponent (e). In Section 2.1.2, we introduce the concept of under and over flow of memory, which is a result of the restriction in the exponent. The restriction on the length of the mantissa is discussed in Section 2.1.3. 2.1.2 Underflow and Overflow of Memory When the value of the exponent e in a floating-point number exceeds the maximum limit of the memory, we encounter the overflow of memory, whereas when this value goes below the minimum of the range, then we encounter underflow. Thus, for a given computing device, there are real numbers m and M such that the exponent e is limited to a range m ≤ e ≤ M. (2.5) During the calculation, if some computed number has an exponent e > M then we say, the memory overflow occurs and if e < m, we say the memory underflow occurs. Remark 2.1.6. In the case of overflow of memory in a floating-point number, a computer will usually produce meaningless results or simply prints the symbol inf or NaN. When your computation involves an undetermined quantity (like 0×∞, ∞−∞, 0/0), then the output of the computed value on a computer will be the symbol NaN (means ‘not a number’). For instance, if X is a sufficiently large number that results in an overflow of memory when stored on a computing device, and x is another number that results in an underflow, then their product will be returned as NaN. On the other hand, we feel that the underflow is more serious than overflow in a computation. Because, when underflow occurs, a computer will simply consider the number as zero without any warning. However, by writing a separate subroutine, one can monitor and get a warning whenever an underflow occurs. S. Baskar and S. Sivaji Ganesh 28 Spring 2017-18
- 41. Section 2.1 Floating-Point Representation Example 2.1.7 [Overflow]. Run the following MATLAB code on a computer with 32-bit intel processor: i=308.25471; fprintf(’%f %fn’,i,10^i); i=308.25472; fprintf(’%f %fn’,i,10^i); We see that the first print command shows a meaningful (but very large) number, whereas the second print command simply prints inf. This is due to the overflow of memory while representing a very large real number. Also try running the following code on the MATLAB: i=308.25471; fprintf(’%f %fn’,i,10^i/10^i); i=308.25472; fprintf(’%f %fn’,i,10^i/10^i); The output will be 308.254710 1.000000 308.254720 NaN If your computer is not showing inf for i = 308.25472, try increasing the value of i till you get inf. Example 2.1.8 [Underflow]. Run the following MATLAB code on a computer with 32-bit intel processor: j=-323.6; if(10^j>0) fprintf(’The given number is greater than zeron’); elseif (10^j==0) fprintf(’The given number is equal to zeron’); else fprintf(’The given number is less than zeron’); end The output will be The given number is greater than zero S. Baskar and S. Sivaji Ganesh 29 Spring 2017-18
- 42. Chapter 2. Error Analysis When the value of j is further reduced slightly as shown in the following program j=-323.64; if(10^j>0) fprintf(’The given number is greater than zeron’); elseif (10^j==0) fprintf(’The given number is equal to zeron’); else fprintf(’The given number is less than zeron’); end the output shows The given number is equal to zero If your computer is not showing the above output, try decreasing the value of j till you get the above output. In this example, we see that the number 10−323.64 is recognized as zero by the com- puter. This is due to the underflow of memory. Note that multiplying any large number by this number will give zero as answer. If a computation involves such an underflow of memory, then there is a danger of having a large difference between the actual value and the computed value. 2.1.3 Chopping and Rounding a Number The number of digits in the mantissa, as given in Definition 2.1.2, is called the pre- cision or length of the floating-point number. In general, a real number can have infinitely many digits, which a computing device cannot hold in its memory. Rather, each computing device will have its own limitation on the length of the mantissa. If a given real number has infinitely many (or sufficiently large number of) digits in the mantissa of the floating-point form as in (2.1), then the computing device converts this number into an n-digit floating-point form as in (2.3). Such an approximation is called the floating-point approximation of a real number. There are many ways to get floating-point approximation of a given real number. Here we introduce two types of floating-point approximation. Definition 2.1.9 [Chopped and Rounded Numbers]. Let x be a real number given in the floating-point representation (2.1) as x = (−1)s × (.d1d2 · · · dndn+1 · · · )β × βe . S. Baskar and S. Sivaji Ganesh 30 Spring 2017-18
- 43. Section 2.1 Floating-Point Representation The floating-point approximation of x using n-digit chopping is given by fl(x) = (−1)s × (.d1d2 · · · dn)β × βe . (2.6) The floating-point approximation of x using n-digit rounding is given by fl(x) = { (−1)s × (.d1d2 · · · dn)β × βe , 0 ≤ dn+1 < β 2 (−1)s × (.d1d2 · · · (dn + 1))β × βe , β 2 ≤ dn+1 < β , (2.7) where (−1)s ×(.d1d2 · · · (dn +1))β ×βe := (−1)s × (.d1d2 · · · dn)β + (. 0 0 · · · 0 (n−1)−times 1)β ×βe . Note As already mentioned, throughout this course, we always take β = 10. Also, we do not assume any restriction on the exponent e ∈ Z. Example 2.1.10. The floating-point representation of π is given by π = (−1)0 × (.31415926 · · · ) × 101 . The floating-point approximation of π using five-digit chopping is fl(π) = (−1)0 × (.31415) × 101 , which is equal to 3.1415. Since the sixth digit of the mantissa in the floating-point rep- resentation of π is a 9, the floating-point approximation of π using five-digit rounding is given by fl(π) = (−1)0 × (.31416) × 101 , which is equal to 3.1416. Remark 2.1.11. Most of the modern processors, including Intel, uses IEEE 754 standard format. This format uses 52 bits in mantissa, (64-bit binary representation), 11 bits in exponent and 1 bit for sign. This representation is called the double precision number. When we perform a computation without any floating-point approximation, we say that the computation is done using infinite precision (also called exact arithmetic). S. Baskar and S. Sivaji Ganesh 31 Spring 2017-18
- 44. Chapter 2. Error Analysis 2.1.4 Arithmetic Using n-Digit Rounding and Chopping In this subsection, we describe the procedure of performing arithmetic operations us- ing n-digit rounding. The procedure of performing arithmetic operation using n-digit chopping is done in a similar way. Let ⊙ denote any one of the basic arithmetic operations ‘+’, ‘−’, ‘×’ and ‘÷’. Let x and y be real numbers. The process of computing x ⊙ y using n-digit rounding is as follows. Step 1: Get the n-digit rounding approximation fl(x) and fl(y) of the numbers x and y, respectively. Step 2: Perform the calculation fl(x) ⊙ fl(y) using exact arithmetic. Step 3: Get the n-digit rounding approximation fl ( fl(x) ⊙ fl(y) ) of fl(x) ⊙ fl(y). The result from step 3 is the value of x ⊙ y using n-digit rounding. Example 2.1.12. Consider the function f(x) = x (√ x + 1 − √ x ) . Let us evaluate f(100000) using a six-digit rounding. We have f(100000) = 100000 (√ 100001 − √ 100000 ) . The evaluation of √ 100001 using six-digit rounding is as follows. √ 100001 ≈ 316.229347 = 0.316229347 × 103 . The six-digit rounded approximation of 0.316229347×103 is given by 0.316229×103 . Therefore, fl( √ 100001) = 0.316229 × 103 . Similarly, fl( √ 100000) = 0.316228 × 103 . The six-digit rounded approximation of the difference between these two numbers is fl ( fl( √ 100001) − fl( √ 100000) ) = 0.1 × 10−2 . S. Baskar and S. Sivaji Ganesh 32 Spring 2017-18
- 45. Section 2.2 Types of Errors Finally, we have fl(f(100000)) = fl(100000) × (0.1 × 10−2 ) = (0.1 × 106 ) × (0.1 × 10−2 ) = 100. Using six-digit chopping, the value of fl(f(100000)) is 200. Definition 2.1.13 [Machine Epsilon]. The machine epsilon of a computer is the smallest positive floating-point number δ such that fl(1 + δ) > 1. For any floating-point number ˆδ < δ, we have fl(1 + ˆδ) = 1, and 1 + ˆδ and 1 are identical within the computer’s arithmetic. Remark 2.1.14. From Example 2.1.8, it is clear that the machine epsilon for a 32-bit intel processor lies between the numbers 10−323.64 and 10−323.6. It is possible to get the exact value of this number, but it is no way useful in our present course, and so we will not attempt to do this here. 2.2 Types of Errors The approximate representation of a real number obviously differs from the actual number, whose difference is called an error. Definition 2.2.1 [Errors]. 1. The error in a computed quantity is defined as Error = True Value - Approximate Value. 2. Absolute value of an error is called the absolute error. 3. The relative error is a measure of the error in relation to the size of the true value as given by Relative Error = Error True Value . Here, we assume that the true value is non-zero. 4. The percentage error is defined as Percentage Error = 100 × Relative Error . S. Baskar and S. Sivaji Ganesh 33 Spring 2017-18
- 46. Chapter 2. Error Analysis Remark 2.2.2. Let xA denote the approximation to the real number x. We use the following notations: E(xA) := Error(xA) = x − xA. (2.8) Ea(xA) := Absolute Error(xA) = |E(xA)| (2.9) Er(xA) := Relative Error(xA) = E(xA) x , x ̸= 0. (2.10) The absolute error has to be understood more carefully because a relatively small difference between two large numbers can appear to be large, and a relatively large difference between two small numbers can appear to be small. On the other hand, the relative error gives a percentage of the difference between two numbers, which is usually more meaningful as illustrated below. Example 2.2.3. Let x = 100000, xA = 99999, y = 1 and yA = 1/2. We have Ea(xA) = 1, Ea(yA) = 1 2 . Although Ea(xA) > Ea(yA), we have Er(xA) = 10−5 , Er(yA) = 1 2 . Hence, in terms of percentage error, xA has only 10−3 % error when compared to x whereas yA has 50% error when compared to y. The errors defined above are between a given number and its approximate value. Quite often we also approximate a given function by another function that can be handled more easily. For instance, a sufficiently differentiable function can be approximated using Taylor’s theorem (Theorem 1.5.2). The error between the function value and the value obtained from the corresponding Taylor’s polynomial is defined as truncation error as defined in Definition 1.5.5. 2.3 Loss of Significance In place of relative error, we often use the concept of significant digits that is closely related to relative error. S. Baskar and S. Sivaji Ganesh 34 Spring 2017-18
- 47. Section 2.3 Loss of Significance Definition 2.3.1 [Significant β-Digits]. Let β be a radix and x ̸= 0. If xA is an approximation to x, then we say that xA approximates x to r significant β-digits if r is the largest non-negative integer such that |x − xA| |x| ≤ 1 2 β−r+1 . (2.11) We also say xA has r significant β-digits in x. Note When β = 10, we refer significant 10-digits by significant digits. Example 2.3.2. 1. For x = 1/3, the approximate number xA = 0.333 has three significant digits, since |x − xA| |x| = 0.001 < 0.005 = 0.5 × 10−2 . Thus, r = 3. 2. For x = 0.02138, the approximate number xA = 0.02144 has three significant digits, since |x − xA| |x| ≈ 0.0028 < 0.005 = 0.5 × 10−2 . Thus, r = 3. 3. For x = 0.02132, the approximate number xA = 0.02144 has two significant digits, since |x − xA| |x| ≈ 0.0056 < 0.05 = 0.5 × 10−1 . Thus, r = 2. 4. For x = 0.02138, the approximate number xA = 0.02149 has two significant digits, since |x − xA| |x| ≈ 0.0051 < 0.05 = 0.5 × 10−1 . Thus, r = 2. 5. For x = 0.02108, the approximate number xA = 0.0211 has three significant digits, since |x − xA| |x| ≈ 0.0009 < 0.005 = 0.5 × 10−2 . Thus, r = 3. S. Baskar and S. Sivaji Ganesh 35 Spring 2017-18
- 48. Chapter 2. Error Analysis 6. For x = 0.02108, the approximate number xA = 0.02104 has three significant digits, since |x − xA| |x| ≈ 0.0019 < 0.005 = 0.5 × 10−2 . Thus, r = 3. Remark 2.3.3. Number of significant digits roughly measures the number of leading non-zero digits of xA that are correct relative to the corresponding digits in the true value x. However, this is not a precise way to get the number of significant digits as it is evident from the above examples. The role of significant digits in numerical calculations is very important in the sense that the loss of significant digits may result in drastic amplification of the relative error as illustrated in the following example. Example 2.3.4. Let us consider two real numbers x = 7.6545428 = 0.76545428 × 101 and y = 7.6544201 = 0.76544201 × 101 . The numbers xA = 7.6545421 = 0.76545421 × 101 and yA = 7.6544200 = 0.76544200 × 101 are approximations to x and y, correct to seven and eight significant digits, respec- tively. The exact difference between xA and yA is zA = xA − yA = 0.12210000 × 10−3 and the exact difference between x and y is z = x − y = 0.12270000 × 10−3 . Therefore, |z − zA| |z| ≈ 0.0049 < 0.5 × 10−2 and hence zA has only three significant digits with respect to z. Thus, we started with two approximate numbers xA and yA which are correct to seven and eight significant S. Baskar and S. Sivaji Ganesh 36 Spring 2017-18
- 49. Section 2.3 Loss of Significance digits with respect to x and y, respectively. But their difference zA has only three significant digits with respect to z. Hence, there is a loss of significant digits in the process of subtraction. A simple calculation shows that Er(zA) ≈ 53581 × Er(xA). Similarly, we have Er(zA) ≈ 375067 × Er(yA). Loss of significant digits is therefore dangerous. The loss of significant digits in the process of calculation is referred to as loss of significance. The loss of significant digits results in accumulation of error during evaluation of a function value at a given point as illustrated in the following example. Example 2.3.5. Consider the function f(x) = x( √ x + 1 − √ x). From Example 2.1.12, the value of f(100000) using six-digit rounding is 100, whereas the true value is 158.113. There is a drastic error in the value of the function, which is due to the loss of significant digits. It is evident that as x increases, the terms √ x + 1 and √ x comes closer to each other and therefore loss of significance in their computed value increases. Such a loss of significance can be avoided by rewriting the given expression of f in such a way that subtraction of near-by non-negative numbers is avoided. For instance, we can rewrite the expression of the function f as f(x) = x √ x + 1 + √ x . With this new form of f, we obtain f(100000) = 158.114000 using six-digit rounding. Example 2.3.6. Consider evaluating the function f(x) = 1 − cos x near x = 0. Since cos x ≈ 1 for x near zero, there will be loss of significance in the process of evaluating f(x) for x near zero. So, we have to use an alternative formula S. Baskar and S. Sivaji Ganesh 37 Spring 2017-18
- 50. Chapter 2. Error Analysis for f(x) such as f(x) = 1 − cos x = 1 − cos2 x 1 + cos x = sin2 x 1 + cos x which can be evaluated quite accurately for small x. Remark 2.3.7. Unlike the above examples, we may not be able to always write an equivalent formula of a given function to avoid loss of significance in the evaluation. In such cases, we have to go for a suitable approximation of the given function by other functions, for instance Taylor’s polynomial of desired degree, that do not involve loss of significance. 2.4 Propagation of Relative Error in Arithmetic Operations Once an error is committed, it affects subsequent results as this error propagates through subsequent calculations. We first study how the results are affected by us- ing approximate numbers instead of actual numbers and then will take up the effect of errors on function evaluation in the next section. Let xA and yA denote the approximate numbers used in the calculation, and let xT and yT be the corresponding true values. We will now see how relative error propagates with the four basic arithmetic operations. 2.4.1 Addition and Subtraction Let xT = xA+ϵ and yT = yA+η be positive real numbers. The relative error Er(xA±yA) is given by Er(xA ± yA) = (xT ± yT ) − (xA ± yA) xT ± yT = (xT ± yT ) − (xT − ϵ ± (yT − η)) xT ± yT Upon simplification, we get Er(xA ± yA) = ϵ ± η xT ± yT . (2.12) S. Baskar and S. Sivaji Ganesh 38 Spring 2017-18
- 51. Section 2.4 Propagation of Relative Error in Arithmetic Operations The above expression shows that there can be a drastic increase in the relative error during subtraction of two approximate numbers whenever xT ≈ yT as we have witnessed in Example 2.3.4 and Example 2.3.5. On the other hand, it is easy to see from (2.12) that |Er(xA + yA)| ≤ |Er(xA)| + |Er(yA)|, which shows that the relative error propagates slowly in addition. Note that such an inequality in the case of subtraction is not possible. 2.4.2 Multiplication The relative error Er(xA × yA) is given by Er(xA × yA) = (xT × yT ) − (xA × yA) xT × yT = (xT × yT ) − ( (xT − ϵ) × (yT − η) ) xT × yT = ηxT + ϵyT − ϵη xT × yT = ϵ xT + η yT − ( ϵ xT ) ( η yT ) Thus, we have Er(xA × yA) = Er(xA) + Er(yA) − Er(xA)Er(yA). (2.13) Taking modulus on both sides, we get |Er(xA × yA)| ≤ |Er(xA)| + |Er(yA)| + |Er(xA)| |Er(yA)| Note that when |Er(xA)| and |Er(yA)| are very small, then their product is negligible when compared to |Er(xA)| + |Er(yA)|. Therefore, the above inequality reduces to |Er(xA × yA)| ⪅ |Er(xA)| + |Er(yA)|, which shows that the relative error propagates slowly in multiplication. S. Baskar and S. Sivaji Ganesh 39 Spring 2017-18
- 52. Chapter 2. Error Analysis 2.4.3 Division The relative error Er(xA/yA) is given by Er(xA/yA) = (xT /yT ) − (xA/yA) xT /yT = xT (yT − η) − yT (xT − ϵ) xT (yT − η) = ϵyT − ηxT xT (yT − η) = yT yT − η (Er(xA) − Er(yA)) Thus, we have Er(xA/yA) = 1 1 − Er(yA) ( Er(xA) − Er(yA) ) . (2.14) The above expression shows that the relative error increases drastically during division whenever Er(yA) ≈ 1. This means that yA has 100% error when compared to y, which is very unlikely because we always expect the relative error to be very small, ie., very close to zero. In this case the right hand side is approximately equal to Er(xA)−Er(yA). Hence, we have |Er(xA/yA)| ⪅ |Er(xA) − Er(yA)| ≤ |Er(xA)| + |Er(yA)|, which shows that the relative error propagates slowly in division. 2.4.4 Total Error In Subsection 2.1.4, we discussed the procedure of performing arithmetic operations using n-digit floating-point approximation. The computed value fl ( fl(x)⊙fl(y) ) involves an error (when compared to the exact value x ⊙ y) which comprises of 1. Error in fl(x) and fl(y) due to n-digit rounding or chopping of x and y, respectively, and 2. Error in fl ( fl(x)⊙fl(y) ) due to n-digit rounding or chopping of the number fl(x)⊙ fl(y). The total error is defined as (x ⊙ y) − fl ( fl(x) ⊙ fl(y) ) = { (x ⊙ y) − ( fl(x) ⊙ fl(y) )} + {( fl(x) ⊙ fl(y) ) − fl ( fl(x) ⊙ fl(y) )} , in which the first term on the right hand side is called the propagated error and the second term is called the floating-point error. The relative total error is obtained by dividing both sides of the above expression by x ⊙ y. S. Baskar and S. Sivaji Ganesh 40 Spring 2017-18
- 53. Section 2.4 Propagation of Relative Error in Arithmetic Operations Example 2.4.1. Consider evaluating the integral In = ∫ 1 0 xn x + 5 dx, for n = 0, 1, · · · , 30. The value of In can be obtained in two different iterative processes, namely, 1. The forward iteration for evaluating In is given by In = 1 n − 5In−1, I0 = ln(6/5). 2. The backward iteration for evaluating In−1 is given by In−1 = 1 5n − 1 5 In, I30 = 0.54046330 × 10−2 . The following table shows the computed value of In using both iterative formulas along with the exact value. The numbers are computed using MATLAB using double precision arithmetic and the final answer is rounded to 6 digits after the decimal point. n Forward Iteration Backward Iteration Exact Value 1 0.088392 0.088392 0.088392 5 0.028468 0.028468 0.028468 10 0.015368 0.015368 0.015368 15 0.010522 0.010521 0.010521 20 0.004243 0.007998 0.007998 25 11.740469 0.006450 0.006450 30 -36668.803026 Not Computed 0.005405 Clearly the backward iteration gives exact value up to the number of digits shown, whereas forward iteration tends to increase error and give entirely wrong values. This is due to the propagation of error from one iteration to the next iteration. In forward iteration, the total error from one iteration is magnified by a factor of 5 at the next iteration. In backward iteration, the total error from one iteration is divided by 5 at the next iteration. Thus, in this example, with each iteration, the total error tends to increase rapidly in the forward iteration and tends to increase very slowly in the backward iteration. S. Baskar and S. Sivaji Ganesh 41 Spring 2017-18
- 54. Chapter 2. Error Analysis 2.5 Propagation of Relative Error in Function Evaluation For a given function f : R → R, consider evaluating f(x) at an approximate value xA rather than at x. The question is how well does f(xA) approximate f(x)? To answer this question, we compare Er(f(xA)) with Er(xA). Assume that f is a C1 function. Using the mean-value theorem, we get f(x) − f(xA) = f′ (ξ)(x − xA), where ξ is an unknown point between x and xA. The relative error in f(xA) when compared to f(x) is given by Er(f(xA)) = f′ (ξ) f(x) (x − xA). Thus, we have Er(f(xA)) = ( f′ (ξ) f(x) x ) Er(xA). (2.15) Since xA and x are assumed to be very close to each other and ξ lies between x and xA, we may make the approximation f(x) − f(xA) ≈ f′ (x)(x − xA). In view of (2.15), we have Er(f(xA)) ≈ ( f′ (x) f(x) x ) Er(xA). (2.16) The expression inside the bracket on the right hand side of (2.16) is the amplifica- tion factor for the relative error in f(xA) in terms of the relative error in xA. Thus, this expression plays an important role in understanding the propagation of relative error in evaluating the function value f(x) and hence motivates the following defini- tion. Definition 2.5.1 [Condition Number of a Function]. The condition number of a continuously differentiable function f at a point x = c is given by f′ (c) f(c) c . (2.17) S. Baskar and S. Sivaji Ganesh 42 Spring 2017-18
- 55. Section 2.5 Propagation of Relative Error in Function Evaluation The condition number of a function at a point x = c can be used to decide whether the evaluation of the function at x = c is well-conditioned or ill-conditioned depending on whether this condition number is smaller or larger as we approach this point. It is not possible to decide a priori how large the condition number should be to say that the function evaluation is ill-conditioned and it depends on the circumstances in which we are working. Definition 2.5.2 [Well-Conditioned and Ill-Conditioned]. The process of evaluating a continuously differentiable function f at a point x = c is said to be well-conditioned if the condition number f′ (c) f(c) c at c is ‘small’. The process of evaluating a function at x = c is said to be ill- conditioned if it is not well-conditioned. In the following example, we illustrate a well-conditioned computation. Example 2.5.3. Consider the function f(x) = √ x, for all x ∈ [0, ∞). Then f′ (x) = 1 2 √ x , for all x ∈ [0, ∞). The condition number of f is f′ (x) f(x) x = 1 2 , for all x ∈ [0, ∞), which shows that taking square roots is a well-conditioned process. From (2.16), we have |Er(f(xA))| ≈ 1 2 |Er(xA)|. Thus, Er(f(xA)) is more closer to zero than Er(xA). We next illustrate an ill-conditioned computation. S. Baskar and S. Sivaji Ganesh 43 Spring 2017-18
- 56. Chapter 2. Error Analysis Example 2.5.4. Consider the function f(x) = 10 1 − x2 , for all x ∈ R. Then f′ (x) = 20x (1 − x2)2 , so that f′ (x) f(x) x = ( 20x (1 − x2)2 ) x 10 (1 − x2) = 2x2 |1 − x2| and this number can be quite large for |x| near 1. Thus, for x near 1 or -1, the process of evaluating this function is ill-conditioned. The above two examples gives us a feeling that if the process of evaluating a function is well-conditioned, then we tend to get less propagating relative error. But, this is not true in general as shown in the following example. Example 2.5.5. Consider the function f(x) = √ x + 1 − √ x, for all x ∈ (0, ∞). For all x ∈ (0, ∞), the condition number of this function is f′ (x) f(x) x = 1 2 ( 1 √ x + 1 − 1 √ x ) √ x + 1 − √ x x = 1 2 x √ x + 1 √ x ≤ 1 2 , (2.18) which shows that the process of evaluating f is well-conditioned for all x ∈ (0, ∞). S. Baskar and S. Sivaji Ganesh 44 Spring 2017-18
- 57. Section 2.5 Propagation of Relative Error in Function Evaluation But, if we calculate f(12345) using six-digit rounding, we find f(12345) = √ 12346 − √ 12345 = 111.113 − 111.108 = 0.005, while, actually, f(12345) = 0.00450003262627751 · · · . The calculated answer has 10% error. The above example shows that a well-conditioned process of evaluating a function at a point is not enough to ensure the accuracy in the corresponding computed value. We need to check for the stability of the computation, which we discuss in the following subsection. 2.5.1 Stable and Unstable Computations Suppose there are n steps to evaluate a function f(x) at a point x = c. Then the total process of evaluating this function is said to have instability if at least one of the n steps is ill-conditioned. If all the steps are well-conditioned, then the process is said to be stable. Example 2.5.6. We continue the discussion in Example 2.5.5 and check the stability in evaluating the function f. Let us analyze the computational process. The function f consists of the following four computational steps in evaluating the value of f at x = x0: x1 := x0 + 1, x2 := √ x1, x3 := √ x0, x4 := x2 − x3. Now consider the last two steps where we already computed x2 and now going to compute x3 and finally evaluate the function f4(t) := x2 − t. At this step, the condition number for f4 is given by f′ 4(t) f4(t) t = t x2 − t . Thus, f4 is ill-conditioned when t approaches x2. Therefore, the above process of evaluating the function f(x) is unstable. S. Baskar and S. Sivaji Ganesh 45 Spring 2017-18
- 58. Chapter 2. Error Analysis Let us rewrite the same function f(x) as ˜f(x) = 1 √ x + 1 + √ x . The computational process of evaluating ˜f at x = x0 is x1 := x0 + 1, x2 := √ x1, x3 := √ x0, x4 := x2 + x3, x5 := 1/x4. It is easy to verify that the condition number of each of the above steps is well- conditioned. For instance, the last step defines ˜f5(t) = 1 x2 + t , and the condition number of this function is approximately, ˜f′ 5(x) ˜f5(x) x = t x2 + t ≈ 1 2 for t sufficiently close to x2. Therefore, this process of evaluating ˜f(x) is stable. Recall from Example 2.3.5 that the above expression gives a more accurate value for sufficiently large x. Remark 2.5.7. As discussed in Remark 2.3.7, we may not be lucky all the time to come out with an alternate expression that lead to stable evaluation for any given function when the original expression leads to unstable evaluation. In such situations, we have to compromise and go for a suitable approximation with other functions with stable evaluation process. For instance, we may try approximating the given function with its Taylor’s polynomial, if possible. S. Baskar and S. Sivaji Ganesh 46 Spring 2017-18
- 59. Section 2.6 Exercises 2.6 Exercises Floating-Point Approximation 1. Let X be a sufficiently large number which result in an overflow of memory on a computing device. Let x be a sufficiently small number which result in underflow of memory on the same computing device. Then give the output of the following operations: (i) x × X (ii) 3 × X (iii) 3 × x (iv) x/X (v) X/x. 2. In the following problems, show all the steps involved in the computation. i) Using 5-digit rounding, compute 37654 + 25.874 − 37679. ii) Let a = 0.00456, b = 0.123, c = −0.128. Using 3-digit rounding, compute (a + b) + c, and a + (b + c). What is your conclusion? iii) Let a = 2, b = −0.6, c = 0.602. Using 3-digit rounding, compute a × (b + c), and (a × b) + (a × c). What is your conclusion? 3. To find the mid-point of an interval [a, b], the formula a+b 2 is often used. Compute the mid-point of the interval [0.982, 0.987] using 3-digit chopping. On the number line represent all the three points. What do you observe? Now use the more geometric formula a + b−a 2 to compute the mid-point, once again using 3-digit chopping. What do you observe this time? Why is the second formula more geometric? 4. In a computing device that uses n-digit rounding binary floating-point arithmetic, show that δ = 2−n is the machine epsilon. What is the machine epsilon in a computing device that uses n-digit rounding decimal floating-point arithmetic? Justify your answer. Types of Errors 5. If fl(x) is the approximation of a real number x in a computing device, and ϵ is the corresponding relative error, then show that fl(x) = (1 − ϵ)x. 6. Let x, y and z be real numbers whose floating point approximations in a com- puting device coincide with x, y and z respectively. Show that the relative er- ror in computing x(y + z) equals ϵ1 + ϵ2 − ϵ1ϵ2, where ϵ1 = Er(fl(y + z)) and ϵ2 = Er(fl(x × fl(y + z))). 7. Let ϵ = Er(fl(x)). Show that i) |ϵ| ≤ 10−n+1 if the computing device uses n-digit (decimal) chopping. ii) |ϵ| ≤ 1 2 10−n+1 if the computing device uses n-digit (decimal) rounding. iii) Can the equality hold in the above inequalities? S. Baskar and S. Sivaji Ganesh 47 Spring 2017-18
- 60. Chapter 2. Error Analysis 8. For small values of x, the approximation sin x ≈ x is often used. Estimate the error in using this formula with the aid of Taylor’s theorem. For what range of values of x will this approximation give an absolute error of at most 1 2 10−6 ? 9. Let xA = 3.14 and yA = 2.651 be obtained from xT and yT using 4-digit rounding. Find the smallest interval that contains (i) xT (ii) yT (iii) xT + yT (iv) xT − yT (v) xT × yT (vi) xT /yT . 10. The ideal gas law is given by PV = nRT where R is the gas constant. We are interested in knowing the value of T for which P = V = n = 1. If R is known only approximately as RA = 8.3143 with an absolute error at most 0.12 × 10−2 . What is the relative error in the computation of T that results in using RA instead of R? Loss of Significance and Propagation of Error 11. Obtain the number of significant digits in xA when compared to x for the following cases: (i) x = 451.01 and xA = 451.023 (ii) x = −0.04518 and xA = −0.045113 (iii) x = 23.4604 and x = 23.4213. 12. Instead of using the true values xT = 0.71456371 and yT = 0.71456238 in cal- culating zT = xT − yT (= 0.133 × 10−5 ), if we use the approximate values xA = 0.71456414 and yA = 0.71456103, and calculate zA = xA − yA(= 0.311 × 10−5 ), then find the loss of significant digits in the process of calculating zA when com- pared to the significant digits in xA. 13. Let x < 0 < y be such that the approximate numbers xA and yA has seven and nine significant digits with x and y respectively. Show that zA := xA − yA has at least six significant digits when compared to z := x − y. Propagation of Relative Error 14. Let x = 0.65385 and y = 0.93263. Obtain the total error in obtaining the product of these two numbers using 3-digit rounding calculation. 15. Let f : R → R and g : R → R be continuously differentiable functions such that • there exists constant M > 0 such that |f′ (x)| ≥ M and |g′ (x)| ≤ M for all x ∈ R, • the process of evaluating f is well-conditioned, and • the process of evaluating g is ill-conditioned. Show that |g(x)| < |f(x)| for all x ∈ R. S. Baskar and S. Sivaji Ganesh 48 Spring 2017-18
- 61. Section 2.6 Exercises 16. Find the condition number at a point x = c for the following functions (i) f(x) = x2 , (ii) g(x) = πx , (iii) h(x) = bx . 17. Let xT be a real number. Let xA = 2.5 be an approximate value of xT with an absolute error at most 0.01. The function f(x) = x3 is evaluated at x = xA instead of x = xT . Estimate the resulting absolute error. 18. Is the process of computing the function f(x) = (ex − 1)/x stable or unstable for x ≈ 0? Justify your answer. 19. Show that the process of evaluating the function f(x) = 1 − cos x x2 for x ≈ 0 is unstable. Suggest an alternate formula for evaluating f for x ≈ 0, and check if the computation process is stable using this new formula. 20. Check for stability of computing the function f(x) = ( x + 1 3 ) − ( x − 1 3 ) for large values of x. 21. Check for stability of computing the function g(x) = ( 3 + x2 3 ) − ( 3 − x2 3 ) x2 for values of x very close to 0. 22. Check for stability of computing the function h(x) = sin2 x 1 − cos2 x for values of x very close to 0. 23. Compute the values of the function sin x √ 1 − sin2 x using your calculator (without simplifying the given formula) at x = 89.9, 89.95, 89.99 degrees. Also compute the values of the function tan x at these values of x and compare the number of significant digits. S. Baskar and S. Sivaji Ganesh 49 Spring 2017-18
- 63. CHAPTER 3 Numerical Linear Algebra In this chapter, we study the methods for solving system of linear equations, and computing an eigenvalue and the corresponding eigen vector for a matrix. The methods for solving linear systems are categorized into two types, namely, the direct methods and the iterative methods. Theoretically, direct methods give exact solution of a linear system and therefore these methods do not involve mathematical error. However, when we implement the direct methods on a computer, because of the presence of arithmetic error, the computed value from a computer will still be an approximate solution. On the other hand, an iterative method generates a sequence of approximate solutions to a given linear system which is expected to converge to the exact solution. An important direct method is the well-known Gaussian elimination method. After a short introduction to linear systems in Section 3.1, we discuss the direct methods in Section 3.2. We recall the Gaussian elimination method in Subsection 3.2.1 and study the effect of arithmetic error in the computed solution. We further count the number of arithmetic operations involved in computing the solution. This operation count revels that this method is more expensive in terms of computational time. In particular, when the given system is of tridiagonal structure, the Gaussian elimination method can be suitably modified so that the resulting method, called the Thomas algorithm, is more efficient in terms of computational time. After introducing the Thomal algorithm in Subsection 3.2.4, we discuss the LU factorization methods for a given matrix and solving a system of linear equation using LU factorization in Subsection 3.2.5. Some matrices are sensitive to even a small error in the right hand side vector of a linear system. Such a matrix can be identified with the help of the condition number of S. Baskar and S. Sivaji Ganesh 51 Spring 2017-18
- 64. Chapter 3. Numerical Linear Algebra the matrix. The condition number of a matrix is defined in terms of the matrix norm. In Section 3.3, we introduce the notion of matrix norms, define condition number of a matrix, and discuss few important theorems that are used in the error analysis of iterative methods. We continue the chapter with the discussion of iterative methods to linear system in Section 3.4, where we introduce two basic iterative methods and discuss the sufficient condition under which the methods converge. We end this section with the definition of the residual error and another iterative method called the residual corrector method. Finally in Section 3.5 we discuss the power method, which is used to capture the domi- nant eigenvalue and a corresponding eigenvectors of a given matrix. We end the chapter with the Gerschgorin’s Theorem and its application to power method. 3.1 System of Linear Equations General form of a system of n linear equations in n variables is a11x1 + a12x2 + · · · + a1nxn = b1, a21x1 + a22x2 + · · · + a2nxn = b2, · · · · · · · · · an1x1 + an2x2 + · · · + annxn = bn. (3.1) Throughout this chapter, we assume that the coefficients aij and the right hand side numbers bi, i, j = 1, 2, · · · , n are real. The above system of linear equations can be written in the matrix notation as a11 a12 · · · a1n a21 a22 · · · a2n ... ... · · · ... an1 an2 · · · ann x1 x2 ... xn = b1 b2 ... bn (3.2) The last equation is usually written in the short form Ax = b, (3.3) where A stands for the n × n matrix with entries aij, x = (x1, x2, · · · , xn)T and the right hand side vector b = (b1, b2, · · · , bn)T . Let us now recall a result from linear algebra concerning the solvability of the system (3.2). S. Baskar and S. Sivaji Ganesh 52 Spring 2017-18
- 65. Section 3.2 Direct Methods for Linear Systems Theorem 3.1.1. Let A be an n×n matrix and b ∈ Rn . Then the following statements concerning the system of linear equations Ax = b are equivalent. 1. det(A) ̸= 0 2. For each right hand side vector b, the system Ax = b has a unique solution x. 3. For b = 0, the only solution of the system Ax = b is x = 0. Note We always assume that the coefficient matrix A is invertible. Any discussion of what happens when A is not invertible is outside the scope of this course. 3.2 Direct Methods for Linear Systems In this section, we discuss two direct methods namely, the Gaussian elimination method and the LU factorization method. We also introduce the Thomas algorithm, which is a particular case of the Gaussian elimination method for tridiagonal systems. 3.2.1 Naive Gaussian Elimination Method Let us describe the Gaussian elimination method to solve a system of linear equations in three variables. The method for a general system is similar. Consider the following system of three linear equations in three variables x1, x2, x3: a11x1 + a12x2 + a13x3 = b1, a21x1 + a22x2 + a23x3 = b2, a31x1 + a32x2 + a33x3 = b3. (3.4) For convenience, we denote the first, second, and third equations by E1, E2, and E3, respectively. Step 1: If a11 ̸= 0, then define m21 = a21 a11 , m31 = a31 a11 . (3.5) We will now obtain a new system that is equivalent to the system (3.4) as follows: S. Baskar and S. Sivaji Ganesh 53 Spring 2017-18
- 66. Chapter 3. Numerical Linear Algebra • Retain the first equation E1 as it is. • Replace the second equation E2 by the equation E2 − m21E1. • Replace the third equation E3 by the equation E3 − m31E1. The new system equivalent to (3.4) is given by a11x1 + a12x2 + a13x3 = b1 0 + a (2) 22 x2 + a (2) 23 x3 = b (2) 2 0 + a (2) 32 x2 + a (2) 33 x3 = b (2) 3 , (3.6) where the coefficients a (2) ij , and b (2) k are given by a (2) ij = aij − mi1a1j, i, j = 2, 3, b (2) k = bk − mk1b1, k = 2, 3. } (3.7) Note that the variable x1 has been eliminated from the last two equations. Step 2: If a (2) 22 ̸= 0, then define m32 = a (2) 32 a (2) 22 . (3.8) We still use the same names E1, E2, E3 for the first, second, and third equations of the modified system (3.6), respectively. We will now obtain a new system that is equivalent to the system (3.6) as follows: • Retain the first two equations in (3.6) as they are. • Replace the third equation by the equation E3 − m32E2. The new system is given by a11x1 + a12x2 + a13x3 = b1 0 + a (2) 22 x2 + a (2) 23 x3 = b (2) 2 0 + 0 + a (3) 33 x3 = b (3) 3 , (3.9) where the coefficient a (3) 33 , and b (3) 3 are given by a (3) 33 = a (2) 33 − m32a (2) 23 , b (3) 3 = b (2) 3 − m32b (2) 2 . Note that the variable x2 has been eliminated from the last equation. This phase of the (Naive) Gaussian elimination method is called forward elimination phase. Step 3: Observe that the system (3.9) is readily solvable for x3 if the coefficient a (3) 33 ̸= 0. Substituting the value of x3 in the second equation of (3.9), we can solve S. Baskar and S. Sivaji Ganesh 54 Spring 2017-18
- 67. Section 3.2 Direct Methods for Linear Systems for x2. Substituting the values of x1 and x2 in the first equation, we can solve for x1. This solution phase of the (Naive) Gaussian elimination method is called backward substitution phase. Note The coefficient matrix of the system (3.9) is an upper triangular matrix given by U = a11 a12 a13 0 a (2) 22 a (2) 23 0 0 a (3) 33 . (3.10) Define a lower triangular matrix L by L = 1 0 0 m21 1 0 m31 m32 1 . (3.11) It is easy to verify that LU = A. Remark 3.2.1 [Why Naive?]. We explain why the word “Naive” is used for the method described above. 1. First of all, we do not know if the method described here can be successfully applied for all systems of linear equations which are uniquely solvable (that is, the coefficient matrix is invertible). 2. Secondly, even when we apply the method successfully, it is not clear if the computed solution is the exact solution. In fact, it is not even clear that the computed solution is close to the exact solution. We illustrate the above points in Example 3.2.2 and Example 3.2.3. Example 3.2.2. Consider the system of equations ( 0 1 1 1 ) ( x1 x2 ) = ( 1 2 ) . (3.12) The Step 1 cannot be started as a11 = 0. Thus the naive Gaussian elimination method fails. S. Baskar and S. Sivaji Ganesh 55 Spring 2017-18
- 68. Chapter 3. Numerical Linear Algebra Example 3.2.3. Let 0 < ϵ ≪ 1. Consider the system of equations ( ϵ 1 1 1 ) ( x1 x2 ) = ( 1 2 ) . (3.13) Since ϵ ̸= 0, after Step 1 of the Naive Gaussian elimination method, we get the system ( ϵ 1 0 1 − ϵ−1 ) ( x1 x2 ) = ( 1 2 − ϵ−1 ) . (3.14) Using backward substitution, the solution is obtained as x2 = 2 − ϵ−1 1 − ϵ−1 , x1 = (1 − x2)ϵ−1 . (3.15) Note that for a sufficiently small ϵ, the computer evaluates 2−ϵ−1 as −ϵ−1 , and 1−ϵ−1 also as −ϵ−1 . Thus, x2 ≈ 1 and as a consequence x1 ≈ 0. However the exact/correct solution is given by x1 = 1 1 − ϵ ≈ 1, x2 = 1 − 2ϵ 1 − ϵ ≈ 1. (3.16) Thus, in this particular example, the solution obtained by the naive Gaussian elimi- nation method is completely wrong. To understand this example better, we instruct the reader to solve the system (3.13) for the cases (1) ϵ = 10−3 , and (2) ϵ = 10−5 using 3-digit rounding. In the following example, we illustrate the need of a pivoting strategy in the Gaussian elimination method. Example 3.2.4. Consider the linear system 6x1 + 2x2 + 2x3 = −2 2x1 + 2 3 x2 + 1 3 x3 = 1 x1 + 2x2 − x3 = 0. (3.17) Let us solve this system using (naive) Gaussian elimination method using 4-digit rounding. S. Baskar and S. Sivaji Ganesh 56 Spring 2017-18
- 69. Section 3.2 Direct Methods for Linear Systems In 4-digit rounding approximation, the above system takes the form 6.000x1 + 2.000x2 + 2.000x3 = −2.000 2.000x1 + 0.6667x2 + 0.3333x3 = 1.000 1.000x1 + 2.000x2 − 1.000x3 = 0.000 After eliminating x1 from the second and third equations, we get (with m21 = 0.3333, m31 = 0.1667) 6.000x1 + 2.000x2 + 2.000x3 = −2.000 0.000x1 + 0.0001x2 − 0.3333x3 = 1.667 0.000x1 + 1.667x2 − 1.333x3 = 0.3334 (3.18) After eliminating x2 from the third equation, we get (with m32 = 16670) 6.000x1 + 2.000x2 + 2.000x3 = −2.000 0.000x1 + 0.0001x2 − 0.3333x3 = 1.667 0.000x1 + 0.0000x2 + 5555x3 = −27790 Using back substitution, we get x1 = 1.335, x2 = 0 and x3 = −5.003, whereas the actual solution is x1 = 2.6, x2 = −3.8 and x3 = −5. The difficulty with this elimination process is that the second equation in (3.18), where the coefficient of x2 should have been zero, but rounding error prevented it and made the relative error very large. The above examples highlight the inadequacy of the Naive Gaussian elimination method. These inadequcies can be overcome by modifying the procedure of Naive Gaussian elim- ination method. There are many kinds of modification. We will discuss one of the most popular modified methods which is called modified Gaussian elimination method with partial pivoting. 3.2.2 Modified Gaussian Elimination Method with Partial Pivoting Consider the following system of three linear equations in three variables x1, x2, x3: a11x1 + a12x2 + a13x3 = b1 a21x1 + a22x2 + a23x3 = b2 a31x1 + a32x2 + a33x3 = b3. (3.19) For convenience, we call the first, second, and third equations by names E1, E2, and E3 respectively. S. Baskar and S. Sivaji Ganesh 57 Spring 2017-18
- 70. Chapter 3. Numerical Linear Algebra Step 1: Define s1 = max { |a11|, |a21|, |a31| }. Note that s1 ̸= 0 (why?). Let k be the least number such that s1 = |ak1|. Interchange the first equation and the kth equation. Let us re-write the system after this modification. a (1) 11 x1 + a (1) 12 x2 + a (1) 13 x3 = b (1) 1 a (1) 21 x1 + a (1) 22 x2 + a (1) 23 x3 = b (1) 2 a (1) 31 x1 + a (1) 32 x2 + a (1) 33 x3 = b (1) 3 . (3.20) where a (1) 11 = ak1, a (1) 12 = ak2, a (1) 13 = ak3, a (1) k1 = a11, a (1) k2 = a12, a (1) k3 = a13; b (1) 1 = bk, b (1) k = b1, (3.21) and rest of the coefficients a (1) ij are same as aij as all equations other than the first and kth remain untouched by the interchange of first and kth equation. Now eliminate the x1 variable from the second and third equations of the system (3.20). Define m21 = a (1) 21 a (1) 11 , m31 = a (1) 31 a (1) 11 . (3.22) We will now obtain a new system that is equivalent to the system (3.20) as follows: • The first equation will be retained as it is. • Replace the second equation by the equation E2 − m21E1. • Replace the third equation by the equation E3 − m31E1. The new system is given by a (1) 11 x1 + a (1) 12 x2 + a (1) 13 x3 = b (1) 1 0 + a (2) 22 x2 + a (2) 23 x3 = b (2) 2 0 + a (2) 32 x2 + a (2) 33 x3 = b (2) 3 , (3.23) where the coefficients a (2) ij , and b (2) k are given by a (2) ij = a (1) ij − mi1a (1) 1j , i, j = 2, 3 b (2) i = b (1) i − mi1b (1) 1 , i = 2, 3. Note that the variable x1 has been eliminated from the last two equations. Step 2: Define s2 = max { |a (2) 22 |, |a (2) 32 | } . Note that s2 ̸= 0 (why?). Let l be the least number such that sl = |a (2) l2 |. Interchange the second row and the lth rows. Let us S. Baskar and S. Sivaji Ganesh 58 Spring 2017-18
- 71. Section 3.2 Direct Methods for Linear Systems rewrite the system after this modification. a (1) 11 x1 + a (1) 12 x2 + a (1) 13 x3 = b (1) 1 0 + a (3) 22 x2 + a (3) 23 x3 = b (3) 2 0 + a (3) 32 x2 + a (3) 33 x3 = b (3) 3 , (3.24) where the coefficients a (3) ij , and b (3) i are given by a (3) 22 = a (2) l2 , a (3) 23 = a (2) l3 , a (3) l2 = a (2) 22 , a (3) l3 = a (2) 23 , b (3) 2 = b (2) l , b (3) l = b (2) 2 We still use the same names E1, E2, E3 for the first, second, and third equations of the modified system (3.24), respectively. In case l = 2, both second and third equations stay as they are. Let us now eliminate x2 from the last equation. Define m32 = a (3) 32 a (3) 22 (3.25) We will now obtain a new system that is equivalent to the system (3.24) as follows: • The first two equations in (3.24) will be retained as they are. • Replace the third equation by the equation E3 − m32E2. The new system is given by a (1) 11 x1 + a (1) 12 x2 + a (1) 13 x3 = b (1) 1 0 + a (3) 22 x2 + a (3) 23 x3 = b (3) 2 0 + 0 + a (4) 33 x3 = b (4) 3 , (3.26) where the coefficient a (4) 33 , and b (4) 3 are given by a (4) 33 = a (3) 33 − m32a (3) 23 , b (4) 3 = b (3) 3 − m32b (3) 2 . Note that the variable x2 has been eliminated from the last equation. This phase of the modified Gaussian elimination method is called forward elimination phase with partial pivoting. Step 3: Now the system (3.26) is readily solvable for x3 if the coefficient a (4) 33 ̸= 0. In fact, it is non-zero (why?). Substituting the value of x3 in the second equation of (3.26), we can solve for x2. Substituting the values of x1 and x2 in the first equation, we can solve for x1. This solution phase of the modified Gaussian elimination method with partial pivoting is called backward substitution phase. S. Baskar and S. Sivaji Ganesh 59 Spring 2017-18
- 72. Chapter 3. Numerical Linear Algebra 3.2.3 Operations Count in Naive Gaussian Elimination Method It is important to know the length of a computation and for that reason, we count the number of arithmetic operations involved in Gaussian elimination method. Let us divide the count into three parts. 1. The elimination step: We now count the additions/subtractions, multiplica- tions and divisions in going from the given system to the triangular system. Step Additions/Subtractions Multiplications Divisions 1 (n − 1)2 (n − 1)2 n − 1 2 (n − 2)2 (n − 2)2 n − 2 . . . . . . . . . . . . n − 1 1 1 1 Total n(n−1)(2n−1) 6 n(n−1)(2n−1) 6 n(n−1) 2 Here we use the formula p ∑ j=1 j = p(p + 1) 2 , p ∑ j=1 j2 = p(p + 1)(2p + 1) 6 , p ≥ 1. Let us explain the first row of the above table. In the first step, computation of m21, m31, · · · , mn1 involve (n − 1) divisions. For each i, j = 2, 3, · · · , n, the computation of a (2) ij involves a multiplication and a subtraction. In total, there are (n − 1)2 multiplications and (n − 1)2 subtractions. Note that we do not count the operations involved in computing the coefficients of x1 in the 2nd to nth equa- tions (namely, a (2) i1 ), as we do not compute them and simply take them as zero. Similarly, other entries in the above table can be accounted for. 2. Modification of the right side: Proceeding as before, we get Addition/Subtraction = (n − 1) + (n − 2) + · · · + 1 = n(n − 1) 2 Multiplication/Division = (n − 1) + (n − 2) + · · · + 1 = n(n − 1) 2 3. The back substitution: Addition/Subtraction = (n − 1) + (n − 2) + · · · + 1 = n(n − 1) 2 Multiplication/Division = n + (n − 1) + · · · + 1 = n(n + 1) 2 S. Baskar and S. Sivaji Ganesh 60 Spring 2017-18
- 73. Section 3.2 Direct Methods for Linear Systems Total number of operations: The total number of operations in obtaining x is Addition/Subtraction = n(n − 1)(2n − 1) 6 + n(n − 1) 2 + n(n − 1) 2 = n(n − 1)(2n + 5) 6 Multiplication/Division = n(n2 + 3n − 1) 3 Even if we take only multiplication and division into consideration, we see that for large value of n, the operation count required for Gaussian elimination is about 1 3 n3 . This means that as n doubled, the cost of solving the linear system goes up by a factor of 8. In addition, most of the cost of Gaussian elimination is in the elimination step. For elimination, we have Multiplication/Division = n(n − 1)(2n − 1) 6 + n(n − 1) 2 = n3 3 ( 1 − 1 n2 ) ≈ 1 3 n3 . The operation count in Elimination step = O(n3 ), RHS modification = O(n2 ), Backward substitution = O(n2 ), as n → ∞. Hence, once the elimination part is completed, it is much less expensive to solve the linear system. 3.2.4 Thomas Method for Tri-diagonal System The Gaussian elimination method can be simplified in the case of a tri-diagonal system so as to increase the efficiency. The resulting simplified method is called the Thomas method. A tri-diagonal system of linear equations is of the form β1x1 +γ1x2 +0x3 +0x4 +0x5 +0x6 + · · · +0xn−2 +0xn−1 +0xn = b1 α2x1 +β2x2 +γ2x3 +0x4 +0x5 +0x6 + · · · +0xn−2 +0xn−1 +0xn = b2 0x1 +α3x2 +β3x3 +γ3x4 +0x5 +0x6 + · · · +0xn−2 +0xn−1 +0xn = b3 0x1 +0x2 +α4x3 +β4x4 +γ4x5 +0x6 + · · · +0xn−2 +0xn−1 +0xn = b4 · · · · · · · · · 0x1 +0x2 +0x3 +0x4 +0x5 +0x6 + · · · +αn−1xn−2 +βn−1xn−1 +γn−1xn = bn−1 0x1 +0x2 +0x3 +0x4 +0x5 +0x6 + · · · +0xn−2 +αnxn−1 +βnxn = bn (3.27) Here, we assume that all αi, i = 2, 3, · · · , n, βi, i = 1, 2, · · · , n, and γi, i = 1, 2, · · · , n−1 are non-zero. S. Baskar and S. Sivaji Ganesh 61 Spring 2017-18
- 74. Chapter 3. Numerical Linear Algebra The first equation of the system reads β1x1 + γ1x2 = b1. This equation can be rewritten as x1 + e1x2 = f1, e1 = γ1 β1 , f1 = b1 β1 . Eliminating x1 from the second equation of the system (3.27) by multiplying the above equation by α2 and subtracting the resulting equation with the second equation of (3.27), we get x2 + e2x3 = f2, e2 = γ2 β2 − α2e1 , f2 = b2 − α2f1 β2 − α2e1 . We now generalize the above procedure by assuming that the jth equation is reduced to the form xj + ejxj+1 = fj, where ej and fj are known quantity, reduce the (j + 1)th equation in the form xj+1 + ej+1xj+2 = fj+1, ej+1 = γj+1 βj+1 − αj+1ej , fj+1 = bj+1 − αj+1fj βj+1 − αj+1ej , for j = 1, 2, · · · , n−2. Note that for j = n−2, we obtain the j +1 = (n−1)th equation as xn−1 + en−1xn = fn−1, en−1 = γn−1 βn−1 − αn−1en−2 , fn−1 = bn−1 − αn−1fn−2 βn−1 − αn−1en−2 . To obtain the reduced form of the nth equation of the system (3.27), eliminate xn−1 from the nth equation by multiplying the above equation by αn and subtracting the resulting equation with the nth equation of (3.27), which gives (αnen−1 − βn)xn = αnfn−1 − bn. Thus, the given tri-diagonal system (3.27) is now reduced to the system x1 +e1x2 +0x3 +0x4 +0x5 +0x6 + · · · +0xn−2 +0xn−1 +0xn = f1 0x1 +x2 +e2x3 +0x4 +0x5 +0x6 + · · · +0xn−2 +0xn−1 +0xn = f2 · · · · · · · · · 0x1 +0x2 +0x3 +0x4 +0x5 +0x6 + · · · +0xn−2 +xn−1 +en−1xn = fn−1 0x1 +0x2 +0x3 +0x4 +0x5 +0x6 + · · · +0xn−2 +0xn−1 +(αnen−1 − βn)xn = αnfn−1 − bn which is an upper triangular matrix and hence, by back substitution we can get the solution. S. Baskar and S. Sivaji Ganesh 62 Spring 2017-18
- 75. Section 3.2 Direct Methods for Linear Systems Remark 3.2.5. If the denominator of any of the ej’s or fj’s is zero, then the Thomas method fails. This is the situation when βj − αjej−1 = 0 which is the coefficient of xj in the reduced equation. A suitable partial pivoting as done in the modified Gaussian elimination method may sometime help us to overcome this problem. 3.2.5 LU Factorization In Theorem 3.1.1, we have stated that when a matrix is invertible, then the correspond- ing linear system can be solved. Let us now ask the next question: ‘Can we give examples of a class(es) of invertible matrices for which the system of linear equations (3.2) given by a11 a12 · · · a1n a21 a22 · · · a2n ... ... · · · ... an1 an2 · · · ann x1 x2 ... xn = b1 b2 ... bn is “easily” solvable?’ There are three types of matrices whose simple structure makes the linear system solv- able “readily”. These matrices are as follows: 1. Invertible Diagonal matrices: These matrices look like d1 0 0 · · · 0 0 d2 0 · · · 0 ... ... · · · · · · ... 0 0 0 · · · dn and di ̸= 0 for each i = 1, 2, · · · , n. In this case, the solution x = (x1, x2, · · · , xn)T is given by x = ( b1 d1 , b2 d2 , · · · , bn dn )T . 2. Invertible Lower triangular matrices: These matrices look like l11 0 0 · · · 0 l21 l22 0 · · · 0 ... ... · · · · · · ... ln1 ln2 ln3 · · · lnn S. Baskar and S. Sivaji Ganesh 63 Spring 2017-18
- 76. Chapter 3. Numerical Linear Algebra and lii ̸= 0 for each i = 1, 2, · · · , n. The linear system takes the form l11 0 0 · · · 0 l21 l22 0 · · · 0 ... ... · · · · · · ... ln1 ln2 ln3 · · · lnn x1 x2 ... xn = b1 b2 ... bn (3.28) From the first equation, we solve for x1 given by x1 = b1 l11 . Substituting this value of x1 in the second equation, we get the value of x2 as x2 = b2 − l21 b1 l11 l22 . Proceeding in this manner, we solve for the vector x. This procedure of obtaining solution may be called the forward substitution. 3. Invertible Upper triangular matrices: These matrices look like u11 u12 u13 · · · u1n 0 u22 u23 · · · u2n ... ... · · · · · · ... 0 0 0 · · · unn and uii ̸= 0 for each i = 1, 2, · · · , n. The linear system takes the form u11 u12 u13 · · · u1n 0 u22 u23 · · · u2n ... ... · · · · · · ... 0 0 0 · · · unn x1 x2 ... xn = b1 b2 ... bn (3.29) From the last equation, we solve for xn given by xn = bn unn . Substituting this value of xn in the penultimate equation, we get the value of xn−1 as xn−1 = bn−1 − un−1,n bn unn un−1,n−1 . Proceeding in this manner, we solve for the vector x. This procedure of obtaining solution may be called the backward substitution. S. Baskar and S. Sivaji Ganesh 64 Spring 2017-18
- 77. Section 3.2 Direct Methods for Linear Systems In general, an invertible matrix A need not be one among the simple structures listed above. However, in certain situations we can always find an invertible lower triangular matrix L and an invertible upper triangular matrix U in such a way that A = LU. In this case, the system Ax = b becomes L ( Ux ) = b. To solve for x, we first solve the lower triangular system Lz = b for the vector z, which can be obtained easily using forward substitution. After ob- taining z, we solve the upper triangular system Ux = z for the vector x, which is again obtained easily using backward substitution. Remark 3.2.6. In Gaussian elimination method discussed in Section 3.2.1, we have seen that a given matrix A can be reduced to an upper triangular matrix U by an elimination procedure and thereby can be solved using backward substitution. In the elim- ination procedure, we also obtained a lower triangular matrix L in such a way that A = LU as remarked in this section. The above discussion motivates us to look for a LU factorization of a given ma- trix. Definition 3.2.7 [LU factorization]. A matrix A is said to have LU factorization (or LU decomposition) if there exists a lower triangular matrix L and an upper triangular matrix U such that A = LU. S. Baskar and S. Sivaji Ganesh 65 Spring 2017-18
- 78. Chapter 3. Numerical Linear Algebra Remark 3.2.8. Clearly if a matrix has an LU decomposition, the matrices L and U are not unique as A = LU = (LD)(D−1 U) = ˜L ˜U for any invertible diagonal matrix D. Note that A = ˜L ˜U is also an LU decompo- sition of A as ˜L is a lower triangular matrix and ˜U is an upper triangular matrix. We discuss three different ways of constructing LU factorization of a given matrix. Doolittle’s factorization Definition 3.2.9 [Doolittle’s factorization]. A matrix A is said to have a Doolittle’s factorization if there exists a lower triangular matrix L with all diagonal elements as 1, and an upper triangular matrix U such that A = LU. We now state the sufficient condition under which the Doolittle’s factorization of a given matrix exists. For this, we need the notion of leading principal minors of a given matrix, which we define first and then state the required theorem. Definition 3.2.10 [Principal Minors of a Matrix]. Let A be an n × n matrix. 1. A sub-matrix of order k (< n) of the matrix A is a k × k matrix obtained by removing n − k rows and n − k columns from A. The determinant of such a sub-matrix of order k of A is called a minor of order k of the matrix A. 2. The principal sub-matrix of order k of the matrix A is obtained by removing the last n − k rows and the last n − k columns from A. The determinant of the leading principal sub-matrix of order k of A is called the principal minor of order k of the matrix A. 3. A principal sub-matrix and the corresponding principal minor are called the leading principal sub-matrix and the leading principal minor of order k, respectively, if k < n. S. Baskar and S. Sivaji Ganesh 66 Spring 2017-18
- 79. Section 3.2 Direct Methods for Linear Systems Example 3.2.11. Consider the 3 × 3 matrix A = a11 a12 a13 a21 a22 a23 a31 a32 a33 . There are two leading principal minors corresponding to the matrix A. • The leading principal minor of order 1 of A is |a11| = a11. • The leading principal minor of order 2 of A is a11 a12 a21 a22 = a11a22 − a12a21. We now state the sufficient condition for the existence of a Doolittle factorization of a given matrix. Theorem 3.2.12. Let n ≥ 2, and A be an n × n invertible matrix such that all of its first n − 1 leading principal minors are non-zero. Then A has an LU-decomposition where L is a unit lower triangular matrix (i.e. all its diagonal elements equal 1). That is, A has a Doolittle’s factorization. We omit the proof of this theorem but illustrate the construction procedure of a Doolit- tle factorization in the case when n = 3. Example 3.2.13 [Computing the Doolittle’s factorization for a 3 × 3 matrix]. Let us illustrate the direct computation of the factors L and U of a 3 × 3 matrix A, whenever its leading principal minors of order 1, 2, and 3 are non-zero. Write A = LU as a11 a12 a13 a21 a22 a23 a31 a32 a33 = 1 0 0 l21 1 0 l31 l32 1 u11 u12 u13 0 u22 u23 0 0 u33 (3.30) S. Baskar and S. Sivaji Ganesh 67 Spring 2017-18
- 80. Chapter 3. Numerical Linear Algebra The right hand matrix multiplication yields a11 = u11, a12 = u12, a13 = u13, a21 = l21u11, a31 = l31u11. (3.31) These gives first column of L and the first row of U. Next multiply row 2 of L times columns 2 and 3 of U, to obtain a22 = l21u12 + u22, a23 = l21u13 + u23. (3.32) These can be solved for u22 and u23. Next multiply row 3 of L to obtain l31u12 + l32u22 = a32, l31u13 + l32u23 + u33 = a33. (3.33) These equations yield values for l32 and u33, completing the construction of L and U. In this process, we must have u11 ̸= 0, u22 ̸= 0 in order to solve for L, which is true because of the assumptions that all the leading principal minors of A are non-zero. The decomposition we have found is the Doolittle’s factorization of A. We now illustrate the above construction for a particular matrix A. Example 3.2.14. Consider the matrix A = 1 1 −1 1 2 −2 −2 1 1 . Using (3.31), we get u11 = 1, u12 = 1, u13 = −1, l21 = a21 u11 = 1, l31 = a31 u11 = −2. Using (3.32) and (3.33), u22 = a22 − l21u12 = 2 − 1 × 1 = 1 u23 = a23 − l21u13 = −2 − 1 × (−1) = −1 l32 = (a32 − l31u12) u22 = (1 − (−2) × 1) 1 = 3 u33 = a33 − l31u13 − l32u23 = 1 − (−2) × (−1) − 3 × (−1) = 2 S. Baskar and S. Sivaji Ganesh 68 Spring 2017-18
- 81. Section 3.2 Direct Methods for Linear Systems Thus we obtain the Doolittle’s factorization of A as A = 1 0 0 1 1 0 −2 3 1 1 1 −1 0 1 −1 0 0 2 . Further, taking b = (1, 1, 1)T , we now solve the system Ax = b using LU factorization, with the matrix A given above. As discussed earlier, first we have to solve the lower triangular system 1 0 0 1 1 0 −2 3 1 z1 z2 z3 = 1 1 1 . Forward substitution yields z1 = 1, z2 = 0, z3 = 3. Keeping the vector z = (1, 0, 3)T as the right hand side, we now solve the upper triangular system 1 1 −1 0 1 −1 0 0 2 x1 x2 x3 = 1 0 3 . Backward substitution yields x1 = 1, x2 = 3/2, x3 = 3/2. Crout’s factorization In Doolittle’s factorization, the lower triangular matrix has special property. If we ask the upper triangular matrix to have special property in the LU decomposition, it is known as the Crout’s factorization. Definition 3.2.15 [Crout’s factorization]. A matrix A is said to have a Crout’s factorization if there exists a lower triangular matrix L, and an upper triangular matrix U with all diagonal elements as 1 such that A = LU. The computation of the Crout’s factorization for a 3×3 matrix can be done in a similar way as done for Doolittle’s factorization in Example 3.2.13. For invertible matrices, one can easily obtain Doolittle’s factorization from Crout’s factorization and vice versa. Cholesky’s factorization Before we introduce Cholesky’s factorization, we recall the concept of a positive definite matrix. S. Baskar and S. Sivaji Ganesh 69 Spring 2017-18
- 82. Chapter 3. Numerical Linear Algebra Definition 3.2.16 [Positive Definite Matrix]. A symmetric matrix A is said to be positive definite if xT Ax > 0, for every non-zero vector x. We recall (from a course on Linear Algebra) a useful theorem concerning positive defi- nite matrices. Lemma 3.2.17. The following statements concerning a symmetric n × n matrix A are equivalent. 1. The matrix A is positive definite. 2. All the principal minors of the matrix A are positive. 3. All the eigenvalues of the matrix A are positive. The statements (1) and (2) are equivalent by definition. The proof of equivalence of (3) with other two statements is out of the scope of this course. We now define Cholesky’s factorization. Definition 3.2.18 [Cholesky’s factorization]. A matrix A is said to have a Cholesky’s factorization if there exists a lower triangular matrix L such that A = LLT . Remark 3.2.19. It is clear from the definition that if a matrix has a Cholesky’s factorization, then the matrix have to be necessarily a symmetric matrix. We now establish a sufficient condition for the existence of the Cholesky’s factoriza- tion. Theorem 3.2.20. If A is an n × n symmetric and positive definite matrix, then A has a unique factor- ization A = LLT , where L is a lower triangular matrix with positive diagonal elements. S. Baskar and S. Sivaji Ganesh 70 Spring 2017-18
- 83. Section 3.2 Direct Methods for Linear Systems Proof. We prove the theorem by induction. The proof of the theorem is trivial when n = 1. Let us assume that the theorem holds for n = k for some k ∈ N. That is, we assume that for every k × k symmetric and positive definite matrix Bk, there exists a unique lower triangular k × k matrix ˜Lk such that Bk = ˜Lk ˜LT k . We prove the theorem for n = k + 1. Let A be a (k + 1) × (k + 1) symmetric positive definite matrix. We first observe that the matrix A can be written as A = ( Ak a aT a(k+1)(k+1) ) , where Ak is the k×k principal sub-matrix of A and a = (a1(k+1), a2(k+1), · · · , ak(k+1))T . Observe that Ak is positive definite and therefore by our assumption, there exists a unique k × k lower triangular matrix Lk such that Ak = LkLT k . Define L = ( Lk 0 lT l(k+1)(k+1) ) , where the real number l(k+1)(k+1) and the vector l = (l1(k+1), l2(k+1), · · · , lk(k+1))T are to be chosen such that A = LLT . That is ( Ak a aT a(k+1)(k+1) ) = ( Lk 0 lT l(k+1)(k+1) ) ( LT k l 0T l(k+1)(k+1) ) . (3.34) Here 0 denotes the zero vector of dimension k. Clearly, Lkl = a, (3.35) which by forward substitution yields the vector l. Here, we need to justify that Lk is invertible, which is left as an exercise. Finally, we have lT l + l2 (k+1)(k+1) = a(k+1)(k+1) and this gives l2 (k+1)(k+1) = a(k+1)(k+1) − lT l, (3.36) provided the positivity of l2 (k+1)(k+1) is justified, which follows from taking determinant on both sides of (3.34) and using the property (3) of Lemma 3.2.17. A complete proof of this justification is left as an exercise. S. Baskar and S. Sivaji Ganesh 71 Spring 2017-18
- 84. Chapter 3. Numerical Linear Algebra Example 3.2.21. Consider the matrix A = 9 3 −2 3 2 3 −2 3 23 . We can check that this matrix is positive definite by any of the equivalent conditions listed in Lemma 3.2.17. Therefore, we expect a unique Cholesky’s factorization for A. For the construction, we follow the proof of Theorem 3.2.20. 1. For n = 1, we have A1 = (9) and therefore let us take L1 = (3). 2. For n = 2, we have A2 = ( 9 3 3 2 ) . Therefore, L2 = ( L1 0 l l22 ) = ( 3 0 l l22 ) . This gives l = 1 and l2 + l2 22 = 2 or l22 = 1. Thus, we have L2 = ( 3 0 1 1 ) . 3. For n = 3, we take L = ( L2 0 lT l33 ) , where lT = (l13, l23) and l33 are to be obtained in such a way that A = LLT . The vector l is obtained by solving the lower triangular system (3.35) with a = (−2, 3)T (by forward substitution), which gives l = (−2/3, 11/3)T . Finally, from (3.36), we have l2 33 = 82/9. Thus, the required lower triangular matrix L is L = 3 0 0 1 1 0 −2/3 11/3 √ 82/3 . It is straightforward to check that A = LLT . It can be observed that the idea of LU factorization method for solving a system of linear equations is parallel to the idea of Gaussian elimination method. However, LU factorization method has an advantage that once the factorization A = LU is done, the solution of the linear system Ax = b is reduced to solving two triangular systems. S. Baskar and S. Sivaji Ganesh 72 Spring 2017-18
- 85. Section 3.3 Matrix Norms and Condition Number of a Matrix Thus, if we want to solve a large set of linear systems where the coefficient matrix A is fixed but the right hand side vector b varies, we can do the factorization once for A and then use this factorization with different b vectors. This is obviously going to reduce the computation cost drastically as we have seen in the operation counting of Gaussian elimination method that the elimination part is the most expensive part of the computation. 3.3 Matrix Norms and Condition Number of a Matrix Given any two real numbers α and β, the distance between them is given by |α − β|. We can also compare their magnitudes |α| and |β|. Vector norm may be thought of as a generalization of the modulus | · | concept to deal with vectors and in general matrices. Definition 3.3.1 [Vector Norm]. A vector norm on Rn is a function ∥ · ∥ : Rn → [0, ∞) having the following properties: 1. ∥x∥ ≥ 0 for all x ∈ Rn . 2. ∥x∥ = 0 if and only if x = 0. 3. ∥αx∥ = |α|∥x∥ for all x ∈ Rn and for all α ∈ R. 4. ∥x + y∥ ≤ ∥x∥ + ∥y∥ for all x, y ∈ Rn . The condition 4 in the above definition is called the triangle inequality. Note We use the following notation for a vector x ∈ Rn in terms of its components: x = (x1, x2, · · · , xn)T . Example 3.3.2. There can be many vector norms on Rn . We define three important vector norms on Rn , which are frequently used in matrix analysis. 1. The Euclidean norm (also called the l2-norm) on Rn is denoted by ∥ · ∥2, and is defined by ∥x∥2 = n∑ i=1 |xi|2. (3.37) S. Baskar and S. Sivaji Ganesh 73 Spring 2017-18
- 86. Chapter 3. Numerical Linear Algebra 2. The l∞-norm, also called maximum norm, on Rn is denoted by ∥ · ∥∞, and is defined by ∥x∥∞ = max { |x1|, |x2|, · · · , |xn| } . (3.38) 3. l1-norm on Rn is denoted by ∥ · ∥1, and is defined by ∥x∥1 = |x1| + |x2| + · · · + |xn|. (3.39) All the three norms defined above are indeed norms; it is easy to verify that they satisfy the defining conditions of a norm given in Definition 3.3.1. Let us illustrate each of the above defined norms with some specific vectors. Example 3.3.3. Let us compute norms of some vectors now. Let x = (4, 4, −4, 4)T , y = (0, 5, 5, 5)T , z = (6, 0, 0, 0)T . Verify that ∥x∥1 = 16, ∥y∥1 = 15, ∥z∥1 = 6; ∥x∥2 = 8, ∥y∥2 = 8.66, ∥z∥2 = 6; ∥x∥∞ = 4, ∥y∥∞ = 5, ∥z∥∞ = 6. From this example we see that asking which vector is big does not make sense. But once the norm is fixed, this question makes sense as the answer depends on the norm used. In this example each vector is big compared to other two but in different norms. Remark 3.3.4. In our computations, we employ any one of the norms depending on convenience. It is a fact that “all vector norms on Rn are equivalent”; we will not elaborate further on this. We can also define matrix norms on the vector space Mn(R) of all n × n real matrices, which helps us in finding distance between two matrices. Definition 3.3.5 [Matrix Norm]. A matrix norm on the vector space of all n × n real matrices Mn(R) is a function ∥ · ∥ : Mn(R) → [0, ∞) having the following properties: 1. ∥A∥ ≥ 0 for all A ∈ Mn(R). 2. ∥A∥ = 0 if and only if A = 0. 3. ∥αA∥ = |α|∥A∥ for all A ∈ Mn(R) and for all α ∈ R. 4. ∥A + B∥ ≤ ∥A∥ + ∥B∥ for all A, B ∈ Mn(R). S. Baskar and S. Sivaji Ganesh 74 Spring 2017-18
- 87. Section 3.3 Matrix Norms and Condition Number of a Matrix Note As in the case of vector norms, the condition 4 in the above definition is called the triangle inequality. For a matrix A ∈ Mn(R), we use the notation A = (aij)1≤i≤n, 1≤j≤n where aij denotes the element in the ith row and jth column of A. There can be many matrix norms on Mn(R). We will describe some of them now. Example 3.3.6. The following define norms on Mn(R). 1. ∥A∥ = n∑ i=1 n∑ j=1 |aij|2 . 2. ∥A∥ = max { |aij| : 1 ≤ i ≤ n, 1 ≤ j ≤ n } . 3. ∥A∥ = n∑ i=1 n∑ j=1 |aij|. All the three norms defined above are indeed norms; it is easy to verify that they satisfy the defining conditions of a matrix norm of Definition 3.3.5. Among matrix norms, there are special ones that satisfy very useful and important properties. They are called matrix norms subordinate to a vector norm. As the name suggests, to define them we need to fix a vector norm. We will give a precise definition now. Definition 3.3.7 [Matrix Norm subordinate to a vector norm]. Let ∥ · ∥ be a vector norm on Rn and let A ∈ Mn(R). The matrix norm of A subordinate to the vector norm ∥ · ∥ is defined by ∥A∥ := sup {∥Ax∥ : x ∈ Rn , ∥x∥ = 1} . (3.40) The formula (3.40) indeed defines a matrix norm on Mn(R). The proof of this fact is beyond the scope of our course. In this course, by matrix norm, we always mean a norm subordinate to some vector norm. An equivalent and more useful formula for the matrix norm subordinate to a vector norm is given in the following lemma. S. Baskar and S. Sivaji Ganesh 75 Spring 2017-18
- 88. Chapter 3. Numerical Linear Algebra Lemma 3.3.8. For any A ∈ Mn(R) and a given vector norm ∥ · ∥, we have ∥A∥ = max z̸=0 ∥Az∥ ∥z∥ . (3.41) Proof. For any z ̸= 0, we have x = z/||z|| as a unit vector. Hence max ||x||=1 ||Ax|| = max ||z||̸=0 A ( z ||z|| ) = max ||z||̸=0 ||Az|| ||z|| . The matrix norm subordinate to a vector norm has additional properties as stated in the following theorem whose proof is left as an exercise. Theorem 3.3.9. Let ∥ · ∥ be a matrix norm subordinate to a vector norm. Then 1. ∥Ax∥ ≤ ∥A∥∥x∥ for all x ∈ Rn . 2. ∥I∥ = 1 where I is the identity matrix. 3. ∥AB∥ ≤ ∥A∥ ∥B∥ for all A, B ∈ Mn(R). Note We do not use different notations for a matrix norm and the subordinate vector norm. These are to be understood depending on the argument. We will now state a few results concerning matrix norms subordinate to some of the vector norms described in Example 3.3.2. We omit their proofs. Theorem 3.3.10 [Matrix norm subordinate to the maximum norm]. The matrix norm subordinate to the l∞-norm given in (3.38) on Rn is denoted by ∥A∥∞ and is given by ∥A∥∞ = max 1≤i≤n n∑ j=1 |aij|, (3.42) which is called the maximum-of-row-sums norm. S. Baskar and S. Sivaji Ganesh 76 Spring 2017-18
- 89. Section 3.3 Matrix Norms and Condition Number of a Matrix Theorem 3.3.11 [Matrix Norm Subordinate to the l1-norm]. The matrix norm subordinate to the l1-norm given in (3.39) on Rn is denoted by ∥A∥1 and is given by ∥A∥1 = max 1≤j≤n n∑ i=1 |aij|. (3.43) The norm ∥A∥1 is called the maximum-of-column-sums norm of A. Description and computation of the matrix norm subordinate to the Euclidean vector norm on Rn is more subtle. Theorem 3.3.12 [Matrix norm subordinate to the Euclidean norm]. The matrix norm subordinate to the l2-norm (Euclidean norm) given by (3.37) on Rn is denoted by ∥A∥2 and is given by ∥A∥2 = √ max 1≤i≤n |λi|, (3.44) where λ1, λ2, · · · , λn are eigenvalues of the matrix AT A. The norm ∥A∥2 given by the formula (3.44) is called the spectral norm of A. Example 3.3.13. Let us now compute ∥A∥∞ and ∥A∥2 for the matrix 1 1 −1 1 2 −2 −2 1 1 . 1. ∥A∥∞ = 5 since 3∑ j=1 |a1j| = |1| + |1| + | − 1| = 3, 3∑ j=1 |a2j| = |1| + |2| + | − 2| = 5, 3∑ j=1 |a3j| = | − 2| + |1| + |1| = 4. 2. ∥A∥2 ≈ 3.5934 as the eigenvalues of AT A are λ1 ≈ 0.0616, λ2 ≈ 5.0256 and λ3 ≈ 12.9128. Hence ∥A∥2 ≈ √ 12.9128 ≈ 3.5934. S. Baskar and S. Sivaji Ganesh 77 Spring 2017-18
- 90. Chapter 3. Numerical Linear Algebra The following theorem motivates the condition number for an invertible matrix which is similar to the condition number defined for a function in Section 2.5.1. Theorem 3.3.14. Let A be an invertible n × n matrix. Let x and ˜x be the solutions of the systems Ax = b and A˜x = ˜b, respectively, where b and ˜b are given vectors. Then ∥x − ˜x∥ ∥x∥ ≤ ∥A∥ ∥A−1 ∥ ∥b − ˜b∥ ∥b∥ (3.45) for any fixed vector norm and the matrix norm subordinate to this vector norm. Proof. Since A is invertible, we have x − ˜x = A−1 ( b − ˜b ) . Taking norms on both sides and using the fact that ∥Ax∥ ≤ ∥A∥∥x∥ (see Theorem 3.3.9) holds for every x ∈ Rn , we get ∥x − ˜x∥ ≤ ∥A−1 ∥∥b − ˜b∥ (3.46) The inequality (3.46) estimates the error in the solution caused by error on the right hand side vector of the linear system Ax = b. The inequality (3.45) is concerned with estimating the relative error in the solution in terms of the relative error in the right hand side vector b. Since Ax = b, we get ∥b∥ = ∥Ax∥ ≤ ∥A∥∥x∥. Therefore ∥x∥ ≥ ∥b∥ ∥A∥ . Using this inequality in (3.46), we get (3.45). Remark 3.3.15. 1. In the above theorem, it is important to note that a vector norm is fixed and the matrix norm used is subordinate to this fixed vector norm. 2. The theorem holds no matter which vector norm is fixed as long as the matrix norm subordinate to it is used. 3. In fact, whenever we do analysis on linear systems, we always fix a vector norm and then use matrix norm subordinate to it. S. Baskar and S. Sivaji Ganesh 78 Spring 2017-18
- 91. Section 3.3 Matrix Norms and Condition Number of a Matrix Notice that the constant appearing on the right hand side of the inequality (3.45) (which is ∥A∥ ∥A−1 ∥) depends only on the matrix A (for a given vector norm). This number is called the condition number of the matrix A. Notice that this condition number depends very much on the vector norm being used on Rn and the matrix norm that is subordinate to the vector norm. Definition 3.3.16 [Condition Number of a Matrix]. Let A be an n×n invertible matrix. Let a matrix norm be given that is subordinate to a vector norm. Then the condition number of the matrix A (denoted by κ(A)) is defined as κ(A) := ∥A∥ ∥A−1 ∥. (3.47) Remark 3.3.17. From Theorem 3.3.14, it is clear that if the condition number is small, then the relative error in the solution will also be small whenever the relative error in the right hand side vector is small. On the other hand, if the condition number is very large, then the relative error could be large even though the relative error in the right hand side vector is small. We illustrate this in the following example. Example 3.3.18. The linear system 5x1 + 7x2 = 0.7 7x1 + 10x2 = 1 has the solution x1 = 0, x2 = 0.1. Let us denote this by x = (0, 0.1)T , and the right hand side vector by b = (0.7, 1)T . The perturbed system 5x1 + 7x2 = 0.69 7x1 + 10x2 = 1.01 has the solution x1 = −0.17, x2 = 0.22, which we denote by ˜x = (−0.17, 0.22)T , and the right hand side vector by ˜b = (0.69, 1.01)T . The relative error between the solutions of the above systems in the l∞ vector norm is given by ∥x − ˜x∥∞ ∥x∥∞ = 1.7, S. Baskar and S. Sivaji Ganesh 79 Spring 2017-18
- 92. Chapter 3. Numerical Linear Algebra which is too high compared to the relative error in the right hand side vector which is given by ∥b − ˜b∥∞ ∥b∥∞ = 0.01. The condition number of the coefficient matrix of the system is 289. Therefore the magnification of the relative error is expected (see the inequality (3.45)). Definition 3.3.19. A matrix with a large condition number is said to be ill conditioned. A matrix with a small condition number is said to be well conditioned. Remark 3.3.20. An immediate question is that how large should the condition number be to declare that a matrix is ill-conditioned. This quantification is very difficult in practical situations as it depends on how large the relative error in the right hand side vector and also the tolerance level of the user. That is, how much error a user can tolerate in the application for which the linear system is solved. For instance, in finance related applications, even 20% of error may be tolerable, whereas in computing the path of a missile even a 0.2% error may lead to fatal disasters. Discussion of condition numbers of matrices is incomplete without the mention of the famous Hilbert matrix. Example 3.3.21. The Hilbert matrix of order n is given by Hn = 1 1 2 1 3 · · · 1 n 1 2 1 3 1 4 · · · 1 n+1 · · · · · · · · · · · · 1 n 1 n+1 1 n+1 · · · 1 2n−1 (3.48) For n = 4, we have κ(H4) = ∥H4∥∞∥H−1 4 ∥∞ = 25 12 13620 ≈ 28000 which may be taken as an ill-conditioned matrix. In fact, as the value of n increases, the corresponding condition number of the Hilbert matrix also increases. S. Baskar and S. Sivaji Ganesh 80 Spring 2017-18
- 93. Section 3.3 Matrix Norms and Condition Number of a Matrix An interesting and important question is that what kind of matrices could have large condition numbers. A partial answer is stated in the following theorem. Theorem 3.3.22. Let A ∈ Mn(R) be non-singular. Then, for any singular n × n matrix B, we have 1 κ(A) ≤ ∥A − B∥ ∥A∥ . (3.49) Proof. We have 1 κ(A) = 1 ∥A∥∥A−1∥ = 1 ∥A∥ 1 max x̸=0 ∥A−1 x∥ ∥x∥ ≤ 1 ∥A∥ 1 ∥A−1 y∥ ∥y∥ where y ̸= 0 is arbitrary. Take y = Az, for some arbitrary vector z. Then we get 1 κ(A) ≤ 1 ∥A∥ ( ∥Az∥ ∥z∥ ) . Let z ̸= 0 be such that Bz = 0 (this is possible since B is singular), we get 1 κ(A) ≤ ∥(A − B)z∥ ∥A∥∥z∥ ≤ ∥(A − B)∥∥z∥ ∥A∥∥z∥ = ∥A − B∥ ∥A∥ . From the above theorem it is apparent that if A is close to a singular matrix, then the reciprocal of the condition number is close to zero, ie., κ(A) is large. Let us illustrate this in the following example. Example 3.3.23. Clearly the matrix B = ( 1 1 1 1 ) is not invertible. For any ϵ > 0, let A = ( 1 1 + ϵ 1 − ϵ 1 ) . S. Baskar and S. Sivaji Ganesh 81 Spring 2017-18
- 94. Chapter 3. Numerical Linear Algebra Since ϵ > 0, A is invertible and we have A−1 = ϵ−2 ( 1 −1 − ϵ −1 + ϵ 1 ) . Let us use the l∞ norm on Rn . Then ∥A∥∞ = 2 + ϵ and ∥A−1 ∥∞ = ϵ−2 (2 + ϵ). Hence κ(A) = ∥A∥∞∥A−1 ∥∞ = ( 2 + ϵ ϵ )2 > 4 ϵ2 . Thus, if ϵ ≤ 0.01, then κ(A) ≥ 40, 000. As ϵ → 0, the matrix A tends to approach the matrix B and consequently, the above inequality says that the condition number κ(A) tends to ∞. Also, we can see that when we attempt to solve the system Ax = b, then the above inequality implies that a small relative perturbation in the right hand side vector b could be magnified by a factor of at least 40, 000 for the relative error in the solution. 3.4 Iterative Methods for Linear Systems In Section 3.2.1 and Section 3.2.5 we have discussed methods that obtain exact solution of a linear system Ax = b in the absence of floating point errors (i.e., exact when arithmetic is used). Such methods are called the direct methods. The solution of a linear system can also be obtained using iterative procedures. Such methods are called iterative methods. There are many iterative procedures out of which Jacobi and Gauss-Seidel methods are the simplest ones. In this section we introduce these methods and discuss their convergence. 3.4.1 Jacobi Method In Section 3.2.5, we have seen that when a linear system Ax = b is such that the coefficient matrix A is a diagonal matrix, then this system can be solved very easily. We explore this idea to build a new method based on iterative procedure. For this, we first rewrite the matrix A as A = D − C, S. Baskar and S. Sivaji Ganesh 82 Spring 2017-18
- 95. Section 3.4 Iterative Methods for Linear Systems where D, C ∈ Mn(R) are such that D = a11 0 · · · 0 0 a22 · · · 0 ... ... · · · ... 0 0 · · · ann , where aii, i = 1, 2, · · · , n are the diagonal elements of the matrix A. Then the given system of linear equations can be rewritten as Dx = Cx + b. (3.50) If we assume that the right hand side vector is fully known to us, then the above system can be solved very easily as D is a diagonal matrix. But obviously, the right hand side vector cannot be a known quantity because it involves the unknown vector x. Rather, if we choose (arbitrarily) some specific value for x, say x = x(0) , then the resulting system Dx = Cx(0) + b can be readily solved. Let us call the solution of this system as x(1) . That is, Dx(1) = Cx(0) + b. Now taking x = x(1) on the right hand side of (3.50) we can obtain the solution of this system, which we denote as x(2) and repeat this procedure to get a general iterative procedure as Dx(k+1) = Cx(k) + b, k = 0, 1, 2, · · · . If D is invertible, then the above iterative procedure can be written as x(k+1) = Bx(k) + c, k = 0, 1, 2, · · · , (3.51) where B = D−1 C and c = D−1 b. The iterative procedure (3.51) is called the Jacobi method. Example 3.4.1. Let us illustrate the Jacobi method in the case of 3 × 3 system a11x1 + a12x2 + a13x3 = b1 a21x1 + a22x2 + a23x3 = b2 a31x1 + a32x2 + a33x3 = b3, where a11 ̸= 0, a22 ̸= 0, and a33 ̸= 0. We can rewrite the above system of linear S. Baskar and S. Sivaji Ganesh 83 Spring 2017-18
- 96. Chapter 3. Numerical Linear Algebra equations as x1 = 1 a11 (b1 − a12x2 − a13x3) x2 = 1 a22 (b2 − a21x1 − a23x3) x3 = 1 a33 (b3 − a31x1 − a32x2) Let x(0) = (x (0) 1 , x (0) 2 , x (0) 3 )T be an initial guess to the true solution x, which is chosen arbitrarily. Define a sequence of iterates (for k = 0, 1, 2, · · · ) by x (k+1) 1 = 1 a11 (b1 − a12x (k) 2 − a13x (k) 3 ) x (k+1) 2 = 1 a22 (b2 − a21x (k) 1 − a23x (k) 3 ) x (k+1) 3 = 1 a33 (b3 − a31x (k) 1 − a32x (k) 2 ). (JS) which is the Jacobi iterative sequence given by (3.51) in the case of 3 × 3 system. Now, the question is that will the sequence of vectors {x(k+1) } generated by the iterative procedure (3.51) always converge to the exact solution x of the given linear system? The following example gives a system for which the Jacobi iterative sequence converges to the exact solution. Example 3.4.2. The Jacobi iterative sequence for the system 6x1 + x2 + 2x3 = −2, x1 + 4x2 + 0.5x3 = 1, −x1 + 0.5x2 − 4x3 = 0. is given by x (k+1) 1 = 1 6 (−2 − x (k) 2 − 2x (k) 3 ), x (k+1) 2 = 1 4 (1 − x (k) 1 − 0.5x (k) 3 ), x (k+1) 3 = − 1 4 (0 + x (k) 1 − 0.5x (k) 2 ). The exact solution (upto 6-digit rounding) of this system is x ≈ (−0.441176, 0.341176, 0.152941). (3.52) S. Baskar and S. Sivaji Ganesh 84 Spring 2017-18
- 97. Section 3.4 Iterative Methods for Linear Systems Choosing the initial guess x(0) = (0, 0, 0), we get x(1) ≈ (−0.333333, 0.250000, 0.000000), x(2) ≈ (−0.375000, 0.333333, 0.114583), x(3) ≈ (−0.427083, 0.329427, 0.135417), x(4) ≈ (−0.433377, 0.339844, 0.147949), x(5) ≈ (−0.439290, 0.339851, 0.150825), and so on. We observe from the above computed results that the sequence {x(k) } seems to be approaching the exact solution. In the following example we discuss a system for which the Jacobi iterative sequence does not converge to the exact solution. Example 3.4.3. Consider the system x1 + 4x2 + 0.5x3 = 1, 6x1 + x2 + 2x3 = −2, −x1 + 0.5x2 − 4x3 = 0. which is exactly the same as the system discussed in Example 3.4.2 but the only difference is the interchange of first and second equation. Hence, the exact solution is same as given in (3.52). The Jacobi iterative sequence for this system is given by x (k+1) 1 = (1 − 4x (k) 2 − 0.5x (k) 3 ), x (k+1) 2 = (−2 − 6x (k) 1 − 2x (k) 3 ), x (k+1) 3 = − 1 4 (0 + x (k) 1 − 0.5x (k) 2 ). Choosing the initial guess x(0) = (0, 0, 0), we get x(1) ≈ (1, −2, 0), x(2) ≈ (9, −8, −0.5), x(3) ≈ (33.25, −55, −3.25), x(4) ≈ (222.625, −195, −15.1875), x(5) ≈ (788.59375, −1307.375, −80.03125), and so on. Here, we observe a diverging trend in the sequence {x(k) }. S. Baskar and S. Sivaji Ganesh 85 Spring 2017-18
- 98. Chapter 3. Numerical Linear Algebra The above two examples shows that the Jacobi sequence need not converge always and so we need to look for a condition on the system for which the Jacobi iterative sequence converges to the exact solution. Define the error in the kth iterate x(k) compared to the exact solution by e(k) = x − x(k) . It follows easily that e(k) satisfies the system e(k+1) = Be(k) , where B is as defined in (3.51). Using any vector norm and the matrix norm subordinate to it in the above equation, we get ∥e(k+1) ∥ = ∥Be(k) ∥ ≤ ∥B∥∥e(k) ∥ ≤ · · · ≤ · · · ≤ · · · ≤ ∥B∥k+1 ∥e(0) ∥. Thus, when ∥B∥ < 1, the iteration method (3.51) always converges for any initial guess x(0) . Again the question is “what are all the matrices A for which the corresponding matrices B in (3.51) have the property ∥B∥ < 1, for some matrix norm subordinate to some vector norm?” One such class of matrices are the diagonally dominant matrices, which we define now. Definition 3.4.4 [Diagonally Dominant Matrices]. A matrix A is said to be diagonally dominant if it satisfies the inequality n∑ j=1,j̸=i |aij| < |aii|, i = 1, 2, · · · , n. We now prove the sufficient condition for the convergence of the Jacobi method. This theorem asserts that if A is a diagonally dominant matrix, then B in (3.51) of the Jacobi method is such that ∥B∥∞ < 1. S. Baskar and S. Sivaji Ganesh 86 Spring 2017-18
- 99. Section 3.4 Iterative Methods for Linear Systems Theorem 3.4.5. If the coefficient matrix A is diagonally dominant, then the Jacobi method (3.51) converges to the exact solution of Ax = b. Proof. Since A is diagonally dominant, the diagonal entries are all non-zero and hence the Jacobi iterating sequence x(k) given by x (k+1) i = 1 aii ( bi − n∑ j=1,j̸=i aijx (k) j ) , i = 1, 2, · · · , n. (3.53) is well-defined. Each component of the error satisfies e (k+1) i = − n∑ j=1 j̸=i aij aii e (k) j , i = 1, 2, · · · , n. (3.54) which gives |e (k+1) i | ≤ n∑ j=1 j̸=i aij aii ∥e(k) ∥∞. (3.55) Define µ = max 1≤i≤n n∑ j=1 j̸=i aij aii . (3.56) Then |e (k+1) i | ≤ µ∥e(k) ∥∞, (3.57) which is true for all i = 1, 2, · · · , n. Therefore, we have ∥e(k+1) ∥∞ ≤ µ∥e(k) ∥∞. (3.58) The matrix A is diagonally dominant if and only if µ < 1. Then iterating the last inequality we get ∥e(k+1) ∥∞ ≤ µk+1 ∥e(0) ∥∞. (3.59) Therefore, if A is diagonally dominant, the Jacobi method converges. S. Baskar and S. Sivaji Ganesh 87 Spring 2017-18
- 100. Chapter 3. Numerical Linear Algebra Remark 3.4.6. Observe that the system given in Example 3.4.2 is diagonally dominant, whereas the system in Example 3.4.3 is not so. 3.4.2 Gauss-Seidel Method Gauss-Seidel method is a modified version of the Jacobi method discussed in Section 3.4.1. We demonstrate the method in the case of a 3 × 3 system and the method for a general n × n system can be obtained in a similar way. Example 3.4.7. Consider the 3 × 3 system a11x1 + a12x2 + a13x3 = b1, a21x1 + a22x2 + a23x3 = b2, a31x1 + a32x2 + a33x3 = b3. When the diagonal elements of this system are non-zero, we can rewrite the above system as x1 = 1 a11 (b1 − a12x2 − a13x3), x2 = 1 a22 (b2 − a21x1 − a23x3), x3 = 1 a33 (b3 − a31x1 − a32x2). Let x(0) = (x (0) 1 , x (0) 2 , x (0) 3 )T be an initial guess to the true solution x. Define a sequence of iterates (for k = 0, 1, 2, · · · ) by x (k+1) 1 = 1 a11 (b1 − a12x (k) 2 − a13x (k) 3 ), x (k+1) 2 = 1 a22 (b2 − a21x (k+1) 1 − a23x (k) 3 ), x (k+1) 3 = 1 a33 (b3 − a31x (k+1) 1 − a32x (k+1) 2 ). (GSS) This sequence of iterates is called the Gauss-Seidel iterative sequence and the method is called Gauss-Seidel Iteration method. S. Baskar and S. Sivaji Ganesh 88 Spring 2017-18
- 101. Section 3.4 Iterative Methods for Linear Systems Remark 3.4.8. Compare (JS) and (GSS). Theorem 3.4.9. If the coefficient matrix is diagonally dominant, then the Gauss-Seidel method con- verges to the exact solution of the system Ax = b. Proof. Since A is diagonally dominant, all the diagonal elements of A are non-zero, and hence the Gauss-Seidel iterative sequence given by x (k+1) i = 1 aii { bi − i−1∑ j=1 aijx (k+1) j − n∑ j=i+1 aijx (k) j } , i = 1, 2, · · · , n. (3.60) is well-defined. The error in each component is given by e (k+1) i = − i−1∑ j=1 aij aii e (k+1) j − n∑ j=i+1 aij aii e (k) j , i = 1, 2, · · · , n. (3.61) For i = 1, 2, · · · , n,, define αi = i−1∑ j=1 aij aii , βi = n∑ j=i+1 aij aii , with the convention that α1 = βn = 0. Note that µ given in (3.56) can be written as µ = max 1≤i≤n (αi + βi) Since A is a diagonally dominant matrix, we have µ < 1. Now |e (k+1) i | ≤ αi∥e(k+1) ∥∞ + βi∥e(k) ∥∞, i = 1, 2, · · · , n. (3.62) Let l be such that ∥e(k+1) ∥∞ = |e (k+1) l |. S. Baskar and S. Sivaji Ganesh 89 Spring 2017-18
- 102. Chapter 3. Numerical Linear Algebra Then with i = l in (3.62), ∥e(k+1) ∥∞ ≤ αl∥e(k+1) ∥∞ + βl∥e(k) ∥∞. (3.63) Since µ < 1, we have αl < 1 and therefore the above inequality gives ∥e(k+1) ∥∞ ≤ βl 1 − αl ∥e(k) ∥∞. (3.64) Define η = max 1≤i≤n βi 1 − αi . (3.65) Then the above inequality yields ∥e(k+1) ∥∞ ≤ η∥e(k) ∥∞. (3.66) Since for each i, (αi + βi) − βi 1 − αi = αi[1 − (αi + βi)] 1 − αi ≥ αi 1 − αi [1 − µ] ≥ 0, (3.67) we have η ≤ µ < 1. (3.68) Thus, when the coefficient matrix A is diagonally dominant, Gauss-Seidel method converges. Remark 3.4.10. A careful observation of the proof of the above theorem reveals that the Gauss- Seidel method converges faster than the Jacobi method by comparing (3.68) and (3.59). 3.4.3 Mathematical Error Let x∗ denote the computed solution using some method. The mathematical error in the approximate solution when compared to the exact solution of a linear system Ax = b is given by e = x − x∗ . (3.69) Recall from Chapter 2 that the mathematical error is due to the approximation made in the numerical method where the computation is done without any floating-point S. Baskar and S. Sivaji Ganesh 90 Spring 2017-18
- 103. Section 3.4 Iterative Methods for Linear Systems approximation ( ie., without rounding or chopping). Observe that to get the mathe- matical error, we need to know the exact solution. But an astonishing feature of linear systems (which is not there in nonlinear equations) is that this error can be obtained exactly without the knowledge of the exact solution. To do this, we first define the residual vector r = b − Ax∗ (3.70) in the approximation of b by Ax∗ . This vector is also referred to as residual error. Since b = Ax, we have r = b − Ax∗ = Ax − Ax∗ = A(x − x∗ ). The above identity can be written as Ae = r. (3.71) This shows that the error e satisfies a linear system with the same coefficient matrix A as in the original system Ax = b, but a different right hand side vector. Thus, by having the approximate solution x∗ in hand, we can obtain the error e without knowing the exact solution x of the system. 3.4.4 Residual Corrector Method When we use a computing device for solving a linear system, irrespective to whether we use direct methods or iterative methods, we always get an approximate solution. An attractive feature (as discussed in the above section) of linear systems is that the error involved in the approximate solution when compared to the exact solution can theoretically be obtained exactly. In this section, we discuss how to use this error to develop an iterative procedure to increase the accuracy of the obtained approximate solution using any other numerical method. There is an obvious difficulty in the process of obtaining e as the solution of the system (3.71), especially on a computer. Since b and Ax∗ are very close to each other, the computation of r involves loss of significant digits which leads to zero residual error, which is of no use. To avoid this situation, the calculation of (3.70) should be carried out at a higher-precision. For instance, if x∗ is computed using single-precision, then r can be computed using double-precision and then rounded back to single precision. Let us illustrate the computational procedure of the residual error in the following example. S. Baskar and S. Sivaji Ganesh 91 Spring 2017-18
- 104. Chapter 3. Numerical Linear Algebra Example 3.4.11 [Computation of residual at a higher precision]. Consider the system 0.729x1 + 0.81x2 + 0.9x3 = 0.6867 x1 + x2 + x3 = 0.8338 1.331x1 + 1.210x2 + 1.100x3 = 1.000 The exact solution of this system with 4-digit rounding is x1 ≈ 0.2245, x2 ≈ 0.2814, x3 ≈ 0.3279. The solution of the system by Gaussian elimination without pivoting using 4-digit rounding leads to x∗ 1 ≈ 0.2251, x∗ 2 ≈ 0.2790, x∗ 3 ≈ 0.3295. As we have used 4-digit rounding arithmetic in obtaining the approximate solution vector x∗ , we use 8-digit rounding arithmetic in computing the residual, which is obtained as r = (0.00006210, 0.0002000, 0.0003519)T . Solving the linear system Ae = r using 8-digit rounding arithmetic, we obtain the approximation as e∗ = [−0.00044710, 0.00215000, −0.00150400]T . Compare this to the true error e = x − x∗ = [−0.0007, 0.0024, −0.0016]T Thus e∗ gives a good idea of the size of the error e in the computed solution x∗ . Let us now propose an iterative procedure by first predicting the error by solving the system (3.71) and then correcting the approximate solution x∗ by adding the predicted error to the vector x∗ . If we take x∗ = x(0) , and define r(0) = b − Ax(0) , then the error e(0) = x − x(0) can be obtained by solving the linear system Ae(0) = r(0) . Now, define x(1) = x(0) + e(0) . S. Baskar and S. Sivaji Ganesh 92 Spring 2017-18
- 105. Section 3.4 Iterative Methods for Linear Systems We expect the vector x(1) is more closer to the exact solution than x(0) . Again compute the residual error vector r(1) using the formula r(1) = b − Ax(1) , and solve the corresponding linear system Ae(1) = r(1) and define x(2) = x(1) + e(1) . Continuing in this way, we can generate a sequence {x(k) } using the formula x(k+1) = x(k) + e(k) . (3.72) where Ae(k) = r(k) with r(k) = b − Ax(k) , for k = 0, 1, · · · . The above iterative procedure is called the residual corrector method (also called the iterative refinement method). Note that in comput- ing r(k) and e(k) , we use a higher precision than the precision used in computing x(k) . Example 3.4.12. Using Gaussian elimination with pivoting and four digit rounding in solving the linear system x1 + 0.5x2 + 0.3333x3 = 1 0.5x1 + 0.3333x2 + 0.25x3 = 0 0.3333x1 + 0.25x2 + 0.2x3 = 0 we obtain the solution as x = (9.062, −36.32, 30.30)T . Let us start with an initial guess of x(0) = (8.968, −35.77, 29.77)T . Using 8-digit rounding arithmetic, we obtain r(0) = (−0.00534100, −0.00435900, −0.00053440)T . After solving the system Ae(0) = r(0) using Gaussian elimination with pivoting using S. Baskar and S. Sivaji Ganesh 93 Spring 2017-18
- 106. Chapter 3. Numerical Linear Algebra 8-digit rounding and the final answer is rounded to four digits, we get e(0) = (0.0922, −0.5442, 0.5239)T . Hence, the corrected solution after the first iteration is x(1) = (9.060, −36.31, 30.29)T . Similarly, we can predict the error in x(1) when compared to the exact solution and correct the solution to obtain the second iterated vector as r(1) = (−0.00065700, −0.00037700, −0.00019800)T , e(2) = (0.0017, −0.0130, 0.0124)T , x(2) = (9.062, −36.32, 30.30)T , and so on. The convergence analysis of this iterative procedure to the exact solution is omitted for this course. 3.4.5 Stopping Criteria In the iterative methods discussed above, we have a sequence {x(k) } that is expected to converge to the exact solution of the given linear system. In practical situation, we cannot go on computing the x(k) indefinitely and we need to terminate our computation once the value of x(k) reaches a desired accuracy for a sufficiently large k. That is, when the error ∥e(k) ∥ = ∥x − x(k) ∥ in the kth iteration in some norm is sufficiently small. Since, we do not know the exact solution x, the error given above cannot be computed easily and needs another linear system (3.71) to be solved. Therefore, the question is how to decide where we have to stop our computation (without solving this linear system)? In other words, how do we know whether the computed vector x(k) at the kth iteration is sufficiently close to the exact solution or not. This can be decided by looking at the residual error vector of the kth iteration defined as r(k) = b − Ax(k) . (3.73) Thus, for a given sufficiently small positive number ϵ, we stop the iteration if ∥r(k) ∥ < ϵ, for some vector norm ∥ · ∥. S. Baskar and S. Sivaji Ganesh 94 Spring 2017-18
- 107. Section 3.5 Eigenvalue Problems 3.5 Eigenvalue Problems Let A be an n × n matrix with real entries. Eigenvalues of A are defined as the roots of the equation det(λI − A) = 0. (3.74) Note that det(λI −A) is a polynomial in λ of degree n, which is known as characteristic polynomial of the matrix A. We know that even if A has real entries, the eigenvalues need not be real numbers. It is also a known fact that for every matrix, there are n eigenvalues λ1, λ2, · · · , λn (in this list, each eigenvalue is repeated as many times as its algebraic multiplicity, i.e., multiplicity of the eigenvalue as a root of the characteristic polynomial). When n = 2 the characteristic polynomial is a quadratic polynomial for which there is a nice formula for computing roots. When n = 3, there is a formula that many of us do not remember. When n = 4, none has a formula. But computing eigenvalues is important for applications. Therefore, numerically approximating the eigenvalues is the only way out. One obvious way of approximating an eigenvalue of a matrix is to first obtain the char- acteristic polynomial (3.74) explicitly in λ and then use one of the numerical methods discussed in the next chapter (on methods for solving nonlinear equations) to compute a root of this polynomial. But this is not an efficient way of computing eigenvalues because of two reasons. One reason is that obtaining explicit form of (3.74) is itself a difficult task when the dimension of the matrix is very large. Secondly, if we make any small error (like floating-point error) in obtaining the explicit form of the polynomial (3.74), the resulting polynomial may have a root which is entirely different from any of the eigenvalues that we are looking for. This is illustrated in the following example by Wilkinson where we see that “the roots of polynomials are extremely sensitive to perturbations in the coefficients”. Example 3.5.1 [Wilkinson’s example]. Let f(x) and g(x) be two polynomials given by f(x) = (x − 1)(x − 2) · · · (x − 10), g(x) = x10 . The roots of the polynomial f(x) are 1, 2, · · · , 10, and all these roots are simple roots. If we perturb this polynomial as F(x) = f(x) + 0.01g(x), then all the roots lie in the interval [1, 3.5] (verified graphically). In fact, the largest root of the polynomial f(x) is 10 and the largest root of the polynomial F(x) is approximately equal to 3.398067. Thus, if the coefficient of x10 is perturbed by a small amount of 0.01, the root 10 of f(x) could move as much a distance as approximately 6.6. S. Baskar and S. Sivaji Ganesh 95 Spring 2017-18
- 108. Chapter 3. Numerical Linear Algebra Due to the two reasons discussed above, we look for an alternate method to compute the eigenvalues of a given matrix. One such method is the power method that can be used to obtain the eigenvalue which is the largest in magnitude among all the other eigenvalues and the corresponding eigen vector. In Subsection 3.5.1, we present the power method and discuss the condition under which this method can be applied. In Subsection 3.5.2 we prove the Gerschgorin theorem which may be used as a tool to find a class of matrices for which power method can be applied successfully. 3.5.1 Power Method There are many variations of power method in the literature. We will present the most elementary form of power method. We always deal with matrices with real entries, all of whose eigenvalues are real numbers. Power method is used to obtain a specific eigenvalue called dominant eigenvalue and a corresponding eigenvector for a given n × n matrix A. The concept of a dominant eigenvalue plays a very important role in many applications. The power method pro- vides an approximation to it under some conditions on the matrix. We now define the concept of a dominant eigenvalue. Definition 3.5.2 [Dominant Eigenvalue of a Matrix]. An eigenvalue λ of an n × n matrix A is said to be a dominant eigenvalue of A if |λ| = max{ |z| : z is an eigenvalue of A }. Remark 3.5.3. 1. If a dominant eigenvalue of a matrix is equal to zero, then all its eigenvalues are zero and an eigenvector can be found by solving the linear system Ax = 0. 2. If λ is a dominant eigenvalue of a matrix A, then there is no other eigenvalue of A whose distance from zero is more than that of λ from zero. 3. Let µ1, µ2, · · · , µn be the eigenvalues of A (repeated according to their alge- braic multiplicities) of an n×n matrix A. These eigenvalues can be renamed (relabelled, reindexed) as λ1, λ2, · · · , λn such that they satisfy the condition: |λ1| ≥ |λ2| ≥ · · · ≥ |λn|. Note that λ1 is a dominant eigenvalue of A. S. Baskar and S. Sivaji Ganesh 96 Spring 2017-18
- 109. Section 3.5 Eigenvalue Problems A matrix may have a unique dominant eigenvalue or more than one dominant eigen- values. Further, even if dominant eigenvalue is unique the corresponding algebraic and geometric multiplicities could be more than one, and also both algebraic and geometric multiplicities may not be the same. All these possibilities are illustrated in the following example. Example 3.5.4. 1. The matrix A = 1 0 0 0 −2 1 0 0 −1 has eigenvalues 1, −1, and −2. The matrix A has a unique dominant eigenvalue, which is −2 as this is the largest in absolute value, of all eigenvalues. Note that the dominant eigenvalue of A is a simple eigenvalue. 2. The matrix B = 1 3 4 0 2 1 0 0 −2 has eigenvalues 1, −2, and 2. According to our definition, the matrix B has two dominant eigenvalues. They are −2 and 2. Note that both the dominant eigenvalues of B are simple eigenvalues. 3. Consider the matrices C1 = 1 3 4 0 2 5 0 0 2 , C2 = ( 2 0 0 2 ) , C3 = ( 2 1 0 2 ) . The matrix C1 has a unique dominant eigenvalue 2, which has algebraic multi- plicity 2 and geometric multiplicity 1. The matrix C2 has a unique dominant eigenvalue 2, whose algebraic and geometric multiplicities equal 2. The matrix C3 has a unique dominant eigenvalue 2, which has algebraic multiplicity 2 and geometric multiplicity 1. As mentioned above the power method is used to compute the dominant eigenvalue and the corresponding eigen vector of a given n × n matrix provided this eigenvalue is unique. Thus, in the above examples, power method can be used for the matrices A but not for B even though B has distinct eigenvalues. Let us now detail the power method. S. Baskar and S. Sivaji Ganesh 97 Spring 2017-18
- 110. Chapter 3. Numerical Linear Algebra Hypotheses for which the power method can work Assume that an n × n matrix A has real eigenvalues λ1, λ2, · · · , and λn (repeated according to their algebraic multiplicities) with the following properties: (H1) The eigenvalues are such that |λ1| > |λ2| ≥ |λ3| ≥ · · · ≥ |λn| (3.75) That is, A has a unique dominant eigenvalue λ1 which is a simple eigenvalue. (H2) There exists a basis of Rn consisting of eigenvectors of A. That is, there exists v1, v2, · · · , vn satisfying Avk = λkvk for k = 1, 2, · · · , n; and such that for each v ∈ Rn there exists unique real numbers c1, c2, · · · , cn such that v = c1v1 + c2v2 + · · · + cnvn. Equivalently, the matrix A is diagonalizable. (H3) An initial guess x(0) ∈ Rn be chosen such that x(0) = n∑ j=1 cjvj, (3.76) for some scalars c1, c2, · · · , cn ∈ R with c1 ̸= 0 and x(0) /∈ ∞∪ k=1 KerAk . Let us now discuss the key idea behind power method. • Choose a non-zero vector x(0) ∈ Rn arbitrarily so that we can find scalars c1, c2, · · · , cn ∈ R such that x(0) = c1v1 + c2v2 + · · · + cnvn, c1 ̸= 0. • Pre-multiplying by A and substituting Avi = λivi, i = 1, · · · , n, we get Ax(0) = c1λ1v1 + · · · + cnλnvn = λ1 ( c1v1 + c2 ( λ2 λ1 ) v2 + · · · + cn ( λn λ1 ) vn ) . Note here that we have assumed λ1 ̸= 0, which follows from the Assumption (1) above. • Pre-multiplying by A again and simplying, we get A2 x(0) = λ2 1 ( c1v1 + c2 ( λ2 λ1 )2 v2 + · · · + cn ( λn λ1 )2 vn ) S. Baskar and S. Sivaji Ganesh 98 Spring 2017-18
- 111. Section 3.5 Eigenvalue Problems • For each k ∈ N, applying A k-times on x(0) yields Ak x(0) = λk 1 ( c1v1 + c2 ( λ2 λ1 )k v2 + · · · + cn ( λn λ1 )k vn ) (3.77) • Using the assumption (3.75), we get |λj/λ1| < 1, for j = 2, · · · , n. Therefore, we have lim k→∞ Ak x(0) λk 1 = c1v1. (3.78) For c1 ̸= 0, the right hand side of the above equation is a scalar multiple of the eigenvector. • From the above expression for Ak x(0) , we also see that lim k→∞ (Ak+1 x(0) )i (Akx(0))i = λ1, (3.79) where i is any index such that the fractions on the left hand side are meaningful (which is the case when x(0) /∈ ∪∞ k=1KerAk ). The power method generates two sequences {µk} and {x(k) }, using the results (3.78) and (3.79), that converge to the dominant eigenvalue λ1 and the corresponding eigenvectors v1, respectively. We now describe the steps involved in the power method for generating these two sequences. Setting up the iterative sequences: Step 1: Choose a vector x(0) arbitrarily and set y(1) := Ax(0) . Step 2: Define µ1 := y (1) i , where i ∈ {1, · · · , n} is the least index such that ∥y(1) ∥∞ = |y (1) i | and set x(1) := y(1) µ1 . Step 3: From x(1) , we can obtain µ2 and x(2) as in step 2. Continue this procedure. General form of the power method iterative sequences: After choosing the initial vector x(0) arbitrarily, we generate the sequences {µ(k) } and {x(k) } using the formulas µk+1 = y (k+1) i , x(k+1) = y(k+1) µk+1 , (3.80) S. Baskar and S. Sivaji Ganesh 99 Spring 2017-18
- 112. Chapter 3. Numerical Linear Algebra where y(k+1) = Ax(k) and i is such that |y (k+1) i | = ∥y(k+1) ∥∞, (3.81) for k = 0, 1, · · · . This iterative procedure is called the power method. Remark 3.5.5. The scaling factor µk introduced in (3.80) makes sure that x(k) has its maximum norm equal to 1, i.e., ∥x(k)∥∞ = 1. This rules out the possibilities of limk→∞ x(k) being 0 or the vector x(k) escaping to infinity. Remark 3.5.6. If the initial guess x(0) ∈ Rn is such that x(0) /∈ ∞∪ k=1 Ker(Ak ) and c1 ̸= 0, then it can be shown (proof omitted) that under the assumptions stated above, we have 1. the sequence {µk} converges to the dominant eigenvalue λ1 and 2. the sequence {x(k)} converges either to +v1/∥v1∥∞ or to −v1/∥v1∥∞ or oscillates between these two vectors. Here, v1 denotes the eigenvector corre- sponding to the eigenvalue λ1. Point 2 means that whatever may be the case (among the three cases stated), the sequence {x(k)} approaches the eigenspace associated with the eigenvalue λ1, as k → ∞. We now give a numerical example illustrating power method procedure. Example 3.5.7. Consider the matrix A = 3 0 0 −4 6 2 16 −15 −5 . The eigenvalues of this matrix are λ1 = 3, λ2 = 1 and λ3 = 0. The corresponding eigen vectors are v1 = (1, 0, 2)T , v2 = (0, 2, −5)T and v3 = (0, 1, −3)T . Thus, the hypothesis (H1) and (H2) are satisfied. Choose the initial guess x0 = (1, 0.5, 0.25)T , which also satisfies the hypothesis (H3). S. Baskar and S. Sivaji Ganesh 100 Spring 2017-18
- 113. Section 3.5 Eigenvalue Problems The first ten terms of the iterative sequence in power method given by (3.80)-(3.81) for the given matrix A are as follows: Iteration No: 1 y1 = Ax0 = (3.000000, −0.500000, 7.250000)T µ1 = 7.250000 x1 = y1 µ1 = (0.413793, −0.068966, 1.000000)T Iteration No: 2 y2 = Ax1 = (1.241379, −0.068966, 2.655172)T µ2 = 2.655172 x2 = y2 µ2 = (0.467532, −0.025974, 1.000000)T Iteration No: 3 y3 = Ax2 = (1.402597, −0.025974, 2.870130)T µ3 = 2.870130 x3 = y3 µ3 = (0.488688, −0.009050, 1.000000)T Iteration No: 4 y4 = Ax3 = (1.466063, −0.009050, 2.954751)T µ4 = 2.954751 x4 = y4 µ4 = (0.496172, −0.003063, 1.000000)T Iteration No: 5 y5 = Ax4 = (1.488515, −0.003063, 2.984686)T µ5 = 2.984686 x5 = y5 µ5 = (0.498717, −0.001026, 1.000000)T Iteration No: 6 y6 = Ax5 = (1.496152, −0.001026, 2.994869)T µ6 = 2.994869 x6 = y6 µ6 = (0.499572, −0.000343, 1.000000)T S. Baskar and S. Sivaji Ganesh 101 Spring 2017-18
- 114. Chapter 3. Numerical Linear Algebra Iteration No: 7 y7 = Ax6 = (1.498715, −0.000343, 2.998287)T µ7 = 2.998287 x7 = y7 µ7 = (0.499857, −0.000114, 1.000000)T Iteration No: 8 y8 = Ax7 = (1.499571, −0.000114, 2.999429)T µ8 = 2.999429 x8 = y8 µ8 = (0.499952, −0.000038, 1.000000)T Iteration No: 9 y9 = Ax8 = (1.499857, −0.000038, 2.999809)T µ9 = 2.999809 x9 = y9 µ9 = (0.499984, −0.000013, 1.000000)T Iteration No: 10 y10 = Ax9 = (1.499952, −0.000013, 2.999936)T µ10 = 2.999936 x10 = y10 µ10 = (0.499995, −0.000004, 1.000000)T These ten iterates suggest that the sequence {µk} converges to the eigenvalue λ1 = 3 and the sequence {x(k) } converges to (0.5, 0, 1) = 1 2 v1. Remark 3.5.8 [Disadvantages of power method]. 1. The Power method requires at the beginning that the matrix has only one dominant eigenvalue, and this information is generally unavailable. 2. Even when there is only one dominant eigenvalue, it is not clear how to choose the initial guess x(0) such that it has a non-zero component (c1 in the notation of the theorem) along the eigenvector v1. S. Baskar and S. Sivaji Ganesh 102 Spring 2017-18
- 115. Section 3.5 Eigenvalue Problems Note that in the above example, all the hypothesis are satisfied. Now let us ask the question “What happens when any of the hypotheses of power method is violated?” We discuss these situations through examples. Example 3.5.9 [Dominant eigenvalue is not unique (Failure of H1)]. Consider the matrix B = 1 3 4 0 2 1 0 0 −2 , which has eigenvalues 1, −2, and 2. Clearly, the matrix B has two dominant eigen- values, namely, −2 and 2. We start with an initial guess x(0) = (1, 1, 1) and the first five iterations generated using power method are given below: Iteration No: 1 y1 = Ax0 = (8.000000, 3.000000, −2.000000)T µ1 = 8.000000 x1 = y1 µ1 = (1.000000, 0.375000, −0.250000)T Iteration No: 2 y2 = Ax1 = (1.125000, 0.500000, 0.500000)T µ2 = 1.125000 x2 = y2 µ2 = (1.000000, 0.444444, 0.444444)T Iteration No: 3 y3 = Ax2 = (4.111111, 1.333333, −0.888889)T µ3 = 4.111111 x3 = y3 µ3 = (1.000000, 0.324324, −0.216216)T Iteration No: 4 y4 = Ax3 = (1.108108, 0.432432, 0.432432)T µ4 = 1.108108 x4 = y4 µ4 = (1.000000, 0.390244, 0.390244)T S. Baskar and S. Sivaji Ganesh 103 Spring 2017-18
- 116. Chapter 3. Numerical Linear Algebra Iteration No: 5 y5 = Ax4 = (3.731707, 1.170732, −0.780488)T µ5 = 3.731707 x5 = y5 µ5 = (1.000000, 0.313725, −0.209150)T It is observed that the sequence oscillates even till 1000 iterations as shown below: Iteration No: 998 y998 = Ax997 = (1.103448, 0.413793, 0.413793)T µ998 = 1.103448 x998 = y998 µ998 = (1.000000, 0.375000, 0.375000)T Iteration No: 999 y999 = Ax998 = (3.625000, 1.125000, −0.750000)T µ999 = 3.625000 x999 = y999 µ999 = (1.000000, 0.310345, −0.206897)T Iteration No: 1000 y1000 = Ax999 = (1.103448, 0.413793, 0.413793)T µ1000 = 1.103448 x1000 = y1000 µ1000 = (1.000000, 0.375000, 0.375000)T and so on. This is a clear indication that the power method is not converging in this case. Thus we conclude that the power method when applied to a matrix which has more than one dominant eigenvalue may not converge. Remark 3.5.10 [Dominant eigenvalue is not simple]. It is not necessary to have the dominant eigenvalue of algebraic multiplicity 1 in order that the iterative sequences of power method converge. The important thing is to have a unique dominant eigenvalue and it is allowed to have an algebraic multiplicity r with r > 1. However it is necessary that the corresponding geometric multiplicity should also be r to satisfy the hypothesis (H2) (also see later on, where we discuss the situation where algebraic and geometric multiplicities do not match). In such a case, power method computes only one eigenvector, as is usual. S. Baskar and S. Sivaji Ganesh 104 Spring 2017-18
- 117. Section 3.5 Eigenvalue Problems Let us now illustrate the situation when the hypothesis (H3) is violated. Example 3.5.11 [Failure of hypothesis (H3): Initial guess x(0) is such that c1 = 0]. Consider the matrix (same as in Example 3.5.7) A = 3 0 0 −4 6 2 16 −15 −5 , The eigenvalues of this matrix are λ1 = 3, λ2 = 1 and λ3 = 0. The corresponding eigenvectors are v1 = (1, 0, 2)T , v2 = (0, 2, −5)T and v3 = (0, 1, −3)T . Thus, the hypothesis (H1) and (H2) are satisfied. Here, we choose a different initial guess x0 = (0, 0.5, 0.25)T . Note that the hypothesis (H3) that c1 ̸= 0 is violated here. However, we can see that c2 ̸= 0. The first four iterations of the power method are as follows: Iteration No: 1 y1 = Ax0 = (0.000000, 3.500000, −8.750000)T µ1 = −8.750000 x1 = y1 µ1 = (−0.000000, −0.400000, 1.000000)T Iteration No: 2 y2 = Ax1 = (0.000000, −0.400000, 1.000000)T µ2 = 1.000000 x2 = y2 µ2 = (0.000000, −0.400000, 1.000000)T Iteration No: 3 y3 = Ax2 = (0.000000, −0.400000, 1.000000)T µ3 = 1.000000 x3 = y3 µ3 = (0.000000, −0.400000, 1.000000)T S. Baskar and S. Sivaji Ganesh 105 Spring 2017-18
- 118. Chapter 3. Numerical Linear Algebra Iteration No: 4 y4 = Ax3 = (0.000000, −0.400000, 1.000000)T µ4 = 1.000000 x4 = y4 µ4 = (0.000000, −0.400000, 1.000000)T Thus, the power method converges to λ2, which is the second dominant eigenvalue of the given matrix. Note that in the chosen initial guess, the first coordinate is zero and therefore, c1 in (3.76) has to be zero. Thus, (3.77) reduces to Ak v = λk 2 ( c2v2 + c3 ( λ3 λ2 )k v3 + · · · + cn ( λn λ2 )k vn ) . This makes the iteration to converge to λ2, which is the next dominant eigenvalue. Remark 3.5.12. It is important that we understand the hypothesis (H3) on the initial guess x(0) correctly. Note that (H3) says that the coefficient of v1 (which was denoted by c1) should be non-zero when x(0) is expressed as x(0) = c1v1 + c2v2 + · · · + cnvn. Note that the coefficients c1, c2, · · · , cn are unique as v1, v2, · · · , vn form a basis for Rn. For such a choice of x(0), it may happen that the first coordinate may be zero. That is, if x(0) is written in coordinate form as x(0) = (x (0) 1 , x (0) 2 , · · · , x (0) n )T , it is possible that x (0) 1 = 0 and c1 ̸= 0. Thus, it is not necessary that the power method will converge to the second dominant eigenvalue if the first coordinate of the initial guess is zero. However, we may expect this to happen if c1 = 0. The following example illustrates this fact. Example 3.5.13. Consider the matrix A = 91.4 −22.0 −44.8000 175.2 −41.0 −86.4 105.2 −26.0 −51.4000 . The eigenvalues of this matrix are λ1 = −5, λ2 = 3 and λ3 = 1. The corresponding eigenvectors are v1 = (3, 5, 4)T , v2 = (2, 6, 1)T and v3 = (1, −2, 3)T . S. Baskar and S. Sivaji Ganesh 106 Spring 2017-18
- 119. Section 3.5 Eigenvalue Problems Note that the matrix A satisfies the hypothesis (H1) since -5 is the unique dominant eigenvalue and it is also a simple eigenvalue. The matrix A satisfies the hypothesis (H2) as all eigenvalues are distinct and hence eigevectors form a basis for R3 . Thus the fate of the power method iterates depends solely on the choice of the initial guess x(0) and whether it satisfies the hypothesis (H3) • Let us take the initial guess x(0) = (1, 0.5, 0.25)T . Note that c1 ̸= 0 for this ini- tial guess. Thus the initial guess satisfies the hypothesis (H3) and the iterative sequences generated by power method converges to the dominant eigenvalue λ1 = −5 and the corresponding eigenvector (with a scalar multiple) 1 5 v1. • Let us take the initial guess x(0) = (0, 0.5, 0.25)T . Note that c1 ̸= 0 for this ini- tial guess. Thus the initial guess satisfies the hypothesis (H3) and the iterative sequences generated by power method converges to the dominant eigenvalue λ1 = −5 and the corresponding eigenvector (with a scalar multiple) 1 5 v1. Com- pare this with Example 3.5.11. In the present case the first coordinate of the initial guess vector is zero, just as in Example 3.5.11. In Example 3.5.11 the power method iterate converged to the second dominant eigenvalue and the corresponding eigenvector, which does not happen in the present case. The reason is that in the Example 3.5.11, c1 = 0 for the initial guess chosen, but in the current example c1 ̸= 0. 3.5.2 Gerschgorin’s Theorem An important tool in eigenvalue approximation is the ability to localize the eigenvalues, and the most important tool in eigenvalue localization is the Gerschgorin’s theorem. Gerschgorin’s Circle theorem helps us in localization of eigenvalues of a matrix. This theorem explicitly constructs n disks in the complex plane with centers at the diagonal elements of the matrix; and all the eigenvalues of the matrix lie in the union of these disks. Theorem 3.5.14 [Gerschgorin’s Circle Theorem]. Let A be an n × n matrix. For each k = 1, 2, · · · , n, define ρk by ρk = n∑ j=1 j̸=k |akj|, and Dk denotes the closed disk in the complex plane with centre akk and radius ρk, i.e., Dk = { z ∈ C : |z − akk| ≤ ρk } . S. Baskar and S. Sivaji Ganesh 107 Spring 2017-18
- 120. Chapter 3. Numerical Linear Algebra 1. Each eigenvalue of A lies in one of the disks Dk. That is, no eigenvalue of A lies in C ∪n k=1Dk. 2. Suppose that among the disks D1, D2, · · · , Dn, there is a collection of m disks whose union (denoted by R1) is disjoint from the union of the rest of the n − m disks (denoted by R2). Then exactly m eigenvalues lie in R1 and n − m eigenvalues lie in R2 (here each eigenvalue is counted as many times as its algebraic multiplicity). The disks Dk, k = 1, 2, . . . , n are called the Gerschgorin’s disks. Proof. We will prove only (i) as it is easy, and the proving (ii) is beyond the scope of this course. Let λ be an eigenvalue of A. Then there exists a v = (v1, v2, · · · , vn) ∈ Rn and v ̸= 0 such that Av = λv (3.82) Let 1 ≤ r ≤ n be such that |vr| = max{|v1|, |v2|, · · · , |vn|}. The rth equation of the system of equations (3.82) is given by (actually, of Av − λv = 0) ar1v1 + · · · + ar,r−1vr−1 + (arr − λ)vr + ar,r+1vr+1 + · · · + arnvn = 0 From the last equation, we get λ − arr = v1 vr ar1 + · · · + vr−1 vr ar,r−1 + vr+1 vr ar,r+1 + · · · + vn vr arn (3.83) Taking modulus on both sides of the equation (3.83), and using the triangle inequality |a + b| ≤ |a| + |b| repeatedly we get |λ − arr| ≤ |v1| |vr| |ar1| + · · · + |vr−1| |vr| |ar,r−1| + |vr+1| |vr| |ar,r+1| + · · · + |vn| |vr| |arn| (3.84) In view of the choice of r, the components of the vector v satisfy |vs| |vr| ≤ 1. The last equation (3.84) becomes |λ − arr| ≤ |ar1| + · · · + |ar,r−1| + |ar,r+1| + · · · + |arn| (3.85) Observe that the right hand side of the inequality (3.85) is ρr. This proves that λ ∈ Dr. S. Baskar and S. Sivaji Ganesh 108 Spring 2017-18
- 121. Section 3.5 Eigenvalue Problems Example 3.5.15. For the matrix 4 1 1 0 2 1 −2 0 9 , the Gerschgorin’s disks are given by D1 = { z ∈ C : |z − 4| ≤ 2 } , D2 = { z ∈ C : |z − 2| ≤ 1 } , D3 = { z ∈ C : |z − 9| ≤ 2 } . Draw a picture of these disks and observe that D3 neither intersects D1 nor D2. By (ii) of Theorem 3.5.14, D3 has one eigenvalue and D1 ∪ D2 has two eigenvalues counting multiplicities. Note that the eigenvalues are approximately 4.6318, 1.8828 ∈ D1 ∪ D2 and 8.4853 ∈ D3. Remark 3.5.16. Gerschgorin’s circle theorem is helpful in finding bound for eigenvalues. For the matrix in Example 3.5.15, any number z in D1 satisfies |z| ≤ 6. Similarly any number z in D2 satisfies |z| ≤ 3, and any number z in D3 satisfies |z| ≤ 11. Since any eigenvalue λ lies in one of three disks, we can conclude that |λ| ≤ 11. Remark 3.5.17. The main disadvantage of the power method discussed in Section 3.5.1 is that if a given matrix has more than one dominant eigenvalues, then the method may not converge. So, for a given matrix, we do not know whether the power method will converge or not. Also, as the power method is reasonably slow (see Example 3.5.13 for an illustration) we may have to perform reasonably large number of iterations to come to know that the method is not actually converging. Thus, a tool to find out whether the given matrix has a unique dominant eigen- value or not is highly desirable. The Gerschgorin theorem (Theorem 3.5.14) can sometime be used to see if power method can be used for a given matrix. For instance, in Example 3.5.15, we see that the power method can be used to obtain an approximation to the dominant eigenvalue. Since a matrix A and its transpose (denoted by AT ) have same eigenvalues, we can apply Gerschgorin Circle Theorem to AT and conclude the following corollary. S. Baskar and S. Sivaji Ganesh 109 Spring 2017-18
- 122. Chapter 3. Numerical Linear Algebra Corollary 3.5.18. Let A be an n × n matrix. For each k = 1, 2, · · · , n, define τk by τk = n∑ j=1 j̸=k |ajk|, and Bk denotes the closed disk in the complex plane with centre akk and radius τk. That is, Bk = { z ∈ C : |z − akk| ≤ τk } . 1. Each eigenvalue of A lies in one of the disks Bk. That is, no eigenvalue of A lies in C ∪n k=1Bk. 2. Suppose that among the disks B1, B2, · · · , Bn, there is a collection of m disks whose union (denoted by C1) is disjoint from the union of the rest of the n − m disks (denoted by C2). Then exactly m eigenvalues lie in C1 and n − m eigenvalues lie in C2 (here each eigenvalue is counted as many times as its algebraic multiplicity). Remark 3.5.19. Advantage of the above corollary is the following: Let A be an n × n matrix. Let R denote the region defined by R = ∪n k=1Dk where Dk are as given by Theorem 3.5.14. Similarly, let C denote the region defined by C = ∪n k=1Bk where Bk are as given by Corollary 3.5.18. It may happen that C ⊂ R, in which case we will get better estimate for eigenvalues of A. It may not happen for every matrix A. However whenever such a thing happens, we get better bounds for eigenvalues of A. Such bounds which we obtain using both informations R and C may be called optimum bounds. Let us illustrate this with two examples. Example 3.5.20. For the matrix 3 0 1 0 −2 2 0 2 −3 the ρk, τk (in the notations of Theorem 3.5.14 and Corollary 3.5.18) are given by ρ1 = 1, ρ2 = 2, ρ3 = 3 and τ1 = 0, τ2 = 2, τ3 = 3 S. Baskar and S. Sivaji Ganesh 110 Spring 2017-18
- 123. Section 3.6 Exercises Therefore, the regions R and C are given by R = ∪n k=1Dk = {z : |z − 3| ≤ 1} ∪ {z : |z + 2| ≤ 2} ∪ {z : |z + 3| ≤ 3} and C = ∪n k=1Bk = {z : |z − 3| ≤ 0} ∪ {z : |z + 2| ≤ 2} ∪ {z : |z + 3| ≤ 3}. Clearly C ⊂ R and hence the bounds obtained using C are optimum bounds. We can also see that any eigenvalue λ of the given matrix satisfies 3 ≤ |λ| ≤ 6. Draw both regions C and R. 3.6 Exercises Gaussian Elimination Methods 1. Solve the following systems of linear equations using Naive Gaussian elimination method, and modified Gaussian elimination method with partial pivoting. i) 6x1 + 2x2 + 2x3 = −2, 2x1 + 0.6667x2 + 0.3333x3 = 1, x1 + 2x2 − x3 = 0. ii) 0.729x1 + 0.81x2 + 0.9x3 = 0.6867, x1 + x2 + x3 = 0.8338, 1.331x1 + 1.21x2 + 1.1x3 = 1. iii) x1 − x2 + 3x3 = 2, 3x1 − 3x2 + x3 = −1, x1 + x2 = 3. 2. Solve the system 0.001x1 + x2 = 1, x1 + x2 = 2 i) using Gaussian elimination with partial pivoting with infinite precision arith- metic, ii) using naive Gaussian elimination with 2-digit rounding, and iii) using modified Gaussian elimination method with partial pivoting, using 2- digit rounding. 3. Let ϵ be such that 0 < ϵ ≪ 1. Solve the linear system x1 + x2 + x3 = 6, 3x1 + (3 + ϵ)x2 + 4x3 = 20, 2x1 + x2 + 3x3 = 13 S. Baskar and S. Sivaji Ganesh 111 Spring 2017-18
- 124. Chapter 3. Numerical Linear Algebra using naive Gaussian elimination method, and using modified Gaussian elimina- tion method with partial pivoting. Obtain the residual error in each case on a computer for which the ϵ is less than its machine epsilon. The residual error vector corresponding to an approximate solution x∗ is defined as r = b − Ax∗ , where A and b are the coefficient matrix and the right side vector, respectively, of the given linear system. 4. A matrix A = (aij) is said to be row-equilibrated if the following condition is satisfied max 1≤j≤n |aij| = 1 ∀i = 1, 2, · · · , n. Consider the following system of linear equations 30.00x1 + 591400x2 = 591700 5.291x1 − 6.130x2 = 46.78 i) Find the exact solution of the given system using Gaussian elimination method with partial pivoting (i.e., with infinite precision arithmetic). ii) Solve the given system using naive Gaussian elimination method using 4- digit rounding. iii) Obtain a system of linear equations Ax = b that is equivalent to the given system, where the matrix A is row-equilibrated. Solve the system Ax = b using naive Gaussian elimination method using 4-digit rounding. iv) Compute the relative errors involved in the two approximations obtained above. 5. Consider the system of linear equations given by a11x1+ · · · +a1nxn = b1, · · · · · · an1x1+ · · · +annxn = bn, where aij = 0 whenever i − j ≥ 2. Write the general form of this system. Use naive Gaussian elimination method to solve it, taking advantage of the elements that are known to be zero. Count the number of operations involved in this com- putation. S. Baskar and S. Sivaji Ganesh 112 Spring 2017-18
- 125. Section 3.6 Exercises 6. Use Thomas method to solve the tri-diagonal system of equations −3x1 + x2 = 2, 2x1 − 3x2 + x3 = 6, 2x2 − 3x3 + x4 = 1, 2x3 − 3x4 = −2. LU Decomposition 7. Prove or disprove the following statements: i) An invertible matrix has at most one Doolittle factorization. ii) If a singular matrix has a Doolittle factorization, then the matrix has at least two Doolittle factorizations. 8. Prove that if an invertible matrix A has an LU-factorization, then all principal minors of A are non-zero. 9. Give an example of a non-invertible 3×3 matrix A such that the leading principal minors of order 1 and 2 are non-zero, and A has Doolittle factorization. 10. Use the Doolittle’s factorization to solve the system 4x1 + x2 + x3 = 4, x1 + 4x2 − 2x3 = 4, 3x1 + 2x2 − 4x3 = 6. 11. Show that the matrix 2 2 1 1 1 1 3 2 1 is invertible but has no LU factorization. Do a suitable interchange of rows to get an invertible matrix, which has an LU factorization. 12. Consider A = 2 6 −4 6 17 −17 −4 −17 −20 . Determine directly the factorization A = LDLT , where D is diagonal and L is a lower triangular matrix with 1’s on its diagonal. S. Baskar and S. Sivaji Ganesh 113 Spring 2017-18
- 126. Chapter 3. Numerical Linear Algebra 13. Let A be an n×n matrix which is positive definite. Let S be a non-empty susbset of {1, 2, · · · , n}. Show that the submatrix AS = (aij)i,j∈S is positive definite. 14. Factor the matrix A = 4 6 2 6 10 3 2 3 5 so that A = LLT , where L is lower triangular. 15. Prove the uniqueness of the factorization A = LLT , where L is a lower triangular matrix all of whose diagonal entries are positive. (Hint: Assume that there are lower triangular matrices L1 and L2 with positive diagonals. Prove that L1L−1 2 = I.) 16. Use Cholesky factorization to solve the system of equations x1 + 3x3 = 1, 2x1 − x2 = 3, x1 + 2x3 = −1. [Hint: Instead of solving for Ax = b, solve for AT Ax = AT b.] Matrix Norms 17. The following inequalities show that the notion of convergent sequences of vectors in Rn is independent of the vector norm. Show that the following inequalities hold for each x ∈ Rn i) ∥x∥∞ ≤ ∥x∥2 ≤ √ n∥x∥∞, ii) ∥x∥∞ ≤ ∥x∥1 ≤ n∥x∥∞, iii) ∥x∥2 ≤ ∥x∥1 ≤ √ n∥x∥2. 18. Show that the norm defined on the set of all n × n matrices by ∥A∥ := max 1≤i≤n 1≤j≤n |aij| is not subordinate to any vector norm on Rn . 19. Let A be an invertible matrix. Show that its condition number κ(A) satisfies κ(A) ≥ 1. S. Baskar and S. Sivaji Ganesh 114 Spring 2017-18
- 127. Section 3.6 Exercises 20. Let A and B be invertible matrices with condition numbers κ(A) and κ(B) re- spectively. Show that κ(AB) ≤ κ(A)κ(B). 21. Let A(α) be a matrix depending on a parameter α ∈ R given by A(α) = ( 0.1α 0.1α 1.0 2.5 ) For each α ∈ R, compute the condition number of A(α). Determine an α0 such that the condition number of A(α0) is the minimum of the set { κ (A(α)) : α ∈ R}. In the computation of condition numbers, use the matrix norm that is subordinate to the maximum vector norm on R2 . 22. Consider the following two systems of linear equations x1 + x2 = 1, x1 + 2x2 = 2. and 10−4 x1 + 10−4 x2 = 10−4 , x1 + 2x2 = 2. Let us denote the first and second systems by A1x = b1 and A2x = b2 respectively. Use maximum-norm for vectors and the matrix norm subordinate to maximum- norm for matrices in your computations. i) Solve each of the above systems using Naive Gaussian elimination method. ii) Compute the condition numbers of A1 and A2. iii) For each of the systems, find an upper bound for the relative error in the solution if the right hand sides are approximated by b1 and b2 respectively. iv) Solve the systems A1x = b1 and A2x = b2 where b1 = (1.02, 1.98)T and b2 = (1.02 × 10−4 , 1.98)T using naive Gaussian elimination method. Compute the relative error in each case. Compare the computed relative errors with the bounds obtained above. Iterative Methods 23. Let A be a diagonally dominant matrix. Show that all the diagonal elements of A are non-zero (i.e., aii ̸= 0 for i = 1, 2, · · · , n.). As a consequence, the iterating sequences of Jacobi and Gauss-Seidel methods are well-defined if the coefficient matrix A in the linear system Ax = b is a diagonally dominant matrix. 24. Find the n×n matrix B and the n-dimensional vector c such that the Gauss-Seidal method can be written in the form x(k+1) = Bx(k) + c, k = 0, 1, 2, · · · . S. Baskar and S. Sivaji Ganesh 115 Spring 2017-18
- 128. Chapter 3. Numerical Linear Algebra 25. For each of the following systems, write down the formula for iterating sequences of Jacobi and Gauss-Seidel methods. Compute three iterates by taking x0 = (0, 0, 0)T . Discuss if you can guarantee that these sequences converge to the exact solution. In case you are not sure about convergence, suggest another iterating sequence that converges to the exact solution if possible; and justify that the new sequence converges to the exact solution. i) 5x1 +2x2 +x3 = 0.12, 1.75x1 +7x2 +0.5x3 = 0.1, x1 +0.2x2 +4.5x3 = 0.5. ii) x1 − 2x2 + 2x3 = 1, x1 + x2 − x3 = 1, 2x1 − 2x2 + x3 = 1. iii) x1 + x2 + 10x3 = −1, 2x1 + 3x2 + 5x3 = −6, 3x1 + 2x2 − 3x3 = 4. [Note: As it is only asked to discuss the guarantee of convergence, it is enough to check the sufficient condition for convergence, ie. to check whether the given system is diagonally dominant or not.] 26. Consider the following system of linear equations 0.8647x1 + 0.5766x2 = 0.2885, 0.4322x1 + 0.2882x2 = 0.1442. i) Find the exact solution of the linear system. ii) Using 4-digit chopping, do one iteration of residual corrector method to obtain x(1) . Use the initial guess to be (0, 0)T . iii) Compute the residual vector for the solution obtained in (ii) above. Eigenvalue Problems 27. The matrix A = −3 0 0 2 1 0 1 0 2 has eigenvalues λ1 = −3, λ = 1 and λ3 = 2 and the corresponding eigenvectors may be taken as v1 = (1, 0, 0)T , v2 = (1, 2, 0)T and v3 = (1, 0, 5)T . Perform 3 iterations to find the eigenvalue and the corresponding eigen vector to which the power method converges when we start the iteration with the initial guess x(0) = (0, 0.5, 0.75)T . Without performing the iteration, find the eigenvalue and the corresponding eigenvector to which the power method converges when we start the iteration with the initial guess x(0) = (0.001, 0.5, 0.75)T . Justify your answer. S. Baskar and S. Sivaji Ganesh 116 Spring 2017-18
- 129. Section 3.6 Exercises 28. Find the value of ∥A∥2 for the matrix A given in the above problem using the power method with x(0) = (0, 0.5, 0.75)T . 29. The matrix A = 5.4 0 0 −113.0233 −0.5388 −0.6461 −46.0567 −6.4358 −0.9612 has eigenvalues λ1 = 5.4, λ2 = 1.3 and λ3 = −2.8 with corresponding eigenvectors v1 = (0.2, −4.1, 2.7)T , v2 = (0, 1.3, −3.7)T and v3 = (0, 2.6, 9.1)T . To which eigenvalue and the corresponding eigenvector does the power method converge if we start with the initial guess x(0) = (0, 1, 1)? Justify your answer. 30. For this question, we take the matrix A = −2.7083 −2.6824 0.4543 0.1913 0.7629 0.1007 −0.3235 −0.4052 5.0453 . Let v1, v2, v3 denote eigenvectors corresponding to the eigenvalues λ1, λ2, λ3 re- spectively of the matrix A which are given by λ1 ≈ 5.0187, λ2 ≈ −2.5313, λ3 ≈ 0.6125 and v1 ≈ (0.25, 0.13, 5.02)T , v2 ≈ (2.53, −0.15, 0.1)T , v3 ≈ (−0.49, 0.61, 0.02)T . Answer the following questions: (End-Sem, Spring 2012) i) Obtain the Gerschgorin disks associated to the matrix A and pictorially represent them in the complex plane. ii) Can Gerschgorin theorem be used to show that the matrix A has a unique dominant eigenvalue? Justify your answer. iii) Define the iterative sequences {µk} and x(k) using power method that con- verge to λ1 and αv1 for some constant α when the initial guess is x(0) = (1, 1, 1)T . Perform one iteration. iv) If we take the initial guess x(0) = (0, 1, 1)T , then show that the iterative sequence obtained by power method converges to λj and Kvj for some j ∈ {1, 2, 3} and for some constant K. What is the value of j and possible values of K? (Hint: x(0) ≈ 0.3060v1 + 0.1864v2 + 1.6748v3. ) v) Give all possible initial guesses for which the sequence {µk} obtained using power method converges to λ2. Justify your answer. vi) Give all possible initial guesses for which the sequence {µk} obtained using power method converges to λ3. Justify your answer. S. Baskar and S. Sivaji Ganesh 117 Spring 2017-18
- 130. Chapter 3. Numerical Linear Algebra 31. Use Gerschgorin’s circle theorem to determine the intervals in which the eigen- values of the matrix A = 0.5 0 0.2 0 3.15 −1 0.57 0 −7.43 . lie, given that all eigenvalues of A are real. Show that power method can be applied for this matrix to find the dominant eigenvalue without computing eigen- values explicitly. Compute the first three iterates of Power method sequences. 32. Use the Gerschgorin Circle theorem to determine bounds for the eigenvalues for the following matrices. Also find optimum bounds wherever possible. Also draw the pictures of all the regions given by Greschgorin circles. 1 0 0 −1 0 1 −1 −1 2 , 4 −1 0 −1 4 −1 −1 −1 4 , 4.75 2.25 −0.25 2.25 4.75 1.25 −0.25 1.25 4.75 , 1 0 −1 1 2 2 −1 1 0 1 3 −2 1 0 1 4 . 33. Show that the imaginary parts of the eigenvalues of 3 1/3 2/3 1 −4 0 1/2 1/2 −1 all lie in the interval [−1, 1]. S. Baskar and S. Sivaji Ganesh 118 Spring 2017-18
- 131. CHAPTER 4 Nonlinear Equations One of the most frequently occurring problems in practical applications is to find the roots of equations of the form f(x) = 0, (4.1) where f : [a, b] → R is a given nonlinear function. It is well-known that not all nonlinear equations can be solved explicitly to obtain the exact value of the roots and hence, we need to look for methods to compute approximate value of the roots. By approximate root to (4.1), we mean a point x∗ ∈ R for which the value f(x) is very near to zero, ie., f(x∗ ) ≈ 0. In this chapter, we introduce various iterative methods to obtain an approximation to a real root of equations of the form (4.1) with f being a continuous nonlinear function. The key idea in approximating the real roots of (4.1) consists of two steps: 1. Starting Step: Take one or more points (arbitrarily or following a procedure) xi ∈ [a, b] (i = 0, 1, · · · , m, m ∈ N) around a root of the equation (4.1). Consider xm as an approximation to the root of (4.1). 2. Improving Step: If xm is not ‘close’ to the required root, then devise a procedure to obtain another point xm+1 that is ‘more close’ to the root than xm. Repeat the improving step until we obtain a point xn (n ≥ m) which is ‘sufficiently close’ to the required root. This process of improving the approximation to the root is called the iterative process (or iterative procedure), and such methods are called iterative methods. In an iterative method, we obtain a sequence of numbers {xn} which is expected to converge S. Baskar and S. Sivaji Ganesh 119 Spring 2017-18
- 132. Chapter 4. Nonlinear Equations to the root of (4.1) as n → ∞. We investigate conditions on the function f, its domain, and co-domain under which the sequence of iterates converge to a root of the equation (4.1). We classify the iterative methods discussed in this chapter into two types, namely, 1. Closed Domain Methods: As the starting step, these methods need the knowl- edge of an interval in which at least one root of the given nonlinear equation ex- ists. Further iterations include the restriction of this interval to smaller intervals in which root lies. These methods are also called bracketing methods. 2. Open Domain Methods: The xi’s mentioned in the starting step above are chosen arbitrarily and the consecutive iterations are based on a formula. In the case of closed domain methods, the difficult part is to locate an interval containing a root. But, once this is done, the iterative sequence will surely converge as we will see in Section 4.1. In this section, we discuss two closed domain methods, namely, the bisection method and the regula-falsi method. In the case of open domain methods, it is easy at the starting step as we can choose the xi’s arbitrarily. But, it is not necessary that the sequence converges. We discuss the secant method, the Newton- Raphson method and the fixed point method in Section 4.3, which are some of the open domain methods. 4.1 Closed Domain Methods The idea behind the closed domain methods is to start with an interval (denoted by [a0, b0]) in which there exists at least one root of the given nonlinear equation and then reduce the length of this interval iteratively with the condition that there is at least one root of the equation at each iteration. Note that the initial interval [a0, b0] can be obtained using the intermediate value theo- rem (as we always assume that the nonlinear function f is continuous) by checking the condition that f(a0)f(b0) < 0. That is, f(a0) and f(b0) are of opposite sign. The closed domain methods differ from each other only by the way they go on reducing the length of this interval at each iteration. In the following subsections we discuss two closed domain methods, namely, (i) the bisection method and (ii) the regula-falsi method. S. Baskar and S. Sivaji Ganesh 120 Spring 2017-18
- 133. Section 4.1 Closed Domain Methods 4.1.1 Bisection Method The most simple way of reducing the length of the interval is to sub-divide the interval into two equal parts and then take the sub-interval that contains a root of the equation and discard the other part of the interval. This method is called the bisection method. Let us explain the procedure of generating the first iteration of this method. Step 1: Define x1 to be the mid-point of the interval [a0, b0]. That is, x1 = a0 + b0 2 . Step 2: Now, exactly one of the following two statements hold. 1. x1 solves the nonlinear equation. That is, f(x1) = 0. 2. Either f(a0)f(x1) < 0 or f(b0)f(x1) < 0. If case (1) above holds, then x1 is a required root of the given equation f(x) = 0 and therefore we stop the iterative procedure. If f(x1) ̸= 0, then case (2) holds as f(a0) and f(b0) are already of opposite signs. In this case, we define a subinterval [a1, b1] of [a0, b0] as follows. [a1, b1] = { [a0, x1], if f(a0)f(x1) < 0, [x1, b0], if f(b0)f(x1) < 0. The outcome of the first iteration of the bisection method is the interval [a1, b1] and the first member of the corresponding iterative sequence is the real number x1. Observe that • the length of the interval [a1, b1] is exactly half of the length of [a0, b0] and • [a1, b1] has at least one root of the nonlinear equation f(x) = 0. Similarly, we can obtain x2 and [a2, b2] as the result of the second iteration and so on. We now present the algorithm for the bisection method. Algorithm 4.1.1 [Bisection Method]. Hypothesis: There exists an interval [a0, b0] such that the function f : [a0, b0] → R is continuous, and the numbers f(a0) and f(b0) have opposite signs. Algorithm: Step 1: For n = 0, 1, 2, · · · , define the iterative sequence of the bisection method as xn+1 = an + bn 2 , which is the midpoint of the interval [an, bn]. S. Baskar and S. Sivaji Ganesh 121 Spring 2017-18
- 134. Chapter 4. Nonlinear Equations Step 2: One of the following two cases hold. 1. xn+1 solves the nonlinear equation. That is, f(xn+1) = 0. 2. Either f(an)f(xn+1) < 0 or f(bn)f(xn+1) < 0. Define the subinterval [an+1, bn+1] of [an, bn] as follows. [an+1, bn+1] = { [an, xn+1], if f(an)f(xn+1) < 0, [xn+1, bn], if f(bn)f(xn+1) < 0. Step 3: Stop the iteration if one of the following happens: • the case (1) in step 2 holds. Then declare the value of xn+1 as the required root. • (bn+1 − an+1) is sufficiently small (less than a pre-assigned positive quantity). Then declare the value of xn+2 as the required root up to the desired accuracy. If any of the above stopping criteria does not hold, then repeat step 1 with n replaced by n + 1. Continue this process till one of the above two stopping criteria is fulfilled. Remark 4.1.2. In practice, one may also use any of the stopping criteria listed in Section 4.2, either single or multiple criteria. Assuming that, for each n = 1, 2, · · · , the number xn is not a root of the nonlinear equation f(x) = 0, we get a sequence of real numbers {xn}. The question is whether this sequence converges to a root of the nonlinear equation f(x) = 0. We now discuss the error estimate and convergence of the iterative sequence generated by the bisection method. Theorem 4.1.3 [Convergence and Error Estimate of Bisection Method]. Hypothesis: Let f : [a0, b0] → R be a continuous function such that the numbers f(a0) and f(b0) have opposite signs. Conclusion: • There exists an r ∈ (a0, b0) such that f(r) = 0 and the iterative sequence {xn} of the bisection method converges to r. • For each n = 0, 1, 2, · · · , we have the following error estimate |xn+1 − r| ≤ ( 1 2 )n+1 (b0 − a0). (4.2) S. Baskar and S. Sivaji Ganesh 122 Spring 2017-18
- 135. Section 4.1 Closed Domain Methods Proof. It directly follows from the construction of the intervals [an, bn] that bn − an = 1 2 (bn−1 − an−1) = · · · = ( 1 2 )n (b0 − a0). As a consequence, we get lim n→∞ (bn − an) = 0. By algebra of limits of sequences, we get lim n→∞ an = lim n→∞ bn. Since for each n = 0, 1, 2, · · · , the number xn+1 is the mid-point of the interval [an, bn], we also have an < xn+1 < bn. Now by sandwich theorem for sequences, we conclude that the sequence {xn} of mid- points also converges to the same limit as the sequences of end-points. Thus we have lim n→∞ an = lim n→∞ bn = lim n→∞ xn = r (say). (4.3) Since for each n = 0, 1, 2, · · · , we have f(an)f(bn) < 0, applying limits on both sides of the inequality and using the continuity of f, we get f(r)f(r) ≤ 0, (4.4) from which we conclude that f(r) = 0. That is, the sequence of mid-points {xn} defined by the bisection method converges to a root of the nonlinear equation f(x) = 0. Since the sequences {an} and {bn} are non-decreasing and non-increasing, respec- tively, for each n = 0, 1, 2, · · · , we have r ∈ [an, bn]. Also, xn+1 is the mid-point of the interval [an, bn]. Therefore, we have |xn+1 − r| ≤ 1 2 (bn − an) = · · · = ( 1 2 )n+1 (b0 − a0), which is the required estimate. S. Baskar and S. Sivaji Ganesh 123 Spring 2017-18
- 136. Chapter 4. Nonlinear Equations Corollary 4.1.4. Let ϵ > 0 be given. Let f satisfy the hypothesis of bisection method with the interval [a0, b0]. Let xn be as in the bisection method, and r be the root of the nonlinear equation f(x) = 0 to which bisection method converges. Then |xn − r| ≤ ϵ whenever n satisfies n ≥ log(b0 − a0) − log ϵ log 2 (4.5) Proof. By the error estimate of bisection method given by (4.2), we are sure that |xn − r| ≤ ϵ, whenever n is such that ( 1 2 )n (b0 − a0) ≤ ϵ. By taking logarithm on both sides of the last inequality, we get the desired estimate on n. Remark 4.1.5. The Corollary 4.1.4 tells us that if we want an approximation xn to the root r of the given equation such that the absolute error is less than a pre-assigned positive quantity, then we have to perform n iterations, where n is the least integer that satisfies the inequality (4.5). It is interesting to observe that to obtain this n, we don’t need to know the root r. Example 4.1.6. Let us find an approximate root to the nonlinear equation sin x + x2 − 1 = 0 using bisection method so that the resultant absolute error is at most ϵ = 0.125. To apply bisection method, we must choose an interval [a0, b0] such that the function f(x) = sin x + x2 − 1 satisfies the hypothesis of bisection method. Note that f satisfies hypothesis of bi- S. Baskar and S. Sivaji Ganesh 124 Spring 2017-18
- 137. Section 4.1 Closed Domain Methods section on the interval [0, 1]. In order to achieve the required accuracy, we should first decide how many iterations are needed. The inequality (4.5), says that required accuracy is achieved provided n satisfies n ≥ log(1) − log(0.125) log 2 = 3 Thus we have to compute x3. We will do it now. Iteration 1: We have a0 = 0, b0 = 1. Thus x1 = 0.5. Since, f(x1) = −0.27 < 0, f(0) < 0, and f(1) > 0, we take [a1, b1] = [x1, b0] = [0.5, 1]. Iteration 2: The mid-point of [0.5, 1] is x2 = 0.75. Since f(x2) = 0.24 > 0, f(0.5) < 0, and f(1) > 0, we take [a2, b2] = [a1, x2] = [0.5, 0.75]. Iteration 3: The mid-point of [0.5, 0.75] is x3 = 0.625. Since f(x3) = −0.024 < 0, f(0.5) < 0 and f(0.75) > 0, we take [a3, b3] = [x3, b2] = [0.625, 0.75]. As suggested by the formula (4.5), we stop the iteration here and take the approximate root of the given equation as the mid-point of the interval [a3, b3], which is x4 ≈ 0.6875. Note that the true value is r ≈ 0.636733. The absolute error is 0.05 which is much less than the required accuracy of 0.125. Remark 4.1.7 [Comments on Bisection method]. 1. Note that the mid-point of an interval [a, b] is precisely the x − coordinate of the point of intersection of the line joining the points (a, sgn(f(a))) and (b, sgn(f(b))) with the x-axis. 2. Let f, and the interval [a0, b0] satisfy the hypothesis of bisection method. Even if the nonlinear equation f(x) = 0 has more than one root in the interval [a, b], the bisection method chooses the root that it tries to approximate. In other words, once we fix the initial interval [a0, b0], the bisection method takes over and we cannot control it to find some specific root than what it chooses to find. S. Baskar and S. Sivaji Ganesh 125 Spring 2017-18
- 138. Chapter 4. Nonlinear Equations 3. Given a function f, choosing an interval [a, b] such that f(a)f(b) < 0 is crucial to bisection method. There is no general procedure to find such an interval. This is one of the main drawbacks of bisection method. 4. Bisection method cannot be used to find zero of functions for which graph touches x-axis but does not cross x-axis. 5. There is a misconception about bisection method’s order of convergence (see Chapter 1 for the definition), which is claimed to be 1. However there is no proof of |xn+1 − r| ≤ c|xn − r|. If the sequence {xn} converges linearly to r, then we get |xn+1 − r| ≤ cn+1 |x0 − r|. In the case of bisection method, we have this inequality with c = 1/2, which is only a necessary condition for the convergence to be linear but not a sufficient condition. 4.1.2 Regula-falsi Method The regula-falsi method is similar to the bisection method. Although the bisection method (discussed in the previous subsection) always converges to the root, the con- vergence is very slow, especially when the length of the initial interval [a0, b0] is very large and the equation has the root very close to the one of the end points. This is because, at every iteration, we are subdividing the interval [an, bn] into two equal parts and taking the mid-point as xn+1 (the (n + 1)th member of the iterative sequence). Therefore, it takes several iterations to reduce the length of the interval to a very small number, and as a consequence the distance between the root and xn+1. The regula-falsi method differs from the bisection method only in the choice of xn+1 in the interval [an, bn] for each n = 0, 1, 2, · · · . Instead of taking the midpoint of the interval, we now take the x-coordinate of the point of intersection of the line joining the points (an, f(an)) and (bn, f(bn)) with the x-axis. Let us now explain the procedure of generating the first iteration of this method. Step 1: Assume the hypothesis of the bisection method and let [a0, b0] be the initial interval. The line joining the points (a0, f(a0)) and (b0, f(b0)) is given by y = f(a0) + f(b0) − f(a0) b0 − a0 (x − a0), The first member x1 of the regula-falsi iterative sequence is the x-coordinate of the point of intersection of the above line with the x-axis. Therefore, x1 satisfies the equation f(a0) + f(b0) − f(a0) b0 − a0 (x1 − a0) = 0 S. Baskar and S. Sivaji Ganesh 126 Spring 2017-18
- 139. Section 4.1 Closed Domain Methods and is given by x1 = a0 − f(a0) b0 − a0 f(b0) − f(a0) , which can also be written as x1 = a0f(b0) − b0f(a0) f(b0) − f(a0) . Step 2: Now, exactly one of the following two statements hold. 1. x1 solves the nonlinear equation. That is, f(x1) = 0. 2. Either f(a0)f(x1) < 0 or f(b0)f(x1) < 0. If case (1) above holds, then x1 is a required root of the given equation f(x) = 0 and therefore we stop the iterative procedure. If f(x1) ̸= 0, then case (2) holds as f(a0) and f(b0) are already of opposite signs. We now define a subinterval [a1, b1] of [a0, b0] as follows. [a1, b1] = { [a0, x1], if f(a0)f(x1) < 0, [x1, b0], if f(b0)f(x1) < 0. The outcome of the first iteration of the regula-falsi method is the interval [a1, b1] and the first member of the corresponding iterative sequence is the real number x1. Observe that • the length of the interval [a1, b1] may be (although not always) much less than half of the length of [a0, b0] and • [a1, b1] has at least one root of the nonlinear equation f(x) = 0. We now summarize the regula-falsi method. Algorithm 4.1.8 [Regula-falsi method]. Hypothesis: Same as bisection method. Algorithm: Step 1: For n = 0, 1, 2, · · · , define the iterative sequence of the regula-falsi method as xn+1 = an − f(an) bn − an f(bn) − f(an) (4.6) or xn+1 = anf(bn) − bnf(an) f(bn) − f(an) , (4.7) which is the x-coordinate of the point of intersection of the line joining the points S. Baskar and S. Sivaji Ganesh 127 Spring 2017-18
- 140. Chapter 4. Nonlinear Equations (an, f(an)) and (bn, f(bn)) (obtained at the nth iteration) with the x-axis. Step 2: One of the following two cases hold. 1. xn+1 solves the nonlinear equation. That is, f(xn+1) = 0. 2. Either f(an)f(xn+1) < 0 or f(bn)f(xn+1) < 0. Define the subinterval [an+1, bn+1] of [an, bn] as follows. [an+1, bn+1] = { [an, xn+1], if f(an)f(xn+1) < 0, [xn+1, bn], if f(bn)f(xn+1) < 0. Step 3: Stop the iteration if the case (1) in step 2 holds and declare the value of xn+1 as the required root. Otherwise repeat step 1 with n replaced by n + 1. Continue this process till a desired accuracy is achieved. Remark 4.1.9 [Stopping criteria]. Unlike in the case of bisection method, there is no clear way of stopping the iteration of regula-falsi method as the length of the interval [an, bn] obtained at the (n+1)th iteration may not converge to zero as n → ∞. This situation occurs especially when the function f is concave or convex in the interval [a0, b0] as illustrated in Example 4.1.11. Remark 4.1.10 [Important observations]. Assuming that, for each n = 1, 2, · · · , the number xn is not a root of the nonlinear equation f(x) = 0, we get a sequence of real numbers {xn}. The question is whether this sequence converges to a root of the nonlinear equation f(x) = 0. Before addressing this question, let us consolidate the information that we have so far. 1. The sequence of left end points of the intervals [an, bn] is a non-decreasing sequence that is bounded above by b. That is, a0 ≤ a1 ≤ · · · ≤ an · · · ≤ b0. Hence the sequence {an} has a limit, i.e., limn→∞ an exists. 2. The sequence of right end points of the intervals [an, bn] is a non-increasing S. Baskar and S. Sivaji Ganesh 128 Spring 2017-18
- 141. Section 4.1 Closed Domain Methods sequence that is bounded below by a. That is, b0 ≥ b1 ≥ · · · ≥ bn · · · ≥ a0. Hence the sequence {bn} has a limit, i.e., limn→∞ bn exists. 3. Since an < bn for all n = 0, 1, 2, · · · , we conclude that lim n→∞ an ≤ lim n→∞ bn. If the lengths of the intervals obtained in regula-falsi method tend to zero, then we have lim n→∞ an = lim n→∞ bn, in which case, we also have by sandwich theorem lim n→∞ an = lim n→∞ xn+1 = lim n→∞ bn, as an < xn+1 < bn for all n = 0, 1, 2, · · · . In this case, the common limit will be a root of the nonlinear equation, as is the case with bisection method. It is important to observe that, it may happen that the lengths of the subintervals chosen by regula-falsi method do not go to zero. In other words, if an → α and bn → β, then it may happen that α < β. Let us illustrate this case by the following example. Example 4.1.11. Consider the nonlinear equation ex − 2 = 0. Note that the function f : R → R defined by f(x) = ex − 2 satisfies the hypothesis of the regula-falsi method on the interval [0, 1]. Let us start the iteration with the initial interval [a0, b0] = [0, 1]. Iteration 1: Using the formula (4.6), we get x1 = 1 e − 1 . Since f(x1) ≈ −0.21043, the subinterval chosen by regula-falsi method is the interval [a1, b1] = [x1, 1]. S. Baskar and S. Sivaji Ganesh 129 Spring 2017-18
- 142. Chapter 4. Nonlinear Equations Iteration 2: We have x2 ≈ 0.67669. Since f(x2) ≈ −0.032645, the subinterval cho- sen by regula falsi method is the interval [a2, b2] = [x2, 1]. By continuing these computations, we see that the subinterval at every iteration chosen by regula-falsi method is the right half of the interval defined in the previous iteration. In other words, for n = 1, 2, · · · , we have [an, bn] = [xn, 1]. This is clearly seen geometrically as the function f is convex. A similar situation occurs when the given function f is concave (at least in the initial interval [a0, b0]). We checked this to be true for n = 1, 2 and a proof for a general n is left as an exercise. Note in this example that β = 1 and α ≤ r ̸= 1, where r is the root of the given equation f(x) = 0 to which the regula-falsi method is expected to converge. The last example ruled out any hopes of proving that α = β. However, it is true that the sequence {xn} converges to a root of the nonlinear equation f(x) = 0, and as can be expected, the proof is by contradiction argument. Let us now state the theorem on regula-falsi method. Theorem 4.1.12 [Convergence of Regula-falsi method]. Hypothesis: Let f : [a0, b0] → R be a continuous function such that the numbers f(a0) and f(b0) have opposite signs. Conclusion: If {xn} is the iterative sequence defined by regula-falsi method. Then the sequence {xn} converges to an r ∈ (a0, b0) such that f(r) = 0. Proof is omitted. Example 4.1.13. Let us find an approximate root to the nonlinear equation sin x + x2 − 1 = 0 using regula-falsi method. Exact root is approximately 0.637. We choose the initial interval as [a0, b0] = [0, 1] as done in Example 4.1.6. Iteration 1: We have a0 = 0, b0 = 1. Thus x1 = 0.54304. Since f(x1) = −0.18837 < 0, f(0) < 0, and f(1) > 0, we take [a1, b1] = [x1, b0] = [0.54304, 1]. Iteration 2: Using the regula-falsi formula (4.6), we get x2 = 0.62662. Since f(x2) = −0.020937 < 0, f(0.54304) < 0, and f(1) > 0, we take [a2, b2] = [x2, 1] = [0.62662, 1]. S. Baskar and S. Sivaji Ganesh 130 Spring 2017-18
- 143. Section 4.2 Stopping Criteria Iteration 3: Using the regula-falsi formula (4.6), we get x3 = 0.63568. Since f(x3) = −0.0021861 < 0, f(0.62662) < 0, and f(1) > 0, we take [a3, b3] = [x3, 1] = [0.63658, 1]. Iteration 4: Using the regula-falsi formula (4.6), we get x4 = 0.63662. and so on. In this example also, we observe that the lengths of the intervals [an, bn] do not seem to be tending to zero as n → ∞. However, we see that the sequence {xn} is approaching the root of the given equation. 4.2 Stopping Criteria The outcome of any iterative method for a given nonlinear equation is a sequence of real numbers that is expected to converges to a root of the equation. When we implement such a methods on a computer, we cannot go on computing the iterations indefinitely and needs to stop the computation at some point. It is desirable to stop computing the iterations when the xn’s are reasonably close to an exact root r for a sufficiently large n. In other words, we want to stop the computation at the nth iteration when the computed value is such that |xn − r| < ϵ for some pre-assigned positive number ϵ. In general, we don’t know the root r of the given nonlinear equation to which the iterative sequence is converging. Therefore, we have no idea of when to stop the iteration as we have seen in the case of regula-falsi method. In fact, this situation will be there for any open domain methods discussed in the next section. An alternate way is to look for some criteria that does not use the knowledge of the root r, but gives a rough idea of how close we are to this root. Such a criteria is called a stopping criteria. We now list some of the commonly used stopping criteria for iterative methods to nonlinear equations. Stopping Criterion 1: Fix a K ∈ N, and ask the iteration to stop after finding xK. This criterion is borne out of fatigue, as it clearly has no mathematical reason why the K fixed at the beginning of the iteration is more important than any other natural number! If we stop the computation using this criterion, we will declare xK to be the approximate root to the nonlinear equation f(x) = 0. Stopping Criterion 2: Fix a real number ϵ > 0 and a natural number N. Ask the iteration to stop after finding xk such that |xk − xk−N | < ϵ. S. Baskar and S. Sivaji Ganesh 131 Spring 2017-18
- 144. Chapter 4. Nonlinear Equations One may interpret this stopping criterion by saying that there is ‘not much’ improve- ment in the value of xk compared to a previous value xk−N . If we stop the computation using this criterion, we will declare xk to be the approximate root of the nonlinear equation f(x) = 0. It is more convenient to take N = 1 in which case, we get the stopping criteria |xk − xk−1| < ϵ. Stopping Criterion 3: Fix a real number ϵ > 0 and a natural number N. Ask the iteration to stop after finding xk such that xk − xk−N xk < ϵ. If we stop the computation using this criterion, we will declare xk to be the approximate root to the nonlinear equation f(x) = 0. As in the above case, it is convenient to take N = 1. Stopping Criterion 4: Fix a real number ϵ > 0 and ask the iteration to stop after finding xk such that |f(xk)| < ϵ. If we stop the computation using this criterion, we will declare xk to be the approximate root to the nonlinear equation f(x) = 0. Sometimes the number |f(xk)| is called the residual error corresponding to the approximate root xk of the nonlinear equation f(x) = 0. In practice, one may use any of the stopping criteria listed above, either single or multiple criteria. Remark 4.2.1. We can also use any of the above stopping criteria in bisection method. 4.3 Open Domain Methods In the closed domain methods described in the previous section, we have seen that the iterative sequences always converge to a root of the nonlinear equation. However, initially to start the iteration, we need to give an interval where at least one root of the equation is known to exist. In many practical situations it may be very difficult to obtain this interval manually. Also, it will be very expensive to find this interval S. Baskar and S. Sivaji Ganesh 132 Spring 2017-18
- 145. Section 4.3 Open Domain Methods using a computer program as we have to adopt a trial and error algorithm for this. Therefore, it is highly desirable to device an iterative method that does not need this information and gives freedom to choose any arbitrary starting point as done in the iterative method for linear systems. This is the objective of an open domain method. The open domain methods do not pre-suppose that a root is enclosed in an inter- val [a0, b0]. These methods always start with a set of initial guesses x0, x1, · · · , xm distributed anywhere on the real line (preferably sufficiently close to a root of the equa- tion) and tell us how to compute the values xm+1, xm+2, · · · . in such a way that the resulting iterative sequence {xn} converges to a root of the equation. In this section we study three open domain iterative methods for approximating a root of a given nonlinear equation. 4.3.1 Secant Method The straightforward modification of the regula-falsi method is the well-known secant method. The only difference in the secant method (when compared to the regula-falsi method) is that here we do not demand the initial guesses a0 and b0 to be on the either side of a root. Let us now present the algorithm of the secant method. Algorithm 4.3.1 [Secant method]. Hypothesis: Given any initial values x0 and x1 (not necessarily on the either side of a root) such that f(x0) ̸= f(x1). Algorithm: Step 1: For n = 1, 2, · · · , the iterative sequence for secant method is given by xn+1 = f(xn)xn−1 − f(xn−1)xn f(xn) − f(xn−1) . (4.8) This expression can also be written as xn+1 = xn − f(xn) xn − xn−1 f(xn) − f(xn−1) (4.9) Step 2: Choose any one of the stopping criteria (or a combination of them) discussed in Section 4.2. If this criterion is satisfied, stop the iteration. Otherwise, repeat the step 1 by replacing n with n + 1 until the criterion is satisfied. Recall that xn+1 for each n = 1, 2, · · · given by (4.8) (or (4.9)) is the x-coordinate of the point of intersection of the secant line joining the points (xn−1, f(xn−1)) and (xn, f(xn)) S. Baskar and S. Sivaji Ganesh 133 Spring 2017-18
- 146. Chapter 4. Nonlinear Equations 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 Figure 4.1: Iterative points of secant method. with the x-axis and hence the name secant method. Remark 4.3.2. It is evident that the secant method fails to determine xn+1 if we have f(xn−1) = f(xn). Observe that such a situation never occurs in regula-falsi method. Example 4.3.3. Consider the equation sin x + x2 − 1 = 0. Let x0 = 0, x1 = 1. Then the iterations from the secant method are given by n xn ϵ 2 0.543044 0.093689 3 0.626623 0.010110 4 0.637072 0.000339 5 0.636732 0.000001 Figure 4.1 shows the iterative points x2 and x3 in black bullet. Recall that the exact value of the root (to which the iteration seems to converge) unto 6 significant digits is r ≈ 0.636733. Obviously, the secant method is much faster than bisection method in this example. We now state the convergence theorem on secant method. S. Baskar and S. Sivaji Ganesh 134 Spring 2017-18
- 147. Section 4.3 Open Domain Methods Theorem 4.3.4 [Convergence of Secant Method]. Hypothesis: 1. Let f : R → R be a C2 (R) function. 2. Let r be a simple root of the nonlinear equation f(x) = 0, that is f′ (r) ̸= 0. Conclusion: Then there exists a δ > 0 such that for every x0, x1 ∈ [r − δ, r + δ], 1. the secant method iterative sequence {xn} is well-defined. 2. the sequence {xn} belongs to the interval [r − δ, r + δ]. 3. lim n→∞ xn = r. 4. Further, we have lim n→∞ |xn+1 − r| |xn − r|α = f′′ (r) 2f′(r) α/(α+1) , (4.10) where α = ( √ 5 + 1)/2 ≈ 1.62. The proof for this is omitted for this course. Remark 4.3.5. The expression (4.10) implies that the order of convergence (see Chapter 1 for the definition) of the iterative sequence of secant method is 1.62. 4.3.2 Newton-Raphson Method In Theorem 4.3.4, we have seen that the secant method has more than linear order of convergence. This method can further be modified to achieve quadratic convergence. To do this, we first observe that when the iterative sequence of the secant method converges, then as n increases, we see that xn−1 approaches xn. Thus, for a sufficiently large value of n, we have f′ (xn) ≈ f(xn) − f(xn−1) xn − xn−1 , provided f is a C1 function. Thus, if f(x) is differentiable, then on replacing in (4.9), the slope of the secant by the slope of the tangent at xn, we get the iteration formula xn+1 = xn − f(xn) f′(xn) (4.11) and is called the Newton-Raphson method. S. Baskar and S. Sivaji Ganesh 135 Spring 2017-18
- 148. Chapter 4. Nonlinear Equations Remark 4.3.6 [Geometric Motivation]. If a function f is differentiable at a point x0, then the tangent (y = g(x)) to the graph of f at the point (x0, f(x0)) is given by g(x) = f′ (x0)(x − x0) + f(x0). We may assume that for x ≈ x0, f(x) ≈ g(x). This can also be interpreted as “If a function f is differentiable at a point, then the graph of f looks like a straight line, for x ≈ x0 on zooming”. Now, if we choose the initial guess x0 very close to the root r of f(x) = 0. That is., if r ≈ x0, we have g(r) ≈ f(r) = 0. This gives (approximately) 0 ≈ f′ (x0)(r − x0) + f(x0). Up on replacing r by x1 and using ‘=’ symbol instead of ‘≈’ symbol, we get the first iteration of the Newton-Raphson’s iterative formula (4.11). Recall in secant method, we need two initial guesses x0 and x1 to start the iteration. In Newton-Raphson method, we need one initial guess x0 to start the iteration. The consecutive iteration x1 is the x-coordinate of the point of intersection of the x-axis and the tangent line at x0, and similarly for the other iterations. This geometrical interpretation of the Newton-Raphson method is clearly observed in Figure 4.2. We now derive the Newton-Raphson method under the assumption that f is a C2 function. Let x0 be given. The Taylor’s polynomial of degree n = 1 with remainder is given by f(x) = f(x0) + f′ (x0)(x − x0) + (x − x0)2 2! f′′ (ξ), where ξ lies between x0 and x. When x0 is very close to x, the last term in the above equation is smaller when compared to the other two terms on the right hand side. By neglecting this term we have f(x) ≈ f(x0) + f′ (x0)(x − x0). (4.12) Notice that the graph of the function g(x) = f(x0) + f′ (x0)(x − x0) is precisely the tangent line to the graph of f at the point (x0, f(x0)). We now define x1 to be the x- coordinate of the point of intersection of this tangent line with the x-coordinate. That is, the point x1 is such that g(x1) = 0, which gives x1 = x0 − f(x0) f′(x0) . (4.13) S. Baskar and S. Sivaji Ganesh 136 Spring 2017-18
- 149. Section 4.3 Open Domain Methods 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −2 −1.5 −1 −0.5 0 0.5 1 Figure 4.2: Iteration Procedure of Newton-Raphson’s method for f(x) = sin(x)+x2 −1. This gives the first member of the iterative sequence of the Newton-Raphson’s method. We now summarize the Newton-Raphson’s method. Algorithm 4.3.7. Hypothesis: 1. Let the function f be C1 and r be the root of the equation f(x) = 0 with f′ (r) ̸= 0. 2. The initial guess x0 is chosen sufficiently close to the root r. Algorithm: Step 1: For n = 0, 1, 2, · · · , the iterative sequence of the Newton-Raphson’s method is given by (4.11) xn+1 = xn − f(xn) f′(xn) . Step 2: Choose any one of the stopping criteria (or a combination of them) discussed in Section 4.2. If this criterion is satisfied, stop the iteration. Otherwise, repeat the step 1 by replacing n with n + 1 until the criterion is satisfied. S. Baskar and S. Sivaji Ganesh 137 Spring 2017-18
- 150. Chapter 4. Nonlinear Equations Remark 4.3.8. It is evident that if the initial guess x0 is such that f′(xn) = 0, for some n ∈ N, then the Newton-Raphson method fails. Geometrically, this means that the tangent line to the graph of f at the point (xn, f(xn)) is parallel to the x-axis. Therefore, this line never intersects x-axis and hence xn+1 never exists. See Remark 4.3.2 for the failure of secant method and compare it with the present case. Example 4.3.9. Consider the equation sin x + x2 − 1 = 0. Let x0 = 1. Then the iterations from the Newton-Raphson method gives n xn ϵ 1 0.668752 0.032019 2 0.637068 0.000335 3 0.636733 0.000000 Figure 4.2 shows the iterative points x2 and x3 in black bullet. Recall that the exact root is x∗ ≈ 0.636733. Obviously, the Newton-Raphson method is much faster than both bisection and secant methods in this example. Let us now discuss the convergence of the Newton-Raphson method. Theorem 4.3.10 [Convergence of Newton-Raphson method]. Hypothesis: 1. Let f : R → R be a C2 (R) function. 2. Let r be a simple root of the nonlinear equation f(x) = 0, that is f′ (r) ̸= 0. Conclusion: Then there exists a δ > 0 such that for every x0 ∈ [r − δ, r + δ], 1. each term of the Newton-Raphson iterative sequence {xn} is well-defined. 2. the sequence {xn} belongs to the interval [r − δ, r + δ]. 3. limn→∞ xn = r. 4. Further, we have lim n→∞ |xn+1 − r| |xn − r|2 = f′′ (r) 2f′(r) . (4.14) The proofs of the conclusions (1)-(3) are omitted. Conclusion (4) implies that the S. Baskar and S. Sivaji Ganesh 138 Spring 2017-18
- 151. Section 4.3 Open Domain Methods Newton-Raphson method converges quadratically. The proof is a direct application of the Taylor’s theorem and is left as an exercise. Theorem on Newton-Raphson method says that if we start near-by a root of the non- linear equation, then Newton-Raphson iterative sequence is well-defined and converges. For increasing convex functions, we need not be very careful in choosing the initial guess. For such functions, the Newton-Raphson iterative sequence always converges, whatever may be the initial guess. This is the content of the next theorem. Theorem 4.3.11 [Convergence Result for Convex Functions]. Hypothesis: Let f : R → R be a twice continuously differentiable function such that 1. f is convex, i.e., f′′ (x) > 0 for all x ∈ R. 2. f is strictly increasing, i.e., f′ (x) > 0 for all x ∈ R. 3. there exists an r ∈ R such that f(r) = 0. Conclusion: Then 1. r is the unique root of f(x) = 0. 2. For every choice of x0, the Newton-Raphson iterative sequence converges to r. Proof. Proof of (1): Since f is strictly increasing, the function cannot take the same value more than once. Thus f(x) = 0 has exactly one root. Proof of (2): From en+1 = − f′′ (ξn) 2f′(xn) e2 n. (4.15) it follows that en+1 ≤ 0. This implies that r ≤ xn+1. Since f is a strictly increasing function, we get f(r) ≤ f(xn+1). Thus f(xn+1) ≥ 0 for every n ∈ N. From (4.11), we get xn+1 ≤ xn. That is, the sequence {xn} is a non-increasing sequence, and is bounded below by r, and hence converges. Let us denote the limit by x∗ . On the other hand, the sequence {en} is a non-decreasing sequence, bounded above by 0. Let e∗ denote the limit of the sequence {en}. Passing to the limit as n → ∞ in the equation en+1 = en + f(xn) f′(xn) , we get the desired result. S. Baskar and S. Sivaji Ganesh 139 Spring 2017-18
- 152. Chapter 4. Nonlinear Equations To following example illustrates the quadratic convergence of the Newton-Raphson method. Example 4.3.12. Start with x0 = −2.4 and use Newton-Raphson iteration to find the root r = −2.0 of the polynomial f(x) = x3 − 3x + 2. The iteration formula is xn+1 = 2x3 n − 2 3x2 n − 3 . It is easy to verify that |r − xn+1|/|r − xn|2 ≈ 2/3, which shows the quadratic con- vergence of Newton-Raphson’s method. 4.3.3 Fixed-Point Iteration Method In fixed point iteration method, the problem of finding a root of a nonlinear equation f(x) = 0 is equivalently viewed as the problem of finding a fixed point of a suitably defined function g. Let us first define the notion of fixed point of a function. Definition 4.3.13 [Fixed point]. Let g be a continuous function. A point α ∈ R is said to be a fixed point of g if α = g(α). The idea behind the choice of g is to rewrite the given nonlinear equation f(x) = 0 in the form x = g(x) for some function g. In general, there may be more than one choice of g with this property as illustrated by the following example. Example 4.3.14. Note that α ∈ R is a root of the equation x2 − x − 2 = 0 if and only if α is a root of each of the following equations. 1. x = x2 − 2 2. x = √ x + 2 3. x = 1 + 2 x . In other words, obtaining a root of the equation f(x) = 0, where f(x) = x2 − x − 2 S. Baskar and S. Sivaji Ganesh 140 Spring 2017-18
- 153. Section 4.3 Open Domain Methods is equivalent to finding the fixed point of any one of the following functions: 1. g1(x) = x2 − 2 2. g2(x) = √ x + 2 3. g3(x) = 1 + 2 x . The fixed-point iteration method for finding a root of g(x) = x consists of a se- quence of iterates {xn}, starting from an initial guess x0, defined by xn = g(xn−1) (4.16) The function g is called the iteration function. As we saw in Example 4.3.14, for a given nonlinear equation, the iteration function is not unique. The crucial point in this method is to choose a good iteration function g(x). A good iteration function should satisfy the following properties: 1. For the given starting point x0, the successive approximations xn given by (4.16) can be calculated. 2. The sequence x1, x2, · · · converges to some point ξ. 3. The limit ξ is a fixed point of g(x), ie., ξ = g(ξ). Not every iteration function has all these properties. The following example shows that for certain iteration functions, even the sequence of iterates need not be de- fined. Example 4.3.15. Consider the equation x2 − x = 0. This equation can be re-written as x = ± √ x. Let us take the iterative function g(x) = − √ x. Since g(x) is defined only for x > 0, we have to choose x0 > 0. For this value of x0, we have g(x0) < 0 and therefore, x1 cannot be calculated. Therefore, the choice of g(x) has to be made carefully so that the sequence of iterates can be calculated. How to choose such an iteration function g(x)? Note that x1 = g(x0), and x2 = g(x1). Thus x1 is defined whenever x0 belongs to the domain of g. Thus we must take the initial guess x0 from the domain of g. For defining S. Baskar and S. Sivaji Ganesh 141 Spring 2017-18
- 154. Chapter 4. Nonlinear Equations Figure 4.3: Fixed-point Iteration Procedure. x2, we need that x1 is in the domain of g once again. In fact the sequence is given by x0, g(x0), g ◦ g(x0), g ◦ g ◦ g(x0), · · · Thus to have a well-defined iterative sequence, we require that Range of the function g is contained in the domain of g. A function with this property is called a self map. We make our first assumption on the iterative function as Assumption 1: a ≤ g(x) ≤ b for all a ≤ x ≤ b. It follows that if a ≤ x0 ≤ b, then for all n, xn ∈ [a, b] and therefore xn+1 = g(xn) is defined and belongs to [a, b]. S. Baskar and S. Sivaji Ganesh 142 Spring 2017-18
- 155. Section 4.3 Open Domain Methods Let us now discuss about the point 3. This is a natural expectation since the expression x = g(x), which is the root of the required equation is precisely the definition of a fixed point. To achieve this, we need g(x) to be a continuous function. For if xn → x∗ then x∗ = lim n→∞ xn = lim n→∞ g(xn−1) = g( lim n→∞ xn−1) = g(x∗ ) Therefore, we need Assumption 2: The iterative function g is continuous. It is easy to prove that a continuous self map on a bounded interval always has a fixed point. However, the question is whether the sequence (4.16) generated by the iterative function g converges, which is the requirement stated in point 2. This point is well understood geometrically. Figure 4.3(a) and Figure 4.3(c) illustrate the convergence of the fixed-point iterations whereas Figure 4.3(b) and Figure 4.3(d) illustrate the diverg- ing iterations. In this geometrical observation, we see that when g′ (x) < 1, we have convergence and otherwise, we have divergence. Therefore, we make the assumption Assumption 3: The iteration function g(x) is differentiable on I = [a, b]. Further, there exists a constant 0 < K < 1 such that |g′ (x)| ≤ K, x ∈ I. (4.17) Such a function is called the contraction map. Let us now present the algorithm of the fixed-point iteration method. Algorithm 4.3.16. Hypothesis: Let g : [a, b] → [a, b] be an iteration function such that Assumptions 1, 2, and 3 stated above hold. Algorithm: Step 1: Choose an initial guess x0 ∈ [a, b]. Step 2: Define the iteration methods as xn+1 = g(xn), n = 0, 1, · · · Step 3: For a pre-assigned positive quantity ϵ, check for one of the (fixed) stop- ping criteria discussed in Section 4.2. If the criterion is satisfied, stop the iteration. Otherwise, repeat the step 1 by replacing n with n + 1 until the criterion is satisfied. S. Baskar and S. Sivaji Ganesh 143 Spring 2017-18
- 156. Chapter 4. Nonlinear Equations Theorem 4.3.17 [Convergence Result for Fixed-Point Iteration Method]. Hypothesis: Let the iterative function g be chosen so that 1. g is defined on the interval [a, b] and a ≤ g(x) ≤ b. That is, g is a self map on [a, b] 2. g is continuously differentiable on [a, b] 3. g is a contraction map. That is, λ = max a≤x≤b |g′ (x)| < 1. (4.18) Conclusion: Then 1. x = g(x) has a unique root r in [a, b]. 2. For any choice of x0 ∈ [a, b], with xn+1 = g(xn), n = 0, 1, · · · , lim n→∞ xn = r. 3. We further have |xn − r| ≤ λn |x0 − r| ≤ λn 1 − λ |x1 − x0| (4.19) and lim n→∞ r − xn+1 r − xn = g′ (r). (4.20) Proof. Proof for (1) is easy. From mean-value theorem and (4.17), we have |r − xn+1| = |g(r) − g(xn)| ≤ λ|r − xn|. (4.21) By induction, we have |r − xn+1| ≤ λn |x0 − r|, n = 0, 1, · · · . Since, as n → ∞, λn → 0, we have xn → r. Further, we have |x0 − r| = |x0 − x1 + x1 − r| ≤ |x0 − x1| + |x1 − r| ≤ λ|x0 − r| + |x0 − x1|. S. Baskar and S. Sivaji Ganesh 144 Spring 2017-18
- 157. Section 4.3 Open Domain Methods Then solving for |x0 − r|, we get (4.19). Now we will prove the rate of convergence (4.20). From Mean-value theorem r − xn+1 = g(r) − g(xn) = g′ (ξn)(r − xn), n = 0, 1, · · · . (4.22) with ξn an unknown point between r and xn. Since xn → r, we must have ξn → r and therefore, lim n→∞ r − xn+1 r − xn = lim n→∞ g′ (ξn) = g′ (r). This completes the proof. Remark 4.3.18. From the inequality (4.21), we see that the fixed point iteration method has linear convergence. In other words, the order of convergence of this method is 1. Example 4.3.19. The nonlinear equation x3 + 4x2 − 10 = 0 has a unique root in [1, 2]. Note that the solution of each of the following fixed-point problems is a root to the given nonlinear equation. 1. x = g1(x), where g1(x) = x − x3 − 4x2 + 10 2. x = g2(x) where g2 = √ 10 x − 4x 3. x = g3(x) where g3 = 1 2 √ 10 − x3 4. x = g4(x) where g4 = √ 10 4 + x 5. x = g5(x) where g5 = x − x3 + 4x2 − 10 3x2 + 8x . We now show that among the five equivalent fixed-point formulations of the given nonlinear equation, only some of them turn out to be good iterative functions. Let us implement fixed-point iteration method with each of the five iterating functions, and compare the results which are tabulated below. S. Baskar and S. Sivaji Ganesh 145 Spring 2017-18
- 158. Chapter 4. Nonlinear Equations n x = g1(x) x = g2(x) x = g3(x) x = g4(x) x = g5(x) 0 1.5 1.5 1.5 1.5 1.5 1 -0.875 0.8165 1.286953768 1.348399725 1.373333333 2 6.732 2.9969 1.402540804 1.367376372 1.365262015 3 -469.7 √ −8.65 1.345458374 1.364957015 1.365230014 4 1.03 × 108 1.375170253 1.365264748 1.365230013 5 1.360094193 1.3652 6 1.367846968 1.365230576 7 1.363887004 1.365229942 8 1.365916734 1.365230022 9 1.364878217 1.365230012 10 1.365410062 1.365230014 15 1.365223680 1.365230013 20 1.365230236 25 1.365230006 30 1.365230013 From the above table, we conclude that the iterative functions g1 and g2 are very bad, while that given by g5 is the best. However iterative functions g3 and g4 are also good but requires more number of iterations compared to g5. 1. Let us start analyzing the iterative sequence generated by the iteration function g1. In this case, the iterative sequence seems to be diverging. The main reason behind this is the fact that g1 is not a contraction map for x ∈ [1, 2], which can be seen easily. Thus the successive iterates using g1 have increasing moduli, which also shows that g1 is not a self map. 2. Let us consider the iterative function g2. It is easy to check that g2 is not a self map of [1, 2] to itself. In our computation above, we see that the entire iterative sequence is not defined as one of the iterates becomes negative, when the initial guess is taken as 1.5. The exact root is approximately equal to r = 1.365. There is no interval containing r on which |g′ 2(x)| < 1. In fact, g′ 2(r) ≈ 3.4 and as a consequence |g′ 2(x)| > 3 on an interval containing r. Thus we don’t expect a convergent iterative sequence even if the sequence is well-defined! 3. Regarding the iterative function g3, note that this iteration function is a de- creasing function on [1, 2] as g′ 3(x) = − 3x2 4 √ 10 − x3 < 0 S. Baskar and S. Sivaji Ganesh 146 Spring 2017-18
- 159. Section 4.3 Open Domain Methods on [1, 2]. Thus maximum of g3 is attained at x = 1, which is 1.5; and the minimum is attained at x = 2 which is approximately equal to 0.707. Thus g3 is a self map of [1, 2]. But |g′ 3(2)| ≈ 2.12. Thus the condition |g′ 3(x)| ≤ λ < 1 is violated on the interval [1, 2]. However by restricting to a smaller interval [1, 1.5], we get that g3 is a self map of [1, 1.5] as g3 is still decreasing function on [1, 1.5] and g3(1) = 1.5, g3(1.5) ≈ 1.28, and also |g′ 3(x)| ≤ |g′ 3(1.5)| ≈ 0.66. Thus g3 satisfies the hypothesis of theorem on fixed-point iteration, and as expected the sequence of iterates converge. 4. Note that the iterative function g4 is given by g4(x) = √ 10 4 + x . We have |g′ 4(x)| = −5 √ 10(4 + x)3/2 ≤ 5 √ 10(5)3/2 < 0.15 for all x ∈ [1, 2]. The bound obtained on the derivative of g4 is considerably smaller when com- pared to that of g3, which explains why the convergence is faster for the iterates obtained using g4, when compared with those obtained by using g3. 5. Finally, let us consider the iterative sequence generated by g5. This sequence converges much more faster compared to g3 and g4. Note that the fixed-point it- erative sequence generated by g5 is nothing but the iterative sequence of Newton- Raphson method for the root of the nonlinear equation f(x) = 0. Example 4.3.20. Consider the equation sin x + x2 − 1 = 0. Take the initial interval as [0, 1]. There are at least three possible choices for the iteration function, namely, 1. g1(x) = sin−1 (1 − x2 ), S. Baskar and S. Sivaji Ganesh 147 Spring 2017-18
- 160. Chapter 4. Nonlinear Equations 2. g2(x) = − √ 1 − sin x, 3. g3(x) = √ 1 − sin x. Here we have g′ 1(x) = −2 √ 2 − x2 . We can see that |g′ 1(x)| > 1. Taking x0 = 0.8 and denoting the absolute error as ϵ, we have n g1(x) ϵ 0 0.368268 0.268465 1 1.043914 0.407181 2 -0.089877 0.726610 3 1.443606 0.806873 The sequence of iterations is diverging as expected. If we take g2(x), clearly the assumption 1 is violated and therefore is not suitable for the iteration process. Let us take g3(x). Here, we have g′ 3(x) = − cos x √ 1 − sin x . Therefore, |g′ 3(x)| = √ 1 − sin2 x 2 √ 1 − sin x = √ 1 + sin x 2 ≤ 1 √ 2 < 1. Taking x0 = 0.8 and denoting the absolute error as ϵ, we have n g3(x) ϵ 0 0.531643 0.105090 1 0.702175 0.065442 2 0.595080 0.041653 3 0.662891 0.026158 The sequence is converging. S. Baskar and S. Sivaji Ganesh 148 Spring 2017-18
- 161. Section 4.4 Comparison and Pitfalls of Iterative Methods 4.4 Comparison and Pitfalls of Iterative Methods Closed domain methods: Bisection and Regula falsi methods 1. In both these methods, where we are trying to find a root of the nonlinear equa- tion f(x) = 0, we are required to find an interval [a, b] such that f(a) and f(b) have opposite signs. This calls for a complete study of the function f. In case the function has no roots on the real line, this search for an interval will be fu- tile. There is no way to realize this immediately, thus necessitating a full fledged understanding of the funtion f. 2. Once it is known that we can start these methods, then surely the iterative se- quences converge to a root of the nonlinear equation. 3. In bisection method, we can keep track of the error by means of an upper bound. But such a thing is not available for regula falsi method. In general convergence of bisection method iterates is slower compared to that of regula falsi method iterates. 4. If the initial interval [a, b] is such that the equation f(x) = 0 has a unique root in it, then both the methods converge to the root. If there are more than one roots in [a, b], then usually both methods find different roots. The only way of finding the desired root is to find an interval in which there is exactly one root to the nonlinear equation. Open domain methods: Secant, Newton-Raphson, and Fixed point methods 1. The main advantage of the open domain methods when compared to closed do- main methods is that we don’t need to locate a root in an interval. Rather, we can start the iteration with an arbitrarily chosen initial guess(es). 2. The disadvantage of the open domain methods is that the iterative sequence may not be well-defined for all initial guesses. Even if the sequence is well-defined, it may not converge. Even if it converges, it may not converge to a specific root of interest. 3. In situations where both open and closed domain methods converge, open domain methods are generally faster compared to closed domain methods. Especially, Newton-Raphson’s method is faster than other methods as the order of conver- gence of this method is 2. In fact, this is the fastest method known today. 4. In these methods, it may happen that we are trying to find a particular root of the S. Baskar and S. Sivaji Ganesh 149 Spring 2017-18
- 162. Chapter 4. Nonlinear Equations −6 −4 −2 0 2 4 6 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 Figure 4.4: Iteration Procedure of Newton-Raphson’s method for f(x) = cos(x). nonlinear equation, but the iterative sequence may converge to a different root. Thus we have to be careful in choosing the initial guess. If the initial guess is far away from the expected root, then there is a danger that the iteration converges to another root of the equation. In the case of Newton-Raphson’s method, this usually happens when the slope f′ (x0) is small and the tangent line to the curve y = f(x) is nearly parallel to the x-axis. Similarly, in the case of secant method, this usually happens when the slope of the secant joining (x0, f(x0)) and (x1, f(x1)) is nearly parallel to the x-axis. For example, if f(x) = cos x and we seek the root x∗ = π/2 and start with x0 = 3, calculation reveals that x1 = −4.01525, x2 = −4.85266, · · · , and the iteration converges to x = −4.71238898 ≈ −3π/2. The iterative sequence for n = 1, 2 is depicted in Figure 4.4. 5. Suppose that f(x) is positive and monotone decreasing on an unbounded interval [a, ∞) and x0 > a. Then the sequence might diverge. For example, if f(x) = xe−x S. Baskar and S. Sivaji Ganesh 150 Spring 2017-18
- 163. Section 4.5 Exercises and x0 = 2, then x1 = 4.0, x2 = 5.333333..., · · · , p15 = 19.72354..., · · · . and the sequence diverges to +∞. This particular function has another suprising problem. The value of f(x) goes to zero rapidly as x gets large, for example f(x15) = 0.0000000536, and it is possible that p15 could be mistaken for a root as per the residual error. Thus, using residual error for iterative methods nonlinear equations is often not preferred. 6. The method can stuck in a cycle. For instance, let us compute the iterative sequence generated by the Newton-Raphson’s method for the function f(x) = x3 − x − 3 with the initial guess x0 = 0. The iterative sequence is x1 = −3.00, x2 = −1.961538, x3 = −1.147176, x4 = −0.006579, x5 = −3.000389, x6 = −1.961818, x7 = −1.147430, · · · and we are stuck in a cycle where xn+4 ≈ xk for k = 0, 1, · · · . But if we start with a value x0 sufficiently close with the root r ≈ 1.6717, then the convergence is obtained. The proof of this is left as an exercise. 7. If f(x) has no real root, then there is no indication by these methods and the iter- ative sequence may simply oscillate. For example compute the Newton-Raphson iteration for f(x) = x2 − 4x + 5. 4.5 Exercises Bisection Method and Regula-falsi Method In the following problems on bisection method, the notation xn is used to denote the mid-point of the interval [an−1, bn−1], and is termed as the bisection method’s nth iterate (or simply, the nth iterate, as the context of bisection method is clear) 1. Let bisection method be used to solve the nonlinear equation (x is in radians) x sin x − 1 = 0 starting with the initial interval [0, 2]. In order to approximate a solution of the nonlinear equation with an absolute error less than or equal to 10−3 , what is the minimum number of iterations required? Also find the corresponding approximate solution. S. Baskar and S. Sivaji Ganesh 151 Spring 2017-18
- 164. Chapter 4. Nonlinear Equations 2. Consider the nonlinear equation 10x + x − 4 = 0 i) Find an interval [a0, b0] such that the function f(x) = 10x + x − 4 satisfies the hypothesis of bisection method. ii) Let r be the solution of the nonlinear equation to which bisection method iterative sequence converges. Find an n such that xn (notation as in the bisection method) approximates r to two significant digits. Also find xn. 3. If bisection method is used with the initial interval [a0, b0] = [2, 3], how many iterations are required to assure that an approximate solution of the nonlinear equation f(x) = 0 is obtained with an absolute error that is at most 10−5 ? 4. In the notation of bisection method, determine (with justification) if the following are possible. i) a0 < a1 < a2 < · · · < an < · · · ii) b0 > b1 > b2 > · · · > bn > · · · iii) a0 = a1 < a2 = a3 < · · · < a2m = a2m+1 < · · · (Hint: First guess what should be the solution found by bisection method in such a case, and then find the simplest function having it as a root! Do not forget the connection between bisection method and binary representation of a number, described in the last problem) 5. Draw the graph of a function that satisfies the hypothesis of bisection method on the interval [0, 1] such that the errors e1, e2, e3 satisfy e1 > e2, and e2 < e3. Give formula for one such function. 6. Draw the graph of a function for which bisection method iterates satisfy x1 = 2, x2 = 0, and x3 = 1 (in the usual notation of bisection method). Indicate in the graph why x1 = 2, x2 = 0, and x3 = 1 hold. Also mention precisely the corre- sponding intervals [a0, b0],[a1, b1], [a2, b2]. 7. Draw the graph of a function (there is no need to give a formula for the func- tion) for which a0, a1, a2, a3 (in the usual notation of bisection method) satisfy a0 < a1 = a2 < a3. (Mark these points clearly on the graph.) Secant Method and Newton-Raphson Method 8. Discuss some instances where the secant method fails. Note that failure of secant method results from one of the following two situations: (i) the iterative sequence is not well-defined, and (ii) the iterative sequence does not converge at all. S. Baskar and S. Sivaji Ganesh 152 Spring 2017-18
- 165. Section 4.5 Exercises 9. Let α be a positive real number. Find formula for an iterative sequence based on Newton-Raphson method for finding √ α and α1/3 . Apply the methods to α = 18 to obtain the results whcih are correct to two significant digits when compared to their exact values. 10. Consider the nonlinear equation 1 3 x3 − x2 + x + 1 = 0. Show that there exists an initial guess x0 ∈ (0, 4) for which x2 of the Newton- Raphson method iterative sequence is not defined. 11. Let a be a real number such that 0 < a ≤ 1. Let {xn}∞ n=1 be the iterative sequence of the Newton-Raphson method to solve the nonlinear equation e−ax = x. If x∗ denotes the exact root of this equation and x0 > 0, then show that |x∗ − xn+1| ≤ 1 2 (x∗ − xn)2 . 12. Newton-Raphson method is to be applied for approximating a root of the nonlin- ear equation x4 − x − 10 = 0. i) How many solutions of the nonlinear equation are there in [1, ∞)? Are they simple? ii) Find an interval [1, b] that contains the smallest positive solution of the nonlinear equation. iii) Compute five iterates of Newton-Raphson method, for each of the initial guesses x0 = 1, x0 = 2, x0 = 100. What are your observations? iv) A solution of the nonlinear equation is approximately equal to 1.85558 Find a δ as in the proof of theorem on Newton-Raphson method, so that iterative sequence of Newton-Raphson method always converges for every initial guess x0 ∈ [1.85558 − δ, 1.85558 + δ]. v) Can we appeal to the theorem for convex functions in the context of Newton- Raphson method? Justify. 13. Newton-Raphson method is to be applied for approximating a root of the equation sin x = 0. i) Find formula for the Newton-Raphson iterative sequence. ii) Let α ∈ (−π/2, π/2) and α ̸= 0 be such that if x0 = α, then the iteration becomes a cycle i.e., α = x0 = x2 = · · · = x2k = x2k+2 = · · · , x1 = x2 = · · · = x2k+1 = x2k+3 = · · · Find a non-linear equation g(y) = 0 whose solution is α. S. Baskar and S. Sivaji Ganesh 153 Spring 2017-18
- 166. Chapter 4. Nonlinear Equations −10 −8 −6 −4 −2 0 2 4 6 8 10 −10 −8 −6 −4 −2 0 2 4 6 8 10 x y=xsinx−1 Figure 4.5: Graph of x sin x − 1 iii) Starting with the initial guess x0 = α, compute x1, x2, x3, x4, x5 using Newton- Raphson method for the equation sin x = 0. iv) Starting with the initial guess y0 = 1, compute y1, y2, y3, y4, y5 using Newton- Raphson method for the equation g(y) = 0 to find an approximate value of α. v) Starting with initial guess x0 = y5, where y5 is obtained in (iv) above, compute x1, x2, x3, x4, x5 using Newton-Raphson method for the equation sin x = 0. 14. Consider the nonlinear equation x sin x − 1 = 0. i) Find an initial guess x0 such that x0 > 1, with the help of the graph de- picted in Figure 4.5, such that the Newton-Raphson method is likely to con- verge to the solution x∗ of the given nonlinear equation lying in the interval (−10, −9). Compute x1, x2, x3, x4 of the corresponding Newton-Raphson it- erative sequence. Explain why the Newton-Raphson iterative sequence so obtained would converge to the desired x∗ . This example shows that even though the initial guess is close to a solution, the corresponding Newton- Raphson iterative sequence might converge to a solution that is far off! ii) Find another initial guess x0 such that x0 > 1, with the help of the graph, S. Baskar and S. Sivaji Ganesh 154 Spring 2017-18
- 167. Section 4.5 Exercises such that the Newton-Raphson method is likely to converge to the small- est positive solution of the given nonlinear equation. Compute x1, x2, x3, x4 of the corresponding Newton-Raphson iterative sequence. Explain why the Newton-Raphson iterative sequence so obtained would converge to the de- sired solution. 15. Draw the graph of a function for which the Newton-Raphson iterates satisfy x0 = x2 = x4 = · · · = 0, and x1 = x3 = x5 = · · · = 2. Indicate in the graph why this happens. 16. Draw the graph of a function for which regula-falsi method iterates satisfy x0 = 0, x1 = 3, and x2 = 1, x3 = 2 (in the usual notation of regula-falsi method). Indicate in the graph why x2 = 1, x3 = 2 hold. Fixed-Point Iteration Method 17. The nonlinear equation f(x) = x2 − 2x − 8 = 0 has two solutions x = −2 and x = 4. Consider the three fixed-point formulations i) x = 8 x − 2 , ii) x = √ 2x + 8, iii) x = x2 − 8 2 . Carry out fixed-point iteration method with two initial guesses x0 = −1, x0 = 3, and for all the three iteration functions. Discuss the convergence or divergence of all the iterative sequences. Can you justify theoretically? 18. To solve the nonlinear equation x − tan x = 0 by fixed-point iteration method, the following fixed-point formulations may be considered. i) x = tan x ii) x = tan−1 x Discuss about convergence of the fixed-point iterative sequences generated by the two formulations. 19. To solve the nonlinear equation e−x − cos x = 0 by fixed-point iteration method, the following fixed-point formulations may be considered. i) x = − ln(cos x) ii) x = cos−1 (e−x ) Discuss about convergence of the fixed-point iterative sequences generated by the two formulations. S. Baskar and S. Sivaji Ganesh 155 Spring 2017-18
- 168. Chapter 4. Nonlinear Equations 20. Show that g(x) = π + 1 2 sin(x/2) has a unique fixed point in [0, 2π]. Use fixed- point iteration method with g as the iteration function and x0 = 0 to find an approximate solution for the equaton 1 2 sin(x/2) − x + π = 0 with the stopping criterion that the residual error is less than 10−4 . 21. Let α ∈ R and β ∈ R be the roots of x2 +ax+b = 0, and such that |α| > |β|. Let g and h be two iterating functions satisfying the hypothesis of the theorem on fixed- point method. Consider the iterative sequences {xn} and {yn} corresponding to two the iterating functions g and h given by xn+1 = − axn + b xn , and yn+1 = − b yn + a respectively. Show that the iterative sequences {xn} and {yn} converge to α and β respectively. 22. Let {xn} ⊂ [a, b] be a sequence generated by a fixed point iteration method with a continuously differentiable iteration function g(x). If this sequence converges to x∗ , then show that |xn+1 − x∗ | ≤ λ 1 − λ |xn+1 − xn|, where λ := max x∈[a,b] |g′ (x)|. (This estimate helps us to decide when to stop iterating if we are using a stopping criterion based on the distance between successive it- erates.) 23. Explain why the sequence of iterates xn+1 = 1 − 0.9x2 n, with initial guess x0 = 0, does not converge to any solution of the quadratic equation 0.9x2 + x − 1 = 0? [Hint: Observe what happens after 25 iterations] 24. Let x∗ be the smallest positive root of the equation 20x3 − 20x2 − 25x + 4 = 0. The following question is concerning the fixed-point formulation of the nonlinear equation given by x = g(x), where g(x) = x3 − x2 − x 4 + 1 5 . i) Show that x∗ ∈ [0, 1]. ii) Does the function g satisfy the hypothesis of theorem on fixed-point method? If yes, we know that x∗ is the only fixed point of g lying in the interval [0, 1]. In the notation of fixed-point iteration method, find an n such that |x∗ − xn| < 10−3 , when the initial guess x0 is equal to 0. 25. Let f : [a, b] → R be a function such that f′ is continuous, f(a)f(b) < 0, and there exists an α > 0 such that f′ (x) ≥ α > 0. S. Baskar and S. Sivaji Ganesh 156 Spring 2017-18
- 169. Section 4.5 Exercises i) Show that f(x) = 0 has exactly one solution in the interval [a, b]. ii) Show that with a suitable choice of the parameter λ, the solution of the nonlinear equation f(x) = 0 can be obtained by applying the fixed-point iteration method applied to the function F(x) = x + λf(x). 26. Let p > 1 be a real number. Show that the following expression has a meaning and find its value. x = √ p + √ p + √ p + · · · Note that the last equation is interpreted as x = limn→∞ xn, where x1 = √ p, x2 = √ p + √ p, · · · . (Hint: Note that xn+1 = √ p + xn, and show that the se- quence {xn} converges using the theorem on fixed-point method.) 27. Draw the graph of a function having the following properties: (i) The function has exactly TWO fixed points. (ii) Give two choices of the initial guess x0 and y0 such that the corresponding sequences {xn} and {yn} have the properties that {xn} converges to one of the fixed points and the sequence {yn} goes away and diverges. Point out the first three terms of both the sequences on the graph. S. Baskar and S. Sivaji Ganesh 157 Spring 2017-18
- 171. CHAPTER 5 Interpolation Let a physical experiment be conducted and the outcome is recorded only at some finite number of times. If we want to know the outcome at some intermediate time where the data is not available, then we may have to repeat the whole experiment once again to get this data. In the mathematical language, suppose that the finite set of values {f(xi) : i = 0, 1, · · · , n} of a function f at a given set of points {xi : i = 0, 1, · · · , n} is known and we want to find the value of f(x), where x ∈ (xj, xk), for some j = 1, 2, · · · , n and k = 1, 2, · · · , n. One way of obtaining the value of f(x) is to compute this value directly from the expression of the function f. Often, we may not know the expression of the function explicitly and only the data {(xi, yi) : i = 0, 1, · · · , n} is known, where yi = f(xi). In terms of the physical experiments, repeating an ex- periment will quite often be very expensive. Therefore, one would like to get at least an approximate value of f(x) (in terms of experiment, an approximate value of the outcome at the desired time). This is achieved by first constructing a function whose value at xi coincides exactly with the value f(xi) for i = 0, 1, · · · , n and then finding the value of this constructed function at the desired points. Such a process is called interpolation and the constructed function is called the interpolating function for the given data. S. Baskar and S. Sivaji Ganesh 159 Spring 2017-18
- 172. Chapter 5. Interpolation In certain circumstances, the function f may be known explicitly, but still too dif- ficult to perform certain operations like differentiation and integration. Thus, it is often preferred to restrict the class of interpolating functions to polynomials, where the differentiation and the integration can be done more easily. In Section 5.1, we introduce the basic problem of polynomial interpolation and prove the existence and uniqueness of polynomial interpolating the given data. There are at least two ways to obtain the unique polynomial interpolating a given data, one is the Lagrange and another one is the Newton. In Section 5.1.2, we introduce Lagrange form of interpolating polynomial, whereas Section 5.1.3 introduces the notion of divided differences and Newton form of interpolating polynomial. The error analysis of the polynomial interpolation is studied in Section 5.3. In certain cases, the interpolating polynomial can differ significantly from the exact function. This is illustrated by Carl Runge and is called the Runge Phenomenon. In Section 5.3.4, we present the example due to Runge and state a few results on convergence of the interpolating polynomials. The concept of piecewise polynomial interpolation and Spline interpolation are discussed in Section 5.5. 5.1 Polynomial Interpolation Polynomial interpolation is a concept of fitting a polynomial to a given data. Thus, to construct an interpolating polynomial, we first need a set of points at which the data values are known. Definition 5.1.1. Any collection of distinct real numbers x0, x1, · · · , xn (not necessarily in increasing order) is called nodes. Definition 5.1.2 [Interpolating Polynomial]. Let x0, x1, · · · , xn be the given nodes and y0, y1, · · · , yn be real numbers. A polyno- mial pn(x) of degree less than or equal to n is said to be a polynomial interpolating the given data or an interpolating polynomial for the given data if pn(xi) = yi, i = 0, 1, · · · n. (5.1) The condition (5.1) is called the interpolation condition. S. Baskar and S. Sivaji Ganesh 160 Spring 2017-18
- 173. Section 5.1 Polynomial Interpolation Remark 5.1.3. Let x0, x1, · · · , xn be given nodes, and y0, y1, · · · , yn be real numbers. Let pn(x) be a polynomial interpolating the given data. Then the graph of pn(x) passes through the set of (n + 1) distinct points in the xy-plane given by the table x x0 x1 x2 x3 · · · xn y y0 y1 y2 y3 · · · yn . We call the set {(xi, yi), i = 0, 1, · · · , n} as data and quite often we represent this set in the above form of a table. 5.1.1 Existence and Uniqueness of Interpolating Polynomial The following result asserts that an interpolating polynomial exists and is unique. Theorem 5.1.4 [Existence and Uniqueness of Interpolating polynomial]. Let x0, x1, · · · , xn be given nodes, and y0, y1, · · · , yn be real numbers. 1. Then there exists a polynomial pn(x) of degree less than or equal to n such that pn(xi) = yi, i = 0, 1, · · · , n. That is, there exists an interpolating polynomial pn(x) for the given data {(xi, yi), i = 0, 1, · · · , n}. 2. Such a polynomial is unique. Proof. Proof of uniqueness: Assume that pn(x) and qn(x) are interpolating polynomials of degree less than or equal to n that satisfies the interpolation condition (5.1). Let rn(x) = pn(x) − qn(x). Then, rn(x) is also a polynomial of degree less than or equal to n, and by the inter- polation condition, we have rn(xi) = 0, for every i = 0, 1, · · · , n. Thus, rn(x) is a polynomial of degree less than or equal to n with n + 1 distinct roots. By the fundamental theorem of algebra, we conclude that rn(x) is the zero polynomial. That is, the interpolating polynomial is unique. S. Baskar and S. Sivaji Ganesh 161 Spring 2017-18
- 174. Chapter 5. Interpolation Proof of existence: Existence of interpolating polynomial is proved using mathe- matical induction. If n = 0, then the constant polynomial p0(x) = y0 is the required polynomial and its degree is less than or equal to 0. Assume that the result is true for n = k. We will now prove that the result is true for n = k + 1. Let the data be given by x x0 x1 x2 x3 · · · xk xk+1 y y0 y1 y2 y3 · · · yk yk+1 By the assumption, there exists a polynomial pk(x) of degree less than or equal to k such that the first k interpolating conditions pk(xi) = yi, i = 0, 1, · · · , k hold. Define a polynomial pk+1(x) of degree less than or equal to k + 1 by pk+1(x) = pk(x) + c(x − x0)(x − x1) · · · (x − xk), (5.2) where the constant c is such that the (k + 1)th interpolation condition pk+1(xk+1) = yk+1 holds. This is achieved by choosing c = yk+1 − pk(xk+1) (xk+1 − x0)(xk+1 − x1) · · · (xk+1 − xk) . Note that pk+1(xi) = yi for i = 0, 1, · · · , k and therefore pk+1(x) is an interpolating polynomial for the given data. This proves the result for n = k + 1. By the principle of mathematical induction, the result is true for any natural number n. Remark 5.1.5. A special case is when the data values yi, i = 0, 1, · · · , n, are the values of a function f at given nodes xi, i = 0, 1, · · · , n. In such a case, a polynomial interpolating the given data x x0 x1 x2 x3 · · · xn y f(x0) f(x1) f(x2) f(x3) · · · f(xn) is said to be the polynomial interpolating the given function or the interpolating polynomial for the given function and has a special significance in applications of Numerical Analysis for computing approximate solutions of differential equations and numerically computing complicated integrals. S. Baskar and S. Sivaji Ganesh 162 Spring 2017-18
- 175. Section 5.1 Polynomial Interpolation Example 5.1.6. Let the following data represent the values of f: x 0 0.5 1 f(x) 1.0000 0.5242 −0.9037 The questions are the following: 1. What is the exact expression for the function f? 2. What is the value of f(0.75)? We cannot get the exact expression for the function f just from the given data, because there are infinitely many functions having same value at the given set of points. Due to this, we cannot expect an exact value for f(0.75), in fact, it can be any real number. On the other hand, if we look for f in the class of polynomials of degree less than or equal to 2, then Theorem 5.1.4 tells us that there is exactly one such polynomial and hence we can obtain a unique value for f(0.75). The interpolating polynomial happens to be p2(x) = −1.9042x2 + 0.0005x + 1 and we have p2(0.75) = −0.0707380. The function used to generate the above table of data is f(x) = sin (π 2 ex ) . With this expression of f, we have (using 7-digit rounding) f(0.75) ≈ −0.1827495. The relative error is given by Er(p2(0.75)) = f(0.75) − p2(0.75) f(0.75) ≈ 0.6129237. That is, at the point x = 0.75 the polynomial approximation to the given function f has more than 61% error. The graph of the function f (blue solid line) and p2 (green dash line) are depicted in Figure 5.1. The blue dots denote the given data, magenta ‘+’ symbol indicates the value of p2(0.75) and the red ‘O’ symbol represents the value of f(0.75). It is also observed from the graph that if we approximate the function f for x ∈ [0, 0.5], then we obtain a better accuracy than approximating f in the interval (0.5, 1). S. Baskar and S. Sivaji Ganesh 163 Spring 2017-18
- 176. Chapter 5. Interpolation 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 x y Figure 5.1: The function f(x) = sin (π 2 ex ) (blue solid line) and p2(x) (green dash line). Blue dots represent the given data, magenta ‘+’ symbol indicates the value of p2(0.75) and the red ‘O’ symbol represents the value of f(0.75). 5.1.2 Lagrange’s Form of Interpolating Polynomial Definition 5.1.7 [Lagrange’s Polynomial]. Let x0, x1, · · · , xn be the given nodes. For each k = 0, 1, · · · , n, the polynomial lk(x) defined by lk(x) = n∏ i=0 i̸=k (x − xi) (xk − xi) (5.3) is called the kth Lagrange polynomial or the kth Lagrange cardinal function. Remark 5.1.8. Note that the kth Lagrange polynomial depends on all the n+1 nodes x0, x1, · · · , xn. S. Baskar and S. Sivaji Ganesh 164 Spring 2017-18
- 177. Section 5.1 Polynomial Interpolation Theorem 5.1.9 [Lagrange’s form of Interpolating Polynomial]. Hypothesis: 1. Let x0, x1, · · · , xn be given nodes. 2. Let the values of a function f be given at these nodes. 3. For each k = 0, 1, · · · , n, let lk(x) be the kth Lagrange polynomial. 4. Let pn(x) (of degree ≤ n) be the polynomial interpolating the function f at the nodes x0, x1, · · · , xn. Conclusion: Then, pn(x) can be written as pn(x) = n∑ i=0 f(xi)li(x). (5.4) This form of the interpolating polynomial is called the Lagrange’s form of In- terpolating Polynomial. Proof. Firstly, we will prove that q(x) := ∑n i=0 f(xi)li(x) is an interpolating polynomial for the function f at the nodes x0, x1, · · · , xn. Since li(xj) = { 1 if i = j 0 if i ̸= j , we get q(xj) = f(xj) for each j = 0, 1, · · · , n. Thus, q(x) is an interpolating polyno- mial. Since interpolating polynomial is unique by Theorem 5.1.4, the polynomial q(x) must be the same as pn(x). This completes the proof of the theorem. Example 5.1.10. Consider the case n = 1 in which we have two distinct points x0 and x1. Thus, we have l0(x) = x − x1 x0 − x1 , l1(x) = x − x0 x1 − x0 and therefore, p1(x) = f(x0)l0(x) + f(x1)l1(x) = f(x0) x − x1 x0 − x1 + f(x1) x − x0 x1 − x0 = f(x0)(x − x1) − f(x1)(x − x0) x0 − x1 S. Baskar and S. Sivaji Ganesh 165 Spring 2017-18
- 178. Chapter 5. Interpolation After a rearrangement of terms, we arrive at p1(x) = f(x0) + f(x1) − f(x0) x1 − x0 (x − x0). (5.5) This is the linear interpolating polynomial of the function f. Similarly, if we are given three nodes with corresponding values, then we can generate the quadratic interpolating polynomial and so on.. Example 5.1.11. Let the values of the function f(x) = ex be given at x0 = 0.82 and x1 = 0.83 by e0.82 ≈ 2.270500, e0.83 ≈ 2.293319. In this example, we would like to obtain an approximate value of e0.826 using the polynomial p1(x) that interpolates f at the nodes x0, x1. The polynomial p1(x) is given by p1(x) ≈ 2.270500 + 2.293319 − 2.270500 0.83 − 0.82 (x − 0.82) = 2.2819x + 0.399342. The approximate value of e0.826 is taken to be p1(0.826), which is given by p1(0.826) ≈ 2.2841914. The true value of e0.826 is e0.826 ≈ 2.2841638. Note that the approximation to e0.826 obtained using the interpolating polynomial p1(x), namely 2.2841914, approximates the exact value to at least five significant digits. If we are given an additional node x2 = 0.84 and the value of f at x2 is taken to be f(x2) ≈ 2.316367, then we would like to use the quadratic interpolating polynomial p2 to obtain an approximate value of e0.826 . In fact, p2(0.826) ≈ 2.2841639. Note that the approximation to e0.826 obtained using the interpolating polynomial p2(x), namely 2.2841639, approximates the exact value to at least eight significant digits. S. Baskar and S. Sivaji Ganesh 166 Spring 2017-18
- 179. Section 5.1 Polynomial Interpolation Remark 5.1.12. The above example gives us a feeling that if we increase the number of nodes, and thereby increasing the degree of the interpolating polynomial, the polynomial approximates the orginal function more accurately. But this is not true in general, and we will discuss this further in Section 5.3.4. Remark 5.1.13. Let x0, x1, · · · , xn be nodes, and f be a function. Recall that computing an in- terpolating polynomial in Lagrange’s form requires us to compute for each k = 0, 1, · · · , n, the kth Lagrange’s polynomial lk(x) which depends on the given nodes x0, x1, · · · , xn. Suppose that we have found the corresponding interpolating poly- nomial pn(x) of f in the Lagrange’s form for the given data. Now if we add one more node xn+1, the computation of the interpolating polynomial pn+1(x) in the Lagrange’s form requires us to compute a new set of Lagrange’s polynomials cor- responding to the set of (n + 1) nodes, and no advantage can be taken of the fact that pn is already available. An alternative form of the interpolating polynomial, namely, Newton’s form of interpolating polynomial, avoids this problem, and will be discussed in the next section. 5.1.3 Newton’s Form of Interpolating Polynomial We saw in the last section that it is easy to write the Lagrange form of the interpolating polynomial once the Lagrange polynomials associated to a given set of nodes have been written. However we observed in Remark 5.1.13 that the knowledge of pn (in Lagrange form) cannot be utilized to construct pn+1 in the Lagrange form. In this section we describe Newton’s form of interpolating polynomial, which uses the knowledge of pn in constructing pn+1. Theorem 5.1.14 [Newton’s form of Interpolating Polynomial]. Hypothesis: 1. Let x0, x1, · · · , xn be given nodes. 2. Let the values of a function f be given at these nodes. 3. Let pn(x) (of degree ≤ n) be the polynomial interpolating the function f at the nodes x0, x1, · · · , xn. S. Baskar and S. Sivaji Ganesh 167 Spring 2017-18
- 180. Chapter 5. Interpolation Conclusion: Then, pn(x) can be written as pn(x) = A0+A1(x−x0)+A2(x−x0)(x−x1)+A3 2∏ i=0 (x−xi)+· · ·+An n−1∏ i=0 (x−xi) (5.6) where A0, A1, · · · , An are constants. This form of the interpolating polynomial is called the Newton’s form of inter- polating polynomial. Proof. Recall that in the proof of Theorem 5.1.4, we proved the existence of an interpolating polynomial using mathematical induction. In fact, we have given an algorithm for constructing interpolating polynomial. The interpolating polynomial given by (5.2) was precisely the Newton’s form of interpolating polynomial. Remark 5.1.15. Let us recall the equation (5.2) from the proof of Theorem 5.1.4 now. 1. It says that for each n ∈ N, we have pn(x) = pn−1(x) + An n−1∏ i=0 (x − xi) (5.7) for some constant An. This shows the recursive nature of computing Newton’s form of interpolating polynomial. 2. Indeed An is given by An = f(xn) − pn−1(xn) (xn − x0)(xn − x1) · · · (xn − xn−1) . (5.8) From the last equality, note that A0 depends only on f(x0). A1 depends on the values of f at x0 and x1 only. In general, An depends on the values of f at x0, x1, x2, · · · , xn only. 3. To compute Newton’s form of interpolating polynomial pn(x), it is enough to compute Ak for k = 0, 1, · · · , n. However note that the formula (5.8) is not well-suited to compute Ak because we need to evaluate all the successive interpolating polynomials pk(x) for k = 0, 1, · · · , n−1 and then evaluate them at the node xk which is computationally costly. It then appears that we are in a similar situation to that of Lagrange’s form of interpolating polynomial as far as computational costs are concerned. But this is not the case, as we shall see shortly that we can compute An directly using the given data (that is, the given values of the function at the nodes), and this will be done in Section 5.2. S. Baskar and S. Sivaji Ganesh 168 Spring 2017-18
- 181. Section 5.2 Newton’s Divided Difference Formulas 5.2 Newton’s Divided Difference Formulas Definition 5.2.1 [Divided Differences]. Let x0, x1, · · · , xn be distinct nodes. Let pn(x) be the polynomial interpolating a function f at the nodes x0, x1, · · · , xn. The coefficient of xn in the polynomial pn(x) is denoted by f[x0, x1, · · · , xn], and is called an nth divided difference of f. Remark 5.2.2. 1. Since the interpolating polynomial for the function f at the nodes x0, x1, · · · , xn is unique, there is one and only one coefficient of xn; even though interpo- lation polynomial may have many forms like Lagrange’s form and Newton’s form. Thus the quantity f[x0, x1, · · · , xn] is well-defined. 2. More generally, the divided difference f[xi, xi+1, · · · , xi+k] is the coefficient of xk in the polynomial interpolating f at the nodes xi, xi+1, · · · , xi+k. 3. The Newton’s form of interpolating polynomial may be written, using divided differences, as pn(x) = f[x0] + n∑ k=1 f[x0, x1, · · · , xk] k−1∏ i=0 (x − xi) (5.9) Example 5.2.3. As a continuation of Example 5.1.10, let us construct the linear interpolating polyno- mial of a function f in the Newton’s form. In this case, the interpolating polynomial is given by p1(x) = f[x0] + f[x0, x1](x − x0), where f[x0] = f(x0), f[x0, x1] = f(x0) − f(x1) x0 − x1 (5.10) are zeroth and first order divided differences, respectively. Observe that this polyno- mial is exactly the same as the interpolating polynomial obtained using Lagrange’s form in Example 5.1.10. The following result is concerning the symmetry properties of divided differences. S. Baskar and S. Sivaji Ganesh 169 Spring 2017-18
- 182. Chapter 5. Interpolation Theorem 5.2.4 [Symmetry]. The divided difference is a symmetric function of its arguments. That is, if z0, z1, · · · , zn is a permutation of x0, x1, · · · , xn, then f[x0, x1, · · · , xn] = f[z0, z1, · · · , zn] (5.11) Proof. Since z0, z1, · · · , zn is a permutation of x0, x1, · · · , xn, which means that the nodes x0, x1, · · · , xn have only been re-labelled as z0, z1, · · · , zn, and hence the polynomial interpolating the function f at both these sets of nodes is the same. By definition f[x0, x1, · · · , xn] is the coefficient of xn in the polynomial interpolating the func- tion f at the nodes x0, x1, · · · , xn, and f[z0, z1, · · · , zn] is the coefficient of xn in the polynomial interpolating the function f at the nodes z0, z1, · · · , zn. Since both the interpolating polynomials are equal, so are the coefficients of xn in them. Thus, we get f[x0, x1, · · · , xn] = f[z0, z1, · · · , zn]. This completes the proof. The following result helps us in computing recursively the divided differences of higher order. Theorem 5.2.5 [Higher-order divided differences]. Divided differences satisfy the equation f[x0, x1, · · · , xn] = f[x1, x2, · · · , xn] − f[x0, x1, · · · , xn−1] xn − x0 (5.12) Proof. Let us start the proof by setting up the following notations. • Let pn(x) be the polynomial interpolating f at the nodes x0, x1, · · · , xn. • Let pn−1(x) be the polynomial interpolating f at the nodes x0, x1, · · · , xn−1. • Let q(x) be the polynomial interpolating f at the nodes x1, x2, · · · , xn. Claim: We will prove the following relation between pn−1, pn, and q: pn(x) = pn−1(x) + x − x0 xn − x0 ( q(x) − pn−1(x) ) (5.13) S. Baskar and S. Sivaji Ganesh 170 Spring 2017-18
- 183. Section 5.2 Newton’s Divided Difference Formulas Since both sides of the equality in (5.13) are polynomials of degree less than or equal to n, and pn(x) is the polynomial interpolating f at the nodes x0, x1, · · · , xn, the equality in (5.13) holds for all x if and only if it holds for x ∈ { x0, x1, · · · , xn } and both sides of the equality reduce to f(x) for x ∈ { x0, x1, · · · , xn }. Let us now verify the equation (5.13) for x ∈ { x0, x1, · · · , xn }. 1. When x = x0, pn−1(x0) + x0 − x0 xn − x0 ( q(x0) − pn−1(x0) ) = pn−1(x0) = f(x0) = pn(x0). 2. When x = xk for 1 ≤ k ≤ n − 1, q(xk) = pn−1(xk) and thus we have pn−1(xk) + xk − x0 xn − x0 ( q(xk) − pn−1(xk) ) = pn−1(xk) = f(xk) = pn(xk). 3. When x = xn, we have pn−1(xn)+ xn − x0 xn − x0 ( q(xn)−pn−1(xn) ) = pn−1(xn)+ ( f(xn)−pn−1(xn) ) = f(xn) = pn(xn). This finishes the proof of the Claim. The coefficient of xn in the polynomial pn(x) is f[x0, x1, · · · , xn]. The coefficient of xn using the right hand side of the equation (5.13) is given by ( coefficient of xn in pn−1(x) ) + 1 xn − x0 ( coefficient of xn in (x−x0) ( q(x)−pn−1(x) )) . On noting that the coefficient of xn−1 in the polynomial pn−1 is f[x0, x1, · · · , xn−1], the coefficient of xn−1 in the polynomial q is f[x1, x2, · · · , xn], and the coefficient of xn in the polynomial pn−1 is zero, we get that the coefficient of xn using the right hand side of the equation (5.13) becomes f[x1, x2, · · · , xn] − f[x0, x1, · · · , xn−1] xn − x0 . Comparing the coefficients of xn in the left and right hand sides of the equation (5.13) yields f[x0, x1, · · · , xn] = f[x1, x2, · · · , xn] − f[x0, x1, · · · , xn−1] xn − x0 . S. Baskar and S. Sivaji Ganesh 171 Spring 2017-18
- 184. Chapter 5. Interpolation Remark 5.2.6. Let i, j ∈ N. Applying Theorem 5.2.5 to a set of nodes xi, xi+1, · · · , xi+j, we conclude f[xi, xi+1, · · · , xi+j] = f[xi+1, xi+2, · · · , xi+j] − f[xi, xi+1, · · · , xi+j−1] xi+j − xi (5.14) Note that the divided differences f[x0, x1, · · · , xn] are defined only for distinct nodes x0, x1, · · · , xn. 5.2.1 Divided Differences Table Given a collection of (n + 1) nodes x0, x1, · · · , xn and the values of the function f at these nodes, we can construct the Newton’s form of interpolating polynomial pn(x) using divided differences. As observed earlier, the Newton’s form of interpolation polynomial has the formula pn(x) = f[x0] + n∑ k=1 f[x0, x1, · · · , xk] k−1∏ i=0 (x − xi) (5.15) One can explicitly write the formula (5.15) for n = 1, 2, 3, 4, 5, · · · . For instance, when n = 5, the formula (5.15) reads p5(x) = f[x0] + f[x0, x1](x − x0) + f[x0, x1, x2](x − x0)(x − x1) + f[x0, x1, x2, x3](x − x0)(x − x1)(x − x2) + f[x0, x1, x2, x3, x4](x − x0)(x − x1)(x − x2)(x − x3) + f[x0, x1, x2, x3, x4, x5](x − x0)(x − x1)(x − x2)(x − x3)(x − x4) . (5.16) For easy computation of the divided differences in view of the formula (5.12), it is convenient to write the divided differences in a table form. For n = 5, the divided difference table is given by S. Baskar and S. Sivaji Ganesh 172 Spring 2017-18
- 185. Section 5.2 Newton’s Divided Difference Formulas x0 f(x0) f[x0, x1] x1 f(x1) f[x0, x1, x2] f[x1, x2] f[x0, x1, x2, x3] x2 f(x2) f[x1, x2, x3] f[x0, x1, x2, x3, x4] f[x2, x3] f[x1, x2, x3, x4] f[x0, x1, x2, x3, x4, x5] x3 f(x3) f[x2, x3, x4] f[x1, x2, x3, x4, x5] f[x3, x4] f[x2, x3, x4, x5] x4 f(x4) f[x3, x4, x5] f[x4, x5] x5 f(x5) Comparing the above divided differences table and the interpolating polynomial p5 given by (5.16), we see that the leading members of each column (denoted in bold font) are the required divided differences used in p5(x). 5.2.2 Divided Difference Formula for Repeated Nodes It follows from the symmetric property (Theorem 5.2.4) that the nth order divided difference formula (5.12) is not well-defined if at least two nodes coincide. But such situations do occur quite often in numerical analysis, for instance, in the error analysis of quadrature formulas. So, we need to interpret the define of the divided difference in such a way that it is still well defined if two or more nodes coincide. For this, the following theorem is useful. Theorem 5.2.7 [Hermite-Genocchi Formula]. Hypothesis: 1. Let x0, x1, · · · , xn ∈ [a, b] be distinct nodes. 2. Let f be n-times continuously differentiable function on the interval [a, b]. Conclusion: Then f[x0, x1, · · · , xn] = ∫ · · · ∫ τn f(n) (t0x0 + · · · + tnxn) dt1 · · · dtn, (5.17) where τn = { (t1, t2, · · · , tn) / ti ≥ 0, i = 1, 2, · · · , n; n∑ i=1 ti ≤ 1 } , t0 = 1 − n∑ i=1 ti. (5.18) S. Baskar and S. Sivaji Ganesh 173 Spring 2017-18
- 186. Chapter 5. Interpolation Proof. We prove the theorem by induction. Step 1 : First, we prove (5.17) for n = 1 . From (5.18), we see that τ1 = [0, 1]. Using the expression for t0 in (5.18), we have 1∫ 0 f′ (t0x0 + t1x1)dt1 = 1∫ 0 f′ (x0 + t1(x1 − x0)) dt1 = 1 x1 − x0 f(x0 + t1(x1 − x0)) t1=1 t1=0 = f(x1) − f(x0) x1 − x0 = f[x0, x1]. Thus, we have proved the result (5.17) for n = 1. Step 2 : Now assume that the formula (5.17) holds for n = k ≥ 1 and prove the formula for n = k + 1. We have ∫ · · · ∫ τk+1 f(k+1) (t0x0 + t1x1 + · · · + tk+1xk+1) dt1 · · · dtk+1 = ∫ · · · ∫ τk 1−(t1+···+tk)∫ 0 f(k+1) (x0 + t1(x1 − x0) + · · · + tk+1(xk+1 − x0)) dtk+1 dt1 · · · dtk = ∫ · · · ∫ τk 1 xk+1 − x0 [ f(k) (x0 + t1(x1 − x0) + · · · + tk+1(xk+1 − x0)) ]tk+1=1−(t1+···+tk) tk+1=0 dt1 · · · dtk = 1 xk+1 − x0 ∫ · · · ∫ τk f(k) (xk+1 + t1(x1 − xk+1) + · · · + tk(xk − xk+1)) dt1 · · · dtk − ∫ · · · ∫ τk f(k) (x0 + t1(x1 − x0) + · · · + tk(xk − x0)) dt1 · · · dtk = f[x1, · · · , xk, xk+1] − f[x0, x1, · · · , xk] xk+1 − x0 = f[x0, x1, · · · , xk+1]. By the principle of mathematical induction, the result is true for all n ∈ N. Remark 5.2.8. Since f is n-times continuously differentiable, the right hand side of (5.17) is mean- ingful for any set of points x0, x1, · · · , xn, which are not necessarily distinct. This gives a meaning, which can be used to define the nth order divided difference when x0, x1, · · · , xn are not distinct (i.e., one or more points are repeated). S. Baskar and S. Sivaji Ganesh 174 Spring 2017-18
- 187. Section 5.2 Newton’s Divided Difference Formulas Following results are direct consequences of the above theorem. Corollary 5.2.9. Hypothesis: 1. Let x0, x1, · · · , xn ∈ [a, b] be a set of points, not necessarily distinct. 2. Let f be n-times continuously differentiable function on the interval [a, b]. Conclusion: 1. The function (x0, x1, · · · , xn) −→ f[x0, x1, · · · , xn] is continuous on Rn+1 . 2. For any x ∈ [a, b], the limit lim (x0,x1,··· ,xn)→(x,x,··· ,x) f[x0, x1, · · · , xn] exists and this limit is the nth -order divided difference f[x, x, · · · , x] (x repeated (n + 1)-times). Further, we have f[x, x, · · · , x] = f(n) (x) n! . (5.19) Proof. 1. The proof is out side the scope of this course and hence omitted. 2. The proof follows from the fact that ∫ · · · ∫ τn 1 dt1 · · · dtn = 1 n! and the Hermite-Genocchi formula (5.17). The following theorem gives an expression for the divided difference of an (n+2)-times differentiable function when two nodes are repeated. Theorem 5.2.10. Hypothesis: 1. Let x0, x1, · · · , xn be given (distinct) nodes in an interval [a, b]. 2. Let x ∈ [a, b], x /∈ {x0, x1, · · · , xn}. Conclusion: The (n + 2)nd -order divided difference f[x0, x1, · · · , xn, x, x] of an (n + 2)-times continuously differentiable function f is given by f[x0, x1, · · · , xn, x, x] = d dx f[x0, x1, · · · , xn, x]. (5.20) S. Baskar and S. Sivaji Ganesh 175 Spring 2017-18
- 188. Chapter 5. Interpolation Proof. It follows from (5.17) that the function F(x) = f[x0, x1, · · · , xn, x] for all x ∈ [a, b] is well-defined and continuous. Therefore, using the symmetry property of the divided differences, we get f[x0, x1, · · · , xn, x, x] = lim h→0 f[x0, x1, · · · , xn, x, x + h] = lim h→0 f[x, x0, x1, · · · , xn, x + h]. As all the points used on the right hand side are distinct, we can use the formula (5.12) to get f[x0, x1, · · · , xn, x, x] = lim h→0 f[x0, x1, · · · , xn, x + h] − f[x, x0, x1, · · · , xn] h = lim h→0 f[x0, x1, · · · , xn, x + h] − f[x0, x1, · · · , xn, x] h = d dx f[x0, x1, · · · , xn, x]. This completes the proof. 5.3 Error in Polynomial Interpolation Let f be a function defined on an interval I = [a, b]. Let pn(x) be a polynomial of degree less than or equal to n that interpolates the function f at n + 1 nodes x0, x1, · · · , xn in I. How well does pn(x) approximate f on the interval I? This question leads to the analysis of interpolation error. As we discussed at the beginning of Chapter 2 (Error Analysis), the error given by MEn(x) = f(x) − pn(x). (5.21) is the mathematical error involved in the interpolating polynomial. But, when we per- form the calculations using finite precision floating-point arithmetic, then the polyno- mial obtained, denoted by ˜pn(x), need not be the same as the interpolating polynomial pn(x). The error AEn(x) = pn(x) − ˜pn(x). (5.22) is the arithmetic error involved in the interpolating polynomial. The total error, denoted by TEn(x), involved in the polynomial interpolation is therefore given by TEn(x) = f(x) − ˜pn(x) = (f(x) − pn(x)) + (pn(x) − ˜pn(x)) = MEn(x) + AEn(x). S. Baskar and S. Sivaji Ganesh 176 Spring 2017-18
- 189. Section 5.3 Error in Polynomial Interpolation In Subsection 5.3.1, we derive the mathematical error involved in polynomial interpola- tion. We analyze the effect of arithmetic error in Subsection 5.3.2. 5.3.1 Mathematical Error The following theorem provides a formula for the interpolation error when we as- sume that the necessary data are given exactly without any floating-point approxi- mation. Theorem 5.3.1 [Mathematical Error in Interpolation]. Let pn(x) be the polynomial interpolating a function f ∈ Cn+1 [a, b] at the nodes x0, x1, · · · , xn lying in I = [a, b]. Then for each x ∈ I, there exists a ξx ∈ (a, b) such that MEn(x) = f(n+1) (ξx) (n + 1)! n∏ i=0 (x − xi) (5.23) Proof. If x is one of the nodes, then (5.23) holds trivially for any choice of ξx ∈ (a, b). Therefore, it is enough to prove the theorem when x is not a node. The idea is to obtain a function having at least n + 2 distinct zeros; and then apply Rolle’s theorem n + 1 times to get the desired conclusion. For a given x ∈ I with x ̸= xi (i = 0, 1, · · · , n), define a new function ψ on the interval I by ψ(t) = f(t) − pn(t) − λ n∏ i=0 (t − xi), t ∈ I, (5.24) where λ is chosen so that ψ(x) = 0. This gives λ = f(x) − pn(x) n∏ i=0 (x − xi) . Note that ψ(xi) = 0 for each i = 0, 1, · · · , n. Thus, ψ has at least n + 2 distinct zeros. By Rolle’s theorem, the function ψ′ has at least n + 1 distinct zeros in (a, b). A repeated application of Rolle’s theorem n + 1 times to the function ψ gives that ψ(n+1) has at least one zero in (a, b); call it ξx. Differentiating (5.24) (n + 1)-times S. Baskar and S. Sivaji Ganesh 177 Spring 2017-18
- 190. Chapter 5. Interpolation and noting that p(n+1) n (t) = 0, ( n∏ i=0 (t − xi) )(n+1) = (n + 1)!, ψ(n+1) (ξx) = 0, we see that ξx satisfies 0 = ψ(n+1) (ξx) = f(n+1) (ξx) − f(x) − pn(x) n∏ i=0 (x − xi) (n + 1) !. Thus, f(n+1) (ξx) − MEn(x) n∏ i=0 (x − xi) (n + 1) ! = 0. The last equation yields (5.23). The following theorem plays an important role in the error analysis of numerical inte- gration. The idea behind the proof of this theorem is similar to the idea used in the above theorem and is left as an exercise. Theorem 5.3.2. If f ∈ Cn+1 [a, b] and if x0, x1, · · · , xn are distinct nodes in [a, b], then for any x ̸= xi, i = 0, 1, · · · , n, there exists a point ξx ∈ (a, b) such that f[x0, x1, · · · , xn, x] = f(n+1) (ξx) (n + 1) ! . (5.25) Remark 5.3.3. It is interesting to note that when all the nodes coincide, then (5.25) reduces to (5.19). Definition 5.3.4 [Infinity Norm]. If f is continuous on a closed interval I = [a, b], then the infinity norm of f, denoted by ∥f∥∞,I, is defined as ∥f∥∞,I = max x∈I |f(x)|. (5.26) S. Baskar and S. Sivaji Ganesh 178 Spring 2017-18
- 191. Section 5.3 Error in Polynomial Interpolation Example 5.3.5. Let us obtain an upper bound of the mathematical error for the linear interpolat- ing polynomial with respect to the infinity norm. As in Example 5.1.10, the linear interpolating polynomial for f at x0 and x1 (x0 < x1) is given by p1(x) = f(x0) + f[x0, x1](x − x0), where f[x0, x1] is given by (5.10). For each x ∈ I := [x0, x1], using the formula (5.23), the error ME1(x) is given by ME1(x) = (x − x0)(x − x1) 2 f′′ (ξx), where ξx ∈ (x0, x1) depends on x. Therefore, |ME1(x)| ≤ |(x − x0)(x − x1)| ∥f′′ ∥∞,I 2 . Note that the maximum value of |(x − x0)(x − x1)| as x varies in the interval [x0, x1], occurs at x = (x0 + x1)/2. Therefore, we have |(x − x0)(x − x1)| ≤ (x1 − x0)2 4 . Using this inequality, we get an upper bound |ME1(x)| ≤ (x1 − x0)2 ∥f′′ ∥∞,I 8 , for all x ∈ [x0, x1]. Note that the above inequality is true for all x ∈ [x0, x1]. In particular, this inequality is true for an x at with the function |ME1| attains its maximum. Thus, we have ∥ME1∥∞,I ≤ (x1 − x0)2 ∥f′′ ∥∞,I 8 . The right hand side quantity, which is a real number, is an upper bound for the mathematical error in linear interpolation with respect to the infinity norm. Example 5.3.6. Let the function f(x) = sin x be approximated by an interpolating polynomial p9(x) of degree less than or equal to S. Baskar and S. Sivaji Ganesh 179 Spring 2017-18
- 192. Chapter 5. Interpolation 9 for f at the nodes x0, x1, · · · , x9 in the interval I := [0, 1]. Let us obtain an upper bound for ∥ME9∥∞,I, where (from (5.23)) ME9(x) = f(10) (ξx) 10! 9∏ i=0 (x − xi). Since |f(10) (ξx)| ≤ 1 and ∏9 i=0 |x − xi| ≤ 1, we have | sin x − p9(x)| = |ME9(x)| ≤ 1 10! < 2.8 × 10−7 , for all x ∈ [0, 1]. Since this holds for all x ∈ [0, 1], we have ∥ME9∥∞,I < 2.8 × 10−7 . The right hand side number is the upper bound for the mathematical error ME9 with respect to the infinity norm on the interval I. 5.3.2 Arithmetic Error Quite often, the polynomial interpolation that we compute is based on the function data subjected to floating-point approximation. In this subsection, we analyze the arithmetic error arising due to the floating-point approximation fl(f(xi)) of f(xi) for each node point xi, i = 0, 1, · · · , n in the interval I = [a, b]. The Lagrange form of interpolating polynomial that uses the values fl(f(xi)) instead of f(xi), i = 0, 1, · · · , n is given by ˜pn(x) = n∑ i=0 fl(f(xi)) li(x). We now analyze the arithmetic error. Denoting ϵi := f(xi) − fl(f(xi); ||ϵ||∞ = max{|ϵi| : i = 0, 1, · · · , n}, we have |AEn(x)| = |pn(x) − ˜pn(x)| = n∑ i=0 ( f(xi) − fl(f(xi)) ) li(x) ≤ ||ϵ||∞ n∑ i=0 ||li||∞,I, for all x ∈ I. Therefore, ∥AEn∥∞,I = ∥pn − ˜pn∥∞,I ≤ ||ϵ||∞ n∑ i=0 ||li||∞,I. (5.27) S. Baskar and S. Sivaji Ganesh 180 Spring 2017-18
- 193. Section 5.3 Error in Polynomial Interpolation 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1 1.05 1.1 1.15 1.2 1.25 x l(x) (a) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1 2 3 4 5 6 7 8 9 10 11 x l(x) (b) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 500 1000 1500 2000 2500 3000 3500 x l(x) (c) 0 2 4 6 8 10 12 14 16 18 20 0 2000 4000 6000 8000 10000 12000 14000 n Mn (d) Figure 5.2: (a) to (c) depicts the graph of function l given by (5.28) for x ∈ [0, 1] when n = 2, 8, and 18. (d) depicts the function n in the x-axis and Mn given by (5.29) in the y-axis. The upper bound in (5.27) might grow quite large as n increases, especially when the nodes are equally spaced as we will study now. Assume that the nodes are equally spaced in the interval [a, b], with x0 = a and xn = b, and xi+1 − xi = h for all i = 0, 1, · · · , n − 1. Note that h = (b − a)/n. We write xi = a + ih, i = 0, 1, · · · , n. Any x ∈ I can be written as x = a + ηh, where 0 ≤ η ≤ n. Here η is not necessarily an integer. Therefore, for any x ∈ I, we have lk(x) = n∏ i=0 i̸=k (x − xi) (xk − xi) = n∏ i=0 i̸=k (η − i) (k − i) , k = 0, · · · , n. Clearly, the Lagrange polynomials are not dependent on the choice of a, b, and h. They depend entirely on n and η (which depends on x). The Figure 5.2 (a), (b) and (c) shows S. Baskar and S. Sivaji Ganesh 181 Spring 2017-18
- 194. Chapter 5. Interpolation the graph of the function l(x) = n∑ i=0 |li(x)| (5.28) for n = 2, 8 and 18. It is observed that as n increases, the maximum of the function l increases. In fact, as n → ∞, the maximum of l tends to infinity and it is observed in Figure 5.2 (d) which depicts n in the x-axis and the function Mn = n∑ i=0 ||li||∞,I (5.29) in the y-axis. This shows that the upper bound of the arithmetic error AEn given in (5.27) tends to infinity as n → ∞. This gives the possibility that the arithmetic may tend to increase as n increases. Thus, as we increase the degree of the interpolating polynomial, the approximation may go worser due to the presence of floating-point approximation. In fact, this behavior of the arithmetic error in polynomial interpolation can also be analyzed theoretically, but this is outside the scope of the present course. 5.3.3 Total Error Let us now estimate the total error, which is given by TEn(x) = f(x) − ˜pn(x) = (f(x) − pn(x)) + (pn(x) − ˜pn(x)). (5.30) Taking infinity norm on both sides of the equation (5.30) and using triangle inequality, we get ∥TEn(x)∥∞,I = ||f − ˜p||∞,I ≤ ||f − pn||∞,I + ||pn − ˜p||∞,I ≤ ||f − pn||∞,I + ||ϵ||∞Mn. It is clear from the Figure 5.2 (d) that Mn increases exponentially with respect to n. This implies that even if ||ϵ||∞ is very small, a large enough n can bring in a significantly large error in the interpolating polynomial. Thus, for a given function and a set of equally spaced nodes, even if the mathematical error is bounded, the presence of floating-point approximation in the given data can lead to significantly large arithmetic error for larger values of n. 5.3.4 Runge Phenomenon In the previous section, we have seen that even a small arithmetic error may lead to a drastic magnification of total error as we go on increasing the degree of the polynomial. S. Baskar and S. Sivaji Ganesh 182 Spring 2017-18
- 195. Section 5.3 Error in Polynomial Interpolation −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 x pn(x) (a) −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 −5 0 5 10 15 20 25 30 x pn(x) (b) Figure 5.3: Runge Phenomenon is illustrated. Figure (a) depicts the graph of the function f given by (5.31) (blue solid line) along with the interpolating polynomial of degree 2 (red dash line) and 8 (magenta dash dot line) with equally spaced nodes. Figure (b) shows the graph of f (blue solid line) along with the interpolating polynomial of degree 18 (red dash line) with equally spaced nodes. This gives us a feeling that if the calculation is done with infinite precision (that is, without any finite digit floating point arithmetic) and the function f is smooth, then we always have a better approximation for a larger value of n. In other words, we expect lim n→∞ ∥f − pn∥∞,I = 0. But this is not true in the case of equally spaced nodes. This was first shown by Carl Runge, where he discovered that there are certain functions for which, as we go on increasing the degree of interpolating polynomial, the total error increases drastically and the corresponding interpolating polynomial oscillates near the boundary of the interval in which the interpolation is done. Such a phenomenon is called the Runge Phenomenon. This phenomenon is well understood by the following example given by Carl Runge. Example 5.3.7 [Runge’s Function]. Consider the Runge’s function defined on the interval [−1, 1] given by f(x) = 1 1 + 25x2 . (5.31) The interpolating polynomials with n = 2, n = 8 and n = 18 are depicted in Figure 5.3. This figure clearly shows that as we increase the degree of the polynomial, the interpolating polynomial differs significantly from the actual function especially, near the end points of the interval. S. Baskar and S. Sivaji Ganesh 183 Spring 2017-18
- 196. Chapter 5. Interpolation In the light of the discussion made in Subsection 5.3.2, we may think that the Runge phenomenon is due to the amplification of the arithmetic error. But, even if the cal- culation is done with infinite precision (that is, without any finite digit floating point arithmetic), we may still have the Runge phenomenon due to the amplification in mathematical error. This can be observed by taking infinity norm on both sides of the formula (5.23). This gives the upper bound of the infinity norm of MEn(x) as ∥MEn∥∞,I ≤ (b − a)n+1 (n + 1)! ∥f(n+1) ∥∞,I. Although the first part, (b−a)n+1 /(n+1)! in the upper bound tends to zero as n → ∞, if the second part, ∥f(n+1) ∥∞,I increases significantly as n increases, then the upper bound can still increase and makes it possible for the mathematical error to be quite high. A more deeper analysis is required to understand the Runge phenomenon more rigor- ously. But this is outside the scope of this course and therefore is omitted. 5.3.5 Convergence of Sequence of Interpolating Polynomials We end this section by stating without proof a negative result and a positive result concerning the convergence of sequence of interpolating polynomials. Theorem 5.3.8 [Faber]. For n ∈ N, let the sequence of nodes a ≤ x (n) 0 < x (n) 1 < · · · < x(n) n ≤ b be given. Then there exists a continuous function f defined on the interval [a, b] such that the polynomials pn(x) that interpolate the function f at these nodes have the property that ∥pn − f∥∞,[a,b] does not tend to zero as n → ∞. Example 5.3.9. In fact, the interpolating polynomial pn(x) for the Runge’s function goes worser and worser as shown in Figure 5.3 for increasing values of n with equally spaced nodes. That is, ∥f − pn∥∞,[−1,1] → ∞ as n → ∞ for any sequence of equally spaced nodes. Let us now state a positive result concerning the convergence of sequence of interpolat- ing polynomials to a given continuous function. S. Baskar and S. Sivaji Ganesh 184 Spring 2017-18
- 197. Section 5.4 Piecewise Polynomial Interpolation Theorem 5.3.10. Let f be a continuous function on the interval [a, b]. Then there exists a sequence of nodes a ≤ x (n) 0 < x (n) 1 < · · · < x(n) n ≤ b, for n ∈ N, such that the polynomials pn(x) that interpolate the function f at these nodes satisfy ∥pn − f∥∞,[a,b] → 0 as n → ∞. The Theorem 5.3.10 is very interesting because it implies that for the Runge’s function, we can find a sequence of nodes for which the corresponding interpolating polynomial yields a good approximation even for a large value of n. Example 5.3.11. For instance, define a sequence of nodes x (n) i = cos ( (2i + 1)π 2(n + 1) ) , i = 0, 1, · · · , n (5.32) for each n = 0, 1, 2, · · · . The nodes x (n) i defined by (5.32) are called Chebyshev nodes. In particular, if n = 4, the nodes are x (4) 0 = cos(π/10), x (4) 1 = cos(3π/10), x (4) 2 = cos(5π/10), x (4) 3 = cos(7π/10), x4 4 = cos(9π/10). Figure 5.4 depicts pn(x) for n = 4, 18, 32, and 64 along with the Runge’s function. From these figures, we observe that the interpolating polynomial pn(x) agrees well with the Runge’s function. 5.4 Piecewise Polynomial Interpolation Quite often polynomial interpolation will be unsatisfactory as an approximation tool. This is true if we insist on letting the order of the polynomial get larger and larger. However, if we keep the order of the polynomial fixed, and use different polynomials over different intervals, with the length of the intervals getting smaller and smaller, then the resulting interpolating function approximates the given function more accurately. S. Baskar and S. Sivaji Ganesh 185 Spring 2017-18
- 198. Chapter 5. Interpolation −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 x f(x),p4(x) (a) −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 xf(x),p18(x) (b) −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 x f(x),p32(x) (c) −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 x f(x),p64(x) (d) Figure 5.4: Runge Phenomenon is illustrated with Chebyshev nodes. Figure (a) to (d) shows the graph of the Runge function (blue solid line) and the interpolating polynomial with Chebyshev nodes (red dash line) for n = 4, 18, 32 and 64 respectively. Note that the two graphs in Figure (c) and (d) are indistinguishable. S. Baskar and S. Sivaji Ganesh 186 Spring 2017-18
- 199. Section 5.4 Piecewise Polynomial Interpolation Let us start with linear interpolation over an interval I = [a, b] which leads to p1(x) = f(a) + f[a, b](x − a) = f(a) + f(b) − f(a) b − a (x − a) = x − b a − b f(a) + x − a b − a f(b), where the nodes are x0 = a, x2 = b. In addition to these two nodes, we now choose one more point x1 such that x0 < x1 < x2. With these three nodes, can obtain a quadratic interpolation polynomial. Instead, we can interpolate the function f(x) in [x0, x1] by a linear polynomial with nodes x0 and x1, and in [x1, x2] by another linear polynomial with nodes x1 and x2. Such polynomials are given by p1,1(x) := x − x1 x0 − x1 f(x0) + x − x0 x1 − x0 f(x1), p1,2(x) := x − x2 x1 − x2 f(x1) + x − x1 x2 − x1 f(x2) and the interpolating function is given by s(x) = { p1,1(x) , x ∈ [x0, x1] p1,2(x) , x ∈ [x1, x2] . Note that s(x) is a continuous function on [x0, x2], which interpolates f(x) and is linear in [x0, x1] and [x1, x2]. Such an interpolating function is called piecewise linear interpolating function. In a similar way as done above, we can also obtain piecewise quadratic, cubic interpo- lating functions and so on. Example 5.4.1. Consider the Example 5.1, where we have obtained the quadratic interpolating poly- nomial for the function f(x) = sin (π 2 ex ) . The piecewise linear polynomial interpolating function for the data x 0 0.5 1 f(x) 1.0000 0.5242 −0.9037 is given by s(x) = { 1 − 0.9516 x , x ∈ [0, 0.5] 1.9521 − 2.8558 x , x ∈ [0.5, 1]. The following table shows the value of the function f at x = 0.25 and x = 0.75 along with the values of p2(x) and s(x) with relative errors. S. Baskar and S. Sivaji Ganesh 187 Spring 2017-18
- 200. Chapter 5. Interpolation 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 x y Figure 5.5: The function f(x) = sin (π 2 ex ) (blue line), p2(x) (green bash line) and the piecewise linear interpolation s(x) (red dash and dot line) are shown. Blue dots represent the given data, blue ‘x’ symbol indicates the value of f(0.25) and f(0.75), green ‘+’ symbol indicates the value of p2(0.25) and p2(0.75), and the red ‘O’ symbol represents the value of s(0.25) and s(0.75). x f(x) p2(x) s(x) Er(p2(x)) Er(s(x)) 0.25 0.902117 0.881117 0.762105 0.023278 0.155203 0.75 -0.182750 -0.070720 -0.189732 0.613022 0.038204 Figure 5.5 depicts the graph of f, p2(x) and s(x). In this figure, we observe that the quadratic polynomial p2(x) agrees well with f(x) than s(x) for x ∈ [0, 0.5], whereas s(x) agrees well with f(x) than p2(x) for x ∈ [0.5, 1]. 5.5 Spline Interpolation Although a piecewise interpolating function introduced in Section 5.4 is continuous, it may not be differentiable at the nodes. We wish to find an interpolating function that is sufficiently smooth and does a better approximation to f(x). This can be achieved by spline interpolation. S. Baskar and S. Sivaji Ganesh 188 Spring 2017-18
- 201. Section 5.5 Spline Interpolation Definition 5.5.1 [Spline Interpolating Function]. A spline interpolating function of degree d with nodes xi, i = 0, 1, · · · , n (x0 < x1 < · · · < xn) is a function s(x) with the following properties 1. On each subinterval [xi−1, xi], i = 1, 2, · · · , n, s(x) is a polynomial of degree less than or equal to d. 2. s(x) and its first (d − 1) derivatives are continous on [x0, xn]. 3. The interpolation conditions s(xi) = f(xi), i = 0, 1, · · · , n are satisfied. We shall now study how we can obtain the interpolation of a function f as spline interpolating functions instead of polynomials. For the sake of simplicity, we restrict only to d = 3, called the cubic spline interpolating function. Remark 5.5.2. Note that in each subinterval [xi−1, xi], i = 1, 2, · · · , n, we only know f(xi−1) and f(xi). But we look for a cubic polynomial in this subinterval. Therefore, we cannot follow the Lagrange’s or Newton’s interpolating formula, as these formulas demand the function values at four distinct nodes in the subinterval. We need to adopt a different method for the construction of the polynomial in each subinterval in order to obtain the spline interpolation. Construction of a Cubic Spline The construction of a cubic spline interpolating function s(x) of a function f(x) is as follows: Step 1: Let us denote by M1, · · · , Mn, Mi = s′′ (xi), i = 0, 1, · · · , n and first obtain s(x) in terms of Mi’s which are unknowns. Step 2: Since s(x) is cubic on each [xi−1, xi], the function s′′ (x) is linear on the interval such that s′′ (xi−1) = Mi−1, s′′ (xi) = Mi. Therefore, it is given by s′′ (x) = (xi − x)Mi−1 + (x − xi−1)Mi xi − xi−1 , xi−1 ≤ x ≤ xi (5.33) Integrating (5.33) two times with respect to x, we get s(x) = (xi − x)3 Mi−1 6(xi − xi−1) + (x − xi−1)3 Mi 6(xi − xi−1) + K1x + K2, S. Baskar and S. Sivaji Ganesh 189 Spring 2017-18
- 202. Chapter 5. Interpolation where K1 and K2 are integrating constants to be determined by using the interpolation conditions s(xi−1) = f(xi−1) and s(xi) = f(xi). We have K1 = f(xi) − f(xi−1) xi − xi−1 − (Mi − Mi−1)(xi − xi−1) 6 K2 = xif(xi−1) − xi−1f(xi) xi − xi−1 − (Mi−1xi − Mixi−1)(xi − xi−1) 6 Substituting these values in the above equation, we get s(x) = (xi − x)3 Mi−1 + (x − xi−1)3 Mi 6(xi − xi−1) + (xi − x)f(xi−1) + (x − xi−1)f(xi) xi − xi−1 − 1 6 (xi − xi−1)[(xi − x)Mi−1 + (x − xi−1)Mi], xi−1 ≤ x ≤ xi (5.34) Formula (5.34) applies to each of the intervals [x1, x2], · · · , [xn−1, xn]. The formulas for adjacent intervals [xi−1, xi] and [xi, xi+1] will agree at their common point x = xi because of the interpolating condition s(xi) = f(xi). This implies that s(x) is continuous over the entire interval [a, b]. Similarly, formula (5.33) for s′′ (x) implies that it is continuous on [a, b]. Step 3: All that remains is to find the values of Mi for all i = 0, 1, · · · , n. This is obtained by ensuring the continuity of s′ (x) over [a, b], ie., the formula for s′ (x) on [xi−1, xi] and [xi, xi+1] are required to give the same value at their common point x = xi, for i = 1, 2, · · · , n − 1. After simplification, we get the system of linear equations for i = 1, 2, · · · n − 1 xi − xi−1 6 Mi−1 + xi+1 − xi−1 3 Mi + xi+1 − xi 6 Mi+1 = f(xi+1) − f(xi) xi+1 − xi − f(xi) − f(xi−1) xi − xi−1 . (5.35) These n − 1 equations together with the assumption that M0 = Mn = 0 (5.36) leads to the values of M0, M1, · · · , Mn and hence to the interpolation function s(x). A spline constructed above is called a natural spline. Example 5.5.3. Calculate the natural cubic spline interpolating the data { (1, 1), ( 2, 1 2 ) , ( 3, 1 3 ) , ( 4, 1 4 )} . S. Baskar and S. Sivaji Ganesh 190 Spring 2017-18
- 203. Section 5.5 Spline Interpolation 1 1.5 2 2.5 3 3.5 4 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 x s(x) Figure 5.6: The function f(x) = 1/x (blue line) and the corresponding cubic natural spline interpolating function s(x) (red dash line) are shown. The data are represented by blue dots. The number of points is n = 3 and xi − xi−1 = 1, for x = 1, 2, 3. Step 1: Here, we have M0 = s′′ (1), M1 = s′′ (2), M2 = s′′ (3), M3 = s′′ (4). Step 2: The function s′′ (x) is given by s′′ (x) = (2 − x)M0 + (x − 1)M1 , x ∈ [1, 2] (3 − x)M1 + (x − 2)M2 , x ∈ [2, 3] (4 − x)M2 + (x − 3)M3 , x ∈ [3, 4] . Integrating s′′ (x) two times with respect to x, we get s(x) = (2 − x)3 M0 6 + (x − 1)3 M1 6 + K1x + K2 , x ∈ [1, 2] (3 − x)3 M1 6 + (x − 2)3 M2 6 + K1x + K2 , x ∈ [2, 3] (4 − x)3 M2 6 + (x − 3)3 M3 6 + K1x + K2 , x ∈ [3, 4] . S. Baskar and S. Sivaji Ganesh 191 Spring 2017-18
- 204. Chapter 5. Interpolation where K1 and K2 are integrating constants to be determined by using the conditions s(xi−1) = f(xi−1) and s(xi) = f(xi). We have K1 = − 1 2 − (M1 − M0) 6 , x ∈ [1, 2] − 1 6 − (M2 − M1) 6 , x ∈ [2, 3] − 1 12 − (M3 − M2) 6 , x ∈ [3, 4] . K2 = 3 2 − 2M0 − M1 6 , x ∈ [1, 2] 5 6 − 3M1 − 2M2 6 , x ∈ [2, 3] 7 12 − 4M2 − 3M3 6 , x ∈ [3, 4] . Substituting these expressions in the expression of s(x), we get the required cubic spline as given in (5.34). Step 3: Since we are constructing the natural spline, we take M0 = M3 = 0. The system (5.35) gives 2 3 M1 + 1 6 M2 = 1 3 , 1 6 M1 + 2 3 M2 = 1 12 . Solving this system ,we get M1 = 1 2 , M2 = 0. Substituting these values into (5.34), we obtain s(x) = 1 12 x3 − 1 4 x2 − 1 3 x + 3 2 , x ∈ [1, 2] − 1 12 x3 + 3 4 x2 − 7 3 x + 17 6 , x ∈ [2, 3] − 1 12 x + 7 12 , x ∈ [3, 4] which is the required natural cubic spline approximation to the given data. A com- parison result is depicted in Figure 5.6. S. Baskar and S. Sivaji Ganesh 192 Spring 2017-18
- 205. Section 5.6 Exercises 5.6 Exercises Polynomial Interpolation 1. Let x0, x1, · · · , xn be distinct nodes. If p(x) is a polynomial of degree less than or equal to n, then show that p(x) = n∑ i=0 p(xi)li(x), where li(x) is the ith Lagrange polynomial. 2. Show that the polynomial 1 + x + 2x2 is an interpolating polynomial for the data x 0 1 2 y 1 4 11 . Find an interpolating polynomial for the new data x 0 1 2 3 y 1 4 11 -2 . Does there exist a quadratic polynomial that satisfies the new data? Justify your answer. 3. The quadratic polynomial p2(x) = 3 4 x2 + 1 4 x + 1 2 interpolates the data x −1 0 1 y 1 1 2 3 2 . Find a node x3 (x3 /∈ {−1, 0, 1}), and a real number y3 such that the polynomial p3(x) interpolating the data x −1 0 1 x3 y 1 1/2 3/2 y3 is a polynomial of degree less than or equal to 2. 4. Let p(x), q(x), and r(x) be interpolating polynomials for the three sets of data x 0 1 y y0 y1 , x 0 2 y y0 y2 , and x 1 2 3 y y1 y2 y3 respectively. Let s(x) be the the interpolating polynomial for the data x 0 1 2 3 y y0 y1 y2 y3 . If p(x) = 1 + 2x, q(x) = 1 + x, and r(2.5) = 3, then find the value of s(2.5). 5. Obtain Lagrange form of interpolating polynomial for equally spaced nodes. 6. Find the Largrange form of interpolating polynomial for the data: x -2 -1 1 3 y -1 3 -1 19 S. Baskar and S. Sivaji Ganesh 193 Spring 2017-18
- 206. Chapter 5. Interpolation 7. Find the Lagrange form of interpolating polynomial p2(x) that interpolates the function f(x) = e−x2 at the nodes x0 = −1, x1 = 0 and x2 = 1. Further, find the value of p2(−0.9) (use 6-digit rounding). Compare the value with the true value f(−0.9) (use 6-digit rounding). Find the percentage error in this calculation. Newton’s Divided Difference Formula 8. For the particular function f(x) = xm (m ∈ N), show that f[x0, x1, · · · , xn] = { 1 if n = m 0 if n > m 9. Let x0, x1, · · · , xn be nodes, and f be a given function. Define w(x) = ∏n i=0(x − xi). Prove that f[x0, x1, · · · , xn] = n∑ i=0 f(xi) w′(xi) . 10. The following data correspond to a polynomial P(x) of unknown degree x 0 1 2 P(x) 2 −1 4 . Determine the coefficient of x in the polynomial P(x) if all the third order divided differences are 1. Error in Polynomial Interpolation 11. Let pn(x) be a polynomial of degree less than or equal to n that interpolates a function f at a set of distinct nodes x0, x1, · · · , xn. If x /∈ { x0, x1, · · · , xn }, then show that the error is given by f(x) − pn(x) = f[x0, x1, · · · , xn, x] n∏ i=0 (x − xi). 12. If f ∈ Cn+1 [a, b] and if x0, x1, · · · , xn are distinct nodes in [a, b], then show that there exists a point ξx ∈ (a, b) such that f[x0, x1, · · · , xn, x] = f(n+1) (ξx) (n + 1) ! 13. Let N be a natural number. Let p1(x) denote the linear interpolating polynomial on the interval [N, N +1] interpolating the function f(x) = x2 at the nodes N and N + 1. Find an upper bound for the mathematical error ME1 using the infinity norm on the interval [N, N + 1] (i.e., ∥ME1∥∞, [N,N+1]). S. Baskar and S. Sivaji Ganesh 194 Spring 2017-18
- 207. Section 5.6 Exercises 14. Let p3(x) denote a polynomial of degree less than or equal to 3 that interpolates the function f(x) = ln x at the nodes x0 = 1, x1 = 4 3 , x2 = 5 3 , x3 = 2. Find a lower bound on the absolute value of mathematical error |ME3(x)| at the point x = 3 2 , using the formula for mathematical error in interpolation. 15. Let f : [0, π 6 ] → R be a given function. The following is the meaning for the Cubic interpolation in a table of function values x x0 x1 · · · xN f(x) f(x0) f(x1) · · · f(xN ) The values of f(x) are tabulated for a set of equally spaced points in [a, b], say xi for i = 0, 1, · · · , N with x0 = 0, xN = π 6 , and h = xi+1 − xi > 0 for every i = 0, 1, · · · , N − 1. For an ¯x ∈ [0, π 6 ] at which the function value f(¯x) is not tabulated, the value of f(¯x) is taken to be the value of p3(¯x), where p3(x) is the polynomial of degree less than or equal to 3 that interpolates f at the nodes xi, xi+1, xi+2, xi+3 where i is the least index such that x ∈ [xi, xi+3]. Take f(x) = sin x for x ∈ [0, π 6 ]; and answer the following questions. i) When ¯x and p3 are as described above, then show that |f(¯x) − p3(¯x)| ≤ h4 48 . ii) If h = 0.005, then show that cubic interpolation in the table of function values yields the value of f(¯x) with at least 10 decimal-place accuracy. 16. Let x0, x1, · · · , xn be n + 1 distinct nodes, and f be a function. For each i = 0, 1, · · · , n, let fl (f(xi)) denote the floating point approximation of f(xi) ob- tained by rounding to 5 decimal places (note this is different from using 5-digit rounding). Assume that 0.1 ≤ f(xi) < 1 for all i = 0, 1, · · · , n. Let pn(x) de- note the Lagrange form of interpolating polynomial corresponding to the data {(xi, f(xi)) : i = 0, 1, · · · , n}. Let ˜pn(x) denote the Lagrange form of interpolat- ing polynomial corresponding to the data {(xi, fl (f(xi))) : i = 0, 1, · · · , n}. Show that the arithmetic error at a point ˜x satisfies the inequality |pn(˜x) − ˜pn(˜x)| ≤ 1 2 10−5 n∑ k=0 |lk(˜x)|. Spline Interpolation 17. Find a natural cubic spline interpolating function for the data x -1 0 1 y 5 7 9 . 18. Determine whether the natural cubic spline function that interpolates the table x -1 0 1 y −3 −1 −1 S. Baskar and S. Sivaji Ganesh 195 Spring 2017-18
- 208. Chapter 5. Interpolation is or is not the function S(x) = { x3 + x − 1 x ∈ [−1, 0], x3 − x − 1 x ∈ [0, 1]. 19. Determine whether the coefficients a, b, c, d exist so that the function S(x) = 1 − 2x x ∈ [−4, −3] a + bx + cx2 + dx3 x ∈ [−3, 4] 157 − 32x x ∈ [4, 5] is a natural cubic spline interpolating function on the interval [−4, 5] for the data x -4 -3 4 5 y 9 7 29 -3 . 20. Does there exist real numbers a and b so that the function S(x) = (x − 2)3 + a(x − 1)2 x ∈ [−1, 2] (x − 2)3 − (x − 3)2 x ∈ [2, 3] (x − 3)3 + b(x − 2)2 x ∈ [3, 5] is a natural cubic spline interpolating function on the interval [−1, 5] for the data x -1 2 3 5 y -31 -1 1 17 ? 21. Show that the natural cubic spline interpolation function for a given data is unique. S. Baskar and S. Sivaji Ganesh 196 Spring 2017-18
- 209. CHAPTER 6 Numerical Integration and Differentiation There are two reasons for approximating derivatives and integrals of a function f(x). One is when the function is very difficult to differentiate or integrate, or only the tabular values are available for the function. Another reason is to obtain solution of a differential or integral equation. In this chapter we introduce some basic methods to approximate integral and derivative of a function given either explicitly or by tabulated values. In Section 6.1, we obtain numerical methods for evaluating the integral of a given integrable function f defined on the interval [a, b]. Section 6.2 introduces various ways to obtain numerical formulas for approximating derivatives of a given differentiable function. 6.1 Numerical Integration In this section we derive and analyze numerical methods for evaluating definite integrals. The problem is to evaluate the number I(f) = b∫ a f(x)dx. (6.1) Most such integrals cannot be evaluated explicitly, and with many others, it is faster to integrate numerically than explicitly. The process of approximating the value of I(f) is usually referred to as numerical integration or quadrature rule. S. Baskar and S. Sivaji Ganesh 197 Spring 2017-18
- 210. Chapter 6. Numerical Integration and Differentiation The idea behind numerical integration is to approximate the integrand f by a simpler function that can be integrated easily. One obvious approximation is the interpolation by polynomials. Thus, we approximate I(f) by I(pn), where pn(x) is the interpolating polynomial for the integrand f at some appropriately chosen nodes x0, · · · , xn. The general form of the approximation is I(f) ≈ I(pn) = w0f(x0) + w1f(x1) + · · · + wnf(xn), where the weights are given by wi = I(li), with li(x) the ith Lagrange polynomial. Without going through interpolation, we now propose a general formula I(f) ≈ w0f(x0) + w1f(x1) + · · · + wnf(xn), (6.2) where x0, · · · , xn are distinct real numbers (called quadrature points) and w0, · · · , wn are real numbers (called quadrature weights). When the quadrature points are equally spaced, the quadrature formula of the form (6.2) is called the Newton-Cotes formula. The Newton-Cotes formula (6.2) gives rise to different quadrature formulas depending on the degree of the interpolating polynomial n and also on the choice of the nodes. We now study few simple quadrature formulas. 6.1.1 Rectangle Rule We now consider the case when n = 0 in (6.2). Then, the corresponding interpolating polynomial is the constant function p0(x) = f(x0), and therefore I(p0) = (b − a)f(x0). From this, we can obtain two quadrature rules depending on the choice of x0. • If x0 = a, then this approximation becomes I(f) ≈ IR(f) := (b − a)f(a) (6.3) and is called the rectangle rule. The geometrical interpretation of the rectangle rule is illustrated in Figure 6.1. • If x0 = (a + b)/2, we get I(f) ≈ IM (f) := (b − a)f ( a + b 2 ) (6.4) and is called the mid-point rule. S. Baskar and S. Sivaji Ganesh 198 Spring 2017-18
- 211. Section 6.1 Numerical Integration 0000000000 0000000000 0000000000 00000000000000000000 0000000000 0000000000 0000000000 1111111111 1111111111 1111111111 11111111111111111111 1111111111 1111111111 1111111111 x y a b y=f(x) Figure 6.1: Geometric interpretation of the rectangle Rule. We now obtain the mathematical error in rectangle rule, given by, MER(f) := I(f) − I(p0). Theorem 6.1.1 [Error in Rectangle Rule]. Let f ∈ C1 [a, b]. The mathematical error MER(f) of the rectangle rule takes the form MER(f) = f′ (η)(b − a)2 2 , (6.5) for some η ∈ (a, b). Proof. For each x ∈ (a, b], the linear interpolating polynomial for f at the nodes a and x is given by f(x) = p0(x) + f[a, x](x − a). Therefore, the mathematical error in the rectangle rule is given by MER(f) = I(f) − IR(f) = b∫ a f[a, x](x − a) dx. S. Baskar and S. Sivaji Ganesh 199 Spring 2017-18
- 212. Chapter 6. Numerical Integration and Differentiation 000000000000000000000000000000000 000000000000000000000000000000000 0000000000000000000000 0000000000000000000000 0000000000000000000000 111111111111111111111111111111111 111111111111111111111111111111111 1111111111111111111111 1111111111111111111111 1111111111111111111111 x y a b y=f(x) Figure 6.2: Geometric interpretation of the trapezoidal Rule. Using mean-value theorem for integrals, we get MER(f) = f[a, ξ] b∫ a (x − a) dx, for some ξ ∈ (a, b). By mean value theorem for derivatives, f[a, ξ] = f′ (η) for some η ∈ (a, ξ). Thus, we get MER(f) = f′ (η)(b − a)2 2 , for some η ∈ (a, b). 6.1.2 Trapezoidal Rule We now consider the case when n = 1. Then p1(x) = f(x0) + f[x0, x1](x − x0), and therefore I(f) ≈ IT (f) := b∫ a (f(x0) + f[x0, x1](x − x0)) dx. Taking x0 = a and x1 = b, we get IT (f) = (b − a) ( f(a) + f(b) 2 ) (6.6) and is called the trapezoidal Rule. The Trapezoidal rule is illustrated in Figure 6.2. S. Baskar and S. Sivaji Ganesh 200 Spring 2017-18
- 213. Section 6.1 Numerical Integration We now obtain the mathematical error in trapezoidal rule, given by MET(f) := I(f) − I(p1). Theorem 6.1.2 [Error in Trapezoidal Rule]. Let f ∈ C2 [a, b]. The mathematical error MET(f) of the trapezoidal rule takes the form MET(f) = − f′′ (η)(b − a)3 12 , (6.7) for some η ∈ (a, b). Proof. We have f(x) = f(a) + f[a, b](x − a) + f[a, b, x](x − a)(x − b). Integrating over the interval [a, b], we get I(f) = IT (f) + b∫ a f[a, b, x](x − a)(x − b)dx. Therefore the mathematical error is given by MET(f) = I(f) − IT (f) = b∫ a f[a, b, x](x − a)(x − b)dx. (6.8) From Corollary 5.2.9 (Conclusion 1), we see that the function x −→ f[a, b, x] is con- tinuous. Therefore, from the mean value theorem for integrals (after noting that (x − a)(x − b) is negative for all x ∈ [a, b]), the expression (6.8) for the mathematical error takes the form MET(f) = f[a, b, η] b∫ a (x − a)(x − b)dx, for some η ∈ (a, b). The formula (6.7) now follows from (5.25) and a direct evaluation of the above integral. S. Baskar and S. Sivaji Ganesh 201 Spring 2017-18
- 214. Chapter 6. Numerical Integration and Differentiation Example 6.1.3. For the function f(x) = 1/(x + 1), we approximate the integral I = 1∫ 0 f(x)dx, using trapezoidal rule to get IT (f) = 1 2 ( 1 + 1 2 ) = 3 4 = 0.75. The true value is I(f) = log(2) ≈ 0.693147. Therefore, the error is MET(f) ≈ −0.0569. Using the formula (6.7), we get the bounds for MET(f) as − 1 6 < MET(f) < − 1 48 which clearly holds in the present case. Composite Trapezoidal Rule We can improve the approximation of trapezoidal rule by breaking the interval [a, b] into smaller subintervals and apply the trapezoidal rule (6.6) on each subinterval. We will derive a general formula for this. Let us subdivide the interval [a, b] into n equal subintervals of length h = b − a n with endpoints of the subintervals as xj = a + jh, j = 0, 1, · · · , n. Then, we get I(f) = b∫ a f(x)dx = xn∫ x0 f(x)dx = n−1∑ j=0 xj+1∫ xj f(x)dx. Using trapezoidal rule (6.6) on the subinterval [xj, xj+1], we get xj+1∫ xj f(x)dx ≈ h ( f(xj) + f(xj+1) 2 ) , j = 0, 1, · · · , n − 1. S. Baskar and S. Sivaji Ganesh 202 Spring 2017-18
- 215. Section 6.1 Numerical Integration Substituting this in the above equation, we get I(f) ≈ h [ f(x0) + f(x1) 2 ] + h [ f(x1) + f(x2) 2 ] + · · · + h [ f(xn−1) + f(xn) 2 ] . The terms on the right hand side can be combined to give the simpler formula In T (f) := h [ 1 2 f(x0) + f(x1) + f(x2) + · · · + f(xn−1) + 1 2 f(xn) ] . (6.9) This rule is called the composite trapezoidal rule. Example 6.1.4. Using composite trapezoidal rule with n = 2, let us approximate the integral I = 1∫ 0 f(x)dx, where f(x) = 1 1 + x . As we have seen in Example 6.1.3, the true value is I(f) = log(2) ≈ 0.693147. Now, the composite trapezoidal rule with x0 = 0, x1 = 1/2 and x2 = 1 gives I2 T (f) ≈ 0.70833. Thus the error is -0.0152. Recall from Example 6.1.3 that with n = 1, the trapezoidal rule gave an error of -0.0569. 6.1.3 Simpson’s Rule We now calculate I(p2(x)) to obtain the formula for the case when n = 2. Let us choose x0 = a, x1 = (a + b)/2 and x2 = b. The Lagrange form of interpolating polynomial is p2(x) = f(x0)l0(x) + f(x1)l1(x) + f(x2)l2(x). Then b∫ a p2(x)dx = f(x0) b∫ a l0(x) dx + f(x1) b∫ a l1(x) dx + f(x2) b∫ a l2(x) dx. Using the change of variable x = b − a 2 t + b + a 2 , S. Baskar and S. Sivaji Ganesh 203 Spring 2017-18
- 216. Chapter 6. Numerical Integration and Differentiation 00000000000 00000000000 0000000000000000000000 00000000000 0000000000000000000000 00000000000 0000000000000000000000 00000000000 00000000000 11111111111 11111111111 1111111111111111111111 11111111111 1111111111111111111111 11111111111 1111111111111111111111 11111111111 11111111111 0000000000 0000000000 00000000000000000000 00000000000000000000 1111111111 1111111111 11111111111111111111 11111111111111111111 x y a b(a+b)/2 y=f(x) y=p (x)2 Figure 6.3: Geometric interpretation of the Simpson’s Rule. we get b∫ a l0(x) dx = b∫ a (x − x1)(x − x2) (x0 − x1)(x0 − x2) dx = 1∫ −1 t(t − 1) 2 b − a 2 dt = b − a 6 . Similarly, we can see b∫ a l1(x) dx = 4 6 (b − a), b∫ a l2(x) dx = b − a 6 . We thus arrive at the formula I(f) ≈ IS(f) := b∫ a p2(x)dx = b − a 6 { f(a) + 4f ( a + b 2 ) + f(b) } (6.10) which is the famous Simpson’s Rule. The Simpson’s rule is illustrated in Figure 6.3. We now obtain the mathematical error in Simpson’s rule, given by, MES(f) := I(f) − I(p2). S. Baskar and S. Sivaji Ganesh 204 Spring 2017-18
- 217. Section 6.1 Numerical Integration Theorem 6.1.5 [Error in Simpson’s Rule]. Let f ∈ C4 [a, b]. The mathematical error MES(f) of the Simpson’s rule takes the form MES(f) = − f(4) (η)(b − a)5 2880 , (6.11) for some η ∈ (a, b). Proof. Consider the collection of nodes a, (a + b)/2, b, x in the interval [a, b]. Let x0 ∈ [a, b] be a given point different from a, (a + b)/2, b, and x. From the fourth order divided difference formula, we have f[a, (a + b)/2, b, x] = f[x0, a, (a + b)/2, b] + f[x0, a, (a + b)/2, b, x](x − x0). The mathematical error in Simpson’s method can therefore be written as MES(f) = b∫ a f[x0, a, (a + b)/2, b]ϕ(x)dx + b∫ a f[x0, a, (a + b)/2, b, x](x − x0)ϕ(x)dx, where ϕ(x) = (x − a)(x − (a + b)/2)(x − b). A direct integration shows b∫ a ϕ(x)dx = b∫ a (x − a) ( x − a + b 2 ) (x − b)dx = 0. Thus, we have MES(f) = b∫ a f[x0, a, (a + b)/2, b, x](x − x0)ϕ(x)dx, Recall in the proof of Theorem 6.1.2, we used mean value theorem at this stage to arrive at the conclusion. But, in the present case, we cannot use this theorem because the function (x − x0)ϕ(x) need not be of one sign on [a, b]. However, the choice x0 = (a + b)/2 makes this function one signed (negative). Now using Corollary 5.2.9 (Conclusion 1) and following the idea of the proof of Theorem 6.1.2, we arrive at the formula (6.11) (this is left as an exercise). S. Baskar and S. Sivaji Ganesh 205 Spring 2017-18
- 218. Chapter 6. Numerical Integration and Differentiation Example 6.1.6. We now use the Simpson’s rule to approximate the integral I(f) = 1∫ 0 f(x) dx, where f(x) = 1 1 + x . The true value is I(f) = log(2) ≈ 0.693147. Using the Simpson’s rule (6.10), we get IS(f) = 1 6 ( 1 + 8 3 + 1 2 ) = 25 36 ≈ 0.694444. Therefore, the error is MES(f) ≈ 0.001297. Composite Simpson’s Rule Let us now derive the composite Simpson’s rule. First let us subdivide the interval [a, b] into n equal parts as we discussed in composite trapezoidal rule (Subsection 6.1.2). Taking a = xi−1, b = xi+1, (note that xi = (xi−1 +xi+1)/2) and xi −xi−1 = h in Simpson rule, we get ∫ xi+1 xi−1 f(x)dx ≈ h 3 {f(xi−1) + 4f(xi) + f(xi+1)} . Summing for i = 1, · · · , N/2, (N even) we get ∫ b a f(x)dx = N/2 ∑ i=1 ∫ x2i x2i−2 f(x)dx ≈ h 3 N/2 ∑ i=1 {f(x2i−2) + 4f(x2i−1) + f(x2i)} . Therefore, the composite Simpson’s rule takes the form I N/2 S (f) = h 3 f(x0) + f(xN ) + 2 N/2 ∑ i=2 f(x2i−2) + 4 N/2 ∑ i=1 f(x2i−1) . (6.12) S. Baskar and S. Sivaji Ganesh 206 Spring 2017-18
- 219. Section 6.1 Numerical Integration 6.1.4 Method of Undetermined Coefficients All the rules so far derived are of the form I(f) = b∫ a f(x)dx ≈ w0f(x0) + w1f(x1) + · · · + wnf(xn) (6.12) where wi’s are weights. In deriving those rules, we have fixed the nodes x0, x1, · · · , xn (n = 0 for rectangle rule, n = 1 for trapezoidal rule and n = 2 for Simpson’s rule), and used interpolating polynomials to obtain the corresponding weights. Instead, we may use another approach in which for a fixed set of nodes weights are determined by imposing the condition that the resulting rule is exact for polynomials of degree less than or equal to n. Such a method is called the method of undetermined coeffi- cients. Example 6.1.7. Let us find w0, w1, and w2 such that the approximate formula b∫ a f(x) dx ≈ w0f(a) + w1f ( a + b 2 ) + w2f(b) (6.13) is exact for all polynomials of degree less than or equal to 2. Since integral of sum of functions is the sum of the integrals of the respective functions, the formula (6.13) is exact for all polynomials of degree less than or equal to 2 if and only if the formula (6.13) is exact for the polynomials 1, x, and x2 . • The condition that the formula (6.13) is exact for the polynomial p(x) = 1 yields b − a = b∫ a 1 dx = w0 + w1 + w2 • The condition that the formula (6.13) is exact for the polynomial p(x) = x yields b2 − a2 2 = b∫ a x dx = aw0 + ( a + b 2 ) w1 + bw2. • The condition that the formula (6.13) is exact for the polynomial p(x) = x2 S. Baskar and S. Sivaji Ganesh 207 Spring 2017-18
- 220. Chapter 6. Numerical Integration and Differentiation yields b3 − a3 3 = b∫ a x2 dx = a2 w0 + ( a + b 2 )2 w1 + b2 w2 Thus, we have a linear system of three equations satisfied by w0, w1, and w2. By solving this system, we get w0 = 1 6 (b − a), w1 = 2 3 (b − a), w2 = 1 6 (b − a), which gives us the familiar Simpson’s rule. This motivates the following definition. Definition 6.1.8 [Degree of Precision]. The degree of precision (also called order of exactness) of a quadrature for- mula is the largest positive integer n such that the formula is exact for all polynomials of degree less than or equal to n. Example 6.1.9. Let us determine the degree of precision of Simpson’s rule. It will suffice to apply the rule over the interval [0, 2] (in fact any interval is good enough and we chose this interval for the sake of having easy computation). 2∫ 0 dx = 2 = 2 6 (1 + 4 + 1), 2∫ 0 xdx = 2 = 2 6 (0 + 4 + 2), 2∫ 0 x2 dx = 8 3 = 2 6 (0 + 4 + 4), 2∫ 0 x3 dx = 4 = 2 6 (0 + 4 + 8), 2∫ 0 x4 dx = 32 5 ̸= 2 6 (0 + 4 + 16) = 20 3 . Therefore, the degree of precision of Simpson’s rule is 3. S. Baskar and S. Sivaji Ganesh 208 Spring 2017-18
- 221. Section 6.1 Numerical Integration Remark 6.1.10. In Example 6.1.7 we have obtained the Simpson’s rule using the method of un- determined coefficients by requiring the exactness of the rule for polynomials of degree less than or equal to 2, the above example shows that the rule is exact for polynomials of degree three as well. 6.1.5 Gaussian Rules In Example 6.1.7 we have fixed the nodes and obtained the weights in the quadrature rule (6.13) such that the rule is exact for polynomials of degree less than or equal to 2. In general, by fixing the nodes, we can obtain the weights in (6.12) such that the rule is exact for polynomials of degree less than or equal to n. But it is also possible to derive a quadrature rule such that the rule is exact for polynomials of degree less than or equal to 2n + 1 by choosing the n + 1 nodes and the weights appropriately. This is the basic idea of Gaussian rules. Let us consider the special case 1∫ −1 f(x)dx ≈ n∑ i=0 wif(xi). (6.14) The weights wi and the nodes xi (i = 0, · · · , n) are to be chosen in such a way that the rule (6.14) is exact, that is 1∫ −1 f(x)dx = n∑ i=0 wif(xi), (6.15) whenever f(x) is a polynomial of degree less than or equal to 2n + 1. Note that (6.15) holds for every polynomial f(x) of degree less than or equal to 2n + 1 if and only if (6.15) holds for f(x) = 1, x, x2 , · · · , x2n+1 . Case 1: (n = 0). In this case, the quadrature formula (6.14) takes the form 1∫ −1 f(x)dx ≈ w0f(x0). The condition (6.15) gives 1∫ −1 1 dx = w0 and 1∫ −1 x dx = w0x0. S. Baskar and S. Sivaji Ganesh 209 Spring 2017-18
- 222. Chapter 6. Numerical Integration and Differentiation These conditions give w0 = 2 and x0 = 0. Thus, we have the formula 1∫ −1 f(x)dx ≈ 2f(0) =: IG0 (f), (6.16) which is the required Gaussian rule for n = 0. Case 2: (n = 1). In this case, the quadrature formula (6.14) takes the form 1∫ −1 f(x)dx ≈ w0f(x0) + w1f(x1). The condition (6.15) gives w0 + w1 = 2, w0x0 + w1x1 = 0, w0x2 0 + w1x2 1 = 2 3 , w0x3 0 + w1x3 1 = 0. A solution of this nonlinear system is w0 = w1 = 1, x0 = −1/ √ 3 and x1 = 1/ √ 3. This lead to the formula 1∫ −1 f(x)dx ≈ f ( − 1 √ 3 ) + f ( 1 √ 3 ) =: IG1(f), (6.17) which is the required Gaussian rule for n = 1. Case 3: (General). In general, the quadrature formula is given by (6.14), where there are 2(n + 1) free parameters xi and wi for i = 0, 1, · · · , n. The condition (6.15) leads to the nonlinear system n∑ j=0 wjxi j = 0 , i = 1, 3, · · · , 2n + 1 2 i + 1 , i = 0, 2, · · · , 2n . These are nonlinear equations and their solvability is not at all obvious and therefore the discussion is outside the scope of this course. So far, we derived Gaussian rule for integrals over [−1, 1]. But this is not a limitation as any integral on the interval [a, b] can easily be transformed to an integral on [−1, 1] S. Baskar and S. Sivaji Ganesh 210 Spring 2017-18
- 223. Section 6.1 Numerical Integration by using the linear change of variable x = b + a + t(b − a) 2 , −1 ≤ t ≤ 1. (6.18) Thus, we have b∫ a f(x)dx = b − a 2 1∫ −1 f ( b + a + t(b − a) 2 ) dt. Example 6.1.11. We now use the Gaussian rule to approximate the integral I(f) = 1∫ 0 f(x) dx, where f(x) = 1 1 + x . Note that the true value is I(f) = log(2) ≈ 0.693147. To use the Gaussian quadrature, we first need to make the linear change of variable (6.18) with a = 0 and b = 1 and we get x = t + 1 2 , −1 ≤ t ≤ 1. Thus the required integral is I(f) = 1∫ 0 dx 1 + x = 1∫ −1 dt 3 + t . We need to take f(t) = 1/(3 + t) in the Gaussian quadrature formula (6.17) and we get 1∫ 0 dx 1 + x = 1∫ −1 dt 3 + t ≈ f ( − 1 √ 3 ) + f ( 1 √ 3 ) ≈ 0.692308 ≈ IG1(f). Therefore, the error is I(f) − IG1(f) ≈ 0.000839. S. Baskar and S. Sivaji Ganesh 211 Spring 2017-18
- 224. Chapter 6. Numerical Integration and Differentiation Theorem 6.1.12. Let f(x) be continous for a ≤ x ≤ b, and let n ≥ 1. Then the absolute error |En(f)| in using Gaussian numerical integration rule to obtain I(f) satisfies |En(f)| ≤ 2(b − a)ρ2n+1(f) (6.19) where ρ2n+1(f) = inf deg q≤2n+1 ∥f − q∥∞ is the minmax error of order 2n + 1 for f(x) on [a, b]. Proof. En(p) = 0 for any polynomial p(x) of degree ≤ 2n + 1. Also, the error function En satisfies En(F + G) = En(F) + En(G) for all F, G ∈ C[a, b]. Let p(x) = q∗ 2n+1(x), the minimax approximation of degree ≤ 2n + 1 to f(x) on [a, b]. Then En(f) = En(f) − En(q∗ 2n+1) = En(f − q∗ 2n+1) = ∫ b a (f(x) − q∗ 2n+1(x))dx − n∑ j=0 wj(f(xj) − q∗ 2n+1(xj)). Therefore, we have |En(f)| ≤ ∥f − q∗ 2n+1∥∞[(b − a) + n∑ j=0 |wj|]. But we have n∑ j=0 |wj| = b − a and therefore, we get the desired result. 6.2 Numerical Differentiation The aim of this section is to obtain formulas to approximate the values of derivatives of a given function at a given point. Such a formula for the first derivative of a function can be obtained directly using the definition of the derivative, namely, the difference quotients of the function. This will be discussed in Subsection 6.2.1. But this idea cannot be adopted for higher order derivatives. Approximating formulas for derivatives can be obtained in at least two ways, namely, S. Baskar and S. Sivaji Ganesh 212 Spring 2017-18
- 225. Section 6.2 Numerical Differentiation 1. Methods based on Interpolation 2. Methods based on Undetermined Coefficients These methods are discussed in Subsection 6.2.2 and 6.2.3, respectively. 6.2.1 Approximations of First Derivative Forward Difference Formula The most simple way to obtain a numerical method for approximating the derivative of a C1 function f is to use the definition of derivative f′ (x) = lim h→0 f(x + h) − f(x) h . The approximating formula can therefore be taken as f′ (x) ≈ f(x + h) − f(x) h =: D+ h f(x) (6.20) for a sufficiently small value of h > 0. The formula D+ h f(x) is called the forward difference formula for the derivative of f at the point x. Theorem 6.2.1. Let f ∈ C2 [a, b]. The mathematical error in the forward difference formula is given by f′ (x) − D+ h f(x) = − h 2 f′′ (η) (6.21) for some η ∈ (x, x + h). Proof. By Taylor’s theorem, we have f(x + h) = f(x) + hf′ (x) + h2 2 f′′ (η) (6.22) for some η ∈ (x, x + h). Using (6.20) and (6.22), we obtain D+ h f(x) = 1 h {[ f(x) + hf′ (x) + h2 2 f′′ (η) ] − f(x) } = f′ (x) + h 2 f′′ (η). This completes the proof. S. Baskar and S. Sivaji Ganesh 213 Spring 2017-18
- 226. Chapter 6. Numerical Integration and Differentiation Remark 6.2.2. If we consider the left hand side of (6.21) as a function of h, i.e., if g(h) = f′ (x) − Dhf(x), then we see that g(h) h = 1 2 |f′′ (η)|. Let M > 0 be such that |f′′(x)| ≤ M for all x ∈ [a, b]. Then we see that g(h) h ≤ M 2 . That is, g = O(h) as h → 0. We say that the forward difference formula D+ h f(x) is of order 1 (order of accuracy). Backward Difference Formula The derivative of a function f is also given by f′ (x) = lim h→0 f(x) − f(x − h) h . Therefore, the approximating formula for the first derivative of f can also be taken as f′ (x) ≈ f(x) − f(x − h) h =: D− h f(x) (6.23) The formula D− h f(x) is called the backward difference formula for the derivative of f at the point x. Deriving the mathematical error for backward difference formula is similar to that of the forward difference formula. It can be shown that the backward difference formula is of order 1. Central Difference Formula The derivative of a function f is also given by f′ (x) = lim h→0 f(x + h) − f(x − h) 2h . Therefore, the approximating formula for the first derivative of f can also be taken as f′ (x) ≈ f(x + h) − f(x − h) 2h =: D0 hf(x), (6.24) S. Baskar and S. Sivaji Ganesh 214 Spring 2017-18
- 227. Section 6.2 Numerical Differentiation . .. xx−h x+h Forward Backward Central x y y=f(x) f’ Figure 6.4: Geometrical interpretation of difference formulae. for a sufficiently small value of h > 0. The formula D0 hf(x) is called the central difference formula for the derivative of f at the point x. The central difference formula is of order 2 as shown in the following theorem. Theorem 6.2.3. Let f ∈ C3 [a, b]. The mathematical error in the central difference formula is given by f′ (x) − D0 hf(x) = − h2 6 f′′′ (η), (6.25) where η ∈ (x − h, x + h). Proof. Using Taylor’s theorem, we have f(x + h) = f(x) + hf′ (x) + h2 2! f′′ (x) + h3 3! f′′′ (η1) and f(x − h) = f(x) − hf′ (x) + h2 2! f′′ (x) − h3 3! f′′′ (η2), S. Baskar and S. Sivaji Ganesh 215 Spring 2017-18
- 228. Chapter 6. Numerical Integration and Differentiation where η1 ∈ (x, x + h) and η2 ∈ (x − h, x). Therefore, we have f(x + h) − f(x − h) = 2hf′ (x) + h3 3! (f′′′ (η1) + f′′′ (η2)). Since f′′′ (x) is continuous, by intermediate value theorem applied to f′′ , we have f′′′ (η1) + f′′′ (η2) 2 = f′′′ (η) for some η ∈ (x − h, x + h). This completes the proof. Geometric interpretation of the three primitive difference formulae (forward, backward, and central) is shown in Figure 6.4. Example 6.2.4. To find the value of the derivative of the function given by f(x) = sin x at x = 1 with h = 0.003906, we use the three primitive difference formulas. We have f(x − h) = f(0.996094) = 0.839354, f(x) = f(1) = 0.841471, f(x + h) = f(1.003906) = 0.843575. 1. Backward difference: D− h f(x) = f(x) − f(x − h) h = 0.541935. 2. Central Difference: D0 hf(x) = f(x + h) − f(x − h) 2h = 0.540303. 3. Forward Difference: D+ h f(x) = f(x + h) − f(x) h = 0.538670. Note that the exact value is f′ (1) = cos 1 = 0.540302. 6.2.2 Methods based on Interpolation Using the polynomial interpolation we can obtain formula for derivatives of any order for a given function. For instance, to calculate f′ (x) at some point x, we use the approximate formula f′ (x) ≈ p′ n(x), where pn(x) denotes the interpolating polynomial for f(x). Many formulas can be obtained by varying n and by varying the placement of the nodes x0, · · · , xn relative to the point x of interest. S. Baskar and S. Sivaji Ganesh 216 Spring 2017-18
- 229. Section 6.2 Numerical Differentiation Let us take n = 1. The linear interpolating polynomial is given by p1(x) = f(x0) + f[x0, x1](x − x0). Hence, we have the formula f′ (x) ≈ p′ 1(x) = f[x0, x1]. (6.26) In particular, • if we take x0 = x and x1 = x + h for a small value h > 0, we obtain the forward difference formula D+ h f(x). • if we take x0 = x−h and x1 = x for small value of h > 0, we obtain the backward difference formula D− h f(x). • if we take x0 = x − h and x1 = x + h for small value of h > 0, we get the central difference formula D0 hf(x). We next prove the formula for mathematical error in approximating the first derivative using interpolating polynomial. Theorem 6.2.5 [Mathematical Error]. Hypothesis: 1. Let f be an (n + 2)-times continuously differentiable function on the interval [a, b]. 2. Let x0, x1, · · · , xn be n + 1 distinct nodes in [a, b]. 3. Let pn(x) denote the polynomial that interpolates f at the nodes x0, x1, · · · , xn. 4. Let x be any point in [a, b] such that x /∈ {x0, x1 · · · , xn}. Conclusion: Then f′ (x) − p′ n(x) = wn(x) f(n+2) (ηx) (n + 2)! + w′ n(x) f(n+1) (ξx) (n + 1)! (6.27) with wn(x) = n∏ i=0 (x − xi) and ξx and ηx are points in between the maximum and minimum of x0, x1 · · · , xn and x, that depend on x. Proof. For any x ∈ [a, b] with x /∈ {x0, x1 · · · , xn}, by Newton’s form of interpolating polynomial, we have f(x) = pn(x) + f[x0, · · · , xn, x]wn(x), S. Baskar and S. Sivaji Ganesh 217 Spring 2017-18
- 230. Chapter 6. Numerical Integration and Differentiation where pn(x) is the polynomial (of degree ≤ n) that interpolates f at x0, · · · , xn. Taking derivative on both sides, we get f′ (x) = p′ n(x) + wn(x) d dx f[x0, · · · , xn, x] + w′ n(x)f[x0, · · · , xn, x]. But we know that (Theorem 5.2.10) d dx f[x0, · · · , xn, x] = f[x0, · · · , xn, x, x]. Therefore, we have f′ (x) = p′ n(x) + wn(x)f[x0, · · · , xn, x, x] + w′ n(x)f[x0, · · · , xn, x]. Further, from Theorem 5.3.2, we see that there exists an ξx ∈ (a, b) such that f[x0, · · · , xn, x] = f(n+1) (ξx) (n + 1)! . Using Theorem 5.3.2 along with the Hermite-Genocchi formula, we see that there exists an ηx ∈ (a, b) such that f[x0, · · · , xn, x, x] = f(n+2) (ηx) (n + 2)! . Therefore, we get f′ (x) − p′ n(x) = wn(x) f(n+2) (ηx) (n + 2)! + w′ n(x) f(n+1) (ξx) (n + 1)! which is what we wish to show. Difference formulas for higher order derivatives and their mathematical error can be obtained similarly. The derivation of the mathematical error for the formulas of higher order derivatives are omitted for this course. Example 6.2.6. Let x0, x1, and x2 be the given nodes. Then, the Newton’s form of interpolating polynomial for f is given by p2(x) = f(x0) + f[x0, x1](x − x0) + f[x0, x1, x2](x2 − (x0 + x1)x + x0x1). Therefore, we take the first derivative of f as f′ (x) ≈ p′ 2(x) = f[x0, x1] + f[x0, x1, x2](2x − x0 − x1). S. Baskar and S. Sivaji Ganesh 218 Spring 2017-18
- 231. Section 6.2 Numerical Differentiation • If we take x0 = x − h, x1 = x, and x2 = x + h for any given x ∈ [a, b], we obtain the central difference formula D0 h(f) and the corresponding error obtained from (6.27) is precisely the error given in (6.25). • If we take x0 = x, x1 = x+h and x2 = x+2h for any given x ∈ [a, b], we obtain the difference formula f′ (x) ≈ −3f(x) + 4f(x + h) − f(x + 2h) 2h with mathematical error obtained using (6.27) as f′ (x) − p′ 2(x) = h2 3 f′′′ (ξ), for some ξ ∈ (x, x + 2h). 6.2.3 Methods based on Undetermined Coefficients Another method to derive formulas for numerical differentiation is called the method of undetermined coefficients. The idea behind this method is similar to the one discussed in deriving quadrature formulas. Suppose we seek a formula for f(k) (x) that involves the nodes x0, x1, · · · , xn. Then, write the formula in the form f(k) (x) ≈ w0f(x0) + w1f(x1) + · · · + wnf(xn) (6.28) where wi, i = 0, 1, · · · , n are free variables that are obtained by imposing the con- dition that this formula is exact for polynomials of degree less than or equal to n. Example 6.2.7. We will illustrate the method by deriving the formula for f′′ (x) at nodes x0 = x − h, x1 = x and x2 = x + h for a small value of h > 0. For a small value of h > 0, let f′′ (x) ≈ D (2) h f(x) := w0f(x − h) + w1f(x) + w2f(x + h) (6.29) where w0, w1 and w2 are to be obtained so that this formula is exact when f(x) is a polynomial of degree less than or equal to 2. This condition is equivalent to the exactness for the three polynomials 1, x and x2 . Step 1: When f(x) = 1 for all x. Then the formula of the form (6.29) is assumed to S. Baskar and S. Sivaji Ganesh 219 Spring 2017-18
- 232. Chapter 6. Numerical Integration and Differentiation be exact and we get w0 + w1 + w2 = 0. (6.30) Step 2: When f(x) = x for all x. Then the formula of the form (6.29) is assumed to be exact and we get w0(x − h) + w1x + w2(x + h) = 0. Using (6.30), we get w2 − w0 = 0. (6.31) Step 3: When f(x) = x2 for all x. Then the formula of the form (6.29) is assumed to be exact and we get w0(x − h)2 + w1x2 + w2(x + h)2 = 2. Using (6.30) and (6.31), we get w0 + w2 = 2 h2 . (6.32) Solving the linear system of equations (6.30), (6.31), and (6.32), we get w0 = w2 = 1 h2 , w1 = − 2 h2 . Substituting these into (6.29), we get D (2) h f(x) = f(x + h) − 2f(x) + f(x − h) h2 , (6.33) which is the required formula. Let us now derive the mathematical error involved in this formula. For this, we use the Taylor’s series f(x ± h) = f(x) ± hf′ (x) + h2 2! f′′ (x) ± h3 3! f(3) (x) + · · · in (6.33) to get D (2) h f(x) = 1 h2 ( f(x) + hf′ (x) + h2 2! f′′ (x) + h3 3! f(3) (x) + h4 4! f(4) (x) + · · · ) − 2 h2 f(x) + 1 h2 ( f(x) − hf′ (x) + h2 2! f′′ (x) − h3 3! f(3) (x) + h4 4! f(4) (x) − · · · ) . S. Baskar and S. Sivaji Ganesh 220 Spring 2017-18
- 233. Section 6.2 Numerical Differentiation After simplification, we get D (2) h f(x) = f′′ (x) + h2 24 [(f(4) (x) + · · · ) + (f(4) (x) − · · · )]. Now treating the fourth order terms on the right hand side as remainders in Taylor’s series, we get D (2) h f(x) = f′′ (x) + h2 24 [f(4) (ξ1) + f(4) (ξ2)], for some ξ1, ξ2 ∈ (x − h, x + h). Using intermediate value theorem for the function f(4) , we get the mathematical error as f′′ (x) − D (2) h f(x) = − h2 12 f(4) (ξ) (6.34) for some ξ ∈ (x − h, x + h), which is the required mathematical error. 6.2.4 Arithmetic Error in Numerical Differentiation Difference formulas are useful when deriving methods for solving differential equations. But they can lead to serious errors when applied to function values that are subjected to floating-point approximations. Let f(xi) = fi + ϵi, i = 0, 1, 2. To illustrate the effect of such errors, we choose the approximation D (2) h f(x) given by (6.33) for the second derivative of f with x0 = x − h, x1 = x and x2 = x + h. Instead of using the exact values f(xi), we use the appoximate values fi in the difference formula (6.33). That is, ¯D (2) h f(x1) = f2 − 2f1 + f0 h2 . The total error committed is f′′ (x1) − ¯D (2) h f(x1) = f′′ (x1) − f(x2) − 2f(x1) + f(x0) h2 + ϵ2 − 2ϵ1 + ϵ0 h2 = − h2 12 f(4) (ξ) + ϵ2 − 2ϵ1 + ϵ0 h2 . Using the notation ϵ∞ := max{|ϵ0|, |ϵ1|, |ϵ3|}, we have |f′′ (x1) − ¯D (2) h f(x1)| ≤ h2 12 |f(4) (ξ)| + 4ϵ∞ h2 . (6.35) The error bound in (6.35) clearly shows that, although the first term (bound of math- ematical error) tends to zero as h → 0, the second term (bound of arithmetic error) S. Baskar and S. Sivaji Ganesh 221 Spring 2017-18
- 234. Chapter 6. Numerical Integration and Differentiation h upperbound * Figure 6.5: A sketch of the upper bound in total error as given in (6.35) as a function of h. The black star indicates the optimal value of h. can tend to infinity as h → 0. This gives a possibility for the total error to be as large as possible when h → 0. In fact, there is an optimal value of h to minimize the right side of (6.35) (as shown in Figure 6.5), which we will illustrate in the following example. Example 6.2.8. In finding f′′ (π/6) for the function f(x) = cos x, if we use the function values fi that has six significant digits when compared to f(xi), then |f(xi) − fi| |f(xi)| ≤ 0.5 × 10−5 . Since |f(xi)| = | cos(xi)| ≤ 1, we have |f(xi) − fi| ≤ 0.5 × 10−5 . We now use the formula ¯D (2) h f(x) to approximate f′′ (x). Assume that other than the approximation in the function values, the formula ¯D (2) h f(x) is calculated exactly. Then the bound for the absolute value of the total error given by (6.35) takes the form |f′′ (π/6) − ¯D (2) h f(π/6)| ≤ h2 12 |f(4) (ξ)| + 4ϵ∞ h2 , where ϵ∞ ≤ 0.5 × 10−5 and ξ ≈ π/6. Thus, we have |f′′ (π/6) − ¯D (2) h f(π/6)| ≤ h2 12 cos (π 6 ) + 4 h2 (0.5 × 10−5 ) ≈ 0.0722h2 + 2 × 10−5 h2 =: E(h). The bound E(h) indicates that there is a smallest value of h, call it h∗ , such that the bound increases rapidly for 0 < h < h∗ when h → 0. To find it, let E′ (h) = 0, with its root being h∗ . This leads to h∗ ≈ 0.129. Thus, for close values of h > h∗ ≈ 0.129, we S. Baskar and S. Sivaji Ganesh 222 Spring 2017-18
- 235. Section 6.3 Exercises have less error bound than the values 0 < h < h∗ . This behavior of E(h) is observed in the following table. Note that the true value is f′′ (π/6) = − cos(π/6) ≈ −0.86603. h ¯D (2) h f(π/6) Total Error E(h) 0.2 -0.86313 -0.0029 0.0034 0.129 -0.86479 -0.0012 0.0024 0.005 -0.80000 -0.0660 0.8000 0.001 0.00000 -0.8660 20 When h is very small, f(x−h), f(x) and f(x+h) are very close numbers and therefore their difference in the numerator of the formula (6.33) tend to have loss of significance. This is clearly observed in the values of ¯D (2) h f(π/6) when compared to the true value where, we are not loosing much number of significant digits for h > 0.129, whereas for h < 0.129, there is a drastic loss of significant digits. 6.3 Exercises Numerical Integration 1. Apply Rectangle, Trapezoidal, Simpson and Gaussian methods to evaluate i) I = π/2∫ 0 cos x 1 + cos2 x dx (exact value ≈ 0.623225) ii) I = π∫ 0 dx 5 + 4 cos x dx (exact value ≈ 1.047198) iii) I = 1∫ 0 e−x2 dx (exact value ≈ 0.746824), iv) I = π∫ 0 sin3 x cos4 x dx (exact value ≈ 0.114286) v) I = 1∫ 0 (1 + e−x sin(4x)) dx. (exact value ≈ 1.308250) Compute the relative error (when compared to the given exact values) in each method. S. Baskar and S. Sivaji Ganesh 223 Spring 2017-18
- 236. Chapter 6. Numerical Integration and Differentiation 2. Write down the errors in the approximation of 1∫ 0 x4 dx and 1∫ 0 x5 dx by the Trapezoidal rule and Simpson’s rule. Find the value of the constant C for which the Trapezoidal rule gives the exact result for the calculation of the integral 1∫ 0 (x5 − Cx4 )dx. 3. A function f has the values shown below: x 1 1.25 1.5 1.75 2 f(x) 10 8 7 6 5 . i) Use trapezoidal rule to approximate ∫ 2 1 f(x) dx. ii) Use Simpson’s rule to approximate ∫ 2 1 f(x) dx. iii) Use composite Simpson’s rule to approximate ∫ 2 1 f(x) dx. 4. Obtain expressions for the arithmetic error in approximating the integral ∫ b a f(x) dx using the trapezoidal and the Simpson’s rules. Also obtain upper bounds. 5. Let a = x0 < x1 < · · · < xn = b be equally spaced nodes (i.e., xk = x0 + kh for k = 1, 2, · · · , n) in the interval [a, b]. Note that h = (b − a)/n. Let f be a twice continuously differentiable function on [a, b]. i) Show that the expression for the mathematical error in approximating the integral ∫ b a f(x) dx using the composite trapezoidal rule, denoted by En T (f), is given by En T (f) = − (b − a)h2 12 f′′ (ξ), for some ξ ∈ (a, b). ii) Show that the mathematical error En T (f) tends to zero as n → ∞ (one uses the terminology composite trapezoidal rule is convergent in such a case). 6. Determine the minimum number of subintervals and the corresponding step size h so that the error for the composite trapezoidal rule is less than 5 × 10−9 for approximating the integral ∫ 7 2 dx/x. 7. Let a = x0 < x1 < · · · < xn = b be equally spaced nodes (i.e., xk = x0 + kh for k = 1, 2, · · · , n) in the interval [a, b], and n is an even natural number. Note that h = (b − a)/n. Let f be a four times continuously differentiable function on [a, b]. i) Show that the expression for the mathematical error in approximating the integral ∫ b a f(x) dx using the composite Simpson rule, denoted by En S(f), is given by En S(f) = − (b − a)h4 180 f(4) (ξ), S. Baskar and S. Sivaji Ganesh 224 Spring 2017-18
- 237. Section 6.3 Exercises for some ξ ∈ (a, b). ii) Show that the mathematical error En S(f) tends to zero as n → ∞ (one uses the terminology composite Simpson rule is convergent in such a case). 8. Use composite Simpson’s and composite Trapezoidal rules to obtain an approxi- mate value for the improper integral ∞∫ 1 1 x2 + 9 dx, with n = 4. 9. Determine the coefficients in the quadrature formula 2h∫ 0 x−1/2 f(x) dx ≈ (2h)1/2 (w0f(0) + w1f(h) + w2f(2h)) such that the formula is exact for all polynomials of degree as high as possible. What is the degree of precision? 10. Use the two-point Gaussian quadrature rule to approximate ∫ 1 −1 dx/(x + 2) and compare the result with the trapezoidal and Simpson’s rules. 11. Assume that xk = x0 + kh are equally spaced nodes. The quadrature formula x3∫ x0 f(x)dx ≈ 3h 8 (f(x0) + 3f(x1) + 3f(x2) + f(x3)) is called the Simpson’s 3 8 rule. Determine the degree of precision of Simpson’s 3 8 rule. Numerical Differentiation 12. In this problem, perform the calculations using 6-digit rounding arithmetic. i) Find the value of the derivative of the function f(x) = sin x at x = 1 using the forward, backward, and central difference formulae with h1 = 0.015625, and h2 = 0.000015. ii) Find f′ (1) directly and compare with the values obtained for each hi (i = 1, 2). 13. Obtain the central difference formula for f′ (x) using polynomial interpolation with nodes at x − h, x, x + h, where h > 0. 14. Given the values of the function f(x) = ln x at x0 = 2.0, x1 = 2.2 and x2 = 2.6, find the approximate value of f′ (2.0) using the method based on quadratic interpolation. Obtain an error bound. S. Baskar and S. Sivaji Ganesh 225 Spring 2017-18
- 238. Chapter 6. Numerical Integration and Differentiation 15. The following data corresponds to the function f(x) = sin x. x 0.5 0.6 0.7 f(x) 0.4794 0.5646 0.6442 Obtain the approximate value of f′ (0.5), f′ (0.6), and f′ (0.7) using forward, back- ward, and central difference formulae whichever are applicable. Compute the relative error in all the three cases. 16. The following data corresponds to the function f(x) = ex − 2x2 + 3x + 1. x 0.0 0.2 0.4 f(x) 0.0 0.7414 1.3718 Obtain the approximate value of f′ (0.0), f′ (0.2), and f′ (0.4) using forward, back- ward, and central difference formulae whichever are applicable. Compute the relative error in all the three cases. 17. Obtain expressions for the arithmetic error in approximating the first derivative of a function using the forward, backward, and central difference formulae. 18. Find an approximation to f′ (x) as a formula involving f(x), f(x + h), and f(x + 2h). Obtain an expression for the mathematical error involved in this approximation. 19. Let h > 0. Use the method of undetermined coefficients to find a numerical differentiation formula for approximating f′′ (x) such that the formula uses values of the function f at each of the following sets of points: i) x + 2h, x + h and x. ii) x + 3h, x + 2h x + h and x. Obtain expressions for mathematical error in both the cases. 20. Show that the formula D(2) f(x) := f(x) − 2f(x − h) + f(x − 2h) h2 gives approximate value for f′′ (x). Find the order of accuracy of this formula. 21. For the method f′ (x) ≈ 4f(x + h) − f(x + 2h) − 3f(x) 2h , obtain an expression for mathematical error, arithmetic error, and hence total error. Find a bound on the absolute value of the total error as function of h. Determine the optimal value of h for which the bound obtained is minimum. 22. Repeat the previous problem when central difference formula is used for numerical differentiation. S. Baskar and S. Sivaji Ganesh 226 Spring 2017-18
- 239. Section 6.3 Exercises 23. Let f(x) = ln(x) (here ln denotes the natural logarithm). Give the formula for approximating f′ (x) using central difference formula. When we use this formula to get an approximate value of f′ (1.5) with the assumption that the function values f(1.5 − h) and f(1.5 + h) are rounded to 2 digits after decimal point, find the value of h such that the total error is minimized. (Final Exam, Autumn 2010 (M.Sc.)) 24. The voltage E = E(t) in an electrical circuit obeys the equation E(t) = L ( dI dt ) + RI(t), where R is resistance and L is inductance. Use L = 0.05 and R = 2 and values for I(t) in the table following. x 1.0 1.1 1.2 1.3 1.4 f(x) 8.2277 7.2428 5.9908 4.5260 2.9122 Find I′ (1.2) using (i) central difference formula, and (ii) the formula given in Problem (21) and use it to compute E(1.2). Compare your answer with I(t) = 10e−t/10 sin(2t). S. Baskar and S. Sivaji Ganesh 227 Spring 2017-18
- 241. CHAPTER 7 Numerical Ordinary Differential Equations Consider a first order ordinary differential equation (ODE) of the form y′ = f(x, y), where y = y(x) is an unknown variable, x ∈ R is an independent variable and the function f : R × R → R is given. Here, we used the notation y′ := dy/dx. The objective is to find the solution y for all x ∈ [a, b] subject to an initial condition y(x0) = y0, x0 ∈ [a, b]. We call the problem of solving the above ODE along with this initial condition, the initial value problem. It is well-known that there are many ODEs of physical interest that cannot be solved exactly although we know that such problems have unique solutions. If one wants the solution of such problems, then the only way is to obtain it approximately. One common way of obtaining an approximate solution to a given initial value problem is to numerically compute the solution using a numerical method (or numerical scheme). In this chapter, we introduce some basic numerical methods for approximating the solution of a given initial value problem. In Section 7.1, we review the exact solvability of a given initial value problem and motivate the need of numerical methods. In Section 7.3, we introduce a basic numerical method called Euler’s method for obtaining approximate solution of an initial value problem and discussed the error involved in this method. We also show in this section that the Euler method is of order 1. Modified forms of Euler method can be obtained S. Baskar and S. Sivaji Ganesh 229 Spring 2017-18
- 242. Chapter 7. Numerical Ordinary Differential Equations using numerical quadrature formulas as discussed in Section 7.4. Taylor approximation with higher order terms can be used to obtain numerical methods of higher order accuracy. However, such methods involve the higher order derivatives of the unknown function and are called Taylor methods. Taylor methods are hard to implement on a computer as they involve higher order derivatives. An alternate way of achieving higher order accuracy without using higher order derivatives of the unknown function is the famous Runge-Kutta methods. Runge-Kutta method of order 2 is derived in full detail in Section 7.5 and the formula for the Runge-Kutta method of order 4 is stated. 7.1 Review of Theory In this section, we will discuss solving an initial value problem (IVP) for an ordinary differential equation. Let f be a continuous function of two variables defined on a domain D ⊂ R2 . Let (x0, y0) ∈ D be given. The initial value problem for an ODE is given by y′ = f(x, y), (x, y) ∈ D, (7.1a) y(x0) = y0. (7.1b) The following result is concerning the equivalence of the initial value problem (7.1) with an integral equation. Lemma 7.1.1. A continuous function y defined on an interval I containing the point x0 is a solution of the initial value problem (7.1) if and only if y satisfies the integral equation y(x) = y0 + ∫ x x0 f(s, y(s)) ds, (7.2) for all x ∈ I. Proof. If y is a solution of the initial value problem (7.1), then we have y′ (x) = f(x, y(x)). (7.3) Integrating the above equation from x0 to x yields the integral equation (7.2). On the other hand let y be a solution of integral equation (7.2). Observe that, due to continuity of the function x → y(x), the function s → f(s, y(s)) is continuous S. Baskar and S. Sivaji Ganesh 230 Spring 2017-18
- 243. Section 7.1 Review of Theory on I. Thus by fundamental theorem of integral calculus, the right hand side of (7.2) is a differentiable function with respect to x and its derivative is given by the function x → f(x, y(x)), which is a continuous function. From (7.2), we see that y is continuously differentiable and hence a direct differentiation of this equation yields the ODE (7.1a). Evaluating (7.2) at x = x0 gives the initial condition y(x0) = y0. Remark 7.1.2. The following are the consequences of Lemma 7.1.1. • If f is a function of x only, ie., f = f(x), which can be integrated explicitly, then the integration on the right hand side of (7.2) can be obtained explicitely to get the solution y of the IVP (7.1a) exactly. For instance, if f(x, y) = ex, then the solution of (7.1a) is y(x) = y0+ex−ex0 . • If f depends on y, ie., f = f(x, y), then (7.2) is an integral equation which in general cannot be solved explicitly. However, there are some particular cases, for instance when (7.1a) is linear, separable or exact, then we can obtain a solution of (7.1a). There are certain other cases where an integral factor can be obtained through which the equation can be made exact. But, there are many ODEs for which none of the above mentioned methods can be used to obtain solution. For instance, the equation y′ = x2 + y2 is clearly neither linear nor separable nor exact. Also any of the standard ways of finding integrating factor will not work for this equation. When standard method cannot be used to obtain solutions exactly, several approximation procedures may be proposed for finding an approximate solu- tion to the initial value problem (7.1). In these procedures, one may use the ODE (7.1a) itself or its equivalent integral formulation (7.2). If we choose the ODE (7.1a), then the derivative may be approximated by a numerical differentiation formula. In case we choose the integral formulation (7.2), then we may use various numerical integration techniques (quadrature rules) to adopt a numerical method for approximating the solution of the initial value problem (7.1). Before going for a numerical method, it is important to ensure that the given initial value problem has a unique solution. Otherwise, we will be solving an initial value problem numerically, which actually does not have a solution or it may have many solutions and we do not know which solution the numerical method obtains. Let us illustrate the non-uniqueness of solution of an initial value problem. S. Baskar and S. Sivaji Ganesh 231 Spring 2017-18
- 244. Chapter 7. Numerical Ordinary Differential Equations Example 7.1.3 [Peano]. Let us consider the initial value problem y′ = 3 y2/3 , y(0) = 0. (7.4) Note that y(x) = 0 for all x ∈ R is clearly a solution for this initial value problem. Also, y(x) = x3 for all x ∈ R. This initial value problem has infinite family of solutions parametrized by c ≥ 0, given by yc(x) = { 0 if x ≤ c, (x − c)3 if x > c, defined for all x ∈ R. Thus, we see that a solution to an initial value problem need not be unique. However by placing additional conditions on f, we can achieve uniqueness as stated in the following theorem. The proof of this theorem is omitted for this course. Theorem 7.1.4 [Cauchy-Lipschitz-Picard’s Existence and Uniqueness Theorem]. Let D ⊆ R2 be a domain and I ⊂ R be an interval. Let f : D → R be a continuous function. Let (x0, y0) ∈ D be a point such that the rectangle R defined by R = {x : |x − x0| ≤ a} × {y : |y − y0| ≤ b} (7.5) is contained in D. Let f be Lipschitz continuous with respect to the variable y on R, i.e., there exists a K > 0 such that |f(x, y1) − f(x, y1)| ≤ K|y1 − y2| ∀(x, y1), (x, y2) ∈ R. Then the initial value problem (7.1) has at least one solution on the interval |x−x0| ≤ δ where δ = min{a, b M }. Moreover, the initial value problem (7.1) has exactly one solution on this interval. Remark 7.1.5. We state without proof that the function f(x, y) = y2/3 in Example 7.1.3 is not a Lipschitz function on any rectangle containing the point (0, 0) in it. While verifying that a given function f is Lipschitz continuous is a little difficult, one can easily give a set of alternative conditions that guarantee Lipschitz continuity, and also these conditions are easy to verify. S. Baskar and S. Sivaji Ganesh 232 Spring 2017-18
- 245. Section 7.1 Review of Theory Lemma 7.1.6. Let D ⊆ R2 be an open set and f : D → R be a continuous function such that its partial derivative with respect to the variable y is also continuous on D, i.e., ∂f ∂y : D → R is a continuous function. Let the rectangle R defined by R = {x : |x − x0| ≤ a} × {y : |y − y0| ≤ b} be contained in D. Then f is Lipschitz continuous with respect to y in R. Proof. Let (x, y1), (x, y2) ∈ R. Applying mean value theorem with respect to the y variable, we get f(x, y1) − f(x, y1) = ∂f ∂y (x, ξ)(y1 − y2), (7.6) for some ξ between y1 and y2. Since we are applying mean value theorem by fixing x, this ξ will also depend on x. However since ∂f ∂y : D → R is a continuous function, it will be bounded on R. That is, there exists a number L > 0 such that ∂f ∂y (x, y) ≤ L for all (x, y) ∈ R. (7.7) Taking modulus in the equation (7.6), and using the last inequality we get |f(x, y1) − f(x, y1)| = ∂f ∂y (x, ξ) |(y1 − y2)| ≤ L |(y1 − y2)|. This finishes the proof of lemma. The following theorem can be used as a tool to check the existence and uniqueness of solution of a given initial value problem (7.1). Corollary 7.1.7 [Existence and Uniqueness Theorem]. Let D ⊆ R2 be a domain and I ⊆ R be an interval. Let f : D → R be a continuous function. Let (x0, y0) ∈ D be a point such that the rectangle R defined by R = {x : |x − x0| ≤ a} × {y : |y − y0| ≤ b} (7.8) is contained in D. If the partial derivative ∂f/∂y is also continuous in D, then there exists a unique solution y = y(x) of the initial value problem (7.1) defined on the interval |x−x0| ≤ δ where δ = min{a, b M }. S. Baskar and S. Sivaji Ganesh 233 Spring 2017-18
- 246. Chapter 7. Numerical Ordinary Differential Equations We state an example (without details) of an initial value problem that has a unique solution but the right hand side function in the differential equation is not a Lipschitz function. Thus this example illustrates the fact that condition of Lipschitz continuity is only a sufficient condition for uniqueness and by no means necessary. Example 7.1.8. The IVP y′ = { y sin 1 y if y ̸= 0 0 if y = 0, y(0) = 0. has unique solution, despite the RHS not being Lipschitz continuous with respect to the variable y on any rectangle containing (0, 0). Remark 7.1.9 [On Existence and Uniqueness Theorem]. 1. Though we have stated existence theorems without proof, we can confirm that their proofs do not give an explicit soluton of the initial value problem being considered. In fact, the proofs give a sequence of functions that converge to a solution of the initial value problem. 2. Even when the RHS of the ordinary differential equation is a function of x only, we do not have explicit solutions. For example, y′ = e−x2 , y(0) = 0. The only alternative is to approximate its solution by a numerical procedure. 7.2 Discretization Notations A numerical method gives approximate value of the solution of the initial value problem (7.1) at only a discrete set of point. Thus, if we are interested in obtaining solution for (7.1a) in an interval [a, b], then we first discretize the interval as a = x0 < x1 < · · · < xn = b, (7.9) where each point xi, i = 0, 1, · · · , n is called a node. Unless otherwise stated, we always assume that the nodes are equally spaced. That is, xj = x0 + jh, j = 0, 1, · · · n (7.10) for a sufficiently small positive real number h. We use the notation for the approximate solution as yj = yh(xj) ≈ y(xj), j = 0, 1, · · · , n. (7.11) S. Baskar and S. Sivaji Ganesh 234 Spring 2017-18
- 247. Section 7.3 Euler’s Method 7.3 Euler’s Method The most simplest numerical method for a first order ordinary differential equation (7.1a) is obtained by replace the first order derivative of the unknown function by its finite difference formula. Assume that we know the value of the unknown function y at a point x = x0. For obtaining the value of y at the point x1 = x0 + h, we use the forward difference formula for the derivative given by y′ (x) ≈ 1 h (y(x + h) − y(x)) in (7.1a) to get 1 h (y(x1) − y(x0)) ≈ f(x0, y(x0)). This can be used to obtain the value of y(x1) in terms of x0 and y(x0) as y(x1) ≈ y(x0) + hf(x0, y(x0)). Since we assumed that we know the value of y(x0), the right hand side is fully known and hence y(x1) can now be computed explicitly. In general, if you know the value of y(xj), j = 0, 1, · · · , n, we can obtain the value of y(xj+1) by using the forward difference formula in (7.1a) at x = xj to get 1 h (y(xj+1) − y(xj)) ≈ f(xj, y(xj)). Denoting the approximate value of y(xj) by yj, we can adopt the formula yj+1 = yj + hf(xj, yj), j = 0, 1, · · · , n − 1 (7.12) for obtaining the values of the solution y at the discrete points xj, j = 1, 2, · · · , n by taking the value of y0 from the initial condition (7.1b). The formula (7.12) is called the forward Euler’s method. The backward Euler’s method can be obtained by using the backward difference formula for the derivative of y in (7.1) and is given by yj−1 = yj − hf(xj, yj), j = 0, −1, · · · , −n + 1. (7.13) By Euler’s method we mean either forward or backward Euler’s method depending on the context. S. Baskar and S. Sivaji Ganesh 235 Spring 2017-18
- 248. Chapter 7. Numerical Ordinary Differential Equations Figure 7.1: Geometrical interpretation of Euler’s method Remark 7.3.1 [Geometric Interpretation]. A geometric insight into Euler’s method is shown in Figure 7.1 The tangent line to the graph of y(x) at x = xj has slope f(xj, yj). The Euler’s method approximates the value of y(xj+1) by the corresponding value of tangent line at the point x = xj+1. Example 7.3.2. Consider the initial value problem y′ = y, y(0) = 1. The Euler method (7.12) for this equation takes the form yj+1 = yj + hyj = (1 + h)yj. Note that the exact solution for the given initial value problem is y(x) = ex . On applying Euler’s method with h = 0.01 and using 7-digit rounding, we get y(0.01) ≈ y1 = 1 + 0.01 = 1.01 y(0.02) ≈ y2 = 1.01 + 0.01(1.01) = 1.0201 y(0.03) ≈ y3 = 1.0201 + 0.01(1.0201) = 1.030301 y(0.04) ≈ y4 = 1.030301 + 0.01(1.030301) = 1.040604 S. Baskar and S. Sivaji Ganesh 236 Spring 2017-18
- 249. Section 7.3 Euler’s Method The numerical results along with the error is presented in the following table for h = 0.01. h x yh(x) Exact Solution Error Relative Error 0.01 0.00 1.000000 1.000000 0.000000 0.000000 0.01 0.01 1.010000 1.010050 0.000050 0.000050 0.01 0.02 1.020100 1.020201 0.000101 0.000099 0.01 0.03 1.030301 1.030455 0.000154 0.000149 0.01 0.04 1.040604 1.040811 0.000207 0.000199 0.01 0.05 1.051010 1.051271 0.000261 0.000248 Since the exact solution of this equation is y = ex , the correct value at x = 0.04 is 1.040811. By taking smaller values of h, we may improve the accuracy in the Euler’s method. The numerical results along with the error is shown in the following table for h = 0.005. h x yh(x) Exact Solution Error Relative Error 0.005 0.00 1.000000 1.000000 0.000000 0.000000 0.005 0.005 1.005000 1.005013 0.000013 0.000012 0.005 0.01 1.010025 1.010050 0.000025 0.000025 0.005 0.015 1.015075 1.015113 0.000038 0.000037 0.005 0.02 1.020151 1.020201 0.000051 0.000050 0.005 0.025 1.025251 1.025315 0.000064 0.000062 0.005 0.03 1.030378 1.030455 0.000077 0.000075 0.005 0.035 1.035529 1.035620 0.000090 0.000087 0.005 0.04 1.040707 1.040811 0.000104 0.000100 0.005 0.045 1.045910 1.046028 0.000117 0.000112 0.005 0.05 1.051140 1.051271 0.000131 0.000125 7.3.1 Error in Euler’s Method In Example (7.3.2), we illustrated that as we reduce the step size h, we tend to get more accurate solution of a given IVP at a given point x = xj. The truncation error confirms this illustration when y′′ is a bounded function. However, the mathematical error which involves the truncation error in the computed solution yj and the propagating error from the computation of the solution at x = xi for i = 0, 1, · · · j − 1. In addition to the mathematical error, we also have arithmetic error due to floating-point approximation in each arithmetic operation. In this section, we study the total error involved in forward Euler’s method. Total error involved in backward Euler’s method can be obtained in a S. Baskar and S. Sivaji Ganesh 237 Spring 2017-18
- 250. Chapter 7. Numerical Ordinary Differential Equations similar way. Using Taylor’s theorem, write y(xj+1) = y(xj) + hy′ (xj) + h2 2 y′′ (ξj) for some xj < ξj < xj+1. Since y(x) satisfies the ODE y′ = f(x, y(x)), we get y(xj+1) = y(xj) + hf(xj, y(xj)) + h2 2 y′′ (ξj). Thus, the local truncation error in forward Euler’s method is Tj+1 = h2 2 y′′ (ξj), (7.14) which is the error involved in obtaining the value y(xj+1) using the exact value y(xj). The forward Euler’s method uses the approximate value yj in the formula and therefore the finally computed value yj+1 not only involves the truncation error but also the propagated error involved in computing yj. Thus, the local mathematical error in the forward Euler’s method is given by ME(yj+1) := y(xj+1) − yj+1 = y(xj) − yj + h(f(xj, y(xj)) − f(xj, yj)) + h2 2 y′′ (ξj). Here, y(xj) − yj + h(f(xj, y(xj)) − f(xj, yj)) is the propagated error. The propagated error can be simplified by applying the mean value theorem to f(x, z) considering it as a functin of z: f(xj, y(xj)) − f(xj, yj) = ∂f(xj, ηj) ∂z [y(xj) − yj], for some ηj lying between y(xj) and yj. Using this, we get the mathematical error ME(yj+1) = [ 1 + h ∂f(xj, ηj) ∂z ] ME(yj) + h2 2 y′′ (ξj) (7.15) for some xj < ξj < xj+1, and ηj lying between y(xj) and yj. We now assume that over the interval of interest, ∂f(xj, y(xj)) ∂z < L, |y′′ (x)| < Y , where L and Y are fixed positive constants. On taking absolute values in (7.15), we obtain |ME(yj+1)| ≤ (1 + hL)|ME(yj)| + h2 2 Y . (7.16) S. Baskar and S. Sivaji Ganesh 238 Spring 2017-18
- 251. Section 7.3 Euler’s Method Applying the above estimate recursively, we get |ME(yj+1)| ≤ (1 + hL)2 |ME(yj−1)| + (1 + (1 + hL)) h2 2 Y ≤ · · · ≤ · · · ≤ (1 + hL)j+1 |ME(y0)| + ( 1 + (1 + hL) + (1 + hL)2 + · · · + (1 + hL)j )h2 2 Y . Using the formulas 1. For any α ̸= 1, 1 + α + α2 + · · · + αj = αj+1 − 1 α − 1 2. For any x ≥ −1, (1 + x)N ≤ eNx , in the above inequality, we have proved the following theorem. Theorem 7.3.3. Let y ∈ C2 [a, b] be a solution of the IVP (7.1) with ∂f(x, y) ∂y < L, |y′′ (x)| < Y , for all x and y, and some constants L > 0 and Y > 0. The mathematical error in the forward Euler’s method at a point xj = x0 + jh satisfies |ME(yj)| ≤ hY 2L ( e(xn−x0)L − 1 ) + e(xn−x0)L |y(x0) − y0| (7.17) Example 7.3.4. Consider the initial value problem y′ = y, y(0) = 1, x ∈ [0, 1]. Let us now find the upper bound for the mathematical error of forward Euler’s method in solving this problem. Here f(x, y) = y. Therefore, ∂f/∂y = 1 and hence we can take L = 1. Since y = ex , y′′ = ex and |y′′ (x)| ≤ e for 0 ≤ x ≤ 1. Therefore, we take Y = e. S. Baskar and S. Sivaji Ganesh 239 Spring 2017-18
- 252. Chapter 7. Numerical Ordinary Differential Equations We now use the estimate (7.17) with x0 = 0 and xn = 1 to obtain |ME(yj)| ≤ he 2 (e − 1) ≈ 2.3354h. Here, we assume that there is no approximation in the initial condition and therefore the second term in (7.17) is zero. To validate the upper bound obtained above, we shall compute the approximate solution of the given IVP using forward Euler’s method. The method for the given IVP reads yj+1 = yj + hf(xj, yj) = (1 + h)yj. The solution of this difference equation satisfing y(0) = 1 is yj = (1 + h)j . Now, if h = 0.1, n = 10, we have yj = (1.1)10 . Therefore, the forward Euler’s method gives y(1) ≈ y10 ≈ 2.5937. But the exact value is y(1) = e ≈ 2.71828. The error is 0.12466, whereas the bound obtained from (7.17) was 0.2354. Remark 7.3.5. The error bound (7.17) is valid for a large family of the initial value problems. But, it usually produces a very poor estimate due to the presence of the exponential terms. For instance, in the above example, if we take xn to be very large, then the corresponding bound will also be very large. The above error analysis assumes that the numbers used are of infinite precision and no floating point approximation is assumed. When we include the floating point ap- proximation that yn = ˜yn + ϵn, then the bound for total error is given in the following theorem. The proof of this theorem is left as an exercise. Theorem 7.3.6. Let y ∈ C2 [a, b] be a solution of the IVP (7.1) with ∂f(x, y) ∂y < L, |y′′ (x)| < Y , for all x and y, and some constants L > 0 and Y > 0. Let yj be the approximate solution of (7.1) computed using the forward Euler’s method (7.12) with infinite S. Baskar and S. Sivaji Ganesh 240 Spring 2017-18
- 253. Section 7.4 Modified Euler’s Methods precision and let ˜yj be the corresponding computed value using finite digit floating- point arithmetic. If yj = ˜yj + ϵj, then the total error TE(yj) := y(xj) − ˜yj in forward Euler’s method at a point xj = x0 + jh satisfies |TE(yj)| ≤ 1 L ( hY 2 + ϵ h ) ( e(xn−x0)L − 1 ) + e(xn−x0)L |ϵ0|, (7.18) where ϵ := max{|ϵi|/i = 0, 1, · · · , n}. 7.4 Modified Euler’s Methods The Euler’s method derived in the previous section can also be derived using the equiv- alent integral form of the IVP (7.1) as discussed in Lemma 7.1.1. Using this integral form in the interval [xj, xj+1], we get y(xj+1) = y(xj) + ∫ xj+1 xj f(s, y)ds. (7.19) The Euler’s method can be obtained by replacing the integral on the right hand side by the rectangle rule. Using the integral form in the interval [xj−1, xj+1] and using the mid-point quadrature formula given by ∫ xj+1 xj−1 f(s, y)ds ≈ f(xj, yj)(xj+1 − xj−1), we get the Euler’s mid-point method yj+1 = yj−1 + 2hf(xj, yj). (7.20) To compute the value of yj+1, we need to know the value of yj−1 and yj. Note that the above formula cannot be used to compute the value of y1. Hence, we need another method to obtain y1 and then yj, for j = 2, 3, · · · , n can be obtained using (7.20). This method belongs to the class of 2-step methods. Example 7.4.1. Consider the initial-value problem y′ = y, y(0) = 1. To obtain the approximate value of y(0.04) with h = 0.01: S. Baskar and S. Sivaji Ganesh 241 Spring 2017-18
- 254. Chapter 7. Numerical Ordinary Differential Equations We first use Euler’s method to get y(0.01) ≈ y1 = 1 + 0.01 = 1.01 Next use Euler’s mid-point method to get y(0.02) ≈ y2 = y0 + 2 × h × y1 = 1 + 2 × 0.01 × 1.01 = 1.0202 y(0.03) ≈ y3 = y1 + 2 × h × y2 = 1.01 + 2 × 0.01 × 1.0202 ≈ 1.030404 y(0.04) ≈ y4 = y2 + 2 × h × y3 = 1.040808 Since the exact solution of this equation is y = ex , the correct value at x = 0.04 is 1.040811. The error is 0.000003. Recall the error in Euler method was 0.000199. The methods derived above are explicit methods in the sense that the value of yj+1 is computed using the known values. If we using the trapezoidal rule for the integration on the right hand side of (7.19), we get yj+1 = yj + h 2 (f(xj, yj) + f(xj+1, yj+1)). (7.21) This method is called the Euler’s Trapezoidal method. Here, we see that the formula (7.21) involves an implicit relation for yj+1. Such methods are referred to as implicit methods. Although the Euler’s Trapezoidal method gives an implicit relation for yj+1, some- times it is explicit to compute the values yj+1 as illustrated in the following exam- ple. Example 7.4.2. Let us use the Euler’s trapezoidal rule with h = 0.2 to obtain the appoximate solution of the initial value problem y′ = xy, y(0) = 1. We have y0 = 1 and y1 = y0 + h 2 (x0y0 + x1y1) = 1 + 0.1(0 + 0.2y1), which gives (1 − 0.02)y1 = 1, and this implies y1 ≈ 1.0204. Similarly, y2 = y1 + h 2 (x1y1 + x2y2) = 1.0204 + 0.1(0.2 × 1.0204 + 0.4y2), S. Baskar and S. Sivaji Ganesh 242 Spring 2017-18
- 255. Section 7.5 Runge-Kutta Methods and y(0.4) ≈ y2 = 1.0408 1 − 0.04 ≈ 1.0842. In general, the Euler’s trapezoidal rule gives a nonlinear equation for yj+1 as illustrated below. Example 7.4.3. Consider the initial value problem y′ = e−y , y(0) = 1. We use the Euler’s trapezoidal rule with h = 0.2 to solve the above problem. We have, y1 = y0 + h 2 (e−y0 + e−y1 ) = 1 + 0.1(e−1 + e−y1 ), which gives the nonlinear equation g(y1) = y1 − 0.1e−y1 − (1 + 0.1e−1 ) = 0, and the solution of this equation is the approximate value of the solution y(x1) of the given initial value problem. 7.5 Runge-Kutta Methods Although Euler’s method is easy to implement, this method is not so efficient in the sense that to get a better approximation, one needs a very small step size. One way to get a better accuracy is to include the higher order terms in the Taylor expansion to get an approximation to y′ . But the higher order terms involve higher derivatives of y. The Runge-Kutta methods attempts to obtain higher order accuracy and at the same time avoid the need for higher derivatives, by evaluating the function f(x, y) at selected points on each subintervals. We first derive the Runge-Kutta method of order 2. The derivation of the Runge-Kutta method of order 4 can be done in a similar way. So, we skip the derivation of this method and present only final formula. 7.5.1 Order Two Let y be a solution of the ODE (7.1a). The Runge-Kutta method of order 2 is obtained by truncating the Taylor expansion of y(x + h) after the quadratic term. We derive now a formula for this method. Taylor expansion of y(x + h) at the point x upto the S. Baskar and S. Sivaji Ganesh 243 Spring 2017-18
- 256. Chapter 7. Numerical Ordinary Differential Equations quadratic term is given by y(x + h) = y(x) + hy′ (x) + h2 2 y′′ (x) + O(h3 ). (7.22) Since y satisfies the given ODE y′ = f(x, y), by differentiating this ODE with respect to x both sides gives y′′ (x) = ∂f ∂x (x, y(x)) + y′ (x) ∂f ∂y (x, y(x)) (7.23) Substituting the values of y′ , y′′ in (7.22), we get y(x + h) = y(x) + hf(x, y(x)) + h2 2 [ ∂f ∂x (x, y(x)) + f(x, y(x)) ∂f ∂y (x, y(x)) ] + O(h3 ). The last equation is re-written as y(x + h) = y(x) + h 2 f(x, y(x)) + h 2 [ f(x, y(x)) + h ∂f ∂x (x, y(x)) + hf(x, y(x)) ∂f ∂y (x, y(x)) ] + O(h3 ) (7.24) Taking x = xj for j = 0, 1, · · · , n − 1 with xj+1 = xj + h in (7.24), we get y(xj+1) = y(xj) + h 2 f(xj, y(xj)) + h 2 [ f(xj, y(xj)) + h ∂f ∂x (xj, y(xj)) + hf(xj, y(xj)) ∂f ∂y (xj, y(xj)) ] + O(h3 ). (7.25) Let us now expand the function f = f(s, t), which is a function of two variables, into its Taylor series at the point (ξ, τ) and truncate the series after the linear term. It is given by f(s, t) = f(ξ, τ) + (s − ξ) ∂f ∂s (ξ, τ) + (t − τ) ∂f ∂t (ξ, τ) + O ( (s − ξ)2 ) + O ( (t − τ)2 ) . Taking (ξ, τ) = (xj, y(xj)) and comparing the above equation with the term in the square brackets in the equation (7.25), we get y(xj+1) = y(xj) + h 2 f(xj, y(xj)) + h 2 [f(xj+1, y(xj) + hf(xj, y(xj)))] + O(h3 ). (7.26) S. Baskar and S. Sivaji Ganesh 244 Spring 2017-18
- 257. Section 7.5 Runge-Kutta Methods Truncating the higher order terms and denoting the approximate value of y(xj+1) as yj+1, we get yj+1 = yj + h 2 f(xj, yj) + h 2 [f(xj+1, yj + hf(xj, yj))] . (7.27) Although the terms dropped from (7.26) to get (7.27) are of order 3 (namely, O(h3 )), the resultant approximation to y′ is of order 2 as is evident from the Taylor’s formula (7.22). The equation (7.27) is therefore known as Runge-Kutta method of order 2. To facilitate easy memorizing, the formula (7.27) may be written as yj+1 = yj + 1 2 (k1 + k2), where k1 = h f(xj, yj) k2 = h f(xj+1, yj + k1). The truncation error of Runge-Kutta method of order 2 is of order O(h3 ) whereas the Euler’s method is of order O(h2 ). Therefore, for a fixed h > 0 we expect to get more accurate result from Runge-Kutta method order 2 when compared to Euler’s method. Example 7.5.1. Consider the initial-value problem y′ = y, y(0) = 1. Using Runge-Kutta method of order 2, we obtain x y k1 k2 0.000000 1.000000 0.010000 0.010100 0.010000 1.010050 0.010000 0.010100 0.020000 1.020201 0.010100 0.010202 0.030000 1.030454 0.010202 0.010304 0.040000 1.040810 0.010305 0.010408 Recall the exact solution is y(x) = ex and y(0.04) ≈ 1.040811. Therefore, the error involved is 0.000001 which is much less than the error (0.000199) obtained in Euler’s method for h = 0.01. S. Baskar and S. Sivaji Ganesh 245 Spring 2017-18
- 258. Chapter 7. Numerical Ordinary Differential Equations 7.5.2 Order Four We state without derivation, the formula for the Runge-Kutta method of order 4. yj+1 = yj + 1 6 (k1 + 2k2 + 2k3 + k4) where k1 = hf(xj, yj), k2 = hf ( xj + h 2 , yj + k1 2 ) , k3 = hf ( xj + h 2 , yj + k2 2 ) , k4 = hf(xj + h, yj + k3) The local truncation error of the 4th order Runge-Kutta Method is of O(h5 ). Example 7.5.2. Consider the initial-value problem y′ = y, y(0) = 1. Using Runge-Kutta method of order 4, we obtain xj yj Exact Solution Error Relative Error 0.00 1.000000 1.000000 0.000000 0.000000 0.01 1.010050 1.010050 0.000000 0.000000 0.02 1.020201 1.020201 0.000000 0.000000 0.03 1.030455 1.030455 0.000000 0.000000 0.04 1.040811 1.040811 0.000000 0.000000 Note that the exact solution is y(x) = ex . 7.6 Exercises 1. Let h > 0 and let xj = x0 + jh (j = 1, 2, · · · , n) be given nodes. Consider the initial value problem y′ (x) = f(x, y), y(x0) = y0, with ∂f(x, y) ∂y ≤ 0, for all x ∈ [x0, xn] and for all y. S. Baskar and S. Sivaji Ganesh 246 Spring 2017-18
- 259. Section 7.6 Exercises i) Using error analysis of the Euler’s method, show that there exists an h > 0 such that |en| ≤ |en−1| + h2 2 f′′ (ξ) for some ξ ∈ (xn−1, xn), where en = y(xn) − yn with yn obtained using Euler method. ii) Applying the conclusion of (i) above recursively, prove that |en| ≤ |e0| + n h2 Y where Y = 1 2 max x0≤x≤xn |y′′ (x)|. (∗∗) 2. The solution of the initial value problem y′ (x) = λy(x) + cos x − λ sin x, y(0) = 0 is y(x) = sin x. For λ = −20, find the approximate value of y(3) using the Euler’s method with h = 0.5. Compute the error bound given in (∗∗), and Show that the actual absolute error exceeds the computed error bound given in (∗∗). Explain why it does not contradict the validity of (∗∗). 3. Derive the backward Euler’s method for finding the approximate value of y(xn) for some xn < 0, where y satisfies the initial value problem y′ (x) = f(x, y), y(0) = y0. 4. Consider the initial value problem y′ = −2y, 0 ≤ x ≤ 1, y(0) = 1. i) Find an upper bound on the error in approximating the value of y(1) com- puted using the Euler’s method (at x = 1) in terms of the step size h. ii) For each h, solve the difference equation which results from the Euler’s method, and obtain an approximate value of y(1). iii) Find the error involved in the approximate value of y(1) obtained in (ii) above by comparing with the exact solution. iv) Compare the error bound obtained in (i) with the actual error obtained in (iii) for h = 0.1, and for h = 0.01. v) If we want the absolute value of the error obtained in (iii) to be at most 0.5 × 10−6 , then how small the step size h should be? 5. Consider the initial value problem y′ = xy, y(0) = 1. Estimate the error involved in the approximate value of y(1) computed using the Euler’s method (with infinite precision) with step size h = 0.01. 6. Find an upper bound for the propagated error in Euler method (with infinite precision) with h = 0.1 for solving the initial value problem y′ = y, y(0) = 1, in the interval i) [0, 1] and S. Baskar and S. Sivaji Ganesh 247 Spring 2017-18
- 260. Chapter 7. Numerical Ordinary Differential Equations ii) [0, 5]. 7. For each n ∈ N, write down the Euler’s method for the solution of the initial value problem y′ = y, y(0) = 1 on the interval [0, 1] with step size h = 1/n. Let the resulting approximation to y(1) be denoted by αn. Show using limiting argument (without using the error bound) that αn → y(1) as n → ∞. 8. In each of the following initial value problems, use the Euler’s method, Runge- Kutta method of order 2 and 4 to find the solution at the specified point with specified step size h: i) y′ = x + y; y(0) = 1. Find y(0.2) (For the Euler’s method take h = 0.1 and for other methods, take h = 0.2). The exact solution is y(x) = −1−x+2ex . ii) y′ = 2 cos x − y, y(0) = 1. Find y(0.6) (For the Euler’s method take h = 0.1 and for other methods, take h = 0.2) The exact solution is y(x) = sin x + cos x. 9. Use the Euler’s, Runge-Kutta methods of order 2 and 4 to solve the IVP y′ = 0.5(x − y) for all x ∈ [0, 3] with initial condition y(0) = 1. Compare the solutions for h = 1, 0.5, 0.25, 0.125 along with the exact solution y(x) = 3e−x/2 + x − 2. 10. Show that the Euler’s and Runge-Kutta methods fail to determine an approxima- tion to the non-trivial solution of the initial value problem y′ = yα , 0 < α < 1, y(0) = 0. Note that the ODE is in separable form and hence a non-trivial solution to initial value problem can be found analytically. 11. Write the formula using the Euler’s method for approximating the solution to the initial value problem y′ = x2 + y2 , y(x0) = y0 at the point x = x1 with h = x0 − x1 > 0. Find the approximate value y1 of the solution to this initial value problem at the point x1 when x0 = 0, y0 = 1, and h = 0.25. Find a bound for the truncation error in obtaining y1. 12. Using the Gaussian rule determine a formula for solving the initial value problem y′ = e−x2 , y(x0) = y0. in the form yj+1 = yj−1 + h ( e −(xj− h√ 3 )2 + e −(xj+ h√ 3 )2 ) when the nodes are equally spaced with spacing h = xj+1 − xj, j ∈ Z. (h > 0). Let x0 = 0, and y0 = 1. Using the method derived above, obtain approximate values of y(−0.1) and y(0.1). S. Baskar and S. Sivaji Ganesh 248 Spring 2017-18
- 261. Index l1-norm, 74 l2-norm, 73 l∞-norm, 74 2-step method, 241 2-step methods, 241 quadrature rule, 197 absolute error, 33 arithmetic error, 25 in interpolating polynomial, 176, 180 backward difference, 214 backward Euler’s method, 235 backward substitution, 55, 59, 64 base, 26, 27 big oh, 18, 19 binary representation, 26 bisection method, 121 algorithm, 121 Bolzano-Weierstrass theorem, 3 bounded sequence, 3 bracketing methods, 120 central difference, 215 Chebyshev nodes, 185 Cholesky’s factorization, 70 chopping a number, 30 closed domain method, 120 composite Simpson’s rule, 206 trapezoidal rule, 203 condition number of a function, 42 of a matrix, 79 continuous function, 6 contraction map, 143 convergent sequence, 2 Crout’s factorization, 69 cubic spline, 189 data, 161 decimal representation, 26 decreasing sequence, 3 degree of precision, 208 derivative of a function, 7 diagonally dominant matrix, 86 difference backward, 214 central, 215 forward, 213 differentiable function, 7 direct method, 51, 53 direct methods, 82 Divided difference higher-order formula, 170 symmetry, 170 divided difference, 169 dominant eigenvalue, 96 Doolittle’s factorization, 66 double precision, 31 eigenvalue dominant, 96 power method, 100 Error in Euler’s method, 237 S. Baskar and S. Sivaji Ganesh 249 Spring 2017-18
- 262. Index in iterative procedure, 86 in Simpson’s rule, 205 in trapezoidal rule, 201 error, 33 absolute, 33 arithmetic, 25 floating-point, 40 in interpolating polynomial, 176 arithmetic, 176, 180 mathematical, 176, 177 total, 176 in rectangle rule, 199 mathematical, 25 percentage, 33 propagated, 40 propagation of, 38 relative, 33 relative total, 40 residual, 91 total, 25, 40 truncation, 14, 34 error estimate, 122 Euclidean norm, 73 Euler’s method, 235 backward, 235 forward, 235 geometric interpretation, 236 mid-point, 241 modified, 241 trapezoidal, 242 exact arithmetic, 31 explicit methods, 242 exponent, 26 Faber’s theorem, 184 first mean value theorem for integrals, 10 fixed point, 22, 140 iteration method, 141 fixed-point iteration method, 141 floating-point n-digit number, 27 approximation, 30 error, 40 representation, 26 forward difference, 213 elimination, 54, 59 substitution, 64 forward Euler’s method, 235 Gauss-Seidel method, 88 Gaussian elimination method modified, 57 naive, 53 operations count, 60 Gaussian rules, 209 Gerschgorin’s circle theorem, 107 disk, 109 Gerschgorin’s disks, 108 Hermite-Gennochi formula, 173 Hilbert matrix, 80 ill conditioned matrix, 80 ill-conditioned function evaluation, 43 implicit methods, 242 increasing sequence, 3 infinite precision, 31 infinity norm, 178 initial condition, 229 value problem, 229 intermediate value theorem, 7 interpolating function, 159 interpolating polynomials convergence, 184 interpolation, 159 S. Baskar and S. Sivaji Ganesh 250 Spring 2017-18
- 263. Section Index condition, 160 error, 176 arithmetic, 176, 180 mathematical, 176, 177 total, 176 linear polynomial, 166 piecewise polynomial, 187 polynomial, 160, 162 existence and uniqueness, 161 Lagrange form, 165 Newton’s form, 168 quadratic polynomial, 166 spline, 188 iteration backward, 41 forward, 41 iteration function, 141 iterative methods, 119 iterative method, 51 fixed-point, 141 Gauss-Seidel, 88 Jacobi, 83 refinement, 93 residual corrector, 93 iterative methods, 82, 119 Jacobi method, 83 Lagrange form of interpolating polynomial, 165 polynomial, 164 limit of a function, 4 of a sequence, 2 left-hand, 4 of a function, 6 right-hand, 4 linear convergence, 20 linear system direct method, 51, 53 Gaussian elimination method, 53, 57 iterative method, 51 LU factorization, 65 Thomas method, 61 little oh, 18, 19 loss of significance, 37 LU factorization/decomposition, 65 Cholesky’s, 70 Crout’s, 69 Doolittle’s, 66 machine epsilon, 33 mantissa, 26 Mathematical error in forward difference formula, 213 mathematical error, 25, 238 in central difference formula, 215 in difference formulas, 217 in Euler’s method, 238 in interpolating polynomial, 176, 177 rectangle rule, 199 Simpson’s rule, 204 trapezoidal rule, 201 matrix Hilbert, 80 ill conditioned, 80 well conditioned, 80 matrix norm, 74 maximum-of-column-sums, 77 maximum-of-row-sums, 76 spectral, 77 subordinate, 75 maximum norm, 74 maximum-of-column-sums norm, 77 maximum-of-row-sums norm, 76 mean value theorem, 42 derivative, 9, 13 integrals, 10, 11 S. Baskar and S. Sivaji Ganesh 251 Spring 2017-18
- 264. Index mid-point rule, 198, 241 modified Gaussian elimination method, 57 monotonic sequence, 3 naive Gaussian elimination method, 53 natural spline, 190 Newton’s divided difference, 169 form of interpolating polynomial, 168 Newton-Cotes formula, 198 Newton-Raphson method, 135 node, 234 nodes, 160 Chebyshev, 185 nonlinear equation, 119 bisection method, 121 fixed-point method, 141 Newton-Raphson method, 135 open domain methods, 132 regula-falsi method, 126 residual error, 132 secant method, 133 norm infinite, 178 matrix, 74 maximum-of-column-sums, 77 maximum-of-row-sums, 76 spectral, 77 subordinate, 75 vector, 73 l1, 74 Euclidean (l2), 73 maximum (l∞), 74 numerical methods, 25 solution, 25 numerical integration, 197 Gaussian rule, 209 mid-point, 198 Newton-Cotes, 198 rectangle, 198 Simpson’s rule, 204, 208 trapezoidal, 200 oh big and little, 18, 19 open domain methods, 120, 132 optimum bound, 110 optimum bounds, 110 order of accuracy, 214 of convergence, 20 order of exactness, 208 ordinary differential equation, 229 overflow, 28 percentage error, 33 piecewise polynomial interpolation, 187 polynomial interpolation, 160, 162 existence and uniqueness, 161 Lagrange form, 165 Newton’s form, 168 piecewise linear, 187 positive definite matrix, 70 power method, 97, 100 precision, 30 degree of, 208 double, 31 infinite, 31 principal minors, 66 leading, 66 principal sub-matrix, 66 propagated error, 40, 238 in Euler’s method, 238 propagation of error, 38 quadratic convergence, 20 quadrature Gaussian, 209 S. Baskar and S. Sivaji Ganesh 252 Spring 2017-18
- 265. Section Index mid-point, 198 rectangle, 198 Simpson’s, 204, 208 trapezoidal, 200 quadrature points, 198 quadrature rule mid-point, 241 Newton-Cotes, 198 quadrature weights, 198 radix, 26 rate of convergence, 20 rectangle rule, 198 regula-falsi method, 126 algorithm, 127 relative error, 33 relative total error, 40 remainder estimate, 14 remainder term in Taylor’s formula, 12 residual error, 91 vector, 91 residual corrector method, 93 residual error, 132 Rolle’s theorem, 9 rounding a number, 30 Runge function, 183 phenomenon, 183 Runge-Kutta method order 2, 245 order 4, 246 Runge-Kutta methods, 243 sandwich theorem, 2, 5 secant method, 133, 134 second mean value theorem for integrals, 11 self map, 142 sequence, 2 bounded, 3 convergent, 2 decreasing (strictly), 3 increasing (strictly), 3 limit, 2 monotonic, 3 sign, 26 significant digits, 35 loss of, 37 number of, 36 Simpson’s Rule, 204 Simpson’s rule, 208 composite, 206 spectral norm, 77 spline interpolation, 188 cubic, 189 natural, 190 stability in function evaluation, 45 stable, 45 stable computation, 45 stopping criteria for nonlinear equations, 131 method for linear systems, 94 strictly decreasing sequence, 3 strictly increasing sequence, 3 subordinate norm, 75 superlinear convergence, 20 Taylor’s formula, 13 polynomial, 11 series, 15 theorem, 12, 34 Thomas method, 61 total error, 25, 40 in interpolating polynomial, 176 in polynomial interpolation, 182 S. Baskar and S. Sivaji Ganesh 253 Spring 2017-18
- 266. Index trapezoidal Rule, 200 trapezoidal rule composite, 203 triangle inequality, 73, 75 truncation error, 14, 34, 238 in Euler’s method, 238 underflow, 28 undetermined coefficients differentiation, 219 integration, 207 unstable computation, 45 vector norm, 73 l1, 74 Euclidean (l2), 73 maximum (l∞), 74 weights, 198 well conditioned matrix, 80 well-conditioned function evaluation, 43 Wilkinson’s example, 95 S. Baskar and S. Sivaji Ganesh 254 Spring 2017-18