![List of Figures
2.1 Discontinuous functions having a minimizing point in [0,1] . . . . . . . . . 12
2.2 Example of an lsc function . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3 Lsc function that is not usc at 0 . . . . . . . . . . . . . . . . . . . . . . . . 23
2.4 Convex function and concave function . . . . . . . . . . . . . . . . . . . . 33
2.5 Example of a quasiconvex function that is not convex . . . . . . . . . . . . 40
2.6 Examples of convex functions: f (not lsc), cl f , and g (lsc) . . . . . . . . 46
2.7 The function f (x,y) = x2/y for y ≥ ε 0 and some small ε . . . . . . . . 47
2.8 Cases y ∈ dom f (left) and y ∈ dom f (right) . . . . . . . . . . . . . . . . 48
3.1 Example of right and left differentiability . . . . . . . . . . . . . . . . . . 71
3.2 Region determined by the functions α and β . . . . . . . . . . . . . . . . . 74
3.3 The function f (x) = |x| in a neighborhood of x = 0 for n = 1 . . . . . . . 81
3.4 Example 3.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.5 Examples 3.6 and 3.8 in logarithmic scale . . . . . . . . . . . . . . . . . . 85
3.6 Example 3.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
3.7 Upper envelope of two functions . . . . . . . . . . . . . . . . . . . . . . . 97
3.8 Upper envelope of three functions . . . . . . . . . . . . . . . . . . . . . . 97
3.9 The two convex functions g1 and g2 on [0,1] of Example 4.1 . . . . . . . . 122
3.10 Function of Exercise 7.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
4.1 Banana-shaped level sets of the Rosenbrock function at 0.25, 1, 4, 9, 16 . . 159
4.2 Basis functions (φ0,...,φi,...,φn) of P 1
n (0,1) . . . . . . . . . . . . . . . . 161
4.3 A nonconvex closed cone in 0 . . . . . . . . . . . . . . . . . . . . . . . . 169
4.4 Closed convex cone in 0 generated by the closed convex set V . . . . . . . 169
4.5 Closed convex cone in 0 generated in R3
by a nonconvex set V . . . . . . . 169
4.6 Convex set U tangent to the level set of f through x ∈ U . . . . . . . . . . 171
4.7 Tangency of the affine subspace A or the linear subspace S
to a level set of f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
4.8 Tangency of U to a level set of the function f at x ∈ U . . . . . . . . . . . 177
4.9 Half-tangent dh(0;+1) to the path h(t) in U at the point h(0) = x. . . . . . 180
4.10 Cusp at x ∈ ∂U: U and TU (x) . . . . . . . . . . . . . . . . . . . . . . . . 182
4.11 First example: U and TU (x) . . . . . . . . . . . . . . . . . . . . . . . . . 183
4.12 Second example: U and TU (x) . . . . . . . . . . . . . . . . . . . . . . . . 183
4.13 Third example: U and TU (x) . . . . . . . . . . . . . . . . . . . . . . . . . 183
4.14 Fourth example: U and TU (x) . . . . . . . . . . . . . . . . . . . . . . . . 184
xi](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-14-320.jpg)
![Preface
A Great and Beautiful Subject
Optimization refers to finding, characterizing, and computing the minima and/or maxima of
a function with respect to a set of admissible points.
Its early steps were intertwined with the ones of the differential calculus and the
mathematical analysis. The first idea of the differential calculus and the rule for the com-
putation of the minima and maxima could be attributed to Fermat in 1638. The concept of
derivative was introduced in that context by Leibniz and Newton almost fifty years later.
So, the condition obtained by Fermat for the extremum of an algebraic function was de
facto generalized in the form f (x) = 0. With the introduction of the notion of differenti-
able function of several variables and of differentiable functions defined on Hilbert and
topological vector spaces, the rule of Fermat remains valid. One of the important areas of
optimization is the calculus of variations, which deals with the minimization/maximization
of functionals, that is, functions of functions. It was also intertwined with the development
of classical analysis and functional analysis.
But, optimization is not just mathematical analysis. Many decision-making prob-
lems in operations research, engineering, management, economics, computer sciences, and
statistics are formulated as mathematical programs requiring the maximization or mini-
mization of an objective function subject to constraints. Such programs1 often have special
structures: linear, quadratic, convex, nonlinear, semidefinite, dynamic, integer, stochastic
programming, etc. This was the source of more theory and efficient algorithms to compute
solutions. With the easier access to increasingly more powerful computers, larger and more
complex problems were tackled thus creating a demand for efficient computer software to
solve large-scale systems.
To give a few landmarks, the modern form of the multipliers rule goes back to La-
grange2 in his path-breaking Mécanique analytique in 1788 and the steepest descent method
to Gauss.3 The simplex algorithm to solve linear programming4 problems was created by
1The term programming in this context does not refer to computer programming. Rather, the term comes from
the use of program by the United States military to refer to proposed training and logistics schedules, which were
the problems that Dantzig was studying at the time.
2Joseph Louis, comte de Lagrange (in Italian Giuseppe Lodovico Lagrangia) (1736–1813).
3Johann Carl Friedrich Gauss (1777–1855).
4Much of the theory had been introduced by Leonid Vitaliyevich Kantorovich (1912–1986) in 1939 (L .V. Kan-
torovich [1, 2]).
xiii](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-16-320.jpg)
![xiv Preface
George Dantzig5 and the theory of the duality was developed by John von Neumann6 both
in 1947. The necessary conditions for inequality-constrained problems were first published
in the Masters thesis of William Karush in 1939, although they became renowned after a
seminal conference paper by Harold W. Kuhn and Albert W. Tucker in 1951.
Intended Audience and Objectives of the Book
This book is intended as a textbook for a one-term course at the undergraduate level for
students in Mathematics, Physics, Engineering, Economics, and other disciplines with a
basic knowledge of mathematical analysis and linear algebra. It is intentionally limited to
the optimization with respect to variables belonging to finite-dimensional spaces. This is
what we call finite-dimensional optimization. It provides a lighter exposition deferring at
the graduate level technical questions of Functional Analysis associated with the Calculus
of Variations. The useful background material has been added at the end of the first chapter
to make the book self-sufficient. The book can also be used for a first year graduate course
or as a companion to other textbooks.
Being limited to one term, choices had to be made. The classical themes of op-
timization are covered emphasizing the semidifferential calculus while staying at a level
accessible to an undergraduate student. In the making of the book, some material has
been added to the original lecture notes. For a one-term basic program the sections and
subsections beginning with the black triangle can be skipped. The book is structured
in such a way that the basic program only requires very basic notions of analysis and the
Hadamard semidifferential that is easily accessible to nonmathematicians as an extension of
their elementary one-dimensional differential calculus. The added material makes the book
more interesting and provides connections with convex analysis and, to a lesser degree,
subdifferentials. Yet, the book does not pretend or aim at covering everything. The added
material is not mathematically more difficult since it only involves more liminf and limsup
in the definitions of lower and upper semidifferentials, but it might be too much for a basic
undergraduate course.
For a first initiation to nondifferentiable optimization, semidifferentials have been
preferred over subdifferentials7 that necessitate a good command of set-valued analysis.
The emphasis will be on Hadamard semidifferentiable8 functions for which the result-
ing semidifferential calculus retains all the nice features of the classical differential cal-
culus, including the good old chain rule. Convex continuous and semiconvex functions
are Hadamard semidifferentiable and an explicit expression of the semidifferential of an
extremum with respect to parameters can be obtained. So, it works well for most non-
differentiable optimization problems including semiconvex or semiconcave problems. The
Hadamard semidifferential calculus readily extends to functions defined on differential man-
ifolds and on groups that naturally occur in optimization problems with respect to the shape
or the geometry.9
5George Bernard Dantzig (1914–2005) (G. B. Dantzig [1, 3]).
6John von Neumann (1903–1957).
7For a treatment of finite-dimensional optimization based on subdifferentials and the generalized gradient, the
reader is referred to the original work of R. T. Rockafellar [1] and F. H. Clarke [2] and to the more recent book
of J. M. Borwein and A. S. Lewis [1].
8The differential in the sense of Hadamard goes back to the beginning of the 20th century. We shall go back to
the original papers of J. Hadamard [2] in 1923 and of M. Fréchet [3] in 1937.
9The reader is referred to the book of M. C. Delfour and J.-P. Zolésio [1].](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-17-320.jpg)
![Preface xv
The book is written in the mathematical style of definitions, theorems, and detailed
proofs. It is not necessary to go through all the proofs, but it was felt important to have
all the proofs in the book. Numerous examples and exercises are incorporated in each
chapter to illustrate and better understand the subject material. In addition, the answer to
all the exercises is provided in Appendix B. This considerably expands the set of examples
and enriches the theoretical content of the book. More exercises along with examples of
applications in various fields can be found in other books such as the ones of S. Boyd and
L. Vandenberghe [1] and F. Bonnans [1].
The purpose of the historical commentaries and landmarks is mainly to put the subject
in perspective and to situate it in time.
Numbering and Referencing Systems
The numbering of equations, theorems, lemmas, corollaries, definitions, examples, and
remarks is by chapter. When a reference to another chapter is necessary it is always followed
by the words in Chapter and the number of the chapter. For instance, “equation (7.5) from
Theorem 7.4(i) of Chapter 2” or “Theorem 5.2 from section 5 in Chapter 3.” The text of
theorems, lemmas, and corollaries is slanted; the text of definitions, examples, and remarks
is normal shape and ended by a square . This makes it possible to aesthetically emphasize
certain words especially in definitions. The bibliography is by author in alphabetical order.
For each author or group of coauthors there is a numbering in square brackets starting
with [1]. A reference to an item by a single author is of the form J. Hadamard [2] and a
reference to an item with several coauthors is of the form H. W. Kuhn andA. W. Tucker [1].
Boxed formulae or statements are used in some chapters for two distinct purposes. First,
they emphasize certain important definitions, results, or identities; second, in long proofs
of some theorems, lemmas, or corollaries they isolate key intermediary results which will
be necessary to more easily follow the subsequent steps of the proof.
Acknowledgments
This book is based on an undergraduate course created in 1975 at the University of Montreal
by Andrzej Manitius (George Mason University) who wrote a first set of lecture notes. The
course and its content were reorganized and modified in 1984 by the author and this book
is the product of their evolution.
The author is grateful to the referees and to Thomas Liebling, Editor-in-Chief of
the MOS-SIAM Series on Optimization, for their very constructive suggestions, without
forgetting the students whose contribution has been invaluable.
Michel Delfour
Montreal, May 16, 2011](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-18-320.jpg)
![2 Chapter 1. Introduction
2 Calculus of Variations and Its Offsprings
The sources of this section are B. van Brunt,1 J. Ferguson,2 and Wikipedia: Calculus of
Variations and Optimization (mathematics).
The first idea of the differential calculus and the rule for the computation of the
minima and maxima3 seems to go back to Fermat4 in 1638. It is generally accepted that
the concept of derivative is due to Leibniz5 who published in 16846 and Newton7 who
published over a longer period of time.8 So, the condition obtained by Fermat for the
extremum of an algebraic function was de facto generalized in the form f (x) = 0. With
the introduction of the concept of differentiable function of several variables of Jacobi and
of differentiable functions defined on Hilbert and topological vector spaces, the rule of
Fermat and Leibniz remains valid. During three centuries, it was applied, justified, adapted,
and generalized in the context of the theory of optimization, of the calculus of variations,
and of the theory of optimal control.
The calculus of variations deals with the minimization/maximization of functionals,
that is, functions of functions. The simplest example of this type of problems is to find the
curve of minimum length between two points. In the absence of constraints, the solution
is a straight line between the points. However, when the curve is constrained to stay on a
surface in the space, the solution is less obvious and not necessarily unique. Such solutions
are called geodesics. A problem of that nature is illustrated by Fermat’s principle in optics:
the light follows the shortest path of optical length between two points, where the optical
length depends on the material of the physical medium. A notion of the same nature in
mechanics is the least action principle. Several important problems involve functions of
several variables. For instance, solutions of boundary value problems for the Laplace
equation satisfy the Dirichlet principle.
According to some historians, the calculus of variations begins with the brachis-
tochrone problem of Johann Bernoulli in 1696. It immediately attracted the attention of
Jakob Bernoulli and of the Marquis de l’Hôpital, but it is Euler who was the first to de-
velop this subject. His contributions began in 1733 and it is his Elementa Calculi Varia-
tionum that gave its name to this discipline. In 1744, Euler published his landmark book
Methodus inveniendi lineas curvas maximi minimive proprietate gaudentes, sive solutio
problematis isoperimetrici latissimo sensu accepti.9 In two papers read to the Académie
des Sciences in 1744, and to the Prussian Academy in 1746, Maupertuis proclaimed the
principle of least action.
1Bruce van Brunt, The Calculus of Variations, Springer-Verlag, New York, 2004.
2James Ferguson, A. Brief Survey of the History of the Calculus of Variations and its Applications, University
of Victoria, Canada, 2004 (arXiv:math/0402357).
3Methodus ad disquirendam Maximam et Minimam, 1638 (see P. de Fermat [1]).
4Pierre de Fermat (1601–1665).
5Gottfried Wilhelm Leibniz (1646–1716).
6Nova methodus pro maximis et minimis, itemque tangentibus, quae nec fractas nec irrationales quantitates
moratur, et singulare pro illis calculi genus (A new method for maxima and minima and their tangents, that are
not limited to fractional or irrational expressions, and a remarkable type of calculus for these), in Acta Eruditorum,
1684, a journal created in Leipzig two years earlier.
7Sir Isaac Newton (1643–1728).
8The Method of Fluxions completed in 1671 and published in 1736 and Philosophiæ Naturalis Principia
Mathematica (Mathematical Principles of Natural Philosophy), often called the Principia (Principles), 1687 and
1726 (third edition).
9A method for discovering curved lines that enjoy a maximum or minimum property, or the solution of the
isoperimetric problem taken in the widest sense.](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-20-320.jpg)
![3. Contents of the Book 3
Lagrange extensively contributed to the theory and Legendre10 in 1786 laid the foun-
dations of the characterization of maxima and minima. In his path-breaking Mécanique
analytique11 in 1788, Lagrange summarized all the work done in the field of classical me-
chanics since Newton.12 It is in that book that he clearly expressed the multipliers rule in
its modern form.
In the 18th century several mathematicians have contributed to this enterprise, but
perhaps the most important work of the century is that of Weierstrass starting in 1870. He
was the first to give a completely correct proof of a sufficient condition for the existence of
a minimum. His celebrated course on the theory is epoch-making, and it may be asserted
that he was the first to place it on a firm and unquestionable foundation. The 20th and the
23rd Hilbert problems published in 1900 enticed further development. In the 20th century,
Noether, Tonelli, Lebesgue, and Hadamard, among others, made significant contributions.
Morse applied the calculus of variations in what is now called Morse theory. Pontryagin,
Rockafellar, and Clarke developed new mathematical tools for the optimal control theory,
a generalization of the calculus of variations.
In problems of geometrical design and shape optimization, the modelling, optimiza-
tion, or control variable is no longer a vector of scalars or functions but the shape of a
geometrical object.13 In this category, we find for instance Plateau’s problem. This prob-
lem was raised by Lagrange in 1760, but it is named after Joseph Plateau who was interested
in soap films around 1873.14 It consists in finding the surface of minimum area with a given
boundary in space. Such surfaces are easy to find experimentally, but their mathematical
interpretation is extremely complex. In this family of problems we also encounter sev-
eral identification problems such as the reconstruction and the processing of images from
two dimensional or three dimensional scanned material or biological (biometrics) objects.
Another area is the enhancing of images such as in the Very Large Telescope project. To
deal with nonparametrized geometrical objects, nonlinear and nonconvex spaces and a shape
differential calculus are required.
3 Contents of the Book
Chapter 2 is devoted to the existence of minimizers of a real-valued function. Natural
notions of lower and upper semicontinuities and several notions of convexity are introduced
to relax the conditions of the classical Weierstrass theorem specialized to the case of the
minimum. Some elements of convex analysis, such as the notion of Fenchel–Legendre
transform, the primal and dual problems, and the Fenchel duality theorem are also included
along with Ekeland’s variational principle and some of its consequences.
Chapter 3 first provides a review of the differentiability of functions of one and several
variables. Notions of semidifferentials and, specifically, of Hadamard semidifferentials are
introduced. The connection is made with the classical Gateaux15 and Fréchet differentials.
A semidifferential calculus that extends the classical differential calculus is developed. In
this framework, Hadamard semidifferentials of lower and upper envelopes of finitely many
10Sur la manière de distinguer les maxima des minima dans le calcul des variations (On the method of distin-
guishing maxima from minima in the calculus of variations).
11J. L. Lagrange [2].
12Philosophiæ naturalis principia mathematica.
13Cf., for instance, M. C. Delfour and J.-P. Zolésio [1].
14Joseph Antoine Ferdinand Plateau (1801–1883). Cf. J. A. Plateau [1].
15Without circumflex accent (see footnote 19 on page 80).](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-21-320.jpg)
![4 Chapter 1. Introduction
Hadamard semidifferentiable functions exist and the chain rule for the composition of
Hadamard semidifferentiable functions remains valid. Moreover, the convex continuous
functions and, more generally, the semiconvex functions are Hadamard semidifferentiable.
Upper and lower notions of semidifferentials are also included in connection with semi-
convex functions. Finally, we give a fairly general theorem on the semidifferentiability
of extrema with respect to a parameter. It is applied to get the explicit expression of
the Hadamard subdifferential of the extremum of quadratic functions. The differentials and
semidifferentials introduced in this chapter are summarized and compared in the last section.
Chapter 4 focuses on optimality conditions to characterize an unconstrained or a
constrained extremum via the semidifferential or differential of the objective function. It first
considers twice differentiable functions without constraints along with several examples.
A special attention is given to the generic example of the least and greatest eigenvalues of a
symmetric matrix. The explicit expression of their Hadamard semidifferentials is provided
with the help of the general theorems of Chapter 3. This is followed by the necessary and
sufficient optimality condition for convex differentiable objective functions and convex sets
of admissible points. It is specialized to linear subspaces, affine subspaces, and convex cones
at the origin. Finally, a necessary and sufficient condition for an arbitrary convex objective
function and an arbitrary set of constraints is given in terms of the lower semidifferential.
Thesecondpartofthechaptergivesageneralnecessaryoptimalityconditionforalocal
minimum using the upper Hadamard semidifferentiability of the objective function and the
Bouligand’s tangent cone of admissible directions. It is quite remarkable that such simple
notions be sufficient to cover most of the so-called nondifferentiable optimization. This
condition is dualized by introducing the notion of dual cone. The dual necessary optimality
condition is then applied to the linear programming problem where the constraints are
specified by a finite number of equalities and inequalities on affine functions. At this
juncture, the Lagrangian is introduced along with its connections to two-person zero-sum
games and to Fenchel’s primal and dual problems of Chapter 2. A general form of Farkas’
lemma is given in preparation of the next chapter. The constructions and results are extended
to the quadratic programming problem and to Fréchet differentiable objective functions. At
the end of this chapter, a glimpse is given into optimization via subdifferentials that involves
set-valued functions.
Chapter 5 is devoted to differentiable optimization where the set of admissible points
is specified by a finite number of differentiable constraint functions. By using the dual
necessary optimality condition, we recover the Lagrange multiplier theorem for equality
constraints, the Karush–Kuhn–Tucker theorem for inequality constraints, and the general
theorem for the mixed case of equalities and inequalities.
4 Some Background Material in Classical Analysis
This section puts together a compact summary of some basic elements of classical analysis
that will be needed in the other chapters. They come from several sources (for instance,
among others, W. H. Fleming [1], W. Rudin [1], or L. Schwartz [1]). The differential
calculus will be completely covered from scratch in Chapter 3 and does not require any
prerequisite. The various notions of convexity that will be needed will be introduced in
each chapter, but the reader is also referred to specialized books such as, for instance,
F.A.Valentine [1], R.T. Rockafellar [1], L. D. Berkovitz [1], S. R. Lay [1], H.Tuy [1],
S. Boyd and L. Vandenberghe [1].](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-22-320.jpg)
![4. Some Background Material in Classical Analysis 7
A Euclidean space is a vector space E that can be identified with Rn via a linear
bijection for some integer n ≥ 1. For instance, we can identify with Rn the space P n−1[0,1]
of polynomials of order less than or equal to n−1 on the interval [0,1]:
p → (p(0),p
(0),...,p(n−1)
(0)) : P n−1
[0,1] → Rn
(p0,p1,...,pn−1) → p(x)
def
=
n−1
i=0
pi
xi
i !
: Rn
→ Pn−1
[0,1].
4.2.2 Open, Closed, and Compact Sets
The notions open and closed sets in Rn can be defined starting from balls.
Ball at x of radius r 0:
open Br(x) = {y ∈ Rn
: y −x r}
closed Br(x) = {y ∈ Rn
: y −x ≤ r}.
Unit ball at 0:
open B = {y ∈ Rn
: y 1}, closed B = {y ∈ Rn
: y ≤ 1}.
Punched open ball at x:
B
r(x) = {y ∈ Rn
: 0 y −x r}
Definition 4.3.
Let U ⊂ Rn.
(i) a ∈ Rn is an interior point of U, if there exists r 0 such that Br(a) ⊂ U.
(ii) The interior of U is the set of all interior points of U. It will be denoted by intU. By
definition intU ⊂ U.
(iii) V (x) is a neighborhood of x if there exists r 0 such that Br(x) ⊂ V (x).
(iv) A is an open subset of Rn if for all x ∈ A, there exists a neighborhood V (x) of x
such that V (x) ⊂ A.
(v) The family T of all open sets in Rn is called the topology on Rn generated by the
norm.
The topology T of Rn is equal to the family of all finite intersections and arbitrary unions
of open balls in Rn.
Definition 4.4. (i) A sequence {xn} in Rn is convergent if there exists a point x ∈ Rn
such that
∀ε 0, ∃N, ∀n N, xn −x Rn ε.
The point x is unique and called the limit point of {xn}.](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-25-320.jpg)
![12 Chapter 2. Existence, Convexities, and Convexification
Example 2.1.
Let U = R and consider the function
f (x) = 1 if x ≤ 0 and f (x) = x if x 0
for which inf f (U) = 0 and f (x) = 0 for all x ∈ U = R.
The Weierstrass1 theorem that will be proved later as Theorem 5.1 is fundamental in
optimization. It gives sufficient conditions on U and f for the existence of minimizers and
maximizers in U.
Theorem 2.1 (Weierstrass). Given a compact nonempty subset U of Rn and a real-valued
function f : U → R that is continuous at U,2
(i) ∃a ∈ U such that f (a) = supx∈U f (x),
(ii) ∃b ∈ U such that f (b) = infx∈U f (x).
But it is a little too strong since it gives the existence of both minimizing and maxi-
mizing points. Indeed, it is not necessary to simultaneously seek the existence of the two
types of points since a supremum can always be formulated as an infimum of the negative
of the function and vice versa: supf (U) = −inf −f (U). It will be sufficient to find con-
ditions for the existence of a minimizing point of f in U. In so doing, it will be possible to
weaken the continuity assumption that is clearly not necessary for the piecewise continuous
functions of Figure 2.1 that reach a minimum at a point of [0,1]. Notice that at the points of
discontinuity, we have chosen to give the function the lower value and not the upper value
that would not have resulted in the existence of a minimizing point.
0 1 0 1
Figure 2.1. Discontinuous functions having a minimizing point in [0,1].
3 Extrema of Functions with Extended Values
The inf f (U) and the supf (U) have been defined for real-valued functions, that is, f (U) ⊂
R. When the set f (U) is unbounded below, inf f (U) = −∞ and when f (U) is unbounded
above, supf (U) = +∞ (see Definition 4.1 and Remark 4.1 of Chapter 1).
1Karl Theodor Wilhelm Weierstrass (1815–1897).
2We can also work with an f : Rn → R continuous on U since, for U closed in Rn, any f : U → R continuous
on U for the relative topology can be extended to a continuous function on Rn.](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-30-320.jpg)
![3. Extrema of Functions with Extended Values 13
The idea of extended real-valued objective functions implicitly having effective do-
mains is due to R. T. Rockafellar3 and J. J. Moreau.4 In order to consider functions
f : Rn → R = R∪{±∞} that are possibly equal to +∞ or −∞ at some points, the defini-
tions of the inf f (U) and the supf (U) have to be extended.
Definition 3.1.
Let f : Rn → R and U ⊂ Rn.
(i) Associate with f its effective domain
dom f
def
=
x ∈ Rn
: −∞ f (x) +∞
. (3.1)
It will also be simply referred to as the domain of f .
(ii) The infimum of f with respect to U is defined as follows:
inf f (U)
def
=
⎧
⎪
⎪
⎪
⎨
⎪
⎪
⎪
⎩
+∞, if U = ∅,
inf f (U), if U = ∅ and f (U) ⊂ R,
+∞, if U = ∅ and ∀x ∈ U, f (x) = +∞,
−∞, if ∃x ∈ U such that f (x) = −∞.
We shall also use the notation infx∈U f (x).
The supremum of f with respect to U is defined as follows:
supf (U)
def
=
⎧
⎪
⎪
⎪
⎨
⎪
⎪
⎪
⎩
−∞, if U = ∅,
supf (U), if U = ∅ and f (U) ⊂ R,
−∞, if U = ∅ and ∀x ∈ U, f (x) = −∞,
+∞, if ∃x ∈ U such that f (x) = +∞.
We shall also use the notation supx∈U f (x).
Infima and suprema constitute the set of extrema of f in U.
(iii) When there exists a ∈ U such that f (a) = inf f (U), f is said to reach its minimum
at a point of U and it is written as
minf (U) or min
x∈U
f (x).
3Ralph Tyrrell Rockafellar (1935– ). “Moreau and I independently in those days at first, but soon in close
exchanges with each other, made the crucial changes in outlook which, I believe, created convex analysis out of
convexity. For instance, he and I passed from the basic objects in Fenchel’s work, which were pairs consisting
of a convex set and a finite convex function on that set, to extended real-valued functions implicitly having
effective domains, for which we moreover introduced set-valued subgradient mappings.” R. T. Rockafellar,
http://www.convexoptimization.com/wikimization/index.php/Rockafellar.
4Jean Jacques Moreau (1923– ) “. . . appears as a rightful heir to the founders of differential calculus and
mechanics through the depth of his thinking in the field of nonsmooth mechanics and the size of his contribution
to the development of nonsmooth analysis. His interest in mechanics has focused on a wide variety of subjects:
singularities in fluid flows, the initiation of cavitation, plasticity, and the statics and dynamics of granular media.
Allied to this is his investment in mathematics in the fields of convex analysis, calculus of variations and differential
measures” (see P. Alart, O. Maisonneuve, and R. T. Rockafellar [1]).](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-31-320.jpg)
![3. Extrema of Functions with Extended Values 15
Theorem 3.2. Let U, ∅ = U ⊂ Rn and f : Rn → R∪{±∞}. Then
inf f (U) = inf fU (U) = inf fU (Rn
) and argminf (U) = U ∩argminfU (Rn
).
If, in addition, inf(U) +∞, then argminf (U) = argminfU (Rn).
Proof. By definition, f (x) ≤ fU (x) for all x ∈ Rn and
inf f (U) = inf fU (U) ≥ inf fU (Rn
).
If inf fU (Rn) = +∞, we trivially have equality. If m
def
= inf fU (Rn) ∈ R, then for all n ∈ N,
there exists xn ∈ Rn such that
m+
1
n
fU (xn) ≥ m.
Since fU (xn) is finite, xn ∈ U and fU (xn) = f (xn). Thence
inf f (U) ≥ m f (xn)−
1
n
≥ inf f (U)−
1
n
.
By letting n go to infinity, we get equality.
As inf f (U) = inf fU (U) = inf fU (Rn) and U ⊂ Rn, we have argminf (U) =
argminfU (U) ⊂ argminfU (Rn). If, in addition, inf f (U) +∞, then, by definition of
fU , argminfU (Rn) ⊂ argminfU (U) = argminf (U).
Remark 3.1.
For a supremum, extend f by −∞ by considering the function
f U
(x)
def
=
f (x), if x ∈ U
−∞, if x ∈ Rn
U
= −(−f )U (x). (3.6)
For an infimum, two cases are trivial:
(i) there exists x ∈ U such that f (x) = −∞ that yields inf fU (Rn) = inf f (U) = −∞
and x ∈ argminf (U);
(ii) for all x ∈ U, f (x) = +∞ that yields inf fU (Rn) = inf f (U) = +∞ and
argminf (U) = U.
In order to exclude cases (i) and (ii) for the infimum, we introduce the notion of proper
function for an infimum that is the natural extension of the notion of proper function for
convex functions (see R. T. Rockafellar [1]). The dual notion of proper function for a
supremum of f is obtained by considering the notion of proper function for the infimum
of −f .
Definition 3.2.
Let f : Rn → R∪{±∞}.
(i) f is said to be proper for the infimum if](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-33-320.jpg)
![4. Lower and Upper Semicontinuities 17
Functions of Figure 2.1 are lower semicontinuous (lsc) in ]0,1[. The function iden-
tically equal to +∞ is lsc and the one identically equal to −∞ is usc on Rn. As we
have seen before, definition (4.1) of the continuity of a function f : Rn → R at a point
a ∈ Rn is equivalent to the two conditions (4.2): the first one is the lower semicontinuity
at a with h = f (a) − ε f (a) and the second one is the upper semicontinuity at a with
k = f (a)+ε f (a).
As for the infimum and the supremum where supf (U) = −inf −f (U), f is usc at x
if and only if −f is lsc at x. So it is sufficient to study the properties of lsc functions.
Theorem 4.1. (i) f : Rn → R∪{−∞} is usc at x if and only if −f : Rn → R∪{+∞}
is lsc at x.
(ii) f : Rn → R∪{+∞} is lsc at x if and only if −f : Rn → R∪{−∞} is usc at x.
Proof. As f : Rn → R∪{−∞}, then −f : Rn → R∪{+∞}. Given h −f (x), then
f (x) −h. As f is usc at x, there exists a neighborhood V (x) of x such that for all
y ∈ V (x), f (y) −h. As a result, for all y ∈ V (x), −f (y) h. By definition, −f is lsc
at x. The proof of the converse is similar.
It is easy to check the following properties of lsc functions (see Exercises 10.1 to
10.4) by using the convention (+∞)+(+∞) = +∞, (+∞)+a = +∞ for all a ∈ R, and
(+∞)a = (a/ a )∞ for all a ∈ R not equal to 0.
Theorem 4.2. (i) For all f : Rn → R∪{+∞} and g : Rn → R∪{+∞} lsc at a ∈ Rn,
the function
(f +g)(x)
def
= f (x)+g(x), ∀x ∈ Rn
,
is lsc at a.
(ii) For all λ ≥ 0 and f : Rn → R∪{+∞} lsc at a ∈ Rn, the function
(λf )(x)
def
=
λf (x), if λ 0
0, if λ = 0
, ∀x ∈ Rn
,
is lsc at a.
(iii) Given a family {fα}α∈A (where A is an index set possibly infinite) of functions fα :
Rn → R∪{+∞} lsc at a ∈ Rn, the upper envelope
sup
α∈A
fα
(x)
def
= sup
α∈A
fα(x), x ∈ Rn
,
is lsc at a ∈ Rn.
(iv) Given a finite family fi : Rn → R∪{+∞}, 1 ≤ i ≤ m, of functions lsc at a ∈ Rn, the
lower envelope
min
1≤i≤m
fi
(x)
def
= min
1≤i≤m
fi(x), x ∈ Rn
,
is lsc at a ∈ Rn.](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-35-320.jpg)
![18 Chapter 2. Existence, Convexities, and Convexification
(v) Given a function f : Rn → R and a point a ∈ Rn,
f is continuous at a ⇐⇒ f is lsc and usc at a.
(vi) Given a linear map A : Rm
→ Rn and a function f : Rn → R∪{+∞} lsc at Ax, then
f ◦A : Rm
→ R∪{+∞} is lsc at x.
Property (iv) is not necessarily true for the lower envelope of an infinite number of
lsc functions as can be seen from the following example.
Example 4.1.
Define for each integer k ≥ 1, the continuous function
fk(x)
def
=
⎧
⎪
⎨
⎪
⎩
1, if x ∈ [0,1],
1−k(x −1), if x ∈ [1,1+1/k],
0, if x ∈ [1+1/k,2].
It is easy to check that
inf
k≥1
fk(x) =
1, if x ∈ [0,1],
0, if x ∈]1,2],
is usc but not lsc at x = 1.
The lower semicontinuity (resp., upper semicontinuity) can also be characterized in
terms of the liminf (resp., limsup).
Definition 4.2.
Given a function f : Rn → R∪{+∞} (resp., f : Rn → R∪{−∞}),
liminf
x→a
f (x)
def
= sup
ε0
inf
x=a
x−a ε
f (x)
⎛
⎜
⎝resp., limsup
x→a
f (x)
def
= inf
ε0
sup
x=a
x−a ε
f (x)
⎞
⎟
⎠.
Theorem 4.3. f : Rn → R∪{+∞} (resp., f : Rn → R∪{−∞}) is lsc (resp., usc) at a if
and only if
liminf
x→a
f (x) ≥ f (a)
resp., limsup
x→a
f (x) ≤ f (a)
.
Proof. (⇒) If f is lsc at a, for all h f (a), there exists a neighborhood V (a) of a such
that for all x ∈ V (a), f (x) h. As V (a) is a neighborhood of a, there exists a ball Bε(a),
ε 0, such that Bε(a) ⊂ V (a) and
∀x ∈ Bε(a), f (x) h ⇒ inf
x∈Bε(a)
x=a
f (x) ≥ h ⇒ sup
ε0
inf
x∈Bε(a)
x=a
f (x) ≥ h.
Since the inequality is true for all h f (a), letting h go to f (a), we get
liminf
x→a
f (x) ≥ h ⇒ liminf
x→a
f (x) ≥ f (a).](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-36-320.jpg)
![5. Existence of Minimizers in U 21
and, by definition, fU is lsc on RnU. As for all a ∈ U, fU (a) = f (a) and f is lsc on U.
So there exists a neighborhood V (a) of a in Rn such that
∀x ∈ V (a)∩U, h f (x) ⇒ ∀x ∈ V (a), h f (x) ≤ fU (x),
since, by construction, fU (x) = +∞ in Rn U. So fU is also lsc on U.
Example 4.2.
The indicator function of a closed subset U of Rn,
IU (x)
def
=
0, if x ∈ U,
+∞, if x /
∈ U,
is lsc on Rn. In fact, IU = fU for the function x → f (x) = 0 : Rn → R.
To complete this section, we introduce the “lower semicontinuous hull” cl f of a
function f , in the terminology of R. T. Rockafellar [1], that corresponds to the lower
semicontinuous regularization of f in I. Ekeland and R. Temam [1].
Definition 4.3. (i) The lsc regularization of a function f : Rn → R∪{+∞} is defined as
the upper envelope of all lsc functions less than or equal to f :
cl f (x)
def
= sup
g lsc and
g≤f on Rn
g(x). (4.7)
If there exists g lsc on Rn such that g ≤ f on Rn, then cl f is lsc on Rn. Otherwise,
set cl f (x) = −∞, by convention.
(ii) The usc regularization of a function f : Rn → R∪{−∞} is defined as the lower
envelope of all usc functions greater than or equal to f :
cl uscf (x)
def
= inf
g usc and
f ≤g on Rn
g(x). (4.8)
If there exists g usc on Rn such that f ≤ g on Rn, then cl uscf is usc on Rn. Otherwise,
set cl uscf (x) = +∞, by convention.
Note that the definition of the usc regularization amounts to cl uscf = −cl (−f ).
5 Existence of Minimizers in U
5.1 U Compact
WecannowweakentheassumptionsoftheWeierstrasstheorem(Theorem 2.1)byseparating
infimum problems from supremum problems.](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-39-320.jpg)
![5. Existence of Minimizers in U 23
|
|
−1 0 1
1
0
Figure 2.3. Lsc function that is not usc at 0.
In general, the infimum over U cannot be replaced by the infimum over U even if f
is lsc. We only have
inf f (U) ≥ inf f (U)
as seen from the following example.
Example 5.1 (see Figure 2.3).
Consider U = [−1,1]{0} and the lsc function
f (x) =
1, if x = 0,
0, if x = 0.
Then U = [−1,1] and
inf f (U) = 1 0 = inf f (U).
The function f is not usc at 0 since for 1/2 0 = f (0), the set
{x ∈ [−1,1] : f (x) 1/2} = {0}
is not a neighborhood of 0.
However, we have the following sufficient condition.
Theorem 5.2. Let U, ∅ = U ⊂ Rn
and let f : Rn → R be usc on U. Then
inf f (U) = inf f (U).
Proof. As U ⊂ U, we have
inf f (U) ≤ inf f (U).
As U = ∅, both inf f (U) and inf f (U) are bounded above.
If inf f (U) = −∞, then inf f (U) = −∞ and the equality is verified. If inf f (U) is
finite, assume that
inf f (U) inf f (U).](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-41-320.jpg)
![24 Chapter 2. Existence, Convexities, and Convexification
By definition of inf f (U), there exists x0 ∈ U such that
inf f (U) ≤ f (x0) inf f (U).
As f is usc, there exists a neighborhood V (x0) of x0 such that
∀x ∈ V (x0), f (x) inf f (U).
But x0 ∈ U is a limit point of U for which V (x0)∩U = ∅. Therefore, there exists u ∈ U
such that f (u) inf f (U). This contradicts the definition of inf f (U).
5.2 U Closed but not Necessarily Bounded
By Theorem 5.1, the functions of Figure 2.1 have minimizers at least at one point of the
compact subset U = [0,1] of R. However, in its present form, this theorem is a little
restrictive since it does not apply to the following simple example:
inf f (Rn
), f (x)
def
= x 2
, x ∈ Rn
.
The difficulty arises from the fact that, as Rn
is not bounded, it is not compact.
Remark 5.1.
Going over the proof of the theorem, it is readily seen that the compactness of U is not really
necessary. Since the family {Fk : k m} of closed subsets of U is an “increasing sequence”
Fk1 ⊂ Fk2 ⊂ U, ∀k2 ≥ k1 m,
then, for each k̄ m, ∩k̄≥kmFk = ∩kmFk. So it is sufficient to find some k̄ ∈ R for which
the lower section Fk̄ = {x ∈ U : f (x) ≤ k̄} is nonempty and bounded (hence compact5)
instead of making the assumption on U.
Definition 5.1.
Let U, ∅ = U ⊂ Rn
.
(i) f : Rn → R∪{+∞} has a bounded lower section in U if there exists k ∈ R such that
the lower section
Fk = {x ∈ U : f (x) ≤ k} (5.3)
is nonempty and bounded.
(ii) f : Rn → R∪{−∞} has a bounded upper section in U if there exists k ∈ R such that
the upper section
Fk = {x ∈ U : f (x) ≥ k} (5.4)
is nonempty and bounded.
5InfinitedimensionasetiscompactifandonlyifitisclosedandboundedbyHeine–Boreltheorem(Theorem 4.2
of Chapter 1).](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-42-320.jpg)
![6. Ekeland’s Variational Principle 29
Theorem 5.6. Let U, ∅ = U ⊂ Rn, be closed and let f : U → R∪{+∞} be lsc on U. Then
argminf (U) is closed (possibly empty). If, in addition, f has a bounded lower section in
U, then argminf (U) is compact and nonempty.
Proof. Since U is closed and f is lsc, the lower sections {Fk} in (5.3) are closed and
argminf (U) is closed as an intersection of closed sets. If, in addition, f has a bounded
lower section (there exists k0 such that Fk0 is nonempty and bounded), then the closed set
Fk0 is compact. Therefore, the set of minimizers
argminf (U) =
$
km
Fk ⊂ Fk0
is compact as a closed subset of the compact set Fk0 .
Example 5.9.
Go back to Examples 5.2 and 5.3 for U, ∅ = U ⊂ Rn. Given x ∈ Rn, it was shown that
dU (x) = dU (x) and that
∃x̂ ∈ U such that dU (x)
def
= inf
y∈U
x −y = x −x̂ . (5.10)
To establish that result we have considered the infimum of the square of the distance
inf
y∈U
f (y), f (y)
def
= y −x 2
for the continuous function f which has a bounded lower section in U. By Theorem 5.6,
argminf (U) is nonempty and compact. Points of argminf (U) are projections of x onto U.
Denote by U (x) this set. If x ∈ U, then U (x) = {x} is a singleton.
6 Ekeland’s Variational Principle
Ekeland’s variational principle8 in 1974 (I. Ekeland [1]) is a basic tool to get existence of
approximate minimizers in the absence of compactness. Its major impact is in the context
of spaces of functions. In this section we provide the finite-dimensional version and some
of its ramifications.
Theorem 6.1. Let f : Rn → R∪{+∞}, dom f = ∅, be lsc and bounded below. Then, for
any ε 0, there exists xε such that
inf
x∈Rn
f (x) ≤ f (xε) inf
x∈Rn
f (x)+ε (6.1)
and, for any η 0, there exists y such that
y −xε η, f (y) ≤ f (xε)−
ε
η
y −xε , (6.2)
and
∀x ∈ Rn
, x = y, f (y) f (x)+
ε
η
y −x . (6.3)
8Ivar Ekeland (1944– ).](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-47-320.jpg)
![30 Chapter 2. Existence, Convexities, and Convexification
This implies that, if f (xε) inf f (Rn), then, in any neighborhood of xε, there exists
a point y such that f (y) f (xε).
Proof. 9 Under the assumption of the theorem, the infimum is finite and, for any ε 0, there
exists xε ∈ dom f such that
inf
x∈Rn
f (x) ≤ f (xε) inf
x∈Rn
f (x)+ε
so that f (xε) is also finite. Given η 0, consider the new lsc function
gε/η(y)
def
= f (y)+
ε
η
y −xε .
The lower section
S
def
= {z ∈ Rn
: gε/η(z) ≤ f (xε}
is not empty, since xε ∈ S, and bounded since
f (z)+
ε
η
z−xε ≤ f (xε)
⇒ inf
x∈Rn
f (x)+
ε
η
z−xε ≤ f (z)+
ε
η
z−xε ≤ f (xε)
⇒
ε
η
z−xε ≤ f (xε)− inf
x∈Rn
f (x)
and f (xε)−infx∈Rn f (x) ≥ 0 is finite. By Theorem 5.3(i), the set of minimizers argmingε/η
is not empty and by Theorem 5.6, it is compact. Since f is lsc, by Theorem 5.1, there exists
y ∈ argmingε/η such that
f (y) = inf
z∈argmingε/η
f (z)
∀x ∈ Rn
, f (y)+
ε
η
y −xε ≤ f (x)+
ε
η
x −xε .
For x = xε,
f (y)+
ε
η
y −xε ≤ f (xε) ⇒ f (y) ≤ f (xε)−
ε
η
y −xε
and, from the definition of xε,
f (y)+
ε
η
y −xε ≤ f (xε) inf
x∈Rn
f (x)+ε ≤ f (y)+ε ⇒ y −xε η.
Consider the new function
g(z)
def
= f (z)+
ε
η
z−y . (6.4)
9From the proof of J. M. Borwein and A. S. Lewis [1, Thm. 7.1.2, pp. 153–154].](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-48-320.jpg)
![6. Ekeland’s Variational Principle 31
By definition, infz∈Rn g(z) ≤ f (y) and
∀z ∈ Rn
, f (z)+
ε
η
z−xε ≤ f (z)+
ε
η
z−y +
ε
η
y −xε
⇒ f (y)++
ε
η
y −xε = inf
z∈Rn
%
f (z)+
ε
η
z−xε
≤ inf
z∈Rn
%
f (z)+
ε
η
z−y
+
ε
η
y −xε
⇒ f (y) ≤ inf
z∈Rn
%
f (z)+
ε
η
z−y
≤ f (y) ⇒ f (y) = inf
z∈Rn
%
f (z)+
ε
η
z−y
.
To complete the proof, it remains to show that y is the unique minimizer of g. By construc-
tion, y is a global minimizer of f on argmingε/η. So, for all y ∈ argmingε/η, y = y,
f (y) ≤ f (y
) f (y
)+
ε
η
y
−y
and for y /
∈ argmingε/η,
f (y)+
ε
η
y −xε f (y
)+
ε
η
y
−xε ≤ f (y
)+
ε
η
y
−y +
ε
η
y −xε
⇒ ∀y
/
∈ argmingε/η, f (y) f (y
)+
ε
η
y
−y .
Combining the two inequalities,
∀y
∈ Rn
, y
= y, f (y) f (y
)+
ε
η
y
−y
and y is the unique minimizer of g.
Ekeland’s variational principle has many interesting ramifications.
Theorem 6.2 (W. Takahashi [1]’s existence theorem). Let f : Rn → R∪{+∞}, dom f =
∅, be lsc and bounded below. Assume that there exists c 0 such that for each x for which
f (x) inf f (Rn),
∃z = x such that f (z) ≤ f (x)−c z−x . (6.5)
Then f has a minimizer in Rn.
Proof. It is sufficient to prove the theorem for the function f (x)/c that amounts to set c = 1.
By contradiction, assume that for all x, f (x) inf f (Rn). By the hypothesis of the theorem,
∃zx = x such that f (zx) f (x)− zx −x .
But, by Ekeland’s variational principle, there exists y such that
∀x ∈ Rn
, x = y, f (y) f (x)+ y −x
(choose ε = 1/n and η = 1 in the theorem). Since zy = y,
f (y) f (zy)+ y −zy and f (zy) ≤ f (y)− zy −y ,
which yields a contradiction.](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-49-320.jpg)
![32 Chapter 2. Existence, Convexities, and Convexification
The above two theorems are intimately related to each other and also related to the
following fixed point theorem.
Theorem 6.3 (J. Caristi [1]’s fixed point theorem).10 Let F : Rn → Rn. If there exists an
lsc function f : Rn → R that is bounded below such that
∀x ∈ Rn
, x −F(x) ≤ f (x)−f (F(x)), (6.6)
then F has a fixed point.
Proof. By Theorem 6.1 with ε/η = 1, there exists y such that
∀x ∈ Rn
x = y, f (y) f (x)+ x −y .
If F(y) = y, then
f (y) f (F(y))+ F(y)−y ⇒ f (y)−f (F(y)) F(y)−y .
But, by assumption, f (y) − f (F(y)) ≥ y − F(y) , which yields a contradiction. Hence
F(y) = y and y is a fixed point of F.
Example 6.1.
For a contracting mapping F,
∃k, 0 k 1, such that ∀x,y ∈ Rn
, F(y)−F(x) ≤ k y −x ,
choose the function
f (x)
def
= x −F(x) /(1−k)
which is continuous and bounded below by 0. Then
∀x ∈ Rn
, x −F(x) = (1−k)f (x) = f (x)−k f (x)
∀x ∈ Rn
, f (F(x)) =
F(x)−F(F(x))
1−k
≤
k
1−k
x −F(x) = k f (x)
⇒ ∀x ∈ Rn
, x −F(x) = (1−k)f (x) = f (x)−k f (x) ≤ f (x)−f (F(x)).
The assumptions of Caristi’s theorem are verified. This is the Banach fixed point
theorem.
7 Convexity, Quasiconvexity, Strict Convexity, and
Uniqueness
7.1 Convexity and Concavity
Different notions of convexity will be introduced to discuss the convexity of argminf (U)
and the uniqueness of minimizers. For the supremum, the dual notion is concavity. As for
10Caristi’s fixed point theorem (also known as the Caristi–Kirk fixed point theorem) generalizes the Banach
fixed point theorem for maps of a complete metric space into itself. Caristi’s fixed point theorem is a variation
of the variational principle of Ekeland. Moreover, the conclusion of Caristi’s theorem is equivalent to metric
completeness, as proved by J. D. Weston [1] (1977). The original result is due to the mathematicians James
Caristi and William Arthur Kirk (see J. Caristi and W. A. Kirk [1]).](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-50-320.jpg)
![7. Convexity, Quasiconvexity, Strict Convexity, and Uniqueness 33
lsc functions, it is advantageous to extend the notion of convexity to functions with values
in R∪{+∞}.
Definition 7.1. (i) A subset U of Rn is said to be convex if
∀λ ∈ [0,1], ∀x,y ∈ U, λx +(1−λ)y ∈ U.
By convention, ∅ is convex.
(ii) Let U, ∅ = U ⊂ Rn, be convex. The function f : Rn → R∪{+∞} is said to be
convex on U if
∀λ ∈]0,1[, ∀x,y ∈ U, f (λx +(1−λ)y) ≤ λf (x)+(1−λ)f (y),
with the convention (+∞) + (+∞) = +∞, (+∞) + a = +∞ for all a ∈ R, and
(+∞)a = (a/ a )∞ for all a ∈ R not equal to 0.
(iii) Let U, ∅ = U ⊂ Rn, be convex. The function f : Rn → R∪{−∞} is said to be
concave on U if −f is convex on U (see Figure 2.4).
(iv) By convention, the function identically equal to −∞ or +∞ is both convex and
concave.
f (x) f (x)
x x
Convex function Concave function
Figure 2.4. Convex function and concave function.
The indicator function IU and the distance function dU for which (dU )U = IU are, respec-
tively, related to the convexity of U and its closure.
Theorem 7.1. Let U ⊂ Rn.
(i) The interior, intU, and the closure, U, of a convex set U are convex.
(ii) The set U is convex if and only if IU is convex.
(iii) The set U is convex if and only if dU is convex.
(iv) Let {Uα}α∈A be a family of convex subsets of Rn, where the set A of indices is
arbitrary and not necessarily finite. Then U = ∩α∈A Uα is convex.
(v) Let U be convex such that intU = ∅. Then intU = U.](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-51-320.jpg)
![34 Chapter 2. Existence, Convexities, and Convexification
Proof. (i) If U = ∅, then ∅ = ∅ is convex by convention. If x,y ∈ U, then there exist
sequences {xn} ⊂ U and {yn} ⊂ U such that xn → x and yn → y. By convexity of U, for
all λ ∈ [0,1],
U λxn +(1−λ)yn → λx +(1−λ)y.
Thus λx +(1−λ)y ∈ U and U is convex.
If intU = ∅, then it is convex by convention. For x,y ∈ intU, there exists Bε(x) and
Bη(y) such that Bε(x) ⊂ U and Bη(y) ⊂ U and, for all λ ∈ [0,1],
λx +(1−λ)y ∈ λBε(x)+(1−λ)Bη(y).
But,
λx +(1−λ)y ∈ Bmin{ε,η}(λx +(1−λ)y) ⊂ λBε(x)+(1−λ)Bη(y) ⊂ U
and λx +(1−λ)y ∈ intU.
(ii) If U = ∅, then IU is identically +∞ which is convex by convention. If U = ∅
is convex, then for all x,y ∈ U and λ ∈]0,1[, λx +(1−λ)y ∈ U and
IU (λx +(1−λ)y) = 0 = λIU (x)+(1−λ)IU (y) = 0.
If either x or y is not in U, then either IU (x) or IU (y) is +∞ and λIU (x)+(1−λ)IU (y) =
+∞, so that
IU (λx +(1−λ)y) ≤ +∞ = λIU (x)+(1−λ)IU (y).
Hence IU is convex.
Conversely, if IU is identically +∞, then U = ∅ which is convex by convention.
Otherwise, for all x,y ∈ U and λ ∈ [0,1],
0 ≤ IU (λx +(1−λ)y) ≤ λIU (x)+(1−λ)IU (y) = 0
and λx +(1−λ)y ∈ U. Hence the convexity of U.
(iii) By convention, U = ∅ implies dU (x) = −∞ that is convex, also by convention.
For U = ∅. Given x and y in Rn, there exist x and y in U such that dU (x) = |x −x| and
dU (y) = |y −y|. By convexity of U, for all λ, 0 ≤ λ ≤ 1, λx +(1−λ)y ∈ U and
dU (λx +(1−λ)y) ≤ λx +(1−λ)y −(λx +(1−λ)y)
≤λ x −x +(1−λ) y −y = λdU (x)+(1−λ)dU (y)
and dU is convex in Rn.
Conversely, if dU (x) = −∞ for some x, then, by convention, U = ∅ which is convex,
also by convention. If dU is finite and dU is convex, then
∀λ ∈ [0,1], ∀x,y ∈ U, dU (λx +(1−λ)y) ≤ λdU (x)+(1−λ)dU (y).](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-52-320.jpg)
![7. Convexity, Quasiconvexity, Strict Convexity, and Uniqueness 35
But x and y in U imply that dU (x) = dU (y) = 0 and hence
∀λ ∈ [0,1], dU (λx +(1−λ)y) = 0.
Thus λx +(1−λ)y ∈ U and U is convex.
(iv) If U is empty, U is convex by definition. If U is not empty, choose
x and y ∈ U = ∩α∈A Uα ⇒ ∀α ∈ A, x ∈ Uα, y ∈ Uα.
For all λ ∈ [0,1] and by convexity of Uα,
∀α ∈ A, λx +(1−λ)y ∈ Uα ⇒ λx +(1−λ)y ∈ ∩α∈AUα = U.
(v) As intU = ∅, pick a point x ∈ intU. By convexity, for all y ∈ ∂U, the segment
[x,y] = {λx + (1 − λ)y : 0 ≤ λ ≤ 1} belongs to U and [x,y[= {λx + (1 − λ)y : 0 λ ≤
1} ⊂ intU. So there exists a sequence yn = x +(y −x)/(n+1) in intU that converges to
y. Hence the result.
The following definitions will also be useful.
Definition 7.2.
Let U, ∅ = U ⊂ Rn.
(i) The convex hull of U is the intersection of all convex subsets of Rn that contain U.
It is denoted coU.
(ii) The closed convex hull of U is the intersection of all closed convex subsets of Rn
that contains U. It is denoted co U.
Theorem 7.2. Let U, ∅ = U ⊂ Rn.
(i) coU is convex and
coU =
k
i=1
λixi :
k
i=1
λi = 1, xi ∈ U, 0 ≤ λi ≤ 1, k ≥ 1
'
. (7.1)
(ii) co U is closed and convex and
co U = coU = coU. (7.2)
Proof. (i) By Theorem 7.1(iv), coU is convex as an intersection of convex sets. Denote by
C the right-hand side of (7.1). We have U ⊂ coU ⊂ C since C is a convex that contains U.
Taking all convex combinations of elements of U, we get C that is entirely contained in
coU and coU = C.
(ii) By Theorem 7.1(iv) and the fact that the intersection of a family of closed sets is
closed.
The convexity of f on a convex U can also be characterized in terms of the convexity
of fU or of its epigraph epifU as in the case when f is lsc (see Lemma 4.1).](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-53-320.jpg)
![36 Chapter 2. Existence, Convexities, and Convexification
Lemma 7.1. (i) If f : Rn → R∪{+∞} is convex on Rn, then dom f is convex and f is
convex on dom f .
(ii) If f : Rn → R∪{+∞} is convex on a convex subset U of Rn, then fU is convex on
Rn, dom fU = U ∩dom f is convex, and fU is convex on dom fU .
Proof. (i) For any convex combination of x,y ∈ dom f , f (x) +∞ and f (y) +∞,
f (λx + (1 − λ)y) ≤ λf (x) + (1 − λ)f (y) +∞, and λx + (1 − λ)y ∈ dom f . Thence,
dom f is convex and, by definition, f : dom f → R is convex.
(ii) By definition, dom fU = U ∩ dom f and as fU (x) = +∞ outside of U, fU is
convex on Rn. Finally, from part (i), dom fU is convex.
Theorem 7.3. Let U, ∅ = U ⊂ Rn, be convex and f : Rn → R∪{+∞}. The following
conditions are equivalent:
(i) f is convex on U;
(ii) fU is convex on Rn;
(iii) epifU is convex on Rn ×R.
Proof. (i) ⇒ (ii) From Lemma 7.1(ii).
(ii) ⇒ (iii) From Lemma 7.1(i), dom fU is convex and fU : dom fU → R is convex.
For all (x,µx) and (y,µy) in epifU , µx ≥ f (x) and µy ≥ f (y) and hence x,y ∈ dom fU .
Given λ ∈ [0,1], consider their convex combination
λ(x,µx)+(1−λ)(y,µy) = (λx +(1−λ)y,λµx +(1−λ)µy).
By construction and convexity of fU , λµx +(1−λ)µy ≥ λf (x)+(1−λ)f (y) ≥ f (λx +
(1 − λ)y). Therefore, f (λx + (1 − λ)y) +∞, λx + (1 − λ)y ∈ dom fU ⊂ U, and
λ(x,µx)+(1−λ)(y,µy) ∈ epifU . So epifU is convex on Rn ×R.
(iii) ⇒ (i) By definition of fU , dom fU ⊂ U and fU = f in dom fU . Therefore, for
all x,y ∈ dom fU , the pairs (x,f (x)) and (y,f (y)) belong to epifU . As epifU is convex,
∀λ ∈ [0,1], λ(x,f (x))+(1−λ)(y,f (y)) ∈ epifU
⇒ (λx +(1−λ)y,λf (x)+(1−λ)f (y)) ∈ epifU
⇒ λf (x)+(1−λ)f (y) ≥ fU (x +(1−λ)y).
Thence, as fU (x + (1 − λ)y) +∞, we get λx + (1 − λ)y ∈ dom fU and the convexity
of dom fU . Finally, fU (x + (1 − λ)y) = f (x + (1 − λ)y) yields λf (x) + (1 − λ)f (y) ≥
f (x +(1−λ)y) and the convexity of the function f in dom fU . The convexity of fU on the
convex subset U ⊃ dom fU is a consequence of Lemma 7.1(i). Finally, as fU = f on U,
f is convex on U.
Except for functions whose domain is a singleton, the convexity of a function forces
its liminf to be less than or equal to its value at each point of its effective domain.
Lemma 7.2. Let f : Rn → R∪{+∞}, dom f = ∅, be a convex function.
(i) If dom f = {x0} is a singleton,
liminf
y→x
f (y) = +∞ = f (x) if x = x0 and liminf
y→x
f (y) = +∞ f (x0). (7.3)](https://image.slidesharecdn.com/2626045-250609213107-9d174860/85/Introduction-To-Optimization-And-Semidifferential-Calculus-Michael-C-Delfour-54-320.jpg)
Introduction To Optimization And Semidifferential Calculus Michael C Delfour
- 1. Introduction To Optimization And Semidifferential Calculus Michael C Delfour download https://ebookbell.com/product/introduction-to-optimization-and- semidifferential-calculus-michael-c-delfour-5252090 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Introduction To Linear Optimization And Extensions With Matlab Kwon https://ebookbell.com/product/introduction-to-linear-optimization-and- extensions-with-matlab-kwon-5145270 Introduction To Optimization For Environmental And Chemical Engineers Behan https://ebookbell.com/product/introduction-to-optimization-for- environmental-and-chemical-engineers-behan-9964914 An Introduction To Continuous Optimization Foundations And Fundamental Algorithms N Andreasson https://ebookbell.com/product/an-introduction-to-continuous- optimization-foundations-and-fundamental-algorithms-n- andreasson-1146222 Practical Mathematical Optimization An Introduction To Basic Optimization Theory And Classical And New Gradientbased Algorithms 1st Edition Jan Snyman https://ebookbell.com/product/practical-mathematical-optimization-an- introduction-to-basic-optimization-theory-and-classical-and-new- gradientbased-algorithms-1st-edition-jan-snyman-1188194
- 3. An Introduction To Optimization With Applications In Machine Learning And Data Analytics Jeffrey Paul Wheeler https://ebookbell.com/product/an-introduction-to-optimization-with- applications-in-machine-learning-and-data-analytics-jeffrey-paul- wheeler-53329220 Introduction To Geological Uncertainty Management In Reservoir Characterization And Optimization Robust Optimization And History Matching Reza Yousefzadeh https://ebookbell.com/product/introduction-to-geological-uncertainty- management-in-reservoir-characterization-and-optimization-robust- optimization-and-history-matching-reza-yousefzadeh-49473592 An Introduction To Structural Optimization Solid Mechanics And Its Applications 1st Edition Peter W Christensen https://ebookbell.com/product/an-introduction-to-structural- optimization-solid-mechanics-and-its-applications-1st-edition-peter-w- christensen-2523464 Introduction To Maintenance Engineering Modelling Optimization And Management Bendaya https://ebookbell.com/product/introduction-to-maintenance-engineering- modelling-optimization-and-management-bendaya-5427022 Vehicle Propulsion Systems Introduction To Modeling And Optimization 3rd Lino Guzzella https://ebookbell.com/product/vehicle-propulsion-systems-introduction- to-modeling-and-optimization-3rd-lino-guzzella-55382860
- 5. IntroductIon to optImIzatIon and SemIdIfferentIal calculuS MO12_Delfour_FM-A.indd 1 1/11/2012 11:28:44 AM
- 6. This series is published jointly by the Mathematical Optimization Society and the Society for Industrial and Applied Mathematics. It includes research monographs, books on applications, textbooks at all levels, and tutorials. Besides being of high scientific quality, books in the series must advance the understanding and practice of optimization. They must also be written clearly and at an appropriate level for the intended audience. Editor-in-Chief Thomas Liebling École Polytechnique Fédérale de Lausanne Editorial Board William Cook, Georgia Tech Gérard Cornuejols, Carnegie Mellon University Oktay Gunluk, IBM T.J. Watson Research Center Michael Jünger, Universität zu Köln Adrian S. Lewis, Cornell University Pablo Parrilo, Massachusetts Institute of Technology Wiliam Pulleyblank, United States Military Academy at West Point Daniel Ralph, University of Cambridge Éva Tardos, Cornell University Ariela Sofer, George Mason University Laurence Wolsey, Université Catholique de Louvain Series Volumes Delfour, M. C., Introduction to Optimization and Semidifferential Calculus Ulbrich, Michael, Semismooth Newton Methods for Variational Inequalities and Constrained Optimization Problems in Function Spaces Biegler, Lorenz T., Nonlinear Programming: Concepts, Algorithms, and Applications to Chemical Processes Shapiro, Alexander, Dentcheva, Darinka, and Ruszczynski, Andrzej, Lectures on Stochastic Programming: Modeling and Theory Conn, Andrew R., Scheinberg, Katya, and Vicente, Luis N., Introduction to Derivative-Free Optimization Ferris, Michael C., Mangasarian, Olvi L., and Wright, Stephen J., Linear Programming with MATLAB Attouch, Hedy, Buttazzo, Giuseppe, and Michaille, Gérard, Variational Analysis in Sobolev and BV Spaces: Applications to PDEs and Optimization Wallace, Stein W. and Ziemba, William T., editors, Applications of Stochastic Programming Grötschel, Martin, editor, The Sharpest Cut: The Impact of Manfred Padberg and His Work Renegar, James, A Mathematical View of Interior-Point Methods in Convex Optimization Ben-Tal, Aharon and Nemirovski, Arkadi, Lectures on Modern Convex Optimization: Analysis, Algorithms, and Engineering Applications Conn, Andrew R., Gould, Nicholas I. M., and Toint, Phillippe L., Trust-Region Methods MOS-SIAM Series on Optimization ´ MO12_Delfour_FM-A.indd 2 1/11/2012 11:28:44 AM
- 7. IntroductIon to optImIzatIon and SemIdIfferentIal calculuS M. C. Delfour Centre de Recherches Mathématiques and Département de Mathématiques et de Statistique Université de Montréal Montréal, Canada Society for Industrial and Applied Mathematics Philadelphia Mathematical Optimization Society Philadelphia MO12_Delfour_FM-A.indd 3 1/11/2012 11:28:44 AM
- 8. Copyright © 2012 by the Society for Industrial and Applied Mathematics and the Mathematical Optimization Society 10 9 8 7 6 5 4 3 2 1 All rights reserved. Printed in the United States of America. No part of this book may be reproduced, stored, or transmitted in any manner without the written permission of the publisher. For information, write to the Society for Industrial and Applied Mathematics, 3600 Market Street, 6th Floor, Philadelphia, PA 19104-2688. Trademarked names may be used in this book without the inclusion of a trademark symbol. These names are used in an editorial context only; no infringement of trademark is intended. Library of Congress Cataloging-in-Publication Data Delfour, Michel C., 1943- Introduction to optimization and semidifferential calculus / M. C. Delfour. p. cm. -- (MOS-SIAM series on optimization) Includes bibliographical references and index. ISBN 978-1-611972-14-6 1. Mathematical optimization. 2. Differential calculus. I. Title. QA402.5.D348 2012 515’.642--dc23 2011040535 is a registered trademark. MO12_Delfour_FM-A.indd 4 1/11/2012 11:28:44 AM
- 9. To Francis and Guillaume MO12_Delfour_FM-A.indd 5 1/11/2012 11:28:45 AM
- 10. Contents List of Figures xi Preface xiii A Great and Beautiful Subject . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Intended Audience and Objectives of the Book . . . . . . . . . . . . . . . . . . . xiv Numbering and Referencing Systems . . . . . . . . . . . . . . . . . . . . . . . . xv Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv 1 Introduction 1 1 Minima and Maxima . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 Calculus of Variations and Its Offsprings . . . . . . . . . . . . . . . . . . . 2 3 Contents of the Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 4 Some Background Material in Classical Analysis . . . . . . . . . . . . . . 4 4.1 Greatest Lower Bound and Least Upper Bound . . . . . . . . . . . 5 4.2 Euclidean Space . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.2.1 Cartesian Product, Balls, and Continuity . . . . . . . . . 6 4.2.2 Open, Closed, and Compact Sets . . . . . . . . . . . . . 7 4.3 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 4.3.1 Definitions and Convention . . . . . . . . . . . . . . . . 9 4.3.2 Continuity of a Function . . . . . . . . . . . . . . . . . . 10 2 Existence, Convexities, and Convexification 11 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2 Weierstrass Existence Theorem . . . . . . . . . . . . . . . . . . . . . . . . 11 3 Extrema of Functions with Extended Values . . . . . . . . . . . . . . . . . 12 4 Lower and Upper Semicontinuities . . . . . . . . . . . . . . . . . . . . . . 16 5 Existence of Minimizers in U . . . . . . . . . . . . . . . . . . . . . . . . . 21 5.1 U Compact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 5.2 U Closed but not Necessarily Bounded . . . . . . . . . . . . . . . 24 5.3 Growth Property at Infinity . . . . . . . . . . . . . . . . . . . . . . 26 5.4 Some Properties of the Set of Minimizers . . . . . . . . . . . . . . 28 6 Ekeland’s Variational Principle . . . . . . . . . . . . . . . . . . . . . . 29 7 Convexity, Quasiconvexity, Strict Convexity, and Uniqueness . . . . . . . . 32 7.1 Convexity and Concavity . . . . . . . . . . . . . . . . . . . . . . . 32 7.2 Quasiconvexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 vii
- 11. viii Contents 7.3 Strict Convexity and Uniqueness . . . . . . . . . . . . . . . . . . . 40 8 Linear and Affine Subspace and Relative Interior . . . . . . . . . . . . . . 43 8.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 8.2 Domain of Convex Functions . . . . . . . . . . . . . . . . . . . . 45 9 Convexification and Fenchel–Legendre Transform . . . . . . . . . . . . . . 46 9.1 Convex lsc Functions as Upper Envelopes of Affine Functions . . . 46 9.2 Fenchel–Legendre Transform . . . . . . . . . . . . . . . . . . . . 51 9.3 Lsc Convexification and Fenchel–Legendre Bitransform . . . . . . 55 9.4 Infima of the Objective Function and of Its lsc Convexified Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 9.5 Primal and Dual Problems and Fenchel Duality Theorem . . . . . . 59 10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3 Semidifferentiability, Differentiability, Continuity, and Convexities 67 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 2 Real-Valued Functions of a Real Variable . . . . . . . . . . . . . . . . . . 69 2.1 Continuity and Differentiability . . . . . . . . . . . . . . . . . . . 72 2.2 Mean Value Theorem . . . . . . . . . . . . . . . . . . . . . . . . . 73 2.3 Property of the Derivative of a Function Differentiable Everywhere . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 2.4 Taylor’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 3 Real-Valued Functions of Several Real Variables . . . . . . . . . . . . . . 76 3.1 Geometrical Approach via the Differential . . . . . . . . . . . . . . 76 3.2 Semidifferentials, Differentials, Gradient, and Partial Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 3.2.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 79 3.2.2 Examples and Counterexamples . . . . . . . . . . . . . . 82 3.2.3 Gradient . . . . . . . . . . . . . . . . . . . . . . . . . . 86 3.2.4 Fréchet Differential . . . . . . . . . . . . . . . . . . . . 88 3.3 Hadamard Differential and Semidifferential . . . . . . . . . . . . . 91 3.4 Operations on Semidifferentiable Functions . . . . . . . . . . . . . 96 3.4.1 Algebraic Operations, Lower and Upper Envelopes . . . 96 3.4.2 Chain Rule for the Composition of Functions . . . . . . . 98 3.5 Lipschitzian Functions . . . . . . . . . . . . . . . . . . . . . . . . 103 3.5.1 Definitions and Their Hadamard Semidifferential . . . . 103 3.5.2 Dini and Hadamard Upper and Lower Semidifferentials . . . . . . . . . . . . . . . . . . . . . . 104 3.5.3 Clarke Upper and Lower Semidifferentials . . . . . . . 105 3.5.4 Properties of Upper and Lower Subdifferentials . . . . 107 3.6 Continuity, Hadamard Semidifferential, and Fréchet Differential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 3.7 Mean Value Theorem for Functions of Several Variables . . . . . . 111 3.8 Functions of Classes C(0) and C(1) . . . . . . . . . . . . . . . . . . 113 3.9 Functions of Class C(p) and Hessian Matrix . . . . . . . . . . . . . 116 4 Convex and Semiconvex Functions . . . . . . . . . . . . . . . . . . . . . . 119 4.1 Directionally Differentiable Convex Functions . . . . . . . . . . . 119 4.2 Semidifferentiability and Continuity of Convex Functions . . . . 122
- 12. Contents ix 4.2.1 Convexity and Semidifferentiability . . . . . . . . . . . 123 4.2.2 Convexity and Continuity . . . . . . . . . . . . . . . . . 126 4.3 Lower Hadamard Semidifferential at a Boundary Point of the Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 4.4 Semiconvex Functions and Hadamard Semidifferentiability . . . 132 5 Semidifferential of a Parametrized Extremum . . . . . . . . . . . . . . . 139 5.1 Semidifferential of an Infimum with respect to a Parameter . . . . . 139 5.2 Infimum of a Parametrized Quadratic Function . . . . . . . . . . . 143 6 Summary of Semidifferentiability and Differentiability . . . . . . . . . . . 148 7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 4 Optimality Conditions 153 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 2 Unconstrained Differentiable Optimization . . . . . . . . . . . . . . . . . 154 2.1 Some Basic Results and Examples . . . . . . . . . . . . . . . . . . 154 2.2 Least and Greatest Eigenvalues of a Symmetric Matrix . . . . . . . 164 2.3 Hadamard Semidifferential of the Least Eigenvalue . . . . . . . 166 3 Optimality Conditions for U Convex . . . . . . . . . . . . . . . . . . . . . 168 3.1 Cones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 3.2 Convex Gateaux Differentiable Objective Function . . . . . . . . . 170 3.3 Semidifferentiable Objective Function . . . . . . . . . . . . . . . . 177 3.4 Arbitrary Convex Objective Fonction . . . . . . . . . . . . . . . 178 4 Admissible Directions and Tangent Cones to U . . . . . . . . . . . . . . . 180 4.1 Set of Admissible Directions or Half-Tangents . . . . . . . . . . . 180 4.2 Properties of the Tangent Cones TU (x) and SU (x) . . . . . . . . . . 184 4.3 Clarke’s and Other Tangent Cones . . . . . . . . . . . . . . . . 187 5 Orthogonality, Transposition, and Dual Cones . . . . . . . . . . . . . . . . 190 5.1 Orthogonality and Transposition . . . . . . . . . . . . . . . . . . . 190 5.2 Dual Cones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 6 Necessary Optimality Conditions for U Arbitrary . . . . . . . . . . . . . . 197 6.1 Necessary Optimality Condition . . . . . . . . . . . . . . . . . . . 197 6.1.1 Hadamard Semidifferentiable Objective Function . . . . 197 6.1.2 Arbitrary Objective Function . . . . . . . . . . . . . . 199 6.2 Dual Necessary Optimality Condition . . . . . . . . . . . . . . . . 200 7 Affine Equality and Inequality Constraints . . . . . . . . . . . . . . . . . . 202 7.1 Characterization of TU (x) . . . . . . . . . . . . . . . . . . . . . . 202 7.2 Dual Cones for Linear Constraints . . . . . . . . . . . . . . . . . . 203 7.3 Linear Programming Problem . . . . . . . . . . . . . . . . . . . . 208 7.4 Some Elements of Two-Person Zero-Sum Games . . . . . . . . . . 217 7.5 Fenchel Primal and Dual Problems and the Lagrangian . . . . . . . 220 7.6 Quadratic Programming Problem . . . . . . . . . . . . . . . . . . 223 7.6.1 Theorem of Frank–Wolfe . . . . . . . . . . . . . . . . . 223 7.6.2 Nonconvex Objective Function . . . . . . . . . . . . . . 226 7.6.3 Convex Objective Function . . . . . . . . . . . . . . . . 228 7.7 Fréchet Differentiable Objective Function . . . . . . . . . . . . . . 234 7.8 Farkas’ Lemma and Its Extension . . . . . . . . . . . . . . . . . . 234
- 13. x Contents 8 Glimpse at Optimality via Subdifferentials . . . . . . . . . . . . . . . . 235 9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 5 Constrained Differentiable Optimization 241 1 Constrained Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 2 Equality Contraints: Lagrange Multipliers Theorem . . . . . . . . . . . . . 242 2.1 Tangent Cone of Admissible Directions . . . . . . . . . . . . . . . 242 2.2 Jacobian Matrix and Implicit Function Theorem . . . . . . . . . . . 243 2.3 Lagrange Multipliers Theorem . . . . . . . . . . . . . . . . . . . . 245 3 Inequality Contraints: Karush–Kuhn–Tucker Theorem . . . . . . . . . . . 256 4 Simultaneous Equality and Inequality Constraints . . . . . . . . . . . . . . 270 5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 A Inverse and Implicit Function Theorems 291 1 Inverse Function Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 291 2 Implicit Function Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 292 B Answers to Exercises 295 1 Exercises of Chapter 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 2 Exercises of Chapter 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 3 Exercises of Chapter 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 4 Exercises of Chapter 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 Elements of Bibliography 339 Index of Notation 349 Index 351
- 14. List of Figures 2.1 Discontinuous functions having a minimizing point in [0,1] . . . . . . . . . 12 2.2 Example of an lsc function . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.3 Lsc function that is not usc at 0 . . . . . . . . . . . . . . . . . . . . . . . . 23 2.4 Convex function and concave function . . . . . . . . . . . . . . . . . . . . 33 2.5 Example of a quasiconvex function that is not convex . . . . . . . . . . . . 40 2.6 Examples of convex functions: f (not lsc), cl f , and g (lsc) . . . . . . . . 46 2.7 The function f (x,y) = x2/y for y ≥ ε 0 and some small ε . . . . . . . . 47 2.8 Cases y ∈ dom f (left) and y ∈ dom f (right) . . . . . . . . . . . . . . . . 48 3.1 Example of right and left differentiability . . . . . . . . . . . . . . . . . . 71 3.2 Region determined by the functions α and β . . . . . . . . . . . . . . . . . 74 3.3 The function f (x) = |x| in a neighborhood of x = 0 for n = 1 . . . . . . . 81 3.4 Example 3.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 3.5 Examples 3.6 and 3.8 in logarithmic scale . . . . . . . . . . . . . . . . . . 85 3.6 Example 3.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 3.7 Upper envelope of two functions . . . . . . . . . . . . . . . . . . . . . . . 97 3.8 Upper envelope of three functions . . . . . . . . . . . . . . . . . . . . . . 97 3.9 The two convex functions g1 and g2 on [0,1] of Example 4.1 . . . . . . . . 122 3.10 Function of Exercise 7.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 4.1 Banana-shaped level sets of the Rosenbrock function at 0.25, 1, 4, 9, 16 . . 159 4.2 Basis functions (φ0,...,φi,...,φn) of P 1 n (0,1) . . . . . . . . . . . . . . . . 161 4.3 A nonconvex closed cone in 0 . . . . . . . . . . . . . . . . . . . . . . . . 169 4.4 Closed convex cone in 0 generated by the closed convex set V . . . . . . . 169 4.5 Closed convex cone in 0 generated in R3 by a nonconvex set V . . . . . . . 169 4.6 Convex set U tangent to the level set of f through x ∈ U . . . . . . . . . . 171 4.7 Tangency of the affine subspace A or the linear subspace S to a level set of f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 4.8 Tangency of U to a level set of the function f at x ∈ U . . . . . . . . . . . 177 4.9 Half-tangent dh(0;+1) to the path h(t) in U at the point h(0) = x. . . . . . 180 4.10 Cusp at x ∈ ∂U: U and TU (x) . . . . . . . . . . . . . . . . . . . . . . . . 182 4.11 First example: U and TU (x) . . . . . . . . . . . . . . . . . . . . . . . . . 183 4.12 Second example: U and TU (x) . . . . . . . . . . . . . . . . . . . . . . . . 183 4.13 Third example: U and TU (x) . . . . . . . . . . . . . . . . . . . . . . . . . 183 4.14 Fourth example: U and TU (x) . . . . . . . . . . . . . . . . . . . . . . . . 184 xi
- 15. xii List of Figures 5.1 Function y = h(x). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 5.2 Production cost ci(pi) as a function of the output power pi . . . . . . . . . 255 5.3 Determination of the output power pi as a function of λ1 . . . . . . . . . . 256 5.4 Region U in Example 3.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 5.5 The set of constraints U of Example 3.3 . . . . . . . . . . . . . . . . . . . 265 5.6 The set of constraints U of Example 3.4 . . . . . . . . . . . . . . . . . . . 268 5.7 The ellipse E and the line L as a function of c . . . . . . . . . . . . . . . . 288
- 16. Preface A Great and Beautiful Subject Optimization refers to finding, characterizing, and computing the minima and/or maxima of a function with respect to a set of admissible points. Its early steps were intertwined with the ones of the differential calculus and the mathematical analysis. The first idea of the differential calculus and the rule for the com- putation of the minima and maxima could be attributed to Fermat in 1638. The concept of derivative was introduced in that context by Leibniz and Newton almost fifty years later. So, the condition obtained by Fermat for the extremum of an algebraic function was de facto generalized in the form f (x) = 0. With the introduction of the notion of differenti- able function of several variables and of differentiable functions defined on Hilbert and topological vector spaces, the rule of Fermat remains valid. One of the important areas of optimization is the calculus of variations, which deals with the minimization/maximization of functionals, that is, functions of functions. It was also intertwined with the development of classical analysis and functional analysis. But, optimization is not just mathematical analysis. Many decision-making prob- lems in operations research, engineering, management, economics, computer sciences, and statistics are formulated as mathematical programs requiring the maximization or mini- mization of an objective function subject to constraints. Such programs1 often have special structures: linear, quadratic, convex, nonlinear, semidefinite, dynamic, integer, stochastic programming, etc. This was the source of more theory and efficient algorithms to compute solutions. With the easier access to increasingly more powerful computers, larger and more complex problems were tackled thus creating a demand for efficient computer software to solve large-scale systems. To give a few landmarks, the modern form of the multipliers rule goes back to La- grange2 in his path-breaking Mécanique analytique in 1788 and the steepest descent method to Gauss.3 The simplex algorithm to solve linear programming4 problems was created by 1The term programming in this context does not refer to computer programming. Rather, the term comes from the use of program by the United States military to refer to proposed training and logistics schedules, which were the problems that Dantzig was studying at the time. 2Joseph Louis, comte de Lagrange (in Italian Giuseppe Lodovico Lagrangia) (1736–1813). 3Johann Carl Friedrich Gauss (1777–1855). 4Much of the theory had been introduced by Leonid Vitaliyevich Kantorovich (1912–1986) in 1939 (L .V. Kan- torovich [1, 2]). xiii
- 17. xiv Preface George Dantzig5 and the theory of the duality was developed by John von Neumann6 both in 1947. The necessary conditions for inequality-constrained problems were first published in the Masters thesis of William Karush in 1939, although they became renowned after a seminal conference paper by Harold W. Kuhn and Albert W. Tucker in 1951. Intended Audience and Objectives of the Book This book is intended as a textbook for a one-term course at the undergraduate level for students in Mathematics, Physics, Engineering, Economics, and other disciplines with a basic knowledge of mathematical analysis and linear algebra. It is intentionally limited to the optimization with respect to variables belonging to finite-dimensional spaces. This is what we call finite-dimensional optimization. It provides a lighter exposition deferring at the graduate level technical questions of Functional Analysis associated with the Calculus of Variations. The useful background material has been added at the end of the first chapter to make the book self-sufficient. The book can also be used for a first year graduate course or as a companion to other textbooks. Being limited to one term, choices had to be made. The classical themes of op- timization are covered emphasizing the semidifferential calculus while staying at a level accessible to an undergraduate student. In the making of the book, some material has been added to the original lecture notes. For a one-term basic program the sections and subsections beginning with the black triangle can be skipped. The book is structured in such a way that the basic program only requires very basic notions of analysis and the Hadamard semidifferential that is easily accessible to nonmathematicians as an extension of their elementary one-dimensional differential calculus. The added material makes the book more interesting and provides connections with convex analysis and, to a lesser degree, subdifferentials. Yet, the book does not pretend or aim at covering everything. The added material is not mathematically more difficult since it only involves more liminf and limsup in the definitions of lower and upper semidifferentials, but it might be too much for a basic undergraduate course. For a first initiation to nondifferentiable optimization, semidifferentials have been preferred over subdifferentials7 that necessitate a good command of set-valued analysis. The emphasis will be on Hadamard semidifferentiable8 functions for which the result- ing semidifferential calculus retains all the nice features of the classical differential cal- culus, including the good old chain rule. Convex continuous and semiconvex functions are Hadamard semidifferentiable and an explicit expression of the semidifferential of an extremum with respect to parameters can be obtained. So, it works well for most non- differentiable optimization problems including semiconvex or semiconcave problems. The Hadamard semidifferential calculus readily extends to functions defined on differential man- ifolds and on groups that naturally occur in optimization problems with respect to the shape or the geometry.9 5George Bernard Dantzig (1914–2005) (G. B. Dantzig [1, 3]). 6John von Neumann (1903–1957). 7For a treatment of finite-dimensional optimization based on subdifferentials and the generalized gradient, the reader is referred to the original work of R. T. Rockafellar [1] and F. H. Clarke [2] and to the more recent book of J. M. Borwein and A. S. Lewis [1]. 8The differential in the sense of Hadamard goes back to the beginning of the 20th century. We shall go back to the original papers of J. Hadamard [2] in 1923 and of M. Fréchet [3] in 1937. 9The reader is referred to the book of M. C. Delfour and J.-P. Zolésio [1].
- 18. Preface xv The book is written in the mathematical style of definitions, theorems, and detailed proofs. It is not necessary to go through all the proofs, but it was felt important to have all the proofs in the book. Numerous examples and exercises are incorporated in each chapter to illustrate and better understand the subject material. In addition, the answer to all the exercises is provided in Appendix B. This considerably expands the set of examples and enriches the theoretical content of the book. More exercises along with examples of applications in various fields can be found in other books such as the ones of S. Boyd and L. Vandenberghe [1] and F. Bonnans [1]. The purpose of the historical commentaries and landmarks is mainly to put the subject in perspective and to situate it in time. Numbering and Referencing Systems The numbering of equations, theorems, lemmas, corollaries, definitions, examples, and remarks is by chapter. When a reference to another chapter is necessary it is always followed by the words in Chapter and the number of the chapter. For instance, “equation (7.5) from Theorem 7.4(i) of Chapter 2” or “Theorem 5.2 from section 5 in Chapter 3.” The text of theorems, lemmas, and corollaries is slanted; the text of definitions, examples, and remarks is normal shape and ended by a square . This makes it possible to aesthetically emphasize certain words especially in definitions. The bibliography is by author in alphabetical order. For each author or group of coauthors there is a numbering in square brackets starting with [1]. A reference to an item by a single author is of the form J. Hadamard [2] and a reference to an item with several coauthors is of the form H. W. Kuhn andA. W. Tucker [1]. Boxed formulae or statements are used in some chapters for two distinct purposes. First, they emphasize certain important definitions, results, or identities; second, in long proofs of some theorems, lemmas, or corollaries they isolate key intermediary results which will be necessary to more easily follow the subsequent steps of the proof. Acknowledgments This book is based on an undergraduate course created in 1975 at the University of Montreal by Andrzej Manitius (George Mason University) who wrote a first set of lecture notes. The course and its content were reorganized and modified in 1984 by the author and this book is the product of their evolution. The author is grateful to the referees and to Thomas Liebling, Editor-in-Chief of the MOS-SIAM Series on Optimization, for their very constructive suggestions, without forgetting the students whose contribution has been invaluable. Michel Delfour Montreal, May 16, 2011
- 19. ! 1 ! 0.8 ! 0.6 ! 0.4 ! 0.2 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 10 0 10 2 10 4 10 6 10 8 10 10 Chapter 1 Introduction 1 Minima and Maxima A first objective is to seek the weakest conditions for the existence of an admissible point achieving the extremum (a minimum or a maximum) of an objective function. This will require notions such as continuity, compactness, and convexity and their relaxation to weaker notions of semicontinuity, bounded lower or upper sections, and quasiconvexity. When the set of admissible points is specified by differentiable constraint functions, dual- izing the necessary optimality condition naturally leads to the introduction of the Lagrange multipliers and the Lagrangian for which the number of unknowns is increased by the number of multipliers associated with each constraint function. A second objective is the characterization of the points achieving an extremum. In most problems this requires the differentiability of the objective function and that the set of admissible points be specified by functions that are also differentiable. Otherwise, it is still possible to obtain a characterization of the extremizers by going to weaker notions of semidifferential and using a local approximation of the set of admissible points by tangent cones. In particular, the convex continuous functions are semidifferentiable. Thus, the minimum of a convex continuous objective function over a convex set of admissible points can be completely characterized by using semidifferentials. When the objective function is not convex, fairly general necessary optimality conditions can also be obtained by using still weaker notions of upper or lower semidifferentials. As it is seldom possible to explicitly solve the equations that characterize an extremum, it is natural to use numerical methods. It is a broad area of activity strongly stimulated by the availability of and access to more and more powerful computers. 1
- 20. 2 Chapter 1. Introduction 2 Calculus of Variations and Its Offsprings The sources of this section are B. van Brunt,1 J. Ferguson,2 and Wikipedia: Calculus of Variations and Optimization (mathematics). The first idea of the differential calculus and the rule for the computation of the minima and maxima3 seems to go back to Fermat4 in 1638. It is generally accepted that the concept of derivative is due to Leibniz5 who published in 16846 and Newton7 who published over a longer period of time.8 So, the condition obtained by Fermat for the extremum of an algebraic function was de facto generalized in the form f (x) = 0. With the introduction of the concept of differentiable function of several variables of Jacobi and of differentiable functions defined on Hilbert and topological vector spaces, the rule of Fermat and Leibniz remains valid. During three centuries, it was applied, justified, adapted, and generalized in the context of the theory of optimization, of the calculus of variations, and of the theory of optimal control. The calculus of variations deals with the minimization/maximization of functionals, that is, functions of functions. The simplest example of this type of problems is to find the curve of minimum length between two points. In the absence of constraints, the solution is a straight line between the points. However, when the curve is constrained to stay on a surface in the space, the solution is less obvious and not necessarily unique. Such solutions are called geodesics. A problem of that nature is illustrated by Fermat’s principle in optics: the light follows the shortest path of optical length between two points, where the optical length depends on the material of the physical medium. A notion of the same nature in mechanics is the least action principle. Several important problems involve functions of several variables. For instance, solutions of boundary value problems for the Laplace equation satisfy the Dirichlet principle. According to some historians, the calculus of variations begins with the brachis- tochrone problem of Johann Bernoulli in 1696. It immediately attracted the attention of Jakob Bernoulli and of the Marquis de l’Hôpital, but it is Euler who was the first to de- velop this subject. His contributions began in 1733 and it is his Elementa Calculi Varia- tionum that gave its name to this discipline. In 1744, Euler published his landmark book Methodus inveniendi lineas curvas maximi minimive proprietate gaudentes, sive solutio problematis isoperimetrici latissimo sensu accepti.9 In two papers read to the Académie des Sciences in 1744, and to the Prussian Academy in 1746, Maupertuis proclaimed the principle of least action. 1Bruce van Brunt, The Calculus of Variations, Springer-Verlag, New York, 2004. 2James Ferguson, A. Brief Survey of the History of the Calculus of Variations and its Applications, University of Victoria, Canada, 2004 (arXiv:math/0402357). 3Methodus ad disquirendam Maximam et Minimam, 1638 (see P. de Fermat [1]). 4Pierre de Fermat (1601–1665). 5Gottfried Wilhelm Leibniz (1646–1716). 6Nova methodus pro maximis et minimis, itemque tangentibus, quae nec fractas nec irrationales quantitates moratur, et singulare pro illis calculi genus (A new method for maxima and minima and their tangents, that are not limited to fractional or irrational expressions, and a remarkable type of calculus for these), in Acta Eruditorum, 1684, a journal created in Leipzig two years earlier. 7Sir Isaac Newton (1643–1728). 8The Method of Fluxions completed in 1671 and published in 1736 and Philosophiæ Naturalis Principia Mathematica (Mathematical Principles of Natural Philosophy), often called the Principia (Principles), 1687 and 1726 (third edition). 9A method for discovering curved lines that enjoy a maximum or minimum property, or the solution of the isoperimetric problem taken in the widest sense.
- 21. 3. Contents of the Book 3 Lagrange extensively contributed to the theory and Legendre10 in 1786 laid the foun- dations of the characterization of maxima and minima. In his path-breaking Mécanique analytique11 in 1788, Lagrange summarized all the work done in the field of classical me- chanics since Newton.12 It is in that book that he clearly expressed the multipliers rule in its modern form. In the 18th century several mathematicians have contributed to this enterprise, but perhaps the most important work of the century is that of Weierstrass starting in 1870. He was the first to give a completely correct proof of a sufficient condition for the existence of a minimum. His celebrated course on the theory is epoch-making, and it may be asserted that he was the first to place it on a firm and unquestionable foundation. The 20th and the 23rd Hilbert problems published in 1900 enticed further development. In the 20th century, Noether, Tonelli, Lebesgue, and Hadamard, among others, made significant contributions. Morse applied the calculus of variations in what is now called Morse theory. Pontryagin, Rockafellar, and Clarke developed new mathematical tools for the optimal control theory, a generalization of the calculus of variations. In problems of geometrical design and shape optimization, the modelling, optimiza- tion, or control variable is no longer a vector of scalars or functions but the shape of a geometrical object.13 In this category, we find for instance Plateau’s problem. This prob- lem was raised by Lagrange in 1760, but it is named after Joseph Plateau who was interested in soap films around 1873.14 It consists in finding the surface of minimum area with a given boundary in space. Such surfaces are easy to find experimentally, but their mathematical interpretation is extremely complex. In this family of problems we also encounter sev- eral identification problems such as the reconstruction and the processing of images from two dimensional or three dimensional scanned material or biological (biometrics) objects. Another area is the enhancing of images such as in the Very Large Telescope project. To deal with nonparametrized geometrical objects, nonlinear and nonconvex spaces and a shape differential calculus are required. 3 Contents of the Book Chapter 2 is devoted to the existence of minimizers of a real-valued function. Natural notions of lower and upper semicontinuities and several notions of convexity are introduced to relax the conditions of the classical Weierstrass theorem specialized to the case of the minimum. Some elements of convex analysis, such as the notion of Fenchel–Legendre transform, the primal and dual problems, and the Fenchel duality theorem are also included along with Ekeland’s variational principle and some of its consequences. Chapter 3 first provides a review of the differentiability of functions of one and several variables. Notions of semidifferentials and, specifically, of Hadamard semidifferentials are introduced. The connection is made with the classical Gateaux15 and Fréchet differentials. A semidifferential calculus that extends the classical differential calculus is developed. In this framework, Hadamard semidifferentials of lower and upper envelopes of finitely many 10Sur la manière de distinguer les maxima des minima dans le calcul des variations (On the method of distin- guishing maxima from minima in the calculus of variations). 11J. L. Lagrange [2]. 12Philosophiæ naturalis principia mathematica. 13Cf., for instance, M. C. Delfour and J.-P. Zolésio [1]. 14Joseph Antoine Ferdinand Plateau (1801–1883). Cf. J. A. Plateau [1]. 15Without circumflex accent (see footnote 19 on page 80).
- 22. 4 Chapter 1. Introduction Hadamard semidifferentiable functions exist and the chain rule for the composition of Hadamard semidifferentiable functions remains valid. Moreover, the convex continuous functions and, more generally, the semiconvex functions are Hadamard semidifferentiable. Upper and lower notions of semidifferentials are also included in connection with semi- convex functions. Finally, we give a fairly general theorem on the semidifferentiability of extrema with respect to a parameter. It is applied to get the explicit expression of the Hadamard subdifferential of the extremum of quadratic functions. The differentials and semidifferentials introduced in this chapter are summarized and compared in the last section. Chapter 4 focuses on optimality conditions to characterize an unconstrained or a constrained extremum via the semidifferential or differential of the objective function. It first considers twice differentiable functions without constraints along with several examples. A special attention is given to the generic example of the least and greatest eigenvalues of a symmetric matrix. The explicit expression of their Hadamard semidifferentials is provided with the help of the general theorems of Chapter 3. This is followed by the necessary and sufficient optimality condition for convex differentiable objective functions and convex sets of admissible points. It is specialized to linear subspaces, affine subspaces, and convex cones at the origin. Finally, a necessary and sufficient condition for an arbitrary convex objective function and an arbitrary set of constraints is given in terms of the lower semidifferential. Thesecondpartofthechaptergivesageneralnecessaryoptimalityconditionforalocal minimum using the upper Hadamard semidifferentiability of the objective function and the Bouligand’s tangent cone of admissible directions. It is quite remarkable that such simple notions be sufficient to cover most of the so-called nondifferentiable optimization. This condition is dualized by introducing the notion of dual cone. The dual necessary optimality condition is then applied to the linear programming problem where the constraints are specified by a finite number of equalities and inequalities on affine functions. At this juncture, the Lagrangian is introduced along with its connections to two-person zero-sum games and to Fenchel’s primal and dual problems of Chapter 2. A general form of Farkas’ lemma is given in preparation of the next chapter. The constructions and results are extended to the quadratic programming problem and to Fréchet differentiable objective functions. At the end of this chapter, a glimpse is given into optimization via subdifferentials that involves set-valued functions. Chapter 5 is devoted to differentiable optimization where the set of admissible points is specified by a finite number of differentiable constraint functions. By using the dual necessary optimality condition, we recover the Lagrange multiplier theorem for equality constraints, the Karush–Kuhn–Tucker theorem for inequality constraints, and the general theorem for the mixed case of equalities and inequalities. 4 Some Background Material in Classical Analysis This section puts together a compact summary of some basic elements of classical analysis that will be needed in the other chapters. They come from several sources (for instance, among others, W. H. Fleming [1], W. Rudin [1], or L. Schwartz [1]). The differential calculus will be completely covered from scratch in Chapter 3 and does not require any prerequisite. The various notions of convexity that will be needed will be introduced in each chapter, but the reader is also referred to specialized books such as, for instance, F.A.Valentine [1], R.T. Rockafellar [1], L. D. Berkovitz [1], S. R. Lay [1], H.Tuy [1], S. Boyd and L. Vandenberghe [1].
- 23. 4. Some Background Material in Classical Analysis 5 4.1 Greatest Lower Bound and Least Upper Bound Let R denote the set of real numbers and let |x| denote the absolute value of x. The following notation will be used for positive and strictly positive real numbers R+ def = {x ∈ R : x ≥ 0} and R+ def = {x ∈ R : x 0} and the notation R = R∪{±∞} for the extended real numbers. Definition 4.1. Let ∅ = A ⊂ R. (a) b0 ∈ R is a least upper bound of A if (i) b0 is an upper bound of A and (ii) for all upper bounds M of A, we have b0 ≤ M. The least upper bound b0 of A is unique and is denoted supA. If A is not bounded above, set supA = +∞. (b) b0 ∈ R is a greatest lower bound of A if (i) b0 is a lower bound of A and (ii) for all lower bounds m of A, we have b0 ≥ m. The greatest lower bound b0 of A is unique and is denoted inf A. If A is not bounded below, set inf A = −∞. Remark 4.1. (i) When A = ∅, we always have −∞ ≤ inf A ≤ supA ≤ +∞. By defi- nition, supA ∈ R if and only if A is bounded above and inf A ∈ R if and only if A is bounded below. (ii) When A = ∅, we write by convention supA = −∞ and inf A = +∞. At first sight it might be shocking to have supA inf A, but, from a mathematical point of view, it is therightchoicesincesupA inf AifandonlyifA = ∅or, equivalently, supA ≥ inf A if and only if A = ∅. We shall often use the following equivalent conditions. Theorem 4.1. Let ∅ = A ⊂ R. (a) b0 is the least upper bound of A if and only if (i) b0 is an upper bound of A and (ii) for all M such that b0 M, there exists x0 ∈ A such that b0 ≥ x0 M. (b) b0 is the greatest lower bound of A if and only if (i) b0 is a lower bound of A and (ii) for all m such that b0 m, there exists x0 ∈ A such that b0 ≤ x0 m. (c) supA = +∞ if and only if, for all M ∈ R, there exists x0 ∈ A such that x0 M. (d) inf A = −∞ if and only if, for all m ∈ R, there exists x0 ∈ A such that x0 m.
- 24. 6 Chapter 1. Introduction 4.2 Euclidean Space Most results in this book remain true in general vector spaces of functions and in groups of transformations of infinite dimension. In this book the scope is limited to vector spaces of finite dimension that will be identified with the Cartesian product Rn. For instance, such spaces include the space of all polynomials of degree less than or equal to n−1, n ≥ 1, an integer. In this section, we recall some definitions, notions, and theorems from Classical Analysis. 4.2.1 Cartesian Product, Balls, and Continuity Given an integer n ≥ 1, let Rn = R×···×R n times (4.1) be the Cartesian product of dimension n with the following notation: an element x = (x1,...,xn) ∈ Rn or in vectorial form x = ⎡ ⎢ ⎣ x1 . . . xn ⎤ ⎥ ⎦ the norm x Rn = n i=1 x2 i 1/2 and the inner product x ·y = n i=1 xiyi. (4.2) The norm will be simply written x when no confusion arises and the arrow on top of the vector x will often be dropped. When n = 1, x R1 coincides with the absolute value |x|. Rn with the scalar multiplication and the addition ∀α ∈ R, x ∈ Rn , α x def = (αx1,...,αxn) ∀x,y ∈ Rn , x +y def = (x1 +y1,...,xn +yn) is a vector space on R of dimension n. Definition 4.2. The canonical orthonormal basis of Rn is the set {en i ∈ Rn : 1 ≤ i ≤ n} defined by (en i )j def = δij , δij def = 1, if i = j 0, if i = j, that is, en 1 = (1,0,0,...,0,0), en 2 = (0,1,0,...,0,0), ..., en n = (0,0,0,...,0,1). In particular, en i ·en j = δij . When no confusion arises, we simply write {ei} without the supersript n.
- 25. 4. Some Background Material in Classical Analysis 7 A Euclidean space is a vector space E that can be identified with Rn via a linear bijection for some integer n ≥ 1. For instance, we can identify with Rn the space P n−1[0,1] of polynomials of order less than or equal to n−1 on the interval [0,1]: p → (p(0),p (0),...,p(n−1) (0)) : P n−1 [0,1] → Rn (p0,p1,...,pn−1) → p(x) def = n−1 i=0 pi xi i ! : Rn → Pn−1 [0,1]. 4.2.2 Open, Closed, and Compact Sets The notions open and closed sets in Rn can be defined starting from balls. Ball at x of radius r 0: open Br(x) = {y ∈ Rn : y −x r} closed Br(x) = {y ∈ Rn : y −x ≤ r}. Unit ball at 0: open B = {y ∈ Rn : y 1}, closed B = {y ∈ Rn : y ≤ 1}. Punched open ball at x: B r(x) = {y ∈ Rn : 0 y −x r} Definition 4.3. Let U ⊂ Rn. (i) a ∈ Rn is an interior point of U, if there exists r 0 such that Br(a) ⊂ U. (ii) The interior of U is the set of all interior points of U. It will be denoted by intU. By definition intU ⊂ U. (iii) V (x) is a neighborhood of x if there exists r 0 such that Br(x) ⊂ V (x). (iv) A is an open subset of Rn if for all x ∈ A, there exists a neighborhood V (x) of x such that V (x) ⊂ A. (v) The family T of all open sets in Rn is called the topology on Rn generated by the norm. The topology T of Rn is equal to the family of all finite intersections and arbitrary unions of open balls in Rn. Definition 4.4. (i) A sequence {xn} in Rn is convergent if there exists a point x ∈ Rn such that ∀ε 0, ∃N, ∀n N, xn −x Rn ε. The point x is unique and called the limit point of {xn}.
- 26. 8 Chapter 1. Introduction (ii) {xn} in Rn is a Cauchy sequence if ∀ε 0, ∃N, ∀n,m N, xn −xm Rn ε. A convergent sequence is a Cauchy sequence. All Cauchy sequences in Rn are convergent to points in Rn. We say that the space Rn is complete for the topology T . The notions of limit point and closed set can be specified in several ways. We do it via the notions of accumulation point and isolated point. Definition 4.5. Let U be a subset of Rn. (i) a ∈ U is an isolated point of U if there exists r 0 such that B r(a)∩U = ∅. (ii) a ∈ Rn is an accumulation point of U if B r(a)∩U = ∅ for all r 0. Definition 4.6. (i) a ∈ Rn is a limit point of U if Br(a)∩U = ∅ for all r 0. (ii) The closure of U is the set of all limit points of U. It will be denoted U. (iii) F is a closed set if it contains all its limit points. Remark 4.2. (i) Equivalently, x is a limit point of U if, for all neighborhoods V (x) of x, V (x)∩U = ∅. (ii) The closure of U is equal to the union of all its isolated and accumulation points. Hence U ⊂ U. (iii) The only subsets of Rn that are both open and closed are ∅ and Rn. Definition 4.7. Let A and B be two subsets of Rn. (i) AB def = {x ∈ A : x / ∈ B}. When A = Rn, we write B or RnB and say that B is the complement of B in Rn. (ii) The boundary of U ⊂ Rn is defined as U ∩U. It will be denoted ∂U. It is easy to check that ∂U = UintU, U = intU ∪∂U, and U = intU ∪∂U. Definition 4.8. (i) A family {Gα} of open subsets of Rn is an open cover of X ⊂ Rn if X ⊂ ∪αGα. (ii) A nonempty subset K of Rn is said to be compact if each open covering {Gα} of K has a finite subcover {Gαi : 1 ≤ i ≤ k}. Theorem 4.2 (Heine–Borel). Let ∅ = U ⊂ Rn. U is compact if and only if U is closed and bounded.16 17 16Heinrich Eduard Heine (1821–1881). 17Félix Edouard Justin Émile Borel (1871–1956).
- 27. 4. Some Background Material in Classical Analysis 9 In a normed vector space E, a compact subset U of E is closed and bounded, but the converse is generally not true except in finite-dimensional normed vector spaces. Theorem 4.3 (Bolzano–Weierstrass). If U is an infinite subset of a compact set K in a metric space, then U has a limit point in K.18 19 As a consequence of this theorem, we have the following useful result. Theorem 4.4. Let K, ∅ = K ⊂ Rn, be compact. Any sequence {xn} in K has a convergent subsequence {xnk } to a point x of K: ∃{xnk } and ∃x ∈ K such that xnk → x ∈ K. 4.3 Functions 4.3.1 Definitions and Convention Let n ≥ 1 and m ≥ 1 be two integers and f : Df → Rm be a function defined on a domain Df in Rn. Since it is always possible to arbitrarily extend the definition of a function from its initial domain of definition Df to all of Rn, we adopt the following convention. Convention 4.1. All functions f : Df ⊂ Rn → Rm in this book have domain Df = Rn. Definition 4.9. (i) A real-valued function of a real variable is a function f : R → R, and a real-valued function of several real variables is a function f : Rn → R, n ≥ 2. (ii) A vector function is a function f : Rn → Rm for some m ≥ 2. Definition 4.10. Let n ≥ 1 and m ≥ 1 be two integers. Denote by {em i ∈ Rm : 1 ≤ j ≤ m} and {en i ∈ Rn : 1 ≤ i ≤ n} the respective canonical orthonormal bases associated with Rm and Rn, respectively. (i) A function A : Rn → Rm is linear if ∀x,y ∈ Rn , ∀α,β ∈ R, A(αx +βy) = α A(x)+β A(y). (4.3) (ii) By convention, the m×n matrix {Aij } associated with A will also be denoted A: Aij def = em i ·Aen j , Ax ·y = m i=1 n j=1 Aij xj yi. (4.4) 18Bernard Placidus Johann Nepomuk Bolzano (1781–1848). 19Karl Theodor Wilhelm Weierstrass (1815–1897) was the leader of a famous school of mathematicians who undertook the systematic revision of various sectors of mathematical analysis.
- 28. 10 Chapter 1. Introduction 4.3.2 Continuity of a Function Definition 4.11. Let f : Rn → Rm for two integers n ≥ 1 and m ≥ 1. The function f is continuous at x ∈ Rn if ∀ε 0, ∃δ(x) 0 such that ∀y such that y −x Rn δ(x), f (y)−f (x) Rm ε. The function f is continuous on U ⊂ Rn if it is continuous at every point of U. The notion of continuity for a function f : Rn → Rm can be defined in terms of open balls. Indeed, the definition involves the open ball Bε(f (x)) in Rm and the open ball Bδ(x)(x) in Rn. The condition becomes: for each open ball Bε(f (x)) of radius ε 0, there exists an open ball Bδ(x)(x) in Rn such that Bδ(x)(x) ⊂ f −1{Bε(f (x))}. This yields the following equivalent criterion in terms of neighborhoods. Theorem 4.5. Let f : Rn → Rm for two integers n ≥ 1 and m ≥ 1. The function f is continuous at x ∈ Rn if and only if for each neighborhood W of f (x) in Rm , f −1{W} is a neighborhood of x in Rn. Proof. For any neighborhood W of f (x), there exists ε 0 such that Bε(f (x)) ⊂ W. If f is continuous at x, then, by definition, there exists δ(x) 0 such that Bδ(x))(x) ⊂ f −1{Bε(f (x))}. Since f −1{Bε(f (x))} ⊂ f −1{W}, f −1{Bε(f (x))} is indeed a neighbor- hood of x by definition. Conversely, for all ε 0, the open ball Bε(f (x)) is a neighborhood of f (x). Then f −1{Bε(f (x))} is a neighborhood of x. So there exists an open ball Bδ(x)(x) of radius δ(x) 0 such that Bδ(x)(x) ⊂ f −1{Bε(f (x))}. Hence we get the ε-δ definition of the continuity of f at x. Theorem 4.6. A linear function A : Rn → Rm is continuous on Rn. Proof. It is sufficient to prove it for a linear function A : Rn → R. Any point x = (x1,...,xn) in Rn can be written x = n i=1 xi ei and, by linearity of A, Ax = n i=1 xi Aei = x ·g, g def = ⎡ ⎢ ⎣ Ae1 . . . Aen ⎤ ⎥ ⎦. The vector g is unique. For any ε 0, choose δ = ε/( g +1). Thence, ∀y, y −x δ ⇒ |Ay −Ax| = |A(y −x)| =|g ·(y −x)| ≤ g y −x g δ = g g +1 ε ε. Therefore, A is continuous on Rn.
- 29. ! 1 ! 0.8 ! 0.6 ! 0.4 ! 0.2 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 10 0 10 2 10 4 10 6 10 8 10 10 Chapter 2 Existence, Convexities, and Convexification 1 Introduction In this chapter, Rn will be the Cartesian product endowed with the scalar product and the norm (4.2) of Chapter 1, f : Rn → R or R∪{+∞} an objective function, and U a nonempty subset of Rn. The Weierstrass theorem provides conditions on U and f for the existence of points in U achieving both the infimum inf f (U) and the supremum supf (U): compactness of U and continuity of f on U. In fact, it is sufficient to consider the minimization problem since the one of maximization can be obtained by minimizing the negative of the objective function. By restricting our analysis to the infimum, the class of objective functions can be enlarged to functions f : Rn → R∪{+∞} and the continuity on U can be relaxed to the weaker notion of lower semicontinuity that includes many discontinuous functions. Growth conditions at infinity will complete the results when U is closed but not bounded. In the absence of compactness, we also give Ekeland’s variational principle and some of its ramifications such as the existence theorem of Takahashi and the fixed point theorem of Caristi. All results are true in finite-dimensional vector spaces and basic ideas and constructions generalize to function spaces. Thelastpartofthechapterisdevotedtoconvexity thatplaysaspecialroleinthecontext of a minimization problem. If, in addition to existence, the convexity of f is strict, then the minimizing point is unique. For convex objective functions all local infima are global and hence equal. This suggests convexifying the objective function and finding the infimum of the convexified function rather than the global infimum of the original function that can have several local infima. This leads to the work of Legendre, Fenchel, and Rockafellar, the introduction of the Fenchel–Legendre transform, the primal and dual problems, and the Fenchel duality theorem that will be seen again in Chapter 4 in the context of linear and quadratic programming. 2 Weierstrass Existence Theorem The fact that inf f (U) is finite does not guarantee the existence of a point a ∈ U that achieves the infimium, f (a) = inf f (U), as illustrated in the following example. 11
- 30. 12 Chapter 2. Existence, Convexities, and Convexification Example 2.1. Let U = R and consider the function f (x) = 1 if x ≤ 0 and f (x) = x if x 0 for which inf f (U) = 0 and f (x) = 0 for all x ∈ U = R. The Weierstrass1 theorem that will be proved later as Theorem 5.1 is fundamental in optimization. It gives sufficient conditions on U and f for the existence of minimizers and maximizers in U. Theorem 2.1 (Weierstrass). Given a compact nonempty subset U of Rn and a real-valued function f : U → R that is continuous at U,2 (i) ∃a ∈ U such that f (a) = supx∈U f (x), (ii) ∃b ∈ U such that f (b) = infx∈U f (x). But it is a little too strong since it gives the existence of both minimizing and maxi- mizing points. Indeed, it is not necessary to simultaneously seek the existence of the two types of points since a supremum can always be formulated as an infimum of the negative of the function and vice versa: supf (U) = −inf −f (U). It will be sufficient to find con- ditions for the existence of a minimizing point of f in U. In so doing, it will be possible to weaken the continuity assumption that is clearly not necessary for the piecewise continuous functions of Figure 2.1 that reach a minimum at a point of [0,1]. Notice that at the points of discontinuity, we have chosen to give the function the lower value and not the upper value that would not have resulted in the existence of a minimizing point. 0 1 0 1 Figure 2.1. Discontinuous functions having a minimizing point in [0,1]. 3 Extrema of Functions with Extended Values The inf f (U) and the supf (U) have been defined for real-valued functions, that is, f (U) ⊂ R. When the set f (U) is unbounded below, inf f (U) = −∞ and when f (U) is unbounded above, supf (U) = +∞ (see Definition 4.1 and Remark 4.1 of Chapter 1). 1Karl Theodor Wilhelm Weierstrass (1815–1897). 2We can also work with an f : Rn → R continuous on U since, for U closed in Rn, any f : U → R continuous on U for the relative topology can be extended to a continuous function on Rn.
- 31. 3. Extrema of Functions with Extended Values 13 The idea of extended real-valued objective functions implicitly having effective do- mains is due to R. T. Rockafellar3 and J. J. Moreau.4 In order to consider functions f : Rn → R = R∪{±∞} that are possibly equal to +∞ or −∞ at some points, the defini- tions of the inf f (U) and the supf (U) have to be extended. Definition 3.1. Let f : Rn → R and U ⊂ Rn. (i) Associate with f its effective domain dom f def = x ∈ Rn : −∞ f (x) +∞ . (3.1) It will also be simply referred to as the domain of f . (ii) The infimum of f with respect to U is defined as follows: inf f (U) def = ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ +∞, if U = ∅, inf f (U), if U = ∅ and f (U) ⊂ R, +∞, if U = ∅ and ∀x ∈ U, f (x) = +∞, −∞, if ∃x ∈ U such that f (x) = −∞. We shall also use the notation infx∈U f (x). The supremum of f with respect to U is defined as follows: supf (U) def = ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ −∞, if U = ∅, supf (U), if U = ∅ and f (U) ⊂ R, −∞, if U = ∅ and ∀x ∈ U, f (x) = −∞, +∞, if ∃x ∈ U such that f (x) = +∞. We shall also use the notation supx∈U f (x). Infima and suprema constitute the set of extrema of f in U. (iii) When there exists a ∈ U such that f (a) = inf f (U), f is said to reach its minimum at a point of U and it is written as minf (U) or min x∈U f (x). 3Ralph Tyrrell Rockafellar (1935– ). “Moreau and I independently in those days at first, but soon in close exchanges with each other, made the crucial changes in outlook which, I believe, created convex analysis out of convexity. For instance, he and I passed from the basic objects in Fenchel’s work, which were pairs consisting of a convex set and a finite convex function on that set, to extended real-valued functions implicitly having effective domains, for which we moreover introduced set-valued subgradient mappings.” R. T. Rockafellar, http://www.convexoptimization.com/wikimization/index.php/Rockafellar. 4Jean Jacques Moreau (1923– ) “. . . appears as a rightful heir to the founders of differential calculus and mechanics through the depth of his thinking in the field of nonsmooth mechanics and the size of his contribution to the development of nonsmooth analysis. His interest in mechanics has focused on a wide variety of subjects: singularities in fluid flows, the initiation of cavitation, plasticity, and the statics and dynamics of granular media. Allied to this is his investment in mathematics in the fields of convex analysis, calculus of variations and differential measures” (see P. Alart, O. Maisonneuve, and R. T. Rockafellar [1]).
- 32. 14 Chapter 2. Existence, Convexities, and Convexification The set of all minimizing points of f in U is denoted argminf (U) def = {a ∈ U : f (a) = inf f (U)}. (3.2) When there exists b ∈ U such that f (b) = supf (U), f is said to reach its maximum at a point of U, and it is written as maxf (U) or max x∈U f (x). The set of all maximizing points of f in U is denoted argmaxf (U) def = {b ∈ U : f (b) = supf (U)}. (3.3) With the above extensions, supf (U) can still be replaced by −inf(−f (U)) and vice versa as can be seen from the next theorem. Theorem 3.1. Let f : Rn → R∪{±∞} and U ⊂ Rn. supf (U) = −inf(−f )(U) and argmaxf (U) = argmin(−f )(U). (3.4) Proof. If U = ∅, then f (U) = ∅. By convention, supf (U) = −∞ and inf(−f )(U) = +∞. Hence supf (U) = −∞ = −inf(−f )(U). Assume now that U = ∅. First eliminate the trivial cases. If there exists x ∈ U such that f (x) = +∞, then supf (U) = +∞, −f (x) = −∞, and inf −f (U) = −∞. The second case is f (x) = −∞ for all x ∈ U which implies that supf (U) = −∞ and, for all x ∈ U, −f (x) = +∞ and inf −f (U) = +∞. The last case is U = ∅ and f : Rn → R∪{−∞} for which there exists x ∈ U such that f (x) −∞. Therefore supf (U) −∞. (i) Assume that b0 = supf (U) ∈ R. From Definition 4.1 of Chapter 1, b0 is an upper bound, that is, for all x ∈ U, f (x) ≤ b0, and all upper bounds M of f (U) are such that b0 ≤ M. Therefore for all x ∈ U, −f (x) ≥ −b0 and −b0 is a lower bound of −f (U) = {−f (x) : x ∈ U}. Let m be a lower bound of −f (U). Then, −m is an upper bound of f (U) and since b0 is the least upper bound, b0 ≤ −m. Thence, −b0 ≥ m and −b0 is the largest lower bound of −f (U). This yields inf −f (U) = −b0 = −supf (U) and −inf −f (U) = b0 = supf (U). (ii) By convention in Remark 4.1 of Chapter 1, the case b0 = supf (U) = +∞ corre- sponds to f (U) not bounded above. Therefore, there exists a sequence {xn} ⊂ U such that f (xn) → +∞. This implies that −f (xn) → −∞ and hence the set −f (U) is not bounded below. By convention, inf −f (U) = −∞ and supf (U) = +∞ = −inf −f (U). Introducing objective functions with values ±∞ makes it possible to replace an infi- mum of f with respect to U by an infimum over all of Rn by introducing the new function x → fU (x) def = f (x), if x ∈ U +∞, if x ∈ Rn U : Rn → R∪{±∞}. (3.5)
- 33. 3. Extrema of Functions with Extended Values 15 Theorem 3.2. Let U, ∅ = U ⊂ Rn and f : Rn → R∪{±∞}. Then inf f (U) = inf fU (U) = inf fU (Rn ) and argminf (U) = U ∩argminfU (Rn ). If, in addition, inf(U) +∞, then argminf (U) = argminfU (Rn). Proof. By definition, f (x) ≤ fU (x) for all x ∈ Rn and inf f (U) = inf fU (U) ≥ inf fU (Rn ). If inf fU (Rn) = +∞, we trivially have equality. If m def = inf fU (Rn) ∈ R, then for all n ∈ N, there exists xn ∈ Rn such that m+ 1 n fU (xn) ≥ m. Since fU (xn) is finite, xn ∈ U and fU (xn) = f (xn). Thence inf f (U) ≥ m f (xn)− 1 n ≥ inf f (U)− 1 n . By letting n go to infinity, we get equality. As inf f (U) = inf fU (U) = inf fU (Rn) and U ⊂ Rn, we have argminf (U) = argminfU (U) ⊂ argminfU (Rn). If, in addition, inf f (U) +∞, then, by definition of fU , argminfU (Rn) ⊂ argminfU (U) = argminf (U). Remark 3.1. For a supremum, extend f by −∞ by considering the function f U (x) def = f (x), if x ∈ U −∞, if x ∈ Rn U = −(−f )U (x). (3.6) For an infimum, two cases are trivial: (i) there exists x ∈ U such that f (x) = −∞ that yields inf fU (Rn) = inf f (U) = −∞ and x ∈ argminf (U); (ii) for all x ∈ U, f (x) = +∞ that yields inf fU (Rn) = inf f (U) = +∞ and argminf (U) = U. In order to exclude cases (i) and (ii) for the infimum, we introduce the notion of proper function for an infimum that is the natural extension of the notion of proper function for convex functions (see R. T. Rockafellar [1]). The dual notion of proper function for a supremum of f is obtained by considering the notion of proper function for the infimum of −f . Definition 3.2. Let f : Rn → R∪{±∞}. (i) f is said to be proper for the infimum if
- 34. 16 Chapter 2. Existence, Convexities, and Convexification (a) for all x ∈ Rn, f (x) −∞ and (b) there exists x ∈ Rn such that f (x) +∞. This is equivalent to f : Rn → R∪{+∞} and dom f = ∅. (ii) f is said to be proper for the supremum if (a) for all x ∈ Rn, f (x) +∞ and (b) there exists x ∈ Rn such that f (x) −∞. This is equivalent to f : Rn → R∪{−∞} and dom f = ∅. Whenever no confusion arises, we shall simply say that the function is proper. Another trivial case occurs when dom f is a singleton; that is, there is only one point where the function f is finite. 4 Lower and Upper Semicontinuities In order to consider the infimum of a discontinuous function, we weaken the notion of continuity by breaking it into two weaker notions. Recall that a real-valued function f : Rn → R is continuous at a ∈ Rn if ∀ε 0, ∃δ 0 such that ∀x ∈ Bδ(a), |f (x)−f (a)| ε. (4.1) The open ball Bδ(a) of radius δ 0 is a neighborhood of a. Letting V (a) = Bδ(a), the condition on f yields the following two conditions: ∀x ∈ V (a), −ε f (x)−f (a) ⇒ f (a)−ε f (x) ∀x ∈ V (a), f (x)−f (a) ε ⇒ f (x) f (a)+ε. (4.2) The first condition says that f (a) is below all limit points of f (x) as x goes to a, while the second one says that f (a) is above, thus yielding the decomposition of the continuity into lower semicontinuity and upper semicontinuity. Definition 4.1. (i) f : Rn → R∪{+∞} is lower semicontinuous at a ∈ Rn if ∀h f (a), ∃ a neighborhood V (a) of a such that ∀x ∈ V (a), h f (x). (4.3) f is lower semicontinuous on U ⊂ Rn if it is lower semicontinuous at every point of U. By convention, the function identically equal to −∞ is lower semicontinuous. (ii) f : Rn → R∪{−∞} is upper semicontinuous at a ∈ Rn if ∀k f (a), ∃ a neighborhood V (a) of a such that ∀x ∈ V (a), k f (x). (4.4) f is upper semicontinuous on U ⊂ Rn if it is upper semicontinuous at every point of U. By convention, the function identically equal to +∞ is upper semicontinuous. In short, we shall write lsc for lower semicontinuous and usc for upper semicontinuous.
- 35. 4. Lower and Upper Semicontinuities 17 Functions of Figure 2.1 are lower semicontinuous (lsc) in ]0,1[. The function iden- tically equal to +∞ is lsc and the one identically equal to −∞ is usc on Rn. As we have seen before, definition (4.1) of the continuity of a function f : Rn → R at a point a ∈ Rn is equivalent to the two conditions (4.2): the first one is the lower semicontinuity at a with h = f (a) − ε f (a) and the second one is the upper semicontinuity at a with k = f (a)+ε f (a). As for the infimum and the supremum where supf (U) = −inf −f (U), f is usc at x if and only if −f is lsc at x. So it is sufficient to study the properties of lsc functions. Theorem 4.1. (i) f : Rn → R∪{−∞} is usc at x if and only if −f : Rn → R∪{+∞} is lsc at x. (ii) f : Rn → R∪{+∞} is lsc at x if and only if −f : Rn → R∪{−∞} is usc at x. Proof. As f : Rn → R∪{−∞}, then −f : Rn → R∪{+∞}. Given h −f (x), then f (x) −h. As f is usc at x, there exists a neighborhood V (x) of x such that for all y ∈ V (x), f (y) −h. As a result, for all y ∈ V (x), −f (y) h. By definition, −f is lsc at x. The proof of the converse is similar. It is easy to check the following properties of lsc functions (see Exercises 10.1 to 10.4) by using the convention (+∞)+(+∞) = +∞, (+∞)+a = +∞ for all a ∈ R, and (+∞)a = (a/ a )∞ for all a ∈ R not equal to 0. Theorem 4.2. (i) For all f : Rn → R∪{+∞} and g : Rn → R∪{+∞} lsc at a ∈ Rn, the function (f +g)(x) def = f (x)+g(x), ∀x ∈ Rn , is lsc at a. (ii) For all λ ≥ 0 and f : Rn → R∪{+∞} lsc at a ∈ Rn, the function (λf )(x) def = λf (x), if λ 0 0, if λ = 0 , ∀x ∈ Rn , is lsc at a. (iii) Given a family {fα}α∈A (where A is an index set possibly infinite) of functions fα : Rn → R∪{+∞} lsc at a ∈ Rn, the upper envelope sup α∈A fα (x) def = sup α∈A fα(x), x ∈ Rn , is lsc at a ∈ Rn. (iv) Given a finite family fi : Rn → R∪{+∞}, 1 ≤ i ≤ m, of functions lsc at a ∈ Rn, the lower envelope min 1≤i≤m fi (x) def = min 1≤i≤m fi(x), x ∈ Rn , is lsc at a ∈ Rn.
- 36. 18 Chapter 2. Existence, Convexities, and Convexification (v) Given a function f : Rn → R and a point a ∈ Rn, f is continuous at a ⇐⇒ f is lsc and usc at a. (vi) Given a linear map A : Rm → Rn and a function f : Rn → R∪{+∞} lsc at Ax, then f ◦A : Rm → R∪{+∞} is lsc at x. Property (iv) is not necessarily true for the lower envelope of an infinite number of lsc functions as can be seen from the following example. Example 4.1. Define for each integer k ≥ 1, the continuous function fk(x) def = ⎧ ⎪ ⎨ ⎪ ⎩ 1, if x ∈ [0,1], 1−k(x −1), if x ∈ [1,1+1/k], 0, if x ∈ [1+1/k,2]. It is easy to check that inf k≥1 fk(x) = 1, if x ∈ [0,1], 0, if x ∈]1,2], is usc but not lsc at x = 1. The lower semicontinuity (resp., upper semicontinuity) can also be characterized in terms of the liminf (resp., limsup). Definition 4.2. Given a function f : Rn → R∪{+∞} (resp., f : Rn → R∪{−∞}), liminf x→a f (x) def = sup ε0 inf x=a x−a ε f (x) ⎛ ⎜ ⎝resp., limsup x→a f (x) def = inf ε0 sup x=a x−a ε f (x) ⎞ ⎟ ⎠. Theorem 4.3. f : Rn → R∪{+∞} (resp., f : Rn → R∪{−∞}) is lsc (resp., usc) at a if and only if liminf x→a f (x) ≥ f (a) resp., limsup x→a f (x) ≤ f (a) . Proof. (⇒) If f is lsc at a, for all h f (a), there exists a neighborhood V (a) of a such that for all x ∈ V (a), f (x) h. As V (a) is a neighborhood of a, there exists a ball Bε(a), ε 0, such that Bε(a) ⊂ V (a) and ∀x ∈ Bε(a), f (x) h ⇒ inf x∈Bε(a) x=a f (x) ≥ h ⇒ sup ε0 inf x∈Bε(a) x=a f (x) ≥ h. Since the inequality is true for all h f (a), letting h go to f (a), we get liminf x→a f (x) ≥ h ⇒ liminf x→a f (x) ≥ f (a).
- 37. 4. Lower and Upper Semicontinuities 19 (⇐) For all h such that f (a) h, by assumption, liminf x→a f (x) = sup ε0 ⎡ ⎣ inf x∈Bε(a) x=a f (x) ⎤ ⎦ ≥ f (a) h. By definition of the sup, for that h, there exists ε 0 such that sup ε0 ⎡ ⎣ inf x∈Bε(a) x=a f (x) ⎤ ⎦ ≥ inf x∈Bε(a) x=a f (x) h ⇒ ∀x ∈ Bε(a), f (x) h. As Bε(a) is a neighborhood of a, f is lsc at a. We have as a corollary the following characterization of the epigraph. Lemma 4.1. f : Rn → R∪{+∞} is lsc on Rn if and only if the epigraph of f , epif def = (x,µ) ∈ Rn ×R : x ∈ dom f and µ ≥ f (x) , (4.5) is closed in Rn ×R. The epigraph epif is nonempty if and only if dom f = ∅, that is, when f is proper for the infimum. Remark 4.1. The effective domain dom f of an lsc function is not necessarily closed, as can be seen from the example of the function f (x) = 1/|x| if x = 0 and +∞ if x = 0, where dom f = R{0}. Proof. If f is lsc on Rn, consider a Cauchy sequence (xn,µn) ∈ epif . By definition, µn ≥ f (xn) and there exists (x,µ) ∈ Rn × R such that xn → x and µn → µ. As f is lsc on Rn, µ = lim n→∞ µn = liminf n→∞ µn ≥ liminf n→∞ f (xn) ≥ f (x) and (x,µ) ∈ epif . Hence the epigraph of f is closed in Rn × R. Conversely, assume that epif is closed in Rn × R. Let x ∈ Rn and h f (x). Then the point (x,h) / ∈ epif . Therefore, there exists a neighborhood W(x,h) such that W(x,h)∩epif = ∅. In particular, there exists a neighborhood V (x) of x such that V (x) × {h} ⊂ W(x,h) and hence for all y ∈ V (x), f (y) h and f is lsc on Rn. We now give a few characterizations of lower semicontinuity in preparation of the proof of Theorem 5.1. Lemma 4.2. f : Rn → R∪{+∞} is lsc at a ∈ Rn if and only if ∀h f (a), Gh def = {x ∈ Rn : f (x) h} (4.6) is a neighborhood of a (see Figure 2.2). Proof. The proof is by definition.
- 38. 20 Chapter 2. Existence, Convexities, and Convexification f (x) h f (a) a x Gh Figure 2.2. Example of an lsc function. Lemma 4.3. Let f : Rn → R∪{+∞}. The following conditions are equivalent: (i) f is lsc on Rn; (ii) ∀k ∈ R, Gk = {x ∈ Rn : f (x) k} is open in Rn; (iii) ∀k ∈ R, Fk = {x ∈ Rn : f (x) ≤ k} is closed in Rn. Proof. (i) ⇒ (ii). If Gk = ∅, Gk is open. If Gk = ∅, for all a ∈ Gk, f (a) k and, as f is lsc at a, there exists a neighborhood V (a) of a such that ∀x ∈ V (a), f (x) k ⇒ V (a) ⊂ Gk. Therefore a ∈ intGk and Gk is open. (ii) ⇒ (i). By definition, Rn = ∪k∈RGk and for each a ∈ Rn, there exists k ∈ R such that k f (a). In particular, a ∈ Gk = ∅. As Gk is open by assumption, Gk is a neighborhood of a. Finally, by definition of Gk, for all x ∈ Gk, f (x) k. Choose V (a) = Gk and, always by definition, f is lsc at a and hence on Rn. (ii) ⇐⇒ (iii) is obvious. To use the above lemmas for a function f : U → R∪{+∞} that is lsc only on the subset U of Rn for the relative topology on U requires the following lemma. Lemma 4.4. Let U, ∅ = U ⊂ Rn, be closed and let f : U → R∪{+∞} be lsc for the relative topology on U. The function fU is lsc on Rn. Proof. Given a ∈ RnU, fU is lsc on RnU. Indeed V (a) = RnU is a nonempty open set containing a since U is closed. So it is a neighborhood of a. For all h f (a) = +∞, ∀x ∈ V (a), h +∞ = f (x)
- 39. 5. Existence of Minimizers in U 21 and, by definition, fU is lsc on RnU. As for all a ∈ U, fU (a) = f (a) and f is lsc on U. So there exists a neighborhood V (a) of a in Rn such that ∀x ∈ V (a)∩U, h f (x) ⇒ ∀x ∈ V (a), h f (x) ≤ fU (x), since, by construction, fU (x) = +∞ in Rn U. So fU is also lsc on U. Example 4.2. The indicator function of a closed subset U of Rn, IU (x) def = 0, if x ∈ U, +∞, if x / ∈ U, is lsc on Rn. In fact, IU = fU for the function x → f (x) = 0 : Rn → R. To complete this section, we introduce the “lower semicontinuous hull” cl f of a function f , in the terminology of R. T. Rockafellar [1], that corresponds to the lower semicontinuous regularization of f in I. Ekeland and R. Temam [1]. Definition 4.3. (i) The lsc regularization of a function f : Rn → R∪{+∞} is defined as the upper envelope of all lsc functions less than or equal to f : cl f (x) def = sup g lsc and g≤f on Rn g(x). (4.7) If there exists g lsc on Rn such that g ≤ f on Rn, then cl f is lsc on Rn. Otherwise, set cl f (x) = −∞, by convention. (ii) The usc regularization of a function f : Rn → R∪{−∞} is defined as the lower envelope of all usc functions greater than or equal to f : cl uscf (x) def = inf g usc and f ≤g on Rn g(x). (4.8) If there exists g usc on Rn such that f ≤ g on Rn, then cl uscf is usc on Rn. Otherwise, set cl uscf (x) = +∞, by convention. Note that the definition of the usc regularization amounts to cl uscf = −cl (−f ). 5 Existence of Minimizers in U 5.1 U Compact WecannowweakentheassumptionsoftheWeierstrasstheorem(Theorem 2.1)byseparating infimum problems from supremum problems.
- 40. 22 Chapter 2. Existence, Convexities, and Convexification Theorem 5.1. Let U, ∅ = U ⊂ Rn, be compact. (i) If f : U → R∪{+∞} is lsc on U, then ∃a ∈ U such that f (a) = inf f (U). (5.1) If U ∩dom f = ∅, then inf f (U) ∈ R. (ii) If f : U → R∪{−∞} is usc on U, then ∃b ∈ U such that f (b) = sup f (U). (5.2) If U ∩dom f = ∅, then supf (U) ∈ R. As U is compact, it is closed and, by Lemma 4.4, fU is lsc on Rn without changing the infimum since inf f (U) = inf fU (Rn) by Theorem 3.2. So we could work with an lsc function f : Rn → R∪{+∞}. Similarly, for f : Rn → R∪{−∞} usc on closed U, the function f U defined in (3.6) of Remark 3.1 is usc on Rn. Proof of Theorem 5.1. Let m = inf f (U). As U is compact, it is closed. By Lemma 4.4, the function fU associated with f defined by (3.5) is lsc on Rn and by Theorem 3.2 we have m = inf f (U) = inf fU (U) = inf fU (Rn). If m = +∞, then f is identically +∞ on U and all points of U are minimizers. If m +∞, then for all reals k m, the set Fk = {x ∈ U : f (x) ≤ k} = {x ∈ Rn : fU (x) ≤ k} is closed by the lower semicontinuity of fU on Rn (Lemma 4.3). It is also nonempty since, by definition of the inf, for all k such that m k, there exists f (x) ∈ f (U) such that m = inf f (U) ≤ f (x) k. Since U is compact, the closed subsets Fk ⊂ U are also compact. By definition of the Fk’s, m k1 ≤ k2 =⇒ Fk1 ⊂ Fk2 and hence any finite family of sets Fk has a nonempty intersection. We claim that ∩km Fk = ∅. By contradiction, if the intersection is empty, then for any K m, FK ∩ ∩mk k=K Fk = ∅ ⇒ FK ⊂ ∩mk k=K Fk = ∪mk k=K Fk. Therefore, {Fk : m k and k = K} is an open cover of the compact FK. So, there exists a finite subcover of FK: FK ⊂ ∪m j=1Fkj = ∩m j=1 Fkj ⇒ FK ∩Fk1 ∩···∩Fkm = ∅. This contradicts the nonempty finite intersection property. So any point a ∈ ∩km Fk ⊂ U, belongs to U and ∀k m, f (a) ≤ k ⇒ f (a) ≤ m = inf x∈U f (x) ≤ f (a). Hence a ∈ U is a minimizer and argminf (U) = ∩km Fk.
- 41. 5. Existence of Minimizers in U 23 | | −1 0 1 1 0 Figure 2.3. Lsc function that is not usc at 0. In general, the infimum over U cannot be replaced by the infimum over U even if f is lsc. We only have inf f (U) ≥ inf f (U) as seen from the following example. Example 5.1 (see Figure 2.3). Consider U = [−1,1]{0} and the lsc function f (x) = 1, if x = 0, 0, if x = 0. Then U = [−1,1] and inf f (U) = 1 0 = inf f (U). The function f is not usc at 0 since for 1/2 0 = f (0), the set {x ∈ [−1,1] : f (x) 1/2} = {0} is not a neighborhood of 0. However, we have the following sufficient condition. Theorem 5.2. Let U, ∅ = U ⊂ Rn and let f : Rn → R be usc on U. Then inf f (U) = inf f (U). Proof. As U ⊂ U, we have inf f (U) ≤ inf f (U). As U = ∅, both inf f (U) and inf f (U) are bounded above. If inf f (U) = −∞, then inf f (U) = −∞ and the equality is verified. If inf f (U) is finite, assume that inf f (U) inf f (U).
- 42. 24 Chapter 2. Existence, Convexities, and Convexification By definition of inf f (U), there exists x0 ∈ U such that inf f (U) ≤ f (x0) inf f (U). As f is usc, there exists a neighborhood V (x0) of x0 such that ∀x ∈ V (x0), f (x) inf f (U). But x0 ∈ U is a limit point of U for which V (x0)∩U = ∅. Therefore, there exists u ∈ U such that f (u) inf f (U). This contradicts the definition of inf f (U). 5.2 U Closed but not Necessarily Bounded By Theorem 5.1, the functions of Figure 2.1 have minimizers at least at one point of the compact subset U = [0,1] of R. However, in its present form, this theorem is a little restrictive since it does not apply to the following simple example: inf f (Rn ), f (x) def = x 2 , x ∈ Rn . The difficulty arises from the fact that, as Rn is not bounded, it is not compact. Remark 5.1. Going over the proof of the theorem, it is readily seen that the compactness of U is not really necessary. Since the family {Fk : k m} of closed subsets of U is an “increasing sequence” Fk1 ⊂ Fk2 ⊂ U, ∀k2 ≥ k1 m, then, for each k̄ m, ∩k̄≥kmFk = ∩kmFk. So it is sufficient to find some k̄ ∈ R for which the lower section Fk̄ = {x ∈ U : f (x) ≤ k̄} is nonempty and bounded (hence compact5) instead of making the assumption on U. Definition 5.1. Let U, ∅ = U ⊂ Rn . (i) f : Rn → R∪{+∞} has a bounded lower section in U if there exists k ∈ R such that the lower section Fk = {x ∈ U : f (x) ≤ k} (5.3) is nonempty and bounded. (ii) f : Rn → R∪{−∞} has a bounded upper section in U if there exists k ∈ R such that the upper section Fk = {x ∈ U : f (x) ≥ k} (5.4) is nonempty and bounded. 5InfinitedimensionasetiscompactifandonlyifitisclosedandboundedbyHeine–Boreltheorem(Theorem 4.2 of Chapter 1).
- 43. 5. Existence of Minimizers in U 25 When U is a nonempty compact subset of Rn (that is, bounded and closed), any function f proper for the infimum has a bounded lower section in U. Theorem 5.3. Let U, ∅ = U ⊂ Rn , be closed. (i) If f : U → R∪{+∞} is lsc on U with a bounded lower section in U, then ∃a ∈ U such that f (a) = inf x∈U f (x) ∈ R. (5.5) (ii) If f : U → R∪{−∞} is usc on U with a bounded upper section in U, then ∃b ∈ U such that f (b) = sup x∈U f (x) ∈ R. (5.6) Example 5.2 (distance function). Let U, ∅ = U ⊂ Rn , be closed. Given x ∈ Rn, there exists x̂ ∈ U such that dU (x) def = inf y∈U x −y = x −x̂ (5.7) and ∀x1, x2 ∈ Rn , |dU (x2)−dU (x1)| ≤ x2 −x1 . (5.8) To show this, consider the infimum inf y∈U f (y), f (y) def = y −x 2 . The function f is continuous and hence lsc on Rn . For any y0 ∈ U and k = y0 −x 2, the lower section Fk def = ! y ∈ U : y −x 2 ≤ y0 −x 2 is not empty, since y0 ∈ Fk, and bounded since ∀y ∈ Fk, y ≤ x + y −x ≤ x + y0 −x ≤ x + √ k. f has a bounded lower section in U. By Theorem 5.3(i), there exists a minimizer x̂ ∈ U. For all y ∈ U, x2 −y ≤ x1 −y + x2 −x1 ⇒ inf y∈U x2 −y ≤ inf y∈U x1 −y + x2 −x1 ⇒ ∀x1, x2 ∈ Rn , dU (x2)−dU (x1) ≤ x2 −x1 . By interchanging the roles of x1 and x2, we get |dU (x2)−dU (x1)| ≤ x2 −x1 . Example 5.3 (distance function). Let U, ∅ = U ⊂ Rn (not necessarily closed) and x ∈ Rn . As in the previous example, define dU (x) def = inf y∈U x −y . (5.9) As U ⊂ U, then dU (x) ≥ dU (x). However, since the function y → y −x is continuous, it is usc and by Theorem 5.2, dU (x) = dU (x).
- 44. 26 Chapter 2. Existence, Convexities, and Convexification 5.3 Growth Property at Infinity A simple condition to ensure that a function has a bounded lower section in an unbounded U is that the function goes to +∞ as the norm of x tends to infinity. Definition 5.2. Let U, ∅ = U ⊂ Rn , be unbounded in Rn. The function f : Rn → R∪{+∞} has the growth property in U if lim x∈U, x →∞ f (x) = +∞. Theorem 5.4. Let U, ∅ = U ⊂ Rn , be unbounded. If f : Rn → R∪{+∞} has the growth property in U, then it has a nonempty lower section in U. Proof. We show that there exists a k ∈ R such that the lower section Fk = {x ∈ U : f (x) ≤ k} is nonempty and bounded. By definition, U = ∪k∈RFk and as U = ∅, ∃k ∈ R such that Fk = ∅. By the growth property in U, ∃R(k) 0 such that ∀x ∈ U and x R(k), f (x) k. As a result, Fk = {x ∈ U : f (x) ≤ k} ⊂ {x ∈ U : x ≤ R(k)} and Fk is nonempty and bounded. Consider a few generic examples. Example 5.4. The functions f (x) = |x| and f (x) = x2 have the growth property in R. Example 5.5. The function f (x) = x −b does not have the growth property in R. Pick a sequence xn = −n of positive integers n going to infinity. Example 5.6. The function f (x) = sin x +(1+x)2 has the growth property in R. Indeed f (x) ≥ −1+(1+x)2 = x2 −2x → +∞ as |x| → ∞. Example 5.7. Consider the function f (x1,x2) = (x1 +x2)2 . f does not have the growth property in R2 : pick the sequence {(n,−n)}, n ≥ 1, f (n,−n) = (n−n)2 = 0 → +∞. However, f has the growth property in U = {(x1,0) : x1 ∈ R}
- 45. 5. Existence of Minimizers in U 27 since f (x) = x2 1 → +∞ as |x1| goes to +∞ in U. Theorem 5.5. Given a ∈ Rn, the functions x → f (x) = x −a p, p ≥ 1, have the growth property in Rn. Proof. For all x = 0, f (x) = x p # # # # x x − a x # # # # p . As x → ∞, the term # # # # x x | − a x # # # # converges to 1 and its pth power also converges to 1. For p ≥ 1, x p → +∞ as x goes to +∞. The limit of f (x) is the product of the two limits. In the next example we use the following technical results. Recall that an n×n matrix is symmetric if A = A, where A denotes the matrix transpose6 of a matrix A. Definition 5.3. A symmetric matrix A is positive definite (resp., positive semidefinite) if ∀x ∈ Rn , x = 0, (Ax)·x 0 (resp., ∀x ∈ Rn , (Ax)·x ≥ 0). This property will be denoted A 0 (resp., A ≥ 0). Lemma 5.1. A symmetric matrix A is positive definite if and only if ∃α 0, ∀x ∈ Rn , (Ax)·x ≥ α x 2 . If A 0, the inverse A−1 exists. Proof. (⇐) If there exists α 0 such that (Ax)·x ≥ α x 2 for all x ∈ Rn , then ∀x ∈ Rn , x = 0, (Ax)·x ≥ α x 2 0 and A 0. (⇒) Conversely, if A 0, then ∀x ∈ Rn , (Ax)·x ≥ 0. Assume that there exists no α 0 such that ∀x ∈ Rn , (Ax)·x ≥ α x 2 . So for each integer k 0, there exists xk such that 0 ≤ Axk ·xk 1 k xk 2 ⇒ xk = 0. 6The notation A will be used for both the linear map A : Rn → Rm and its associated m×n matrix. Similarly, the notation A will be used for both the linear map A : Rm → Rn and its associated n×m matrix.
- 46. 28 Chapter 2. Existence, Convexities, and Convexification By dividing by xk 2 we get 0 ≤ A xk xk · xk xk 1 k ⇒ lim k→∞ A xk xk · xk xk = 0. The points sk = xk/ xk belong to the sphere S = {x ∈ Rn : x = 1} which is compact in Rn. By the theorem of Bolzano–Weierstrass (see Chapter 1, Theorem 4.4), there exists a subsequence {sk } of {sk} that converges to a point s of S: ∃s, s = 1 such that 0 = lim →∞ Ask ·sk = As ·s. Therefore, ∃s = 0 such that As ·s = 0 =⇒ A ≯ 0. This contradicts the assumption thatA 0 and proves the result. To show thatA is invertible, it is sufficient to check that under the assumption A 0, the linear map A : Rn → Rn is injective, that is, Ax = 0 implies x = 0. From the previous result, it is readily seen that Ax = 0 ⇒ 0 = Ax ·x ≥ α x 2 ⇒ x = 0. Example 5.8. Let A 0 be an n × n symmetric matrix. Associate with A and b ∈ Rn the following real-valued function:7 f (x) = 1 2 (Ax)·x +b ·x, x ∈ Rn . By Lemma 5.1 applied to A, there exists α 0 such that ∀x ∈ Rn , Ax ·x ≥ α x 2 and f (x) = 1 2 (Ax)·x +b ·x =⇒ f (x) ≥ 1 2 α x 2 − b x . The function f has the growth property in Rn . Since f is polynomial, it is continuous and as U = Rn is closed, there exists a minimizer. 5.4 Some Properties of the Set of Minimizers Recall that, in the proof of Theorem 5.1 for the infimum of a function f with respect to U, the set of minimizers argminf (U) is given by argminf (U) = $ km Fk, where m = inf f (U) and Fk = {x ∈ U : f (x) ≤ k}. So a number of properties of argminf (U) can be obtained from those of the Fk’s. 7If a vector v ∈ Rn is considered as an n × 1 matrix, the product (Ax) · x can be rewritten as the product of three matrices xAx and b ·x can be written as the product of two matrices bx.
- 47. 6. Ekeland’s Variational Principle 29 Theorem 5.6. Let U, ∅ = U ⊂ Rn, be closed and let f : U → R∪{+∞} be lsc on U. Then argminf (U) is closed (possibly empty). If, in addition, f has a bounded lower section in U, then argminf (U) is compact and nonempty. Proof. Since U is closed and f is lsc, the lower sections {Fk} in (5.3) are closed and argminf (U) is closed as an intersection of closed sets. If, in addition, f has a bounded lower section (there exists k0 such that Fk0 is nonempty and bounded), then the closed set Fk0 is compact. Therefore, the set of minimizers argminf (U) = $ km Fk ⊂ Fk0 is compact as a closed subset of the compact set Fk0 . Example 5.9. Go back to Examples 5.2 and 5.3 for U, ∅ = U ⊂ Rn. Given x ∈ Rn, it was shown that dU (x) = dU (x) and that ∃x̂ ∈ U such that dU (x) def = inf y∈U x −y = x −x̂ . (5.10) To establish that result we have considered the infimum of the square of the distance inf y∈U f (y), f (y) def = y −x 2 for the continuous function f which has a bounded lower section in U. By Theorem 5.6, argminf (U) is nonempty and compact. Points of argminf (U) are projections of x onto U. Denote by U (x) this set. If x ∈ U, then U (x) = {x} is a singleton. 6 Ekeland’s Variational Principle Ekeland’s variational principle8 in 1974 (I. Ekeland [1]) is a basic tool to get existence of approximate minimizers in the absence of compactness. Its major impact is in the context of spaces of functions. In this section we provide the finite-dimensional version and some of its ramifications. Theorem 6.1. Let f : Rn → R∪{+∞}, dom f = ∅, be lsc and bounded below. Then, for any ε 0, there exists xε such that inf x∈Rn f (x) ≤ f (xε) inf x∈Rn f (x)+ε (6.1) and, for any η 0, there exists y such that y −xε η, f (y) ≤ f (xε)− ε η y −xε , (6.2) and ∀x ∈ Rn , x = y, f (y) f (x)+ ε η y −x . (6.3) 8Ivar Ekeland (1944– ).
- 48. 30 Chapter 2. Existence, Convexities, and Convexification This implies that, if f (xε) inf f (Rn), then, in any neighborhood of xε, there exists a point y such that f (y) f (xε). Proof. 9 Under the assumption of the theorem, the infimum is finite and, for any ε 0, there exists xε ∈ dom f such that inf x∈Rn f (x) ≤ f (xε) inf x∈Rn f (x)+ε so that f (xε) is also finite. Given η 0, consider the new lsc function gε/η(y) def = f (y)+ ε η y −xε . The lower section S def = {z ∈ Rn : gε/η(z) ≤ f (xε} is not empty, since xε ∈ S, and bounded since f (z)+ ε η z−xε ≤ f (xε) ⇒ inf x∈Rn f (x)+ ε η z−xε ≤ f (z)+ ε η z−xε ≤ f (xε) ⇒ ε η z−xε ≤ f (xε)− inf x∈Rn f (x) and f (xε)−infx∈Rn f (x) ≥ 0 is finite. By Theorem 5.3(i), the set of minimizers argmingε/η is not empty and by Theorem 5.6, it is compact. Since f is lsc, by Theorem 5.1, there exists y ∈ argmingε/η such that f (y) = inf z∈argmingε/η f (z) ∀x ∈ Rn , f (y)+ ε η y −xε ≤ f (x)+ ε η x −xε . For x = xε, f (y)+ ε η y −xε ≤ f (xε) ⇒ f (y) ≤ f (xε)− ε η y −xε and, from the definition of xε, f (y)+ ε η y −xε ≤ f (xε) inf x∈Rn f (x)+ε ≤ f (y)+ε ⇒ y −xε η. Consider the new function g(z) def = f (z)+ ε η z−y . (6.4) 9From the proof of J. M. Borwein and A. S. Lewis [1, Thm. 7.1.2, pp. 153–154].
- 49. 6. Ekeland’s Variational Principle 31 By definition, infz∈Rn g(z) ≤ f (y) and ∀z ∈ Rn , f (z)+ ε η z−xε ≤ f (z)+ ε η z−y + ε η y −xε ⇒ f (y)++ ε η y −xε = inf z∈Rn % f (z)+ ε η z−xε ≤ inf z∈Rn % f (z)+ ε η z−y + ε η y −xε ⇒ f (y) ≤ inf z∈Rn % f (z)+ ε η z−y ≤ f (y) ⇒ f (y) = inf z∈Rn % f (z)+ ε η z−y . To complete the proof, it remains to show that y is the unique minimizer of g. By construc- tion, y is a global minimizer of f on argmingε/η. So, for all y ∈ argmingε/η, y = y, f (y) ≤ f (y ) f (y )+ ε η y −y and for y / ∈ argmingε/η, f (y)+ ε η y −xε f (y )+ ε η y −xε ≤ f (y )+ ε η y −y + ε η y −xε ⇒ ∀y / ∈ argmingε/η, f (y) f (y )+ ε η y −y . Combining the two inequalities, ∀y ∈ Rn , y = y, f (y) f (y )+ ε η y −y and y is the unique minimizer of g. Ekeland’s variational principle has many interesting ramifications. Theorem 6.2 (W. Takahashi [1]’s existence theorem). Let f : Rn → R∪{+∞}, dom f = ∅, be lsc and bounded below. Assume that there exists c 0 such that for each x for which f (x) inf f (Rn), ∃z = x such that f (z) ≤ f (x)−c z−x . (6.5) Then f has a minimizer in Rn. Proof. It is sufficient to prove the theorem for the function f (x)/c that amounts to set c = 1. By contradiction, assume that for all x, f (x) inf f (Rn). By the hypothesis of the theorem, ∃zx = x such that f (zx) f (x)− zx −x . But, by Ekeland’s variational principle, there exists y such that ∀x ∈ Rn , x = y, f (y) f (x)+ y −x (choose ε = 1/n and η = 1 in the theorem). Since zy = y, f (y) f (zy)+ y −zy and f (zy) ≤ f (y)− zy −y , which yields a contradiction.
- 50. 32 Chapter 2. Existence, Convexities, and Convexification The above two theorems are intimately related to each other and also related to the following fixed point theorem. Theorem 6.3 (J. Caristi [1]’s fixed point theorem).10 Let F : Rn → Rn. If there exists an lsc function f : Rn → R that is bounded below such that ∀x ∈ Rn , x −F(x) ≤ f (x)−f (F(x)), (6.6) then F has a fixed point. Proof. By Theorem 6.1 with ε/η = 1, there exists y such that ∀x ∈ Rn x = y, f (y) f (x)+ x −y . If F(y) = y, then f (y) f (F(y))+ F(y)−y ⇒ f (y)−f (F(y)) F(y)−y . But, by assumption, f (y) − f (F(y)) ≥ y − F(y) , which yields a contradiction. Hence F(y) = y and y is a fixed point of F. Example 6.1. For a contracting mapping F, ∃k, 0 k 1, such that ∀x,y ∈ Rn , F(y)−F(x) ≤ k y −x , choose the function f (x) def = x −F(x) /(1−k) which is continuous and bounded below by 0. Then ∀x ∈ Rn , x −F(x) = (1−k)f (x) = f (x)−k f (x) ∀x ∈ Rn , f (F(x)) = F(x)−F(F(x)) 1−k ≤ k 1−k x −F(x) = k f (x) ⇒ ∀x ∈ Rn , x −F(x) = (1−k)f (x) = f (x)−k f (x) ≤ f (x)−f (F(x)). The assumptions of Caristi’s theorem are verified. This is the Banach fixed point theorem. 7 Convexity, Quasiconvexity, Strict Convexity, and Uniqueness 7.1 Convexity and Concavity Different notions of convexity will be introduced to discuss the convexity of argminf (U) and the uniqueness of minimizers. For the supremum, the dual notion is concavity. As for 10Caristi’s fixed point theorem (also known as the Caristi–Kirk fixed point theorem) generalizes the Banach fixed point theorem for maps of a complete metric space into itself. Caristi’s fixed point theorem is a variation of the variational principle of Ekeland. Moreover, the conclusion of Caristi’s theorem is equivalent to metric completeness, as proved by J. D. Weston [1] (1977). The original result is due to the mathematicians James Caristi and William Arthur Kirk (see J. Caristi and W. A. Kirk [1]).
- 51. 7. Convexity, Quasiconvexity, Strict Convexity, and Uniqueness 33 lsc functions, it is advantageous to extend the notion of convexity to functions with values in R∪{+∞}. Definition 7.1. (i) A subset U of Rn is said to be convex if ∀λ ∈ [0,1], ∀x,y ∈ U, λx +(1−λ)y ∈ U. By convention, ∅ is convex. (ii) Let U, ∅ = U ⊂ Rn, be convex. The function f : Rn → R∪{+∞} is said to be convex on U if ∀λ ∈]0,1[, ∀x,y ∈ U, f (λx +(1−λ)y) ≤ λf (x)+(1−λ)f (y), with the convention (+∞) + (+∞) = +∞, (+∞) + a = +∞ for all a ∈ R, and (+∞)a = (a/ a )∞ for all a ∈ R not equal to 0. (iii) Let U, ∅ = U ⊂ Rn, be convex. The function f : Rn → R∪{−∞} is said to be concave on U if −f is convex on U (see Figure 2.4). (iv) By convention, the function identically equal to −∞ or +∞ is both convex and concave. f (x) f (x) x x Convex function Concave function Figure 2.4. Convex function and concave function. The indicator function IU and the distance function dU for which (dU )U = IU are, respec- tively, related to the convexity of U and its closure. Theorem 7.1. Let U ⊂ Rn. (i) The interior, intU, and the closure, U, of a convex set U are convex. (ii) The set U is convex if and only if IU is convex. (iii) The set U is convex if and only if dU is convex. (iv) Let {Uα}α∈A be a family of convex subsets of Rn, where the set A of indices is arbitrary and not necessarily finite. Then U = ∩α∈A Uα is convex. (v) Let U be convex such that intU = ∅. Then intU = U.
- 52. 34 Chapter 2. Existence, Convexities, and Convexification Proof. (i) If U = ∅, then ∅ = ∅ is convex by convention. If x,y ∈ U, then there exist sequences {xn} ⊂ U and {yn} ⊂ U such that xn → x and yn → y. By convexity of U, for all λ ∈ [0,1], U λxn +(1−λ)yn → λx +(1−λ)y. Thus λx +(1−λ)y ∈ U and U is convex. If intU = ∅, then it is convex by convention. For x,y ∈ intU, there exists Bε(x) and Bη(y) such that Bε(x) ⊂ U and Bη(y) ⊂ U and, for all λ ∈ [0,1], λx +(1−λ)y ∈ λBε(x)+(1−λ)Bη(y). But, λx +(1−λ)y ∈ Bmin{ε,η}(λx +(1−λ)y) ⊂ λBε(x)+(1−λ)Bη(y) ⊂ U and λx +(1−λ)y ∈ intU. (ii) If U = ∅, then IU is identically +∞ which is convex by convention. If U = ∅ is convex, then for all x,y ∈ U and λ ∈]0,1[, λx +(1−λ)y ∈ U and IU (λx +(1−λ)y) = 0 = λIU (x)+(1−λ)IU (y) = 0. If either x or y is not in U, then either IU (x) or IU (y) is +∞ and λIU (x)+(1−λ)IU (y) = +∞, so that IU (λx +(1−λ)y) ≤ +∞ = λIU (x)+(1−λ)IU (y). Hence IU is convex. Conversely, if IU is identically +∞, then U = ∅ which is convex by convention. Otherwise, for all x,y ∈ U and λ ∈ [0,1], 0 ≤ IU (λx +(1−λ)y) ≤ λIU (x)+(1−λ)IU (y) = 0 and λx +(1−λ)y ∈ U. Hence the convexity of U. (iii) By convention, U = ∅ implies dU (x) = −∞ that is convex, also by convention. For U = ∅. Given x and y in Rn, there exist x and y in U such that dU (x) = |x −x| and dU (y) = |y −y|. By convexity of U, for all λ, 0 ≤ λ ≤ 1, λx +(1−λ)y ∈ U and dU (λx +(1−λ)y) ≤ λx +(1−λ)y −(λx +(1−λ)y) ≤λ x −x +(1−λ) y −y = λdU (x)+(1−λ)dU (y) and dU is convex in Rn. Conversely, if dU (x) = −∞ for some x, then, by convention, U = ∅ which is convex, also by convention. If dU is finite and dU is convex, then ∀λ ∈ [0,1], ∀x,y ∈ U, dU (λx +(1−λ)y) ≤ λdU (x)+(1−λ)dU (y).
- 53. 7. Convexity, Quasiconvexity, Strict Convexity, and Uniqueness 35 But x and y in U imply that dU (x) = dU (y) = 0 and hence ∀λ ∈ [0,1], dU (λx +(1−λ)y) = 0. Thus λx +(1−λ)y ∈ U and U is convex. (iv) If U is empty, U is convex by definition. If U is not empty, choose x and y ∈ U = ∩α∈A Uα ⇒ ∀α ∈ A, x ∈ Uα, y ∈ Uα. For all λ ∈ [0,1] and by convexity of Uα, ∀α ∈ A, λx +(1−λ)y ∈ Uα ⇒ λx +(1−λ)y ∈ ∩α∈AUα = U. (v) As intU = ∅, pick a point x ∈ intU. By convexity, for all y ∈ ∂U, the segment [x,y] = {λx + (1 − λ)y : 0 ≤ λ ≤ 1} belongs to U and [x,y[= {λx + (1 − λ)y : 0 λ ≤ 1} ⊂ intU. So there exists a sequence yn = x +(y −x)/(n+1) in intU that converges to y. Hence the result. The following definitions will also be useful. Definition 7.2. Let U, ∅ = U ⊂ Rn. (i) The convex hull of U is the intersection of all convex subsets of Rn that contain U. It is denoted coU. (ii) The closed convex hull of U is the intersection of all closed convex subsets of Rn that contains U. It is denoted co U. Theorem 7.2. Let U, ∅ = U ⊂ Rn. (i) coU is convex and coU = k i=1 λixi : k i=1 λi = 1, xi ∈ U, 0 ≤ λi ≤ 1, k ≥ 1 ' . (7.1) (ii) co U is closed and convex and co U = coU = coU. (7.2) Proof. (i) By Theorem 7.1(iv), coU is convex as an intersection of convex sets. Denote by C the right-hand side of (7.1). We have U ⊂ coU ⊂ C since C is a convex that contains U. Taking all convex combinations of elements of U, we get C that is entirely contained in coU and coU = C. (ii) By Theorem 7.1(iv) and the fact that the intersection of a family of closed sets is closed. The convexity of f on a convex U can also be characterized in terms of the convexity of fU or of its epigraph epifU as in the case when f is lsc (see Lemma 4.1).
- 54. 36 Chapter 2. Existence, Convexities, and Convexification Lemma 7.1. (i) If f : Rn → R∪{+∞} is convex on Rn, then dom f is convex and f is convex on dom f . (ii) If f : Rn → R∪{+∞} is convex on a convex subset U of Rn, then fU is convex on Rn, dom fU = U ∩dom f is convex, and fU is convex on dom fU . Proof. (i) For any convex combination of x,y ∈ dom f , f (x) +∞ and f (y) +∞, f (λx + (1 − λ)y) ≤ λf (x) + (1 − λ)f (y) +∞, and λx + (1 − λ)y ∈ dom f . Thence, dom f is convex and, by definition, f : dom f → R is convex. (ii) By definition, dom fU = U ∩ dom f and as fU (x) = +∞ outside of U, fU is convex on Rn. Finally, from part (i), dom fU is convex. Theorem 7.3. Let U, ∅ = U ⊂ Rn, be convex and f : Rn → R∪{+∞}. The following conditions are equivalent: (i) f is convex on U; (ii) fU is convex on Rn; (iii) epifU is convex on Rn ×R. Proof. (i) ⇒ (ii) From Lemma 7.1(ii). (ii) ⇒ (iii) From Lemma 7.1(i), dom fU is convex and fU : dom fU → R is convex. For all (x,µx) and (y,µy) in epifU , µx ≥ f (x) and µy ≥ f (y) and hence x,y ∈ dom fU . Given λ ∈ [0,1], consider their convex combination λ(x,µx)+(1−λ)(y,µy) = (λx +(1−λ)y,λµx +(1−λ)µy). By construction and convexity of fU , λµx +(1−λ)µy ≥ λf (x)+(1−λ)f (y) ≥ f (λx + (1 − λ)y). Therefore, f (λx + (1 − λ)y) +∞, λx + (1 − λ)y ∈ dom fU ⊂ U, and λ(x,µx)+(1−λ)(y,µy) ∈ epifU . So epifU is convex on Rn ×R. (iii) ⇒ (i) By definition of fU , dom fU ⊂ U and fU = f in dom fU . Therefore, for all x,y ∈ dom fU , the pairs (x,f (x)) and (y,f (y)) belong to epifU . As epifU is convex, ∀λ ∈ [0,1], λ(x,f (x))+(1−λ)(y,f (y)) ∈ epifU ⇒ (λx +(1−λ)y,λf (x)+(1−λ)f (y)) ∈ epifU ⇒ λf (x)+(1−λ)f (y) ≥ fU (x +(1−λ)y). Thence, as fU (x + (1 − λ)y) +∞, we get λx + (1 − λ)y ∈ dom fU and the convexity of dom fU . Finally, fU (x + (1 − λ)y) = f (x + (1 − λ)y) yields λf (x) + (1 − λ)f (y) ≥ f (x +(1−λ)y) and the convexity of the function f in dom fU . The convexity of fU on the convex subset U ⊃ dom fU is a consequence of Lemma 7.1(i). Finally, as fU = f on U, f is convex on U. Except for functions whose domain is a singleton, the convexity of a function forces its liminf to be less than or equal to its value at each point of its effective domain. Lemma 7.2. Let f : Rn → R∪{+∞}, dom f = ∅, be a convex function. (i) If dom f = {x0} is a singleton, liminf y→x f (y) = +∞ = f (x) if x = x0 and liminf y→x f (y) = +∞ f (x0). (7.3)
- 55. Random documents with unrelated content Scribd suggests to you:
- 56. I am much obliged to you, Mr. Inglesant, for the great trouble you have taken. I assure you I shall not forget it. Lady Fentham, as Sir Richard will so soon be here—he stopped suddenly as an idea struck him, and looking full at Inglesant, said slowly and with marked emphasis, Supposing Mr. Inglesant to—to have spoken the truth he would have said, but Johnny's perfectly courteous attitude of calm politeness, the utter absence of any tangible ground of offence, and his own instincts as a gentleman, checked him, and he continued,—has not been misinformed, you will not need my protection any further. I will leave you with Mr. Inglesant; probably Lady Cardiff will be back before long. He took his leave with equal courtesy both to the lady and Inglesant, and went down to his men. Ann Fentham sank into her chair, and began to sob bitterly, saying,— What shall I say to my husband, Mr. Inglesant? He will be here directly, and will find me alone. What would have happened to me if you had not come? If I may offer any advice, madam, I should say, Tell your husband everything exactly as it happened. Nothing has happened of which you have need to be ashamed. Sir Richard will doubtless see that you have been shamefully deceived by your friends, as far as I understand the matter. You can trust to his sympathy and kindness. She did not reply, and Inglesant, who found his situation far more awkward than before, said, Shall I seek for Lady Cardiff, madam, and bring her to you?
- 57. No, don't leave me, Mr. Inglesant, she said, springing up and coming to him; I shall bless your name for ever for what you have done for me this day. Inglesant stayed with the lady until it was plain Lord H—— had left the house with his servants, and he then left her and went into the garden to endeavour to find his brother and Lady Cardiff; but in this he was not successful, and returned to the house, where he ordered some dinner—for he had eaten nothing since the morning— and seated himself at the window to wait for Sir Richard. He had sat there about an hour when the latter arrived, and drew his rein before the house before dismounting. Inglesant greeted him and went out to him in the porch. Fentham returned his greeting warmly. Your wife is upstairs, Sir Richard, Inglesant said; she came down with Lady Isabella Thynne, and is waiting for her to take her back. Fentham left his horse with the servant and ran upstairs straight to his wife, and as Inglesant followed him into the house he met Lady Cardiff and his brother, who came in from the garden. Eustace Inglesant was radiant, and introduced Lady Cardiff to his brother as his future wife. He took them into a private room, and called for wine and cakes. Johnny thought it best not to tell them what had occurred, but merely said that Sir Richard and his wife were upstairs; upon which Eustace sent a servant up with his compliments, asking them to come and join them. Both Lady Cardiff and Eustace appeared conscious, however, that some blame attached to them, for they expressed great surprise at the absence of Lady Isabella, and took pains to inform Johnny that they had left Lady Fentham with her, and had no idea she was going away. Sir
- 58. Richard and Lady Fentham joined the party, and appeared composed and happy, and they had not sat long before Lady Isabella's coach appeared before the door, and her ladyship came in. The ladies returned to Oxford in the coach, and the gentlemen on horseback. Nothing was said by the latter as to what had occurred until after they had left Eustace at his lodgings, and Johnny was parting with Fentham at the door of Lord Falkland, to whom he was going. Then Sir Richard said,— Mr. Inglesant, my wife has told me all, and has told me that she owes everything to you, even to this last blessing, that there is no secret between us. I beg you to believe two things,—first, that nothing I can do or say can ever repay the obligation that I owe to you; secondly, that the blame of this matter rests mostly with me, in that I have left my wife too much. Inglesant waited for several days in expectation of hearing from Lord H——, but no message came. They met several times and passed each other with the usual courtesies. At last Eustace Inglesant heard from one of his lordship's friends that the latter had been very anxious to meet Johnny, but had been dissuaded. You have not the slightest tangible ground of offence against young Inglesant, they told him, and you have every cause to keep this affair quiet, out of which you have not emerged with any great triumph. Inglesant has shown by the line of conduct he adopted that he desires to keep it close. None of the rest of the party will speak of it for their own sakes. Were it known, it would ruin you at once with the King, and damage you very much in the estimation of all the principal men here, who are Sir Richard's friends, and such as are not would resent such conduct towards a man engaged on his
- 59. master's business. Besides this you are not a remarkably good fencer, whereas John Inglesant is a pupil of the Jesuits, and master of all their arts and tricks of stabbing. That he could kill you in five minutes if he chose, there can be no doubt. These and other similar arguments finally persuaded Lord H—— to restrain his desire of revenge, which was the easier for him to do as Inglesant always treated him when they met with marked deference and courtesy. The marriage of Lady Cardiff and Eustace Inglesant was hurried forward, and took place at Oxford some weeks after the foregoing events; the King and Queen being present at the ceremony. It was indeed very important to attach this wealthy couple unmistakably to the royal party, and no efforts were spared for the purpose. Lady Cardiff and her husband, however, did not manifest any great enthusiasm in the royal cause. The music of the wedding festival was interrupted by the cannon of Newbury, where Lord Falkland was killed, together with a sad roll of gentlemen of honour and repute. Lord Clarendon says, —Such was always the unequal fate that attended this melancholy war, that while some obscure, unheard-of colonel or officer was missing on the enemy's side, and some citizen's wife bewailed the loss of her husband, there were on the other above twenty officers of the field and persons of honour and public name slain upon the place, and more of the same quality hurt. In this battle Inglesant was more fortunate than in his first, for he was not hurt, though he rode in the Lord Biron's regiment, the same in which Lord Falkland was also a volunteer.
- 60. The King returned to Oxford, where Inglesant found every one in great dejection of mind; the conduct of the war was severely criticized, the army discontented, and the chief commanders engaged in reproaches and recriminations. One afternoon Inglesant was sent for to Merton College, where the Queen lay, and where the King spent much of his time; where he found the Jesuit standing with the King in one of the windows, and Mr. Jermyn, who had just been made a baron, talking to the Queen. The King motioned Inglesant to approach him, and the Jesuit explained the reason he had been sent for. The trial of Archbishop Laud was commencing, and in order to incite the people against him Mr. Prynne had published the particulars of a popish plot in a pamphlet which contained the names of many gentlemen, both Protestant and Catholic, the publication of which at such a moment excited considerable uneasiness among their relations and friends. I wish you, Mr. Inglesant, said the King, to ride to London. Mr. Hall has provided passes for you, and letters to several of his friends. The new French Ambassador is landing; I wish to know how far the French Court is true to me. Prynne's wit has overreached himself. His charges have frightened so many, that a reaction is setting in in favour of the Archbishop, and many are willing to testify in his favour in order to exonerate themselves. You will be of great use in finding out these people. Seek every one who is mentioned in Prynne's libel; many of them are men of influence. Your familiar converse with Papists, in other respects unfortunate, may be of use here.
- 61. Inglesant spent some time in London, and was in constant communication with Mr. Bell, the Archbishop's secretary. He was successful in procuring evidence from among the Papists of their antipathy to Laud, and in various other ways in providing Bell with materials for defence. Laud was informed of these acts of friendship, and being in a very low and broken state, was deeply touched that a comparative stranger, and one who had been under no obligation to him, should show so much attachment, and exert himself so much in his service, at a time when the greatest danger attended any one so doing, and when he seemed deserted both by his royal master and by those on whom he had showered benefits in the time of his prosperity. He sent his blessing and grateful thanks, the thanks of an old and dying man, which would be all the more valuable as they never could be accompanied by any earthly favour. Inglesant's name was associated with that of the Archbishop, and the Jesuit's aim in sending him to London was accomplished. CHAPTER X. Inglesant was of so much use in gaining information, and managed to live on such confidential terms with many in London in the confidence of members of the Parliament, that he remained there during all the early part of the year, and would have stayed longer; but the enemies of the Archbishop, who pursued him with a malignant and remorseless activity, set their eyes at last upon the young envoy, and he was advised to leave London, at any rate till
- 62. the trial was over. He was very unwilling to leave the Archbishop, but dared not run the risk of being imprisoned and thwarting the Jesuit's schemes, and therefore left London about the end of May, and returned straight to Oxford. He left London only a few days before the allied armies of Sir W. Waller and the Earl of Essex, and had no sooner arrived in Oxford than the news of the advance of the Parliamentary forces caused the greatest alarm. The next day Abingdon was vacated by some mistake, and the rebels took possession of the whole of the country to the east and south of Oxford; Sir William Waller being on the south, and the Earl of Essex on the east. It was reported in London that the King intended to surrender to the Earl's army, and such a proposition was seriously made to the King by his own friends a few days afterwards in Oxford. The royal army was massed about the city, most of the foot being on the north side; Inglesant served with the foot in Colonel Lake's regiment of musketeers and pikes, taking a pike in the front rank. It was a weapon which the gentlemen of that day frequently practised, and of which he was a master. Several other gentlemen volunteers were in the front rank with him. The Earl's army was drawn up at Islip, on the other side of the river Cherwell, having marched by Oxford the day before, in open file, drums beating and colours flying, so that the King had a full view of them on the bright fine day. The Earl himself, with a party of horse, came within cannon shot of the city, and the King's horse charged him several times without any great hurt on either side. It was a gay and brilliant scene to any one who could look upon it with careless and indifferent eyes.
- 63. The next morning a strong party of the Earl's army endeavoured to pass the Cherwell at Gosford bridge, where Sir Jacob Astley commanded, and where the regiment in which Inglesant served was stationed. The bridge was barricaded with breastworks and a bastion, but the Parliamentarian army attempted to cross the stream both above and below. They succeeded in crossing opposite to Colonel Lake's regiment, under a heavy fire from the musketeers, who advanced rank by rank between the troops of pikes and a little in advance of them, and after giving their fire, wheeled off to the right and left, and took their places again in the rear. The rebels reserved their fire, their men falling at every step; but they still advanced, supported by troops of horse, till they reached the Royalists, when they delivered their fire, closed their ranks, and charged, their horse charging the pikes at the same time. The ranks of the royal musketeers halted and closed up, and the pikes drew close together shoulder to shoulder, till the rapiers of their officers met across the front. The shock was very severe, and the struggle for a moment undecided; but the pikes standing perfectly firm, owing in a great measure to the number of gentlemen in the front ranks, and the musketeers fighting with great courage, the enemy began to give way, and having been much broken before they came to the charge fell into disorder, and were driven back across the stream, the Royalists following them to the opposite bank, and even pursuing them up the slope. Inglesant had noticed an officer on the opposite side who was fighting with great courage, and as they crossed the river he saw him stumble and nearly fall, though he appeared to struggle forward on the opposite slope to where an old thorn tree broke the rank of the pikes. Johnny came close to him,
- 64. and recognized him as the Mr. Thorne whom he had known at Gidding. As he knew the regiment would be halted immediately, he fell out of his rank, leaving his file to the bringer-up or lieutenant behind him, and stooped over his old rival, who evidently was desperately hurt. He raised his head, and gave him some aqua vitæ from his flask. The other knew him at once, and tried to speak; but his strength was too far gone, and his utterance failed him. He seemed to give over the effort, and lay back in Inglesant's arms, staining his friend with his blood. Inglesant asked him if he had any mission he would wish performed, but the other shook his head, and seemed to give himself to prayer. After a minute or two he seemed to rally, and his face became very calm. Opening his eyes, he looked at Johnny steadily and with affection, and said, slowly and with difficulty, but still with a look of rest and peace,— Mr. Inglesant, you spoke to me once of standing together in a brighter dawn; I did not believe you, but it was true; the dawn is breaking—and it is bright. As he spoke a volley of musketry shook the hill-side, and the regiment came down the slope at a run, and carrying Inglesant with them, crossed the river, and, halting on the other side, wheeled about and faced the passage in the same order in which they had stood at first. This dangerous manoeuvre was executed only just in time, for the enemy advanced in great force to the river-side; but the Royalists being also very strong, they did not attempt to pass. After facing each other for some time, the fighting having ceased all along the line, Inglesant spoke to his officer, and got leave to cross the river with a flag of truce to seek his friend. An officer from the other side met him, most of the enemy's troops having fallen back
- 65. some distance from the river. He was an old soldier, evidently a Low- country officer, and not much of a Puritan, and he greeted Inglesant politely as a fellow-soldier. Inglesant told him his errand, and that he was anxious to find out his friend's body, if, as he feared, he would be found to have breathed his last. They went to the old thorn, where, indeed, they found Mr. Thorne quite dead. Several of the rebel officers gathered round. Mr. Thorne was evidently well known, and they spoke of him with respect and regard. Inglesant stopped, looking down on him for a few minutes, and then turned to go. Gentlemen, he said, raising his hat, I leave him in your care. He was, as you have well said, a brave and a good man. I crossed his path twice—once in love and once in war—and at both times he acted as a gallant gentleman and a man of God. I wish you good day. He turned away, and went down to the river, from which his regiment had by this time also fallen back, the others looking after him as he went. Who is that? said a stern and grim-looking Puritan officer. He does not speak as the graceless Cavaliers mostly do. His name is Inglesant, said a quiet, pale man, in dark and plain clothes; he is one of the King's servants, a concealed Papist, and, they say, a Jesuit. I have seen him often at Whitehall. Thou wilt not see him much longer, brother, said the other grimly, either at Whitehall or elsewhere. It were a good deed to prevent his further deceiving the poor and ignorant folk, and he raised his piece to fire.
- 66. Scarcely, said the other quietly, since he came to do us service and courtesy. But he made no effort to restrain the Puritan, looking on, indeed, with a sort of quiet interest as to what would happen. Thou art enslaved over much to the customs of this world, brother, said the other, still with his grave smile; knowest thou not that it is the part of the saints militant to root out iniquity from the earth? He arranged his piece to fire, and would no doubt have done so; but the Low-country officer, who had been looking on in silence, suddenly threw himself upon the weapon, and wrested it out of his hand. By my soul, Master Fight-the-fight, he said, that passes a joke. The good cause is well enough, and the saints militant and triumphant, and all the rest of it; but to shoot a man under a flag of truce was never yet required of any saint, whether militant or triumphant. The other looked at him severely as he took back his weapon. Thou art in the bonds of iniquity thyself, he said, and in the land of darkness and the shadow of death. The Lord's cause will never prosper while it puts trust in such as thou. But he made no further attempt against Inglesant, who, indeed, by this time had crossed the river, and was out of musket shot on the opposite bank. A few days afterwards the King left Oxford and went into the West. Inglesant remained in garrison, and took his share in all the expeditions of any kind that were undertaken. The Roman Catholics were at this time very strong in Oxford; they celebrated mass every day, and had frequent sermons, at which many of the Protestants
- 67. attended; but it was thought among the Church people to be an extreme thing to do, and any of the commanders who did it excited suspicion thereby. The Church of England people were by this time growing jealous of the power and unrestrained license of the Catholics, and the Jesuit warned Inglesant to attach himself more to the English Church party, and avoid being much seen with extreme Papists. Colonel Gage, a Papist, was appointed governor by the King; but being a very prudent man and a general favourite, as well as an excellent officer, the appointment did not give much offence. Inglesant was present at Cropredy Bridge, which battle or skirmish was fought after the King returned to Oxford from his hasty march through Worcestershire, and was wounded severely in the head by a sword cut—a wound which he thought little of at the time, but which long afterwards made itself felt. Notwithstanding this wound he intended following the King into the West, for His Majesty had latterly shown a greater kindness to him, and a wish to keep him near his person; but Father St. Clare, after an interview with the King, told Inglesant that he had a mission for him to perform in London, and so kept him in Oxford. The trial of the Archbishop was dragging slowly on through the year, and the Jesuit procured Inglesant another pass, and directed him to endeavour in every way to assist the Archbishop in his trial, without fear of his prosecutors, telling him that he could procure his liberation even if he were put in prison, which he did not believe he would be. Inglesant, therefore, on his return to London, gave himself heartily to assisting the counsel and secretary of the Archbishop, and found himself perfectly unmolested in so doing. He lodged at a druggist's over against the Goat Tavern, near Toy Bridge
- 68. in the Strand, and frequented the ordinary at Haycock's, near the Palsgrave's Head Tavern, where the Parliament men much resorted. Here he met among others Sir Henry Blount, who had been a gentleman pensioner of the King's, and had waited on him in his turn to York and Edgehill fight, but then, returning to London, walked into Westminster Hall, with his sword by his side, so coolly as to astonish the Parliamentarians. He was summoned before the Parliament, but pleading that he only did his duty as a servant, was acquitted. This man, who was a man of judgment and experience, was of great use to Inglesant in many ways, and put him in the way of finding much that might assist the Archbishop; but it occurred to Inglesant more than once to doubt whether the latter would benefit much by his advocacy, a known pupil of the Papists as he was. This caused him to keep more quiet than he otherwise would have done; but what was doubtless the Jesuit's chief aim was completely answered; for the Church people, both in London and the country, who regarded the Archbishop as a martyr, becoming aware of the sincere and really useful exertions that Inglesant had made with such untiring energy, attached themselves entirely to him, and took him completely into their confidence, so that he could at this time have depended on any of them for assistance and support. The different parties were at this time so confused and intermixed—the Papists playing in many cases a double game—that it would have been difficult for Inglesant, who was partly in the confidence of all, to know which way to act, had he stood alone. He saw now, more than he had ever done, the intrigues of that party among the Papists who favoured the Parliament, and was astonished at their skill and duplicity. At last the Commons, failing to find the Archbishop guilty of
- 69. anything worthy of death, passed a Bill of Attainder, as they had done with Lord Strafford, and condemned him with no precedence of law. The Lords hesitated to pass the Bill, and on Christmas Eve, 1644, demanded a conference with the Commons. The next day was the strangest Christmas Day Inglesant had ever spent. The whole city was ordered to fast in the most solemn way by a special ordinance of Parliament, and strict inquisition was made to see that this ordinance was carried out by the people. Inglesant was well acquainted with Mr. Hale, afterwards Chief Justice Hale, one of the Archbishop's counsel, then a young lawyer in Lincoln's Inn, who, it was said, had composed the defence which Mr. Hern, the senior counsel, had spoken before the Lords. Johnny spent part of the morning with this gentleman, and in the afternoon walked down to the Tower from Lincoln's Inn. The streets were very quiet, the shops closed, and a feeling of sadness and dread hung over all—at any rate in Inglesant's mind. At the turnstile at Holborn he went into a bookseller's shop kept by a man named Turner, a Papist, who sold popish books and pamphlets. Here he found an apothecary, who also was useful to the Catholics, making Hosts for them. These both immediately began to speak to Inglesant about the Archbishop and the Papists, expressing their surprise that he should exert himself so much in his favour, telling him that the Papists, to a man, hated him and desired his death, and that a gentleman lately returned from Italy had that very day informed the bookseller that the news of the Archbishop's execution was eagerly expected in Rome. The Lords were certain to give way, they said, and the Archbishop was as good as dead already. They were evidently very anxious to extract from Inglesant whether he acted on his own
- 70. responsibility or from the directions of the Jesuit; but Inglesant was much too prudent to commit himself in any way. When he had left them he went straight to the Tower, where he was admitted to the Archbishop, whom he found expecting him. He gave him all the intelligence he could, and all the gossip of the day which he had picked up, including the sayings of the wits at the taverns and ordinaries respecting the trial and the Archbishop, of whom all men's minds were full. Laud was inclined to trust somewhat to the Lords' resistance, and Inglesant had scarcely the heart to refute his opinion. He told him the feeling of the Papists, and his fear that even the Catholics at Oxford were not acting sincerely with him. After the failure of the King's pardon, Laud entertained little hope from any other efforts Charles might be disposed to make; but Inglesant promised him to ride to Oxford, and see the Jesuit again. This he did the next day, before the Committee of the Commons met the Lords, which they did not do till the 2d of January. He had a long interview with the Jesuit, and urged as strongly as he could the cruelty and impolicy of letting the Archbishop die without an effort to save him. What can be done? said the Jesuit; the King can do nothing. All that he can do in the way of pardon he has done: besides, I never see the King; the feeling against the Catholics is now so strong, that His Majesty dare not hold any communications with me. Inglesant inquired what the policy of the Roman Catholic Church really was; was it favourable to the King and the English Church, or against it? The Jesuit hesitated, but then, with that appearance of frankness which always won upon his pupil, he confessed that the
- 71. policy of the Papal Court had latterly gone very much more in favour of the party who wished to destroy the English Church than it had formerly done; and that at present the Pope and the Catholic powers abroad were only disposed to help the King on such terms as he could not accept, and at the same time retain the favour of the Church and Protestant party; and he acknowledged that he had himself under-estimated the opposition of the bulk of English people to Popery. He then requested Inglesant to return to London, and continue to show himself openly in support of the Archbishop, assuring him that in this way alone could he fit himself for performing a most important service to the King, which, he said, he should be soon able to point out to him. The old familiar charm, which had lost none of its power over Johnny, would, of itself, have been sufficient to make him perfectly pliant to the Jesuit's will. He returned to London, but was refused admission to the Archbishop until after the Committee of the Commons had met the Lords, and on the 3d of January the Lords passed the Bill of Attainder. When the news of this reached the Archbishop, he broke off his history, which he had written from day to day, and prepared himself for death. He petitioned that he might be beheaded instead of hanged, and the Commons at last, after much difficulty, granted this request. On the 6th of January it was ordered by both Houses that he should suffer on the 10th. On the same day Inglesant received a special message from the Jesuit in these words, in cypher:—Apply for admission to the scaffold; it will be granted you. Very much surprised, Inglesant went to Alderman Pennington, and requested admission to attend the Archbishop to the scaffold,
- 72. pleading that he was one of the King's household, and attached to the Archbishop from a boy. Pennington examined him concerning his being in London, his pass, and place of abode, but Inglesant thought more from curiosity than from any other motive; for it was evident that he knew all about him, and his behaviour in London. He asked him many questions about Oxford and the Catholics, and seemed to enjoy any embarrassment that Inglesant was put to in replying. Finally he gave him the warrant of admission, and dismissed him. But as he left the room he called him back, and said with great emphasis,— I would warn you, young man, to look very well to your steps. You are treading a path full of pitfalls, few of which you see yourself. All your steps are known, and those are known who are leading you. They think they hold the wires in their own hands, and do not know that they are but the puppets themselves. If you are not altogether in the snare of the destroyer, come out from them, and escape both destruction in this world and the wrath that is to come. Inglesant thanked him and took his leave. He could not help thinking that there was much truth in the alderman's description of his position. The next three days the Archbishop spent in preparing for death and composing his speech; and on the day on which he was to die, Inglesant found when he reached the Tower, that he was at his private prayers, at which he continued until Pennington arrived to conduct him to the scaffold. When he came out and found Inglesant there, he seemed pleased, as well he might, for excepting Stern, his chaplain, the only one who was allowed to attend him, he was alone amongst his enemies. He ascended the scaffold with a brave and
- 73. cheerful courage, some few of the vast crowd assembled reviling him, but the greater part preserving a decent and respectful silence. The chaplain and Inglesant followed him close, and it was well they did so, for a crowd of people, whether by permission or not is not known, pressed up upon the scaffold, as Dr. Heylyn said, upon the theatre to see the tragedy, so that they pressed upon the Archbishop, and scarcely gave him room to die. Inglesant had never seen such a wonderful sight before—once afterwards he saw one like it, more terrible by far. The little island of the scaffold, surrounded by a surging, pressing sea of heads and struggling men, covering the whole extent of Tower Hill; the houses and windows round full of people, the walls and towers behind covered too. People pressed underneath the scaffold; people climbed up the posts and hung suspended by the rails that fenced it round; people pressed up the steps till there was scarcely room within the rails to stand. The soldiers on guard seemed careless what was done, probably feeling certain that there was no fear of any attempt to rescue the hated priest. Inglesant recognized many Churchmen and friends of the Archbishop among the crowd, and saw that they recognized him, and that his name was passed about among both friends and enemies. The Archbishop read his speech with great calmness and distinctness, the opening moving many to tears, and when he had finished, gave the papers to Stern to give to his other chaplains, praying God to bestow His mercies and blessings upon them. He spoke to a man named Hind, who sat taking down his speech, begging him not to do him wrong by mistaking him. Then begging the crowd to stand back and give him room, he knelt down to the
- 74. block; but seeing through the chinks of the boards the people underneath, he begged that they might be removed, as he did not wish that his blood should fall upon the heads of the people. Surely no man was ever so crowded upon and badgered to his death. Then he took off his doublet, and would have addressed himself to prayer, but was not allowed to do so in peace; one Sir John Clotworthy, an Irishman, pestering him with religious questions. After he had answered one or two meekly, he turned to the executioner and forgave him, and kneeling down, after a very short prayer, to which Hind listened with his head down and wrote word for word, the axe with a single blow cut off his head. He was buried in All Hallows Barking, a great crowd of people attending him to the grave in silence and great respect,—the Church of England service read over him without interruption, though it had long been discontinued in all the Churches in London. News of his death spread rapidly over England, and was received by all Church people with religious fervour as the news of a martyrdom; and wherever it was told, it was added that Mr. John Inglesant, the King's servant, who had used every effort to aid the Archbishop on his trial, was with him on the scaffold to the last. Inglesant returned to Oxford, where the Jesuit received him cordially. He had, it would have seemed, failed in his mission, for the Archbishop was dead; nevertheless, the Jesuit's aim was fully won. On the King's leaving Oxford, before the advance of General Fairfax, Inglesant accompanied him, and was present at the battle of Naseby, so fatal to the royal cause. No mention of this battle, however, is to be found among the papers from which these memoirs are compiled; and the fact that Inglesant was present at it
- 75. is known only by an incidental reference to it at a later period. Amid the confusion of the flight, and the subsequent wanderings of the King before he returned to Oxford, it is impossible to follow less important events closely, and it does not seem clear whether Inglesant met with the Jesuit immediately after the battle or not. Acting, however, there can be no doubt, with his approval, if not by his direction, he appears very soon after to have found his way to Gidding, where he remained during several weeks.
- 76. CHAPTER XI. The autumn days passed quickly over, and with them the last peaceful hours that Inglesant would know for a long time, and that youthful freshness and bloom and peace which he would never know again. Such a haven as this, such purity and holiness, such rest and repose, lovely as the autumn sunshine resting on the foliage and the grass, would never be open to him again. It was long before rest and peace came to him at all, and when they did come, under different skies and an altered life, it was a rest after a stern battle that left its scars deep in his very life; it was apart from every one of his early friends; it was unblest by first love and early glimpse of heaven. It was about the end of October that he received a message from the Jesuit, which was the summons to leave this paradise, sanctified to him by the holiest moments of his life. The family were at evening prayers in the Church when the messenger arrived, and Inglesant, as usual, was kneeling where he could see Mary Collet, and probably was thinking more of her than of the prayers. Nevertheless he remembered afterwards, when he thought during the long lonely hours of every moment spent at Gidding, that the third collect was being read, and that at the words Lighten our darkness he looked up at some noise, and saw the sunshine from the west window shining into the Church upon Mary Collet and the kneeling women, and, beyond them, standing in the dark shadow under the window, the messenger of the Jesuit, whom he knew. He
- 77. got up quietly and went out. From his marriage feast, nay, from the table of the Lord, he would have got up all the same had that summons come to him. His whole life from his boyhood had been so formed upon the idea of some day proving himself worthy of the confidence reposed in him (that perfect unexpressed confidence which won his very nature to a passionate devotion capable of the supreme action, whatever it might be, to which all his training had tended), that to have faltered at any moment would have been more impossible to him than suicide, than any self-contradictory action could have been —as impossible as for a proud man to become suddenly naturally humble, or a merciful man cruel. That there might have been found in the universe a power capable of overmastering this master passion is possible; hitherto, however, it had not been found. Outside the Church the messenger gave him a letter from the Jesuit, which, as usual, was very short. Johnny, come to me at Oxford as soon as you can. The time for which we have waited is come. The service which you and none other can perform, and which I have always foreseen for you, is waiting to be accomplished. I depend on you. Inglesant ordered some refreshment to be given to the messenger, and his own horses to be got out. Then he went back into the Church, and waited till the prayers were over. The family expressed great regret at parting with him; they were in a continual state of apprehension from their Puritan neighbours; but Inglesant's presence was no defence but rather the contrary, and it is possible that some of them may have been glad that he was going.
- 78. Mary Collet looked sadly and wistfully at him as they stood before the porch of the house in the setting sunlight, the long shadows resting on the grass, the evening wind murmuring in the tall trees and shaking down the falling leaves. Do you know what this service is? she said at last. I cannot make the slightest guess, he answered. Whatever it is you will do it? she asked again. Certainly; to do otherwise would be to contradict the tenor of my life. It may be something that your conscience cannot approve, she said. It is too late to think of that, he said, smiling; I should have thought of that years ago, when I was a boy at Westacre, and this man came to me as an angel of light—to me a weak, ignorant, country lad—to me, who owe him everything that I am, everything that I know, everything—even the power that enables me to act for him. Did she remember how he had once offered himself without reserve to her, then at least without any reservation in favour of this man? Did she regret that she had not encouraged this other attraction, or did she see that the same thing would have happened whether she had accepted him or no? She gave no indication of either of these thoughts. I think you owe something to another, she said, softly; to One who knew you before this Jesuit; to One who was leading you onward before he came across your path; to One who gave you high and noble qualities, without which the Jesuit could have given you nothing; to One whom you have professed to love; to One for whose
- 79. Divine Voice you have desired to listen. Johnny, will you listen no longer for it? He never forgot her, standing before him with her hands clasped and her eyes raised to his,—the flush of eager speaking on her face, —those great eyes, moistened again with tears, that pierced through him to his very soul,—her trembling lip,—the irresistible nobleness of her whole figure,—her winning manner, through which the love she had confessed for him spoke in every part. He never saw her again but once—then in how different a posture and scene; and the beauty of this sight never went out of his life, but it produced no effect upon his purpose; indeed how could it, when his purpose was not so much a part of him as he was a part of it? He looked at her in silence, and his love and admiration spoke out so unmistakably in his look that Mary never afterwards doubted that he had loved her. He had not power to explain his conduct; he could not have told himself why he acted as he did. Amid the distracting purposes which tore his heart in twain he could say nothing but,— It may not be so bad as you think. Mary gave him her hand, turned from him, and went into the house; and he let her go—her of whom the sight must have been to him as that of an angel—he let her go without an effort to stay her, even to prolong the sight. His horses were waiting, and one of his servants would follow with his mails; he mounted and rode away. The sun had set in a cloud, and the autumn evening was dark and gloomy, yet he rode along without any appearance of depression, steadily and quietly, like a man going about some business he has long expected to perform. I cannot even say he was sad: that moment had come to him which from his boyhood he had looked
- 80. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com