SlideShare a Scribd company logo
Chapter 1
Introduction




Recent advances in technology have led to dynamical systems with increasing com-
plexity, which in turn demand for more and more efficient and reliable control sys-
tems. It has been widely recognized that the requirements of specific behaviors and
stringent performances call for the inclusion of possible failure prevention in a mod-
ern control design. Therefore, either due to security reasons or efficiency necessity,
a system failure is a critical issue to be considered in the design of a controller in
modern technology. In view of this, dynamical systems that are subject to abrupt
changes have been a theme of increasing investigation in recent years and a vari-
ety of different approaches to analyze this class of systems has emerged over the
last decades. A particularly interesting class of models within this framework is the
so-called Markov jump linear systems (MJLS), which is the subject matter of this
book. The goal of this first chapter is to highlight, in a rather informal way, some
of the main characteristics of MJLS, through some illustrative examples of possible
applications of this class of systems.



1.1 Markov Jump Linear Systems

One of the main challenges when modeling a dynamical system is to find the best
trade-off between the mathematical complexity of the resulting equations and the
capability of obtaining a tractable problem. Thus, it is of overriding importance to
reach a proper balance in the settling of this issue while keeping in mind the per-
formance requirements. For instance, in many situations the use of robust control
techniques and some classical sensitivity analysis for time-invariant linear models
can handle in a simple and straightforward way the control problem of dynamical
systems subject to changes in their dynamics. However, if these changes signifi-
cantly alter the dynamical behavior of the system, these approaches may no longer
be adequate to meet the performance requirements. Within this scenario, the intro-
duction of some degree of specialization on the modeling of the dynamical system
in order to accommodate these changes is inevitable and perhaps even desirable.

O.L.V. Costa et al., Continuous-Time Markov Jump Linear Systems,                    1
Probability and Its Applications, DOI 10.1007/978-3-642-34100-7_1,
© Springer-Verlag Berlin Heidelberg 2013
2                                                                         1   Introduction

    The modeling of dynamic systems subject to abrupt changes in their dynamics
has been receiving lately a great deal of attention. These changes can be due, for in-
stance, to abrupt environmental disturbances, to actuator or sensor failure or repairs,
to the switching between economic scenarios, to abrupt changes in the operation
point for a non-linear plant, etc. Therefore, it is important to introduce mathemati-
cal models that take into account these kind of events and to develop control systems
capable of maintaining an acceptable behavior and meeting some performance re-
quirements even in the presence of abrupt changes in the system dynamics.
    In the case in which the dynamics of the system is subject to abrupt changes, one
can consider, for instance, that these changes are due to switching (jump) among
well-defined models. To illustrate this situation, consider a continuous-time dynam-
ical system that is, at a certain moment, well described by a model G1 . Suppose that,
after a certain amount of time, this system is subject to abrupt changes that cause
it to be described by a different model, say G2 . More generally, one can imagine
that the system is subject to a series of possible qualitative changes that make it to
switch, over time, among a countable set of models, for example, {G1 , G2 , . . . , GN }.
One can associate each of these models to an operation mode of the system, or just
mode, and say that the system jumps from one mode to the other or that there are
transitions between them. A central issue on this approach is how to append the
jumps into the model. A first step is to consider, for instance, that the jumps occur in
a random way, i.e., the mechanism that rules the switching between the aforemen-
tioned models is random. In addition, one can assume that this process (hereinafter
θ (t)) just indicates which model, among the {Gi , i = 1, 2, . . . , N }, is running the
system, i.e., θ (t) = i means that Gi is running the system. Furthermore, it would be
desirable to have some a priori information on the way in which the system jumps
from one operation mode to another (the transition mechanism). A random process
which bears these features and has been used with a great deal of success for mod-
eling these situations is the Markov chain.
    Within this framework, a particularly interesting class of models is the so-called
Markov jump linear systems (from now on MJLS). Since its introduction, this class
of models has an intimate connection with systems which are vulnerable to abrupt
changes in their structure, and the associated literature surrounding this subject is
now fairly extensive. Due, in part, to a large coherent body of theoretical results
on these systems, MJLS has been used recently in a number of applications on a
variety of fields, including robotics, air vehicles, economics, and some issues in
wireless communication, among others. For instance, it was mentioned in [266] that
the results achieved by MJLS, when applied to the synthesis problem of wing de-
ployment of an uncrewed air vehicle, were quite encouraging. As mentioned before,
the basic idea is to consider a family of continuous-time linear systems, which will
represent the possible modes of operation of the real system. The modal transition
is given by a Markov chain represented, as before, by θ (t), which is also known
in the literature as the operation mode (or the Markov state). This class of systems
will be the focus of investigation of the present book. We will restrict ourselves in
this book to the case in which all operation modes are continuous-time linear mod-
els and the jumps from one mode of operation to another follow a continuous-time
1.1 Markov Jump Linear Systems                                                          3

Markov chain taking values in a finite set {1, . . . , N}. In this scenario, it is possible
to develop a unified and coherent body of concepts and results for stability, filtering,
and control as well as to present controller and filter design procedures.
   In its most simple form, continuous-time MJLS are described as

                                   x(t) = Aθ(t) x(t).
                                   ˙                                                (1.1)

For a more general situation, known as the case with partial observations, continuous-
time MJLS are represented by
           ⎧
           ⎪ dx(t) = Aθ(t) (t)x(t) dt + Bθ(t) (t)u(t) dt + Jθ(t) (t) dw(t),
           ⎪
           ⎪
           ⎪
           ⎪
           ⎨ dy(t) = Hθ(t) (t)x(t) dt + Gθ(t) (t) dw(t),
      G=                                                                         (1.2)
           ⎪ z(t) = Cθ(t) (t)x(t) + Dθ(t) (t)u(t),
           ⎪
           ⎪
           ⎪
           ⎪
           ⎩
              x(0) = x0 ,      θ (0) = θ0 ,

with x(t) standing for the state variable of the system, u(t) the control variable, y(t)
the measured variable available to the controller, z(t) the output of the system, and
w(t) is a Wiener process.
   Although MJLS seem, at first sight, just an extension of linear systems, they dif-
fer from the latter in many instances. This is due, in particular, to some peculiar
properties of these systems that can be included in the class of complex systems
(roughly speaking, a system composed of interconnected parts that as a whole ex-
hibit one or more properties not obvious from the properties of the individual parts).
In order to give a small glimpse of this, let us consider a typical situation in which
one of these peculiar properties is unveiled. Consider a homogeneous Markov chain
θ with state space S = {1, 2} and transition rate matrix

                                    −β     β
                             Π=               ,       β > 0.
                                    β      −β

Some sample paths of θ , which in this case corresponds to a telegraph process with
exponentially distributed waiting times, are shown in Fig. 1.1. Clearly, an increase
in the transition rate gives rise to more frequent jumps (fast switching), with the
converse applying in the slow switching case.
   In this setting, consider system (1.1) evolving in R2 , with initial condition x(0) =
[1 −1] and state matrices
                               1
                                   −1                   −2 −1
                      A1 =     2      ,        A2 =         1 .                     (1.3)
                              0    −2                   0   2

The state trajectory of the system, corresponding to the realizations of θ depicted in
Fig. 1.1, is shown in Fig. 1.2. From these figures it is evident that the overall system
behavior may vary considerably, depending on how fast the switching occurs. Even
more remarkable is the fact that the state converges to the origin in the case β = 3/2,
4                                                                           1   Introduction




Fig. 1.1 Sample paths of θ for β = 3/2 (top) and β = 3/100 (bottom)




Fig. 1.2 Sample paths of x(t) corresponding to β = 3/2 (top) and β = 3/100 (bottom) in (1.3)


in spite of neither A1 nor A2 being stable. Suppose now that the system matrices
in (1.1) are replaced by

                              −1 10                      −1 0
                      A1 =          ,            A2 =          ,                      (1.4)
                              0 −1                       10 −1

which are both stable, i.e., have all the eigenvalues with negative real parts. In this
case the sample paths of x(t) corresponding to the same trajectories of θ are
depicted in Fig. 1.3. As these two situations suggest, Markovian switching between
stable (unstable) systems may produce unstable (stable) dynamics. In fact, as it will
1.1 Markov Jump Linear Systems                                                            5




Fig. 1.3 Sample paths of x(t) corresponding to β = 3/2 (top) and β = 3/100 (bottom) in (1.4)


be explicitly shown in Chap. 3, in the first case stability is equivalent to β > 4/3,
whereas in the second situation the overall system is stable if and only if β < 1/24.
   Some features which distinguish MJLS from the classical linear systems are:
• The stochastic process {x(t)} alone is no longer a Markov process.
• As seen in the previous examples, the stability (instability) for each mode of
  operation does not guarantee the stability (instability) of the system as a whole.
• The optimal filter for the case in which (x(t), θ (t)) are unknown is nonlinear and
  infinite-dimensional (in the filtering sense).
• Contrary to the linear time-invariant case, there is a fundamental limitation on
  the degree of robustness against linear perturbation that the H∞ control of MJLS
  may offer (see [284]).
If the state space of the Markov chain is infinitely countable, there are even further
distinctions as illustrated below:
• Mean-square stability is no longer equivalent to L2 -stability (see [148]).
• One cannot guarantee anymore that the maximal solution of the Riccati equation
  associated to the quadratic optimal control problem is a strong solution (see [17]).
   In order to develop a theory for MJLS using an operator theoretical approach, the
following steps are adopted:
(S.1) “Markovianize” the problem by considering as the state of the model the pair
      (x(t), θ (t)).
(S.2) Establish a connection between x(t) and measurable functions of the pair
      (x(t), θ (t)).
(S.3) Devise adequate operators from (S.2) by considering the first and second mo-
      ments.
6                                                                             1   Introduction

    These steps were followed in [152] (see also [77]), using the identities

                  N                                       N
         x(t) =         x(t)1{θ(t)=i}   and x(t)x(t)∗ =         x(t)x(t)∗ 1{θ(t)=i} ,
                  i=1                                     i=1

where 1{θ(t)=i} represents the Dirac measure over the set {θ (t) = i} (see (2.2)), and
∗ the transpose conjugate. The key point here is to work with x(t)1
                                                                            {θ(t)=i} and
x(t)x(t)∗ 1{θ(t)=i} , which are measurable functions of (x(t), θ (t)), and obtain dif-
ferential equations for the first and second moments in terms of some appropriate
operators (see Sect. 3.3). This approach has uncovered many new differences be-
tween the MJLS and its linear classical counterpart, and has allowed the develop-
ment of several new theoretical results and applications for MJLS. For instance, it
is possible to devise a spectral criterion for mean-square stability and study stability
radius in the same spirit as the one found in the linear system theory. What unifies
the body of results in this book, and has certainly helped us to choose its content, is
this particular approach, which we will call the analytical point of view (from now
on APV).
    It is worth mentioning two other distinctive approaches, which, together with the
APV, have given rise to a host of important results on various topics of MJLS: the
so-called Multiple Model (MM) approach and the Hidden Markov Model (HMM)
approach. In the MM approach the idea is, roughly speaking, to devise strategies that
decide in an efficient way which mode is running the system and work with the lin-
ear system associated to this mode (see, e.g., [22] for a comprehensive treatment on
this subject). The HMM approach focuses on what is known in the control literature
as a class of partially observed stochastic dynamical systems. The basic framework
for the HMM consists of a Markov process θ (t) that is not directly observed but is
hidden in a noisy observation process y(t). Roughly speaking, the aim is to estimate
the Markov process, given the related observations, and from this estimation to de-
rive a control strategy for the hidden Markov process (usually the transition matrix
of the chain depends on the control variable u(t)). See, e.g., [130] for a modern
treatment on this topic.
    The present book can be seen, roughly speaking, as a continuation of the book
[81], which dealt with the discrete-time case, to the continuous-time case. From
a practical viewpoint, perhaps the most important aspect which favors the study
of continuous-time systems is that many models in science are based upon spe-
cific relations between the system variables and their instantaneous rates of varia-
tion (some examples being Newton’s second law of classical mechanics, Maxwell’s
nonstationary equations of electromagnetics, the law of continuity of fluid transport
phenomena, Verhulst’s logistic equation, and Gompertz’s model of tumor growth,
among many others). In these cases, the direct application of discrete-time systems
theory may be a potential cause for distortion to the model in the form, e.g., of
spurious effects such as artificial energy dissipation, instability, oscillations, and a
potentially cumbersome dependence on the discretization parameters (which may
even be plagued by the curse of dimensionality, as pointed out in [114]). Hence, it
1.2 Some Applications of MJLS                                                       7

Fig. 1.4 Transitions between
the different scenarios of a
country’s economy




is fair to say that, prior to sampling, a cautious study of the underlying continuous-
time process should be carried out in order to ensure that such adversities would not
hinder the desired specifications to be achieved in a given application.



1.2 Some Applications of MJLS
Since their inception in the early 1960s, MJLS have found many applications in a
great variety of fields. These include unmanned air vehicles [266], solar power sta-
tions [275], satellite dynamics [226], economics [32, 33, 52, 108, 268], flight sys-
tems [174, 175], power systems [206, 211, 212, 290, 291], communication systems
[1, 2, 208, 243], among many others. This section is devoted to a brief exposition of
some selected topics regarding applications of MJLS, with special attention to those
in continuous time.
    The earliest application of MJLS to economics, introduced in the discrete-time
scenario in [33], was based on Samuelson’s multiplier–accelerator macroeconomic
model (see [252, 286, 299]). The model studied in [33] consists of a very simpli-
fied relation for the dynamical evolution of a country’s national income in terms of
the governmental expenditure, which is weighted by two parameters (the marginal
propensity to save and the accelerator coefficient). Based on the historical data ob-
tained from the U.S. Department of Commerce from years 1929 until 1971, [33]
assumed that the state of the economy could be roughly lumped in three possible
operation modes (“normal”, “boom”, and “slump”) and that the switching between
them could be modeled as a homogeneous Markov chain, with the transition rates
depicted in Fig. 1.4. The subsequent problem considered in [33] corresponds to
an MJLS version of the optimal linear quadratic control setup. This application,
which was also analyzed in [81] for the discrete-time case, will be further studied in
Chap. 10.
    Another practical application of MJLS, which, in the discrete-time case, has been
extensively discussed in [81], is the control of a solar power receiver. In contin-
uous time, this situation was initially considered in [275] with θ (t) representing
8                                                                        1   Introduction




Fig. 1.5 Simplified boiler configuration


abrupt environmental changes between “sunny” and “cloudy” conditions, which are
measured by sensors located on the plant. The role of control is to determine the
feedwater flow rate which will enter the boiler, in such a way as to regulate the
outflow temperature at the desired level. The boiler flow rate is strongly dependent
upon the receiving insolation, and, as a result of this abrupt variability, several lin-
earized models are required to characterize the evolution of the boiler when clouds
interfere with the sun’s rays. The control law described in [275] makes use of the
state feedback and a measurement of θ (t) through the use of flux sensors on the
receiver panels. In Fig. 1.5 a very simplified representation of the system is shown.
Although several important blocks such as the boiler internal models for steam and
metal dynamics, thermal couplings, and feedforward compensation loops are not
displayed, their abstraction evidences the dependence of the control system on the
abrupt variability of the received insolation.
    Robotic manipulator arms are employed in a great deal of modern applications,
which span areas as diverse as deep sea engineering, manufacturing processes, space
technology, or teleoperated medicine, for instance. The adequate operation of such
devices, however, is severely compromised by the occurrence of failures, which may
be intolerable in safety-critical applications, for example. Furthermore, repairing the
faulty arm may frequently turn out to be a difficult task during the course of opera-
tion (which may occur if, e.g., the robot is functioning in a hazardous environment).
On this regard, a great deal of research has been carried out on the control of robotic
arms with less actuators than degrees of freedom, commonly referred to as under-
actuated manipulators in the specialized literature [11, 12, 92]. In practical terms, a
manipulator arm is said to be underactuated whenever the motor on at least one of
its joints is in a passive state. The basic principle in the operation of these devices
is then to explore the dynamic coupling that the active joints impose on the passive
ones, in such a way as to drive the arm to the desired position in spite of its faulty
condition.
    The introduction of MJLS theory to tackle the control of underactuated robotic
arms was made by Siqueira and Terra in [261], in the discrete-time setting. This
approach was subsequently described in [81] and, more recently, brought to the
continuous-time setting in [262, 263]. In Chap. 10 the robust control of the planar
3-link underactuated arm depicted in Fig. 1.6 is further studied by means of the
results devised in this book.
1.2 Some Applications of MJLS                                                         9

Fig. 1.6 Geometric modeling
of the planar 3-link arm




Fig. 1.7 Birth–death process considered in [175]


   In yet another field, MJLS were employed in [175] for the stability analysis of
controlled flight systems affected by electromagnetic disturbances. This problem
stems from the susceptibility of electronic devices against external disturbances
such as lightning, thermal noise, and radio signals, for instance, which is recog-
nized as a potential cause for computer upsets in digital controllers. In order to cope
with the resulting adversities, which may range from random bit errors to permanent
computer failures, the strategy in [175] is to model the accumulative effect of exter-
nal disturbances on the system by means of a continuous-time birth–death Markov
process of the form indicated in Fig. 1.7. The rationale for this choice has been, as
pointed out in [175], that by doing so the arrival of new disturbances is dictated by
a Poisson process with exponentially distributed sojourn times.
   An important issue in this situation is that, while the aircraft dynamics treated
in [175, Sect. IV.B] evolves in continuous time, the occurrence of electromagnetic
disturbances takes place in the sampled-data digital controller implementation. This
leaves open an interesting conjecture of whose aspects of continuous- and discrete-
time MJLS theory should be relevant to the problem at hand.
   In [211, 212] the modeling and control of power systems subject to Markov
jumps has been addressed in the continuous-time scenario. In this case the switching
mechanism is used to model random changes in the load, generating unit outages,
and transmission line faults, for instance. As shown in [212], intermittent couplings
between electrical machines operating in a network can render the overall system
unstable in a stochastic sense, a result which somewhat resembles the analysis previ-
ously carried out for (1.1) in the cases (1.3) and (1.4). In [211] the main results were
10                                                                       1   Introduction

applied to the problem of dynamic security assessment, which amounts to determin-
ing, with desired probability, whether certain parameters of the electrical system are
guaranteed to remain within a safe region of operation at a given period. A “security
measure” was defined, which corresponds to a quantitative indicator of the vulner-
ability of the current system state and network topology to stochastic contingency
events. Two types of switchings are considered therein: primary ones, driven by a
continuous-time Markov process taking values in a finite set, and secondary events,
which are modeled by state-dependent (controlled) jumps. Recent advances in the
control of power systems have also been reported in [206, 290, 291] by means of
decentralized control methods and the S-procedure. An alternative account of this
problem via robust control methods is presented in details in Sect. 10.3.
    The modeling of communication systems via MJLS is by now another promis-
ing trend in the applications front. In discrete-time this is boldly motivated by the
connection between MJLS and the Gilbert–Elliott model for burst communication
channels (see [129, 167, 172, 173, 186, 244, 254]), which in its simplest form corre-
sponds to a two-state Markov chain. A convenient feature of these models (besides
their relative simplicity) is that they are capable of describing the fact that eventual
packet losses typically occur during intervals of time (e.g., while a wireless link
is obstructed). In other words, isolated packet losses are not common events, and
it usually takes a while before communication, once lost, is restored. As pointed
out in [167, 173, 186, 254], large packet-loss rates imply poor performance or even
instability, and therefore controllers implemented within a network may provide
considerably better results if their design takes into account the probabilistic nature
of the network.
    In the continuous-time scenario, references [1, 2] treated the problem of dynamic
routing in mobile networks. In loose terms, this amounts to determining a route
within the network topology constraints, through which information packets will
travel from one given node to another. Of course, many of these routing operations
will typically occur at the same time and between different nodes and directions, so
that the routing algorithm must be able to provide a satisfactory quality of service
for all customers (for example, by delivering packets in the shortest time, with as lit-
tle lag as possible, and with a very low packet loss rate), without violating physical
constraints such as link capacity, memory size (queueing length), or power avail-
ability. The approach in [1, 2] considers that sudden variations, modeled by MJLS,
occur in the network due to, e.g., mobility and topological variations. The consid-
eration of time delay in the underlying model is of major importance, owing to the
time that packets must wait on a queue before being processed, together with the
fact that the network nodes are geographically separated. Furthermore, this latter
constraint motivated the consideration of a decentralized control scheme. The ulti-
mate problem considered in [1, 2] was the minimization of the worst-case queueing
length, with the aid of H∞ control methods.
1.3 Prerequisites and General Remarks                                               11

1.3 Prerequisites and General Remarks

As prerequisite for this book, it is desirable some knowledge on the classical linear
control theory (stability results, the linear quadratic Gaussian (LQG) control prob-
lem, H∞ control problem, and Riccati equations), some familiarity with continuous-
time Markov chains and probability theory, and some basic knowledge of operator
theory.
   In this book we follow an approach that combines probability and operator theory
to develop the results. By doing this we believe that the book provides a unified and
rigorous treatment of recent results for the control theory of continuous-time MJLS.
Most of the material included in the book was published after 1990. The goal is to
provide a complete and unified picture on the main topics of the control theory of
continuous-time MJLS such as mean-square stability, quadratic control and H2 con-
trol for the complete and partial observations (also called partial information) cases,
associated coupled differential and algebraic Riccati equations, linear filtering, and
H∞ control. The book also intends to present some design algorithms mainly based
on linear matrix inequalities (LMIs) optimization tool packages.
   One of the objectives of the book is to introduce, as far as possible in a friendly
way, a bent of the MJLS theory that we have named here as the analytical point of
view. We believe (and do hope) that experts in linear systems with Markov jump
parameters will find in this book the minimal essential tools to follow this approach.
Moreover, the stochastic control problems for MJLS considered in this book provide
one of those few cases in the stochastic control field in which explicit solutions can
be obtained, being a useful material for a course and for introducing students into an
interesting and active research area. In addition, we do hope to motivate the reader,
especially the graduate students, in such a way that this book could be a starting
point for further developments and applications of continuous-time MJLS. From the
application point of view we believe that the book provides a powerful theory with
potential application in systems whose dynamics are subject to abrupt changes, as
those found in safety-critical and high-integrity systems, industrial plants, economic
systems, etc.



1.4 Overview of the Chapters

We next present a brief overview of the contents of the chapters of the book.
    Chapter 2 is dedicated to present some background material needed throughout
the book, as the notation, norms, and spaces that are appropriate for our approach.
It also presents some important auxiliary results, especially related to the stability
concepts to be considered in Chap. 3. A few facts on Markov chains and the bear
essential of infinitesimal generator is also included. We also recall some basic facts
regarding LMIs, which are useful for the design techniques and the H∞ control
problem.
12                                                                      1   Introduction

    Chapter 3 deals with mean-square stability for continuous-time MJLS. It is
shown that mean-square stability is equivalent to the maximal real part of the eigen-
values of an augmented matrix being less than zero or to the existence of a solution
of a Lyapunov equation. As aforementioned, stability of all modes of operation is
neither necessary nor sufficient for global stability of the system. The criterion based
on the eigenvalues of an augmented matrix reveals that a balance between the modes
and the transition rate matrix is essential for mean-square stability.
    It is worth pointing out that the setup of Chap. 3 is on the domain of complex
matrices and vectors. By doing this we can use a useful result on the decomposition
of a matrix into some positive semidefinite matrices, which is very important to
prove the equivalence on stability results. But it is shown later on in this chapter
that the results obtained are valid even in the setup of real matrices and vectors. In
order to simplify the notation and proofs, in the remaining chapters of the book we
consider just the real case.
    Chapter 4 analyzes the quadratic optimal control problem for MJLS in the usual
finite- and infinite-time horizon framework. We consider in this chapter that the con-
troller has access to both the state variable x(t) and the jump parameter θ (t). The
case in which the controller has access only to an output y(t) and θ (t) is considered
in Chap. 6 and called the partial observation (or partial information) case. The so-
lution for the quadratic optimal control problems of Chap. 4 relies, in part, on the
study of a finite set of coupled differential and algebraic Riccati equations (CDRE
and CARE, respectively). These equations are studied in the Appendix A.
    Chapter 5 restudies the infinite-horizon quadratic optimal control for MJLS but
now from another point of view, usually known in the literature of linear systems as
the H2 control. The advantage of the H2 approach is that it allows one to consider
parametric uncertainties and solve the problem using LMIs optimization tools.
    Chapter 6 deals with the finite-horizon quadratic optimal control problem and
the H2 control problem of continuous-time MJLS for the partial information case.
The main result shown is that the optimal control is obtained from two sets of cou-
pled differential (for the finite-horizon case) and algebraic (for the H2 case) Riccati
equations, one set associated with the optimal control problem when the state vari-
able is available, as analyzed in Chaps. 4 and 5, and the other set associated with
the optimal filtering problem. This establishes the so-called separation principle for
continuous-time MJLS.
    Chapter 7 aims to derive the best linear mean square estimator of continuous-
time MJLS assuming that only an output y(t) is available. It is important to em-
phasize that in this chapter we assume that the jump parameter θ (t) is not known.
The idea is to derive a filter which bears those desirable properties of the Kalman
filter: a recursive scheme suitable for computer implementation which allows some
offline computation that alleviates the computational burden. The linear filter has
dimension N n (where n denotes the dimension of the state vector, and N the num-
ber of states of the Markov chain). Both the finite-horizon and stationary cases are
considered.
    Chapter 8 is devoted to the H∞ control of Markov jump linear systems, in the
infinite-horizon setting. The statement of a bounded real lemma is the starting point
1.5 Historical Remarks                                                                13

toward a complete LMIs characterization of static state feedback, as well as of
full-order dynamic output feedback stabilizing controllers that guarantee that a pre-
scribed closed-loop H∞ performance is attained. The main results include explicit
formulas and the corresponding algorithms for designing the controllers of interest.
    Chapters 9 and 10 are intended to conclude the book assembling some problems
in the Markov jump context and the tools to solve them.
    In the Appendix A some results on coupled differential and algebraic Riccati
equations (CDRE and CARE, respectively) associated to the control problem are
presented. Initially, we consider the problem of uniqueness, existence, positive defi-
niteness, and continuity of the solution of the CDRE. After that we study the CARE,
dealing essentially with conditions for the existence of solutions and asymptotic
convergence, based on the concepts of mean-square stabilizability and detectability
seen in Chap. 3. Regarding the existence of a solution, we are particularly interested
in maximal and stabilizing solutions.
    In the Appendix B we derive some auxiliary results related to an adjoint operator
for MJLS used in the separation principle presented in Chap. 6.



1.5 Historical Remarks

The study of dynamical systems with random parameters can be traced back at least
to [196, 200–202], and [142]. Due to the importance and complexity of the theme, a
variety of techniques has emerged, and an extraordinary burst of publications asso-
ciated with terminologies such as multiple model, switching systems, hidden Markov
models and Markov jump linear systems, inter alia, has appeared in the specialized
literature. Even for those approaches which embraced the idea of modeling the ran-
dom parameter as a Markov chain, the methodologies were different. Each one of
these topics have charted its own course, and the associated literature is by now
huge. Therefore, it is out of the scope here to go into details on all of these topics
(see, e.g., [22, 130, 232] and references therein for an account on multiple model,
hidden Markov model, and switching systems, respectively). We focus instead on
the MJLS case.
    Regarding MJLS, the seminal papers [270] and [303] set the stage for future
research in this area. In the first one, the jump linear quadratic (JLQ) control problem
was considered for the finite-horizon setting, via a maximum principle approach
(see also [269]). In the other one, dynamic programming was used, and the infinite-
horizon case was also treated. In this case, although the objective had been carried
out successfully, a technical inconvenience was related to the choice of the adopted
stability criteria, which did not seem to be fully adequate. This, in turn, has entailed
a great deal of challenges for future research. In the 1970s we can mention the
papers [32, 248, 273, 274] and [33] dealing with the JLQ control problem, where
the latter seems to be the first one that treated the discrete-time version of the optimal
quadratic control problem for the finite-time horizon case (see also [20] for the MM
approach).
14                                                                       1   Introduction

    Due, in part, to the lack of a suitable concept for stability, it took a while for
the theory flourish. For instance, it elapsed more than ten years to appear some of
the key papers that put mean-square stability (stochastic stability, or L2 -stability),
mean square stabilizability, and mean square detectability for MJLS in a solid
ground. Without any intention of being exhaustive here, we mention, for instance,
[43, 48, 71, 77, 79, 118, 137, 147, 148, 150–152, 156, 164, 170, 170, 187, 189–
191, 210, 221, 230, 231, 245, 278, 282, 284], for a sample of papers dealing with
stability and stabilizability for MJLS. For the case in which the state space of the
Markov chain is countably infinite, it was shown in [78] and [148] that mean-square
and stochastic stability (L2 -stability) are no longer equivalent (see also [147]). An-
other critical issue was an adequate concept for the solution of the Riccati equa-
tion associated to MJLS (coupled Riccati equations). As far as the authors are
aware of, it was in [156] where the concept of mean-square solution for the cou-
pled Riccati equations was first proposed. For a glimpse on some issues dealing
with coupled Riccati equations, see, for instance, [4–6, 17, 66, 67, 149, 156, 246].
For control problems (optimal, adaptive, H∞ , H2 , robust, receding horizon, singu-
larly perturbed, partial observations, etc.), we mention, for instance, [18, 38, 44, 46,
47, 59, 60, 69, 70, 73–75, 78, 85, 87, 93, 94, 101, 104, 108, 111, 119, 120, 122,
125, 134, 143–146, 158, 159, 171, 176, 188, 190, 224, 229, 240, 258, 279, 307].
For the filtering problem, the reader is referred, for instance, to [9, 35–37, 65, 82–
84, 126, 127, 132, 155, 160, 217, 321, 322]. The case with delay was treated,
for instance, in [24, 44, 45, 55, 165, 209, 218, 257, 297]. The case with un-
certainty (including uncertainty in the transition matrix of the Markov chain)
was considered in [15, 26, 42, 256, 305, 306, 316–319]. Separation principles
were derived, for instance, in [80, 89, 90, 153]. Structural properties such as
controllability, detectability, and observability have been studied, for instance, in
[61, 63, 187, 189, 222]. See [287, 288] for problems related to detection and iden-
tification for MJLS. In addition, there is by now a growing conviction that MJLS
provide models of wide applicability (see, e.g., [14] and [266]). The evidence in
favor of such a proposition has been amassing rapidly over the last decades. We
mention [1, 2, 8, 14, 21, 28, 32, 39, 52, 57, 108, 124, 133, 141, 173–175, 184, 186,
197, 198, 208, 213, 219, 220, 225, 226, 228, 241, 243, 249, 253, 254, 266, 275]
and [323], as works dealing with applications of this class of systems (see also the
books [22, 40, 41, 81, 121, 223, 271, 309] and references therein).
http://www.springer.com/978-3-642-34099-4

More Related Content

PDF
Lec markov note
PDF
Finite Time Stability of Linear Control System with Constant Delay in the State
PDF
IFAC2008art
PDF
Detecting Assignable Signals Via Decomposition Of Newma Statistic
PDF
Introduction to finite element analysis
PPTX
Presentation on GMM
PDF
Tutorial marzo2011 villen
PDF
A New Method of Determining Instability of Linear System
Lec markov note
Finite Time Stability of Linear Control System with Constant Delay in the State
IFAC2008art
Detecting Assignable Signals Via Decomposition Of Newma Statistic
Introduction to finite element analysis
Presentation on GMM
Tutorial marzo2011 villen
A New Method of Determining Instability of Linear System

What's hot (9)

PDF
A High Order Continuation Based On Time Power Series Expansion And Time Ratio...
PDF
Invariant Manifolds, Passage through Resonance, Stability and a Computer Assi...
PDF
An alternative scheme for approximating a periodic function
PDF
Hall 2006 problems-encounteredfromuserayleighdamping
PDF
Comparisons
PDF
Gmm chapter6
PDF
Machiwal, D. y Jha, MK (2012). Modelado estocástico de series de tiempo. En A...
PPT
Introduction to finite element method
PPT
Fem in geotech engineering
A High Order Continuation Based On Time Power Series Expansion And Time Ratio...
Invariant Manifolds, Passage through Resonance, Stability and a Computer Assi...
An alternative scheme for approximating a periodic function
Hall 2006 problems-encounteredfromuserayleighdamping
Comparisons
Gmm chapter6
Machiwal, D. y Jha, MK (2012). Modelado estocástico de series de tiempo. En A...
Introduction to finite element method
Fem in geotech engineering
Ad

Similar to Continuous time markov jump linear system (20)

PDF
solver (1)
PPTX
Discrete state space model 9th &10th lecture
PDF
Modern Control - Lec 02 - Mathematical Modeling of Systems
PDF
Introduction to Hybrid Vehicle System Modeling and Control - 2013 - Liu - App...
PPTX
control system, open and closed loop engineering
PPTX
Introduction to mathematical control theory - Dr. Purnima Pandit
PDF
Compit 2013 - Torsional Vibrations under Ice Impact
PDF
VasasElekMarkus_RSAR_JSPI_final
PDF
Modern Control System (BE)
PDF
Chaos Suppression and Stabilization of Generalized Liu Chaotic Control System
PPT
Transfer Function Cse ppt
PPTX
Time response of discrete systems 4th lecture
PPT
Dynamic Process Modeling
PPT
Dynamic Process Modeling
PDF
Adaptive Projective Lag Synchronization of T and Lu Chaotic Systems
PDF
SYNERGETIC CONTROL AND SYNCHRONISATION OF CHAOTIC SYSTEMS
PDF
Test Generation for Analog and Mixed-Signal Circuits Using Hybrid System Mode...
PDF
TEST GENERATION FOR ANALOG AND MIXED-SIGNAL CIRCUITS USING HYBRID SYSTEM MODELS
PDF
1 Aminullah Assagaf_Estimation-of-domain-of-attraction-for-the-fract_2021_Non...
PPT
lecture1ddddgggggggggggghhhhhhh (11).ppt
solver (1)
Discrete state space model 9th &10th lecture
Modern Control - Lec 02 - Mathematical Modeling of Systems
Introduction to Hybrid Vehicle System Modeling and Control - 2013 - Liu - App...
control system, open and closed loop engineering
Introduction to mathematical control theory - Dr. Purnima Pandit
Compit 2013 - Torsional Vibrations under Ice Impact
VasasElekMarkus_RSAR_JSPI_final
Modern Control System (BE)
Chaos Suppression and Stabilization of Generalized Liu Chaotic Control System
Transfer Function Cse ppt
Time response of discrete systems 4th lecture
Dynamic Process Modeling
Dynamic Process Modeling
Adaptive Projective Lag Synchronization of T and Lu Chaotic Systems
SYNERGETIC CONTROL AND SYNCHRONISATION OF CHAOTIC SYSTEMS
Test Generation for Analog and Mixed-Signal Circuits Using Hybrid System Mode...
TEST GENERATION FOR ANALOG AND MIXED-SIGNAL CIRCUITS USING HYBRID SYSTEM MODELS
1 Aminullah Assagaf_Estimation-of-domain-of-attraction-for-the-fract_2021_Non...
lecture1ddddgggggggggggghhhhhhh (11).ppt
Ad

More from Springer (20)

PDF
The chemistry of the actinide and transactinide elements (set vol.1 6)
PDF
Transition metal catalyzed enantioselective allylic substitution in organic s...
PDF
Total synthesis of natural products
PDF
Solid state nmr
PDF
Mass spectrometry
PDF
Higher oxidation state organopalladium and platinum
PDF
Principles and applications of esr spectroscopy
PDF
Inorganic 3 d structures
PDF
Field flow fractionation in biopolymer analysis
PDF
Thermodynamics of crystalline states
PDF
Theory of electroelasticity
PDF
Tensor algebra and tensor analysis for engineers
PDF
Springer handbook of nanomaterials
PDF
Shock wave compression of condensed matter
PDF
Polarization bremsstrahlung on atoms, plasmas, nanostructures and solids
PDF
Nanostructured materials for magnetoelectronics
PDF
Nanobioelectrochemistry
PDF
Modern theory of magnetism in metals and alloys
PDF
Mechanical behaviour of materials
PDF
Magnonics
The chemistry of the actinide and transactinide elements (set vol.1 6)
Transition metal catalyzed enantioselective allylic substitution in organic s...
Total synthesis of natural products
Solid state nmr
Mass spectrometry
Higher oxidation state organopalladium and platinum
Principles and applications of esr spectroscopy
Inorganic 3 d structures
Field flow fractionation in biopolymer analysis
Thermodynamics of crystalline states
Theory of electroelasticity
Tensor algebra and tensor analysis for engineers
Springer handbook of nanomaterials
Shock wave compression of condensed matter
Polarization bremsstrahlung on atoms, plasmas, nanostructures and solids
Nanostructured materials for magnetoelectronics
Nanobioelectrochemistry
Modern theory of magnetism in metals and alloys
Mechanical behaviour of materials
Magnonics

Continuous time markov jump linear system

  • 1. Chapter 1 Introduction Recent advances in technology have led to dynamical systems with increasing com- plexity, which in turn demand for more and more efficient and reliable control sys- tems. It has been widely recognized that the requirements of specific behaviors and stringent performances call for the inclusion of possible failure prevention in a mod- ern control design. Therefore, either due to security reasons or efficiency necessity, a system failure is a critical issue to be considered in the design of a controller in modern technology. In view of this, dynamical systems that are subject to abrupt changes have been a theme of increasing investigation in recent years and a vari- ety of different approaches to analyze this class of systems has emerged over the last decades. A particularly interesting class of models within this framework is the so-called Markov jump linear systems (MJLS), which is the subject matter of this book. The goal of this first chapter is to highlight, in a rather informal way, some of the main characteristics of MJLS, through some illustrative examples of possible applications of this class of systems. 1.1 Markov Jump Linear Systems One of the main challenges when modeling a dynamical system is to find the best trade-off between the mathematical complexity of the resulting equations and the capability of obtaining a tractable problem. Thus, it is of overriding importance to reach a proper balance in the settling of this issue while keeping in mind the per- formance requirements. For instance, in many situations the use of robust control techniques and some classical sensitivity analysis for time-invariant linear models can handle in a simple and straightforward way the control problem of dynamical systems subject to changes in their dynamics. However, if these changes signifi- cantly alter the dynamical behavior of the system, these approaches may no longer be adequate to meet the performance requirements. Within this scenario, the intro- duction of some degree of specialization on the modeling of the dynamical system in order to accommodate these changes is inevitable and perhaps even desirable. O.L.V. Costa et al., Continuous-Time Markov Jump Linear Systems, 1 Probability and Its Applications, DOI 10.1007/978-3-642-34100-7_1, © Springer-Verlag Berlin Heidelberg 2013
  • 2. 2 1 Introduction The modeling of dynamic systems subject to abrupt changes in their dynamics has been receiving lately a great deal of attention. These changes can be due, for in- stance, to abrupt environmental disturbances, to actuator or sensor failure or repairs, to the switching between economic scenarios, to abrupt changes in the operation point for a non-linear plant, etc. Therefore, it is important to introduce mathemati- cal models that take into account these kind of events and to develop control systems capable of maintaining an acceptable behavior and meeting some performance re- quirements even in the presence of abrupt changes in the system dynamics. In the case in which the dynamics of the system is subject to abrupt changes, one can consider, for instance, that these changes are due to switching (jump) among well-defined models. To illustrate this situation, consider a continuous-time dynam- ical system that is, at a certain moment, well described by a model G1 . Suppose that, after a certain amount of time, this system is subject to abrupt changes that cause it to be described by a different model, say G2 . More generally, one can imagine that the system is subject to a series of possible qualitative changes that make it to switch, over time, among a countable set of models, for example, {G1 , G2 , . . . , GN }. One can associate each of these models to an operation mode of the system, or just mode, and say that the system jumps from one mode to the other or that there are transitions between them. A central issue on this approach is how to append the jumps into the model. A first step is to consider, for instance, that the jumps occur in a random way, i.e., the mechanism that rules the switching between the aforemen- tioned models is random. In addition, one can assume that this process (hereinafter θ (t)) just indicates which model, among the {Gi , i = 1, 2, . . . , N }, is running the system, i.e., θ (t) = i means that Gi is running the system. Furthermore, it would be desirable to have some a priori information on the way in which the system jumps from one operation mode to another (the transition mechanism). A random process which bears these features and has been used with a great deal of success for mod- eling these situations is the Markov chain. Within this framework, a particularly interesting class of models is the so-called Markov jump linear systems (from now on MJLS). Since its introduction, this class of models has an intimate connection with systems which are vulnerable to abrupt changes in their structure, and the associated literature surrounding this subject is now fairly extensive. Due, in part, to a large coherent body of theoretical results on these systems, MJLS has been used recently in a number of applications on a variety of fields, including robotics, air vehicles, economics, and some issues in wireless communication, among others. For instance, it was mentioned in [266] that the results achieved by MJLS, when applied to the synthesis problem of wing de- ployment of an uncrewed air vehicle, were quite encouraging. As mentioned before, the basic idea is to consider a family of continuous-time linear systems, which will represent the possible modes of operation of the real system. The modal transition is given by a Markov chain represented, as before, by θ (t), which is also known in the literature as the operation mode (or the Markov state). This class of systems will be the focus of investigation of the present book. We will restrict ourselves in this book to the case in which all operation modes are continuous-time linear mod- els and the jumps from one mode of operation to another follow a continuous-time
  • 3. 1.1 Markov Jump Linear Systems 3 Markov chain taking values in a finite set {1, . . . , N}. In this scenario, it is possible to develop a unified and coherent body of concepts and results for stability, filtering, and control as well as to present controller and filter design procedures. In its most simple form, continuous-time MJLS are described as x(t) = Aθ(t) x(t). ˙ (1.1) For a more general situation, known as the case with partial observations, continuous- time MJLS are represented by ⎧ ⎪ dx(t) = Aθ(t) (t)x(t) dt + Bθ(t) (t)u(t) dt + Jθ(t) (t) dw(t), ⎪ ⎪ ⎪ ⎪ ⎨ dy(t) = Hθ(t) (t)x(t) dt + Gθ(t) (t) dw(t), G= (1.2) ⎪ z(t) = Cθ(t) (t)x(t) + Dθ(t) (t)u(t), ⎪ ⎪ ⎪ ⎪ ⎩ x(0) = x0 , θ (0) = θ0 , with x(t) standing for the state variable of the system, u(t) the control variable, y(t) the measured variable available to the controller, z(t) the output of the system, and w(t) is a Wiener process. Although MJLS seem, at first sight, just an extension of linear systems, they dif- fer from the latter in many instances. This is due, in particular, to some peculiar properties of these systems that can be included in the class of complex systems (roughly speaking, a system composed of interconnected parts that as a whole ex- hibit one or more properties not obvious from the properties of the individual parts). In order to give a small glimpse of this, let us consider a typical situation in which one of these peculiar properties is unveiled. Consider a homogeneous Markov chain θ with state space S = {1, 2} and transition rate matrix −β β Π= , β > 0. β −β Some sample paths of θ , which in this case corresponds to a telegraph process with exponentially distributed waiting times, are shown in Fig. 1.1. Clearly, an increase in the transition rate gives rise to more frequent jumps (fast switching), with the converse applying in the slow switching case. In this setting, consider system (1.1) evolving in R2 , with initial condition x(0) = [1 −1] and state matrices 1 −1 −2 −1 A1 = 2 , A2 = 1 . (1.3) 0 −2 0 2 The state trajectory of the system, corresponding to the realizations of θ depicted in Fig. 1.1, is shown in Fig. 1.2. From these figures it is evident that the overall system behavior may vary considerably, depending on how fast the switching occurs. Even more remarkable is the fact that the state converges to the origin in the case β = 3/2,
  • 4. 4 1 Introduction Fig. 1.1 Sample paths of θ for β = 3/2 (top) and β = 3/100 (bottom) Fig. 1.2 Sample paths of x(t) corresponding to β = 3/2 (top) and β = 3/100 (bottom) in (1.3) in spite of neither A1 nor A2 being stable. Suppose now that the system matrices in (1.1) are replaced by −1 10 −1 0 A1 = , A2 = , (1.4) 0 −1 10 −1 which are both stable, i.e., have all the eigenvalues with negative real parts. In this case the sample paths of x(t) corresponding to the same trajectories of θ are depicted in Fig. 1.3. As these two situations suggest, Markovian switching between stable (unstable) systems may produce unstable (stable) dynamics. In fact, as it will
  • 5. 1.1 Markov Jump Linear Systems 5 Fig. 1.3 Sample paths of x(t) corresponding to β = 3/2 (top) and β = 3/100 (bottom) in (1.4) be explicitly shown in Chap. 3, in the first case stability is equivalent to β > 4/3, whereas in the second situation the overall system is stable if and only if β < 1/24. Some features which distinguish MJLS from the classical linear systems are: • The stochastic process {x(t)} alone is no longer a Markov process. • As seen in the previous examples, the stability (instability) for each mode of operation does not guarantee the stability (instability) of the system as a whole. • The optimal filter for the case in which (x(t), θ (t)) are unknown is nonlinear and infinite-dimensional (in the filtering sense). • Contrary to the linear time-invariant case, there is a fundamental limitation on the degree of robustness against linear perturbation that the H∞ control of MJLS may offer (see [284]). If the state space of the Markov chain is infinitely countable, there are even further distinctions as illustrated below: • Mean-square stability is no longer equivalent to L2 -stability (see [148]). • One cannot guarantee anymore that the maximal solution of the Riccati equation associated to the quadratic optimal control problem is a strong solution (see [17]). In order to develop a theory for MJLS using an operator theoretical approach, the following steps are adopted: (S.1) “Markovianize” the problem by considering as the state of the model the pair (x(t), θ (t)). (S.2) Establish a connection between x(t) and measurable functions of the pair (x(t), θ (t)). (S.3) Devise adequate operators from (S.2) by considering the first and second mo- ments.
  • 6. 6 1 Introduction These steps were followed in [152] (see also [77]), using the identities N N x(t) = x(t)1{θ(t)=i} and x(t)x(t)∗ = x(t)x(t)∗ 1{θ(t)=i} , i=1 i=1 where 1{θ(t)=i} represents the Dirac measure over the set {θ (t) = i} (see (2.2)), and ∗ the transpose conjugate. The key point here is to work with x(t)1 {θ(t)=i} and x(t)x(t)∗ 1{θ(t)=i} , which are measurable functions of (x(t), θ (t)), and obtain dif- ferential equations for the first and second moments in terms of some appropriate operators (see Sect. 3.3). This approach has uncovered many new differences be- tween the MJLS and its linear classical counterpart, and has allowed the develop- ment of several new theoretical results and applications for MJLS. For instance, it is possible to devise a spectral criterion for mean-square stability and study stability radius in the same spirit as the one found in the linear system theory. What unifies the body of results in this book, and has certainly helped us to choose its content, is this particular approach, which we will call the analytical point of view (from now on APV). It is worth mentioning two other distinctive approaches, which, together with the APV, have given rise to a host of important results on various topics of MJLS: the so-called Multiple Model (MM) approach and the Hidden Markov Model (HMM) approach. In the MM approach the idea is, roughly speaking, to devise strategies that decide in an efficient way which mode is running the system and work with the lin- ear system associated to this mode (see, e.g., [22] for a comprehensive treatment on this subject). The HMM approach focuses on what is known in the control literature as a class of partially observed stochastic dynamical systems. The basic framework for the HMM consists of a Markov process θ (t) that is not directly observed but is hidden in a noisy observation process y(t). Roughly speaking, the aim is to estimate the Markov process, given the related observations, and from this estimation to de- rive a control strategy for the hidden Markov process (usually the transition matrix of the chain depends on the control variable u(t)). See, e.g., [130] for a modern treatment on this topic. The present book can be seen, roughly speaking, as a continuation of the book [81], which dealt with the discrete-time case, to the continuous-time case. From a practical viewpoint, perhaps the most important aspect which favors the study of continuous-time systems is that many models in science are based upon spe- cific relations between the system variables and their instantaneous rates of varia- tion (some examples being Newton’s second law of classical mechanics, Maxwell’s nonstationary equations of electromagnetics, the law of continuity of fluid transport phenomena, Verhulst’s logistic equation, and Gompertz’s model of tumor growth, among many others). In these cases, the direct application of discrete-time systems theory may be a potential cause for distortion to the model in the form, e.g., of spurious effects such as artificial energy dissipation, instability, oscillations, and a potentially cumbersome dependence on the discretization parameters (which may even be plagued by the curse of dimensionality, as pointed out in [114]). Hence, it
  • 7. 1.2 Some Applications of MJLS 7 Fig. 1.4 Transitions between the different scenarios of a country’s economy is fair to say that, prior to sampling, a cautious study of the underlying continuous- time process should be carried out in order to ensure that such adversities would not hinder the desired specifications to be achieved in a given application. 1.2 Some Applications of MJLS Since their inception in the early 1960s, MJLS have found many applications in a great variety of fields. These include unmanned air vehicles [266], solar power sta- tions [275], satellite dynamics [226], economics [32, 33, 52, 108, 268], flight sys- tems [174, 175], power systems [206, 211, 212, 290, 291], communication systems [1, 2, 208, 243], among many others. This section is devoted to a brief exposition of some selected topics regarding applications of MJLS, with special attention to those in continuous time. The earliest application of MJLS to economics, introduced in the discrete-time scenario in [33], was based on Samuelson’s multiplier–accelerator macroeconomic model (see [252, 286, 299]). The model studied in [33] consists of a very simpli- fied relation for the dynamical evolution of a country’s national income in terms of the governmental expenditure, which is weighted by two parameters (the marginal propensity to save and the accelerator coefficient). Based on the historical data ob- tained from the U.S. Department of Commerce from years 1929 until 1971, [33] assumed that the state of the economy could be roughly lumped in three possible operation modes (“normal”, “boom”, and “slump”) and that the switching between them could be modeled as a homogeneous Markov chain, with the transition rates depicted in Fig. 1.4. The subsequent problem considered in [33] corresponds to an MJLS version of the optimal linear quadratic control setup. This application, which was also analyzed in [81] for the discrete-time case, will be further studied in Chap. 10. Another practical application of MJLS, which, in the discrete-time case, has been extensively discussed in [81], is the control of a solar power receiver. In contin- uous time, this situation was initially considered in [275] with θ (t) representing
  • 8. 8 1 Introduction Fig. 1.5 Simplified boiler configuration abrupt environmental changes between “sunny” and “cloudy” conditions, which are measured by sensors located on the plant. The role of control is to determine the feedwater flow rate which will enter the boiler, in such a way as to regulate the outflow temperature at the desired level. The boiler flow rate is strongly dependent upon the receiving insolation, and, as a result of this abrupt variability, several lin- earized models are required to characterize the evolution of the boiler when clouds interfere with the sun’s rays. The control law described in [275] makes use of the state feedback and a measurement of θ (t) through the use of flux sensors on the receiver panels. In Fig. 1.5 a very simplified representation of the system is shown. Although several important blocks such as the boiler internal models for steam and metal dynamics, thermal couplings, and feedforward compensation loops are not displayed, their abstraction evidences the dependence of the control system on the abrupt variability of the received insolation. Robotic manipulator arms are employed in a great deal of modern applications, which span areas as diverse as deep sea engineering, manufacturing processes, space technology, or teleoperated medicine, for instance. The adequate operation of such devices, however, is severely compromised by the occurrence of failures, which may be intolerable in safety-critical applications, for example. Furthermore, repairing the faulty arm may frequently turn out to be a difficult task during the course of opera- tion (which may occur if, e.g., the robot is functioning in a hazardous environment). On this regard, a great deal of research has been carried out on the control of robotic arms with less actuators than degrees of freedom, commonly referred to as under- actuated manipulators in the specialized literature [11, 12, 92]. In practical terms, a manipulator arm is said to be underactuated whenever the motor on at least one of its joints is in a passive state. The basic principle in the operation of these devices is then to explore the dynamic coupling that the active joints impose on the passive ones, in such a way as to drive the arm to the desired position in spite of its faulty condition. The introduction of MJLS theory to tackle the control of underactuated robotic arms was made by Siqueira and Terra in [261], in the discrete-time setting. This approach was subsequently described in [81] and, more recently, brought to the continuous-time setting in [262, 263]. In Chap. 10 the robust control of the planar 3-link underactuated arm depicted in Fig. 1.6 is further studied by means of the results devised in this book.
  • 9. 1.2 Some Applications of MJLS 9 Fig. 1.6 Geometric modeling of the planar 3-link arm Fig. 1.7 Birth–death process considered in [175] In yet another field, MJLS were employed in [175] for the stability analysis of controlled flight systems affected by electromagnetic disturbances. This problem stems from the susceptibility of electronic devices against external disturbances such as lightning, thermal noise, and radio signals, for instance, which is recog- nized as a potential cause for computer upsets in digital controllers. In order to cope with the resulting adversities, which may range from random bit errors to permanent computer failures, the strategy in [175] is to model the accumulative effect of exter- nal disturbances on the system by means of a continuous-time birth–death Markov process of the form indicated in Fig. 1.7. The rationale for this choice has been, as pointed out in [175], that by doing so the arrival of new disturbances is dictated by a Poisson process with exponentially distributed sojourn times. An important issue in this situation is that, while the aircraft dynamics treated in [175, Sect. IV.B] evolves in continuous time, the occurrence of electromagnetic disturbances takes place in the sampled-data digital controller implementation. This leaves open an interesting conjecture of whose aspects of continuous- and discrete- time MJLS theory should be relevant to the problem at hand. In [211, 212] the modeling and control of power systems subject to Markov jumps has been addressed in the continuous-time scenario. In this case the switching mechanism is used to model random changes in the load, generating unit outages, and transmission line faults, for instance. As shown in [212], intermittent couplings between electrical machines operating in a network can render the overall system unstable in a stochastic sense, a result which somewhat resembles the analysis previ- ously carried out for (1.1) in the cases (1.3) and (1.4). In [211] the main results were
  • 10. 10 1 Introduction applied to the problem of dynamic security assessment, which amounts to determin- ing, with desired probability, whether certain parameters of the electrical system are guaranteed to remain within a safe region of operation at a given period. A “security measure” was defined, which corresponds to a quantitative indicator of the vulner- ability of the current system state and network topology to stochastic contingency events. Two types of switchings are considered therein: primary ones, driven by a continuous-time Markov process taking values in a finite set, and secondary events, which are modeled by state-dependent (controlled) jumps. Recent advances in the control of power systems have also been reported in [206, 290, 291] by means of decentralized control methods and the S-procedure. An alternative account of this problem via robust control methods is presented in details in Sect. 10.3. The modeling of communication systems via MJLS is by now another promis- ing trend in the applications front. In discrete-time this is boldly motivated by the connection between MJLS and the Gilbert–Elliott model for burst communication channels (see [129, 167, 172, 173, 186, 244, 254]), which in its simplest form corre- sponds to a two-state Markov chain. A convenient feature of these models (besides their relative simplicity) is that they are capable of describing the fact that eventual packet losses typically occur during intervals of time (e.g., while a wireless link is obstructed). In other words, isolated packet losses are not common events, and it usually takes a while before communication, once lost, is restored. As pointed out in [167, 173, 186, 254], large packet-loss rates imply poor performance or even instability, and therefore controllers implemented within a network may provide considerably better results if their design takes into account the probabilistic nature of the network. In the continuous-time scenario, references [1, 2] treated the problem of dynamic routing in mobile networks. In loose terms, this amounts to determining a route within the network topology constraints, through which information packets will travel from one given node to another. Of course, many of these routing operations will typically occur at the same time and between different nodes and directions, so that the routing algorithm must be able to provide a satisfactory quality of service for all customers (for example, by delivering packets in the shortest time, with as lit- tle lag as possible, and with a very low packet loss rate), without violating physical constraints such as link capacity, memory size (queueing length), or power avail- ability. The approach in [1, 2] considers that sudden variations, modeled by MJLS, occur in the network due to, e.g., mobility and topological variations. The consid- eration of time delay in the underlying model is of major importance, owing to the time that packets must wait on a queue before being processed, together with the fact that the network nodes are geographically separated. Furthermore, this latter constraint motivated the consideration of a decentralized control scheme. The ulti- mate problem considered in [1, 2] was the minimization of the worst-case queueing length, with the aid of H∞ control methods.
  • 11. 1.3 Prerequisites and General Remarks 11 1.3 Prerequisites and General Remarks As prerequisite for this book, it is desirable some knowledge on the classical linear control theory (stability results, the linear quadratic Gaussian (LQG) control prob- lem, H∞ control problem, and Riccati equations), some familiarity with continuous- time Markov chains and probability theory, and some basic knowledge of operator theory. In this book we follow an approach that combines probability and operator theory to develop the results. By doing this we believe that the book provides a unified and rigorous treatment of recent results for the control theory of continuous-time MJLS. Most of the material included in the book was published after 1990. The goal is to provide a complete and unified picture on the main topics of the control theory of continuous-time MJLS such as mean-square stability, quadratic control and H2 con- trol for the complete and partial observations (also called partial information) cases, associated coupled differential and algebraic Riccati equations, linear filtering, and H∞ control. The book also intends to present some design algorithms mainly based on linear matrix inequalities (LMIs) optimization tool packages. One of the objectives of the book is to introduce, as far as possible in a friendly way, a bent of the MJLS theory that we have named here as the analytical point of view. We believe (and do hope) that experts in linear systems with Markov jump parameters will find in this book the minimal essential tools to follow this approach. Moreover, the stochastic control problems for MJLS considered in this book provide one of those few cases in the stochastic control field in which explicit solutions can be obtained, being a useful material for a course and for introducing students into an interesting and active research area. In addition, we do hope to motivate the reader, especially the graduate students, in such a way that this book could be a starting point for further developments and applications of continuous-time MJLS. From the application point of view we believe that the book provides a powerful theory with potential application in systems whose dynamics are subject to abrupt changes, as those found in safety-critical and high-integrity systems, industrial plants, economic systems, etc. 1.4 Overview of the Chapters We next present a brief overview of the contents of the chapters of the book. Chapter 2 is dedicated to present some background material needed throughout the book, as the notation, norms, and spaces that are appropriate for our approach. It also presents some important auxiliary results, especially related to the stability concepts to be considered in Chap. 3. A few facts on Markov chains and the bear essential of infinitesimal generator is also included. We also recall some basic facts regarding LMIs, which are useful for the design techniques and the H∞ control problem.
  • 12. 12 1 Introduction Chapter 3 deals with mean-square stability for continuous-time MJLS. It is shown that mean-square stability is equivalent to the maximal real part of the eigen- values of an augmented matrix being less than zero or to the existence of a solution of a Lyapunov equation. As aforementioned, stability of all modes of operation is neither necessary nor sufficient for global stability of the system. The criterion based on the eigenvalues of an augmented matrix reveals that a balance between the modes and the transition rate matrix is essential for mean-square stability. It is worth pointing out that the setup of Chap. 3 is on the domain of complex matrices and vectors. By doing this we can use a useful result on the decomposition of a matrix into some positive semidefinite matrices, which is very important to prove the equivalence on stability results. But it is shown later on in this chapter that the results obtained are valid even in the setup of real matrices and vectors. In order to simplify the notation and proofs, in the remaining chapters of the book we consider just the real case. Chapter 4 analyzes the quadratic optimal control problem for MJLS in the usual finite- and infinite-time horizon framework. We consider in this chapter that the con- troller has access to both the state variable x(t) and the jump parameter θ (t). The case in which the controller has access only to an output y(t) and θ (t) is considered in Chap. 6 and called the partial observation (or partial information) case. The so- lution for the quadratic optimal control problems of Chap. 4 relies, in part, on the study of a finite set of coupled differential and algebraic Riccati equations (CDRE and CARE, respectively). These equations are studied in the Appendix A. Chapter 5 restudies the infinite-horizon quadratic optimal control for MJLS but now from another point of view, usually known in the literature of linear systems as the H2 control. The advantage of the H2 approach is that it allows one to consider parametric uncertainties and solve the problem using LMIs optimization tools. Chapter 6 deals with the finite-horizon quadratic optimal control problem and the H2 control problem of continuous-time MJLS for the partial information case. The main result shown is that the optimal control is obtained from two sets of cou- pled differential (for the finite-horizon case) and algebraic (for the H2 case) Riccati equations, one set associated with the optimal control problem when the state vari- able is available, as analyzed in Chaps. 4 and 5, and the other set associated with the optimal filtering problem. This establishes the so-called separation principle for continuous-time MJLS. Chapter 7 aims to derive the best linear mean square estimator of continuous- time MJLS assuming that only an output y(t) is available. It is important to em- phasize that in this chapter we assume that the jump parameter θ (t) is not known. The idea is to derive a filter which bears those desirable properties of the Kalman filter: a recursive scheme suitable for computer implementation which allows some offline computation that alleviates the computational burden. The linear filter has dimension N n (where n denotes the dimension of the state vector, and N the num- ber of states of the Markov chain). Both the finite-horizon and stationary cases are considered. Chapter 8 is devoted to the H∞ control of Markov jump linear systems, in the infinite-horizon setting. The statement of a bounded real lemma is the starting point
  • 13. 1.5 Historical Remarks 13 toward a complete LMIs characterization of static state feedback, as well as of full-order dynamic output feedback stabilizing controllers that guarantee that a pre- scribed closed-loop H∞ performance is attained. The main results include explicit formulas and the corresponding algorithms for designing the controllers of interest. Chapters 9 and 10 are intended to conclude the book assembling some problems in the Markov jump context and the tools to solve them. In the Appendix A some results on coupled differential and algebraic Riccati equations (CDRE and CARE, respectively) associated to the control problem are presented. Initially, we consider the problem of uniqueness, existence, positive defi- niteness, and continuity of the solution of the CDRE. After that we study the CARE, dealing essentially with conditions for the existence of solutions and asymptotic convergence, based on the concepts of mean-square stabilizability and detectability seen in Chap. 3. Regarding the existence of a solution, we are particularly interested in maximal and stabilizing solutions. In the Appendix B we derive some auxiliary results related to an adjoint operator for MJLS used in the separation principle presented in Chap. 6. 1.5 Historical Remarks The study of dynamical systems with random parameters can be traced back at least to [196, 200–202], and [142]. Due to the importance and complexity of the theme, a variety of techniques has emerged, and an extraordinary burst of publications asso- ciated with terminologies such as multiple model, switching systems, hidden Markov models and Markov jump linear systems, inter alia, has appeared in the specialized literature. Even for those approaches which embraced the idea of modeling the ran- dom parameter as a Markov chain, the methodologies were different. Each one of these topics have charted its own course, and the associated literature is by now huge. Therefore, it is out of the scope here to go into details on all of these topics (see, e.g., [22, 130, 232] and references therein for an account on multiple model, hidden Markov model, and switching systems, respectively). We focus instead on the MJLS case. Regarding MJLS, the seminal papers [270] and [303] set the stage for future research in this area. In the first one, the jump linear quadratic (JLQ) control problem was considered for the finite-horizon setting, via a maximum principle approach (see also [269]). In the other one, dynamic programming was used, and the infinite- horizon case was also treated. In this case, although the objective had been carried out successfully, a technical inconvenience was related to the choice of the adopted stability criteria, which did not seem to be fully adequate. This, in turn, has entailed a great deal of challenges for future research. In the 1970s we can mention the papers [32, 248, 273, 274] and [33] dealing with the JLQ control problem, where the latter seems to be the first one that treated the discrete-time version of the optimal quadratic control problem for the finite-time horizon case (see also [20] for the MM approach).
  • 14. 14 1 Introduction Due, in part, to the lack of a suitable concept for stability, it took a while for the theory flourish. For instance, it elapsed more than ten years to appear some of the key papers that put mean-square stability (stochastic stability, or L2 -stability), mean square stabilizability, and mean square detectability for MJLS in a solid ground. Without any intention of being exhaustive here, we mention, for instance, [43, 48, 71, 77, 79, 118, 137, 147, 148, 150–152, 156, 164, 170, 170, 187, 189– 191, 210, 221, 230, 231, 245, 278, 282, 284], for a sample of papers dealing with stability and stabilizability for MJLS. For the case in which the state space of the Markov chain is countably infinite, it was shown in [78] and [148] that mean-square and stochastic stability (L2 -stability) are no longer equivalent (see also [147]). An- other critical issue was an adequate concept for the solution of the Riccati equa- tion associated to MJLS (coupled Riccati equations). As far as the authors are aware of, it was in [156] where the concept of mean-square solution for the cou- pled Riccati equations was first proposed. For a glimpse on some issues dealing with coupled Riccati equations, see, for instance, [4–6, 17, 66, 67, 149, 156, 246]. For control problems (optimal, adaptive, H∞ , H2 , robust, receding horizon, singu- larly perturbed, partial observations, etc.), we mention, for instance, [18, 38, 44, 46, 47, 59, 60, 69, 70, 73–75, 78, 85, 87, 93, 94, 101, 104, 108, 111, 119, 120, 122, 125, 134, 143–146, 158, 159, 171, 176, 188, 190, 224, 229, 240, 258, 279, 307]. For the filtering problem, the reader is referred, for instance, to [9, 35–37, 65, 82– 84, 126, 127, 132, 155, 160, 217, 321, 322]. The case with delay was treated, for instance, in [24, 44, 45, 55, 165, 209, 218, 257, 297]. The case with un- certainty (including uncertainty in the transition matrix of the Markov chain) was considered in [15, 26, 42, 256, 305, 306, 316–319]. Separation principles were derived, for instance, in [80, 89, 90, 153]. Structural properties such as controllability, detectability, and observability have been studied, for instance, in [61, 63, 187, 189, 222]. See [287, 288] for problems related to detection and iden- tification for MJLS. In addition, there is by now a growing conviction that MJLS provide models of wide applicability (see, e.g., [14] and [266]). The evidence in favor of such a proposition has been amassing rapidly over the last decades. We mention [1, 2, 8, 14, 21, 28, 32, 39, 52, 57, 108, 124, 133, 141, 173–175, 184, 186, 197, 198, 208, 213, 219, 220, 225, 226, 228, 241, 243, 249, 253, 254, 266, 275] and [323], as works dealing with applications of this class of systems (see also the books [22, 40, 41, 81, 121, 223, 271, 309] and references therein).