Reinforcement learning for data-driven optimisation

Reinforcement learning for data-driven optimisation
Part I - Application for decision-making problems for electrical systems
Prof. Damien ERNST
EES-UETP Advanced Data Analytics for Energy System
September 3rd-5th, 2018
INESC Technology and Science, Porto, Portugal

Agent
Environment
state
reward
action
numerical value
Reinforcement learning agent

The battery
controller
State: (i) the battery
level (ii)
Everything you know
about the market
Reward: The money
you make during the
market period.
The battery setting
for the next market
period.
+ the energy
market

Table taken from: “Reinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives”. M. Glavic, R. Fonteneau and D. Ernst. Proceedings of the
20th IFAC World Congress.

Learning:
Exploration/exploitation: Not always
take the action that is believed to be
optimal to allow exploration.
Generalization: Generalize the
experience gained in some states to
other states.

Learning
phase
Effect of the
resulting control
policy
First control law for stabilizing power systems every computed using reinforcement learning. More at: “Reinforcement Learning Versus Model Predictive Control: A Comparison on a Power
System Problem”. D. Ernst, M. Glavic, F.Capitanescu, and L. Wehenkel. IEEE Transactions on Syestems, Man, An Cybernetics—PART B: Cybernetics, Vol. 39, No. 2, April 2009.

Reinforcement learning for trading in the intraday market
More: “Intra-day Bidding Strategies for Storage Devices Using Deep Reinforcement”. I. Boukas, D. Ernst, A. Papavasiliou, and B. Cornélusse. Proceedings of the 2018 15th International
Conference on the European Energy Market (EEM).
Complex problem:
• Adversarial environment
• Highly dimensional
• Partially observable
Best results obtained with optimisation
of strategies based on past data
together with supervised learning to
learn from the optimised
strategies (imitative-learning type of
approach)

Part II - Building your intelligent agent using
reinforcement learning
1

Artificial autonomous intelligent agent: formal definition
An agent is anything that is capable of acting upon information it
perceives.
An intelligent agent is an agent capable of making decisions about
how it acts based on experience, that is of learning decision from
experience.
An autonomous intelligent agent is an intelligent agent that is free
to choose between different actions.
An artificial autonomous intelligent agent is anything we create that
is capable of actions based on information it perceives, its own
experience, and its own decisions about which actions to perform.
Since “artificial autonomous intelligent agent” is quite mouthful, we
follow the convention of using “intelligent agent” or “autonomous
agent” for short.
2

Application of intelligent agents
Intelligent agents are applied in variety of areas: project
management, electronic commerce, robotics, information retrieval,
military, networking, planning and scheduling, etc.
Examples:
• A web search agent that may have the goal of obtaining web site
addresses that would match the query of history made by customer.
It could operate in the background and deliver recommendations to
the customer on a weekly basis.
• A robot agent that would learn to fulfill some specific tasks
through experience such as playing soccer, cleaning, etc.
• An intelligent flight control system
• An agent for placing advertisements on the web.
• A intelligent agent for playing computer games that does rely on
some predefined strategies for playing but that would learn them by
interacting with some opponents.
3

Machine learning and reinforcement learning: definitions
Machine learning is a broad subfield of artificial intelligence is
concerned with the development of algorithms and techniques that
allow computers to ”learn”.
Reinforcement Learning (RL in short) refers to a class of problems
in machine learning which postulate an autonomous agent exploring
an environment in which the agent perceives information about its
current state and takes actions. The environment, in return,
provides a reward signal (which can be positive or negative). The
agent has as objective to maximize the (expected) cumulative
reward signal over the course of the interaction.
4

The policy of an agent determines the way the agent selects its
action based on the information it has. A policy can be either
deterministic or stochastic.
Research in reinforcement learning aims at designing policies which
lead to large (expected) cumulative reward.
Where does the intelligence come from ? The policies process in an
“intelligent way” the information to select “good actions”.
5

An RL agent interacting with its environment
6

Some generic difficulties with designing intelligent agents
• Inference problem. The environment dynamics and the mechanism
behind the reward signal are (partially) unknown. The policies need
to be able to infer from the information the agent has gathered from
interaction with the system, “good control actions”.
• Computational complexity. The policy must be able to process the
history of the observation within limited amount of computing times
and memory.
• Tradeoff between exploration and exploitation.∗ To obtain a lot of
reward, a reinforcement learning agent must prefer actions that it
has tried in the past and found to be effective in producing reward.
But to discover such actions, it has to try actions that it has not
selected before.
∗May be seen as a subproblem of the general inference problem. This problem is
often referred to in the “classical control theory” as the dual control problem.
7

The agent has to exploit what it already knows in order to obtain
reward, but it also has to explore in order to make better action
selections in the future. The dilemma is that neither exploration nor
exploitation can be pursued exclusively without failing at the task.
The agent must try a variety of actions and progressively favor those
that appear to be best. On a stochastic task, each action must be
tried many times to gain a reliable estimate its expected reward.
• Exploring safely the environment. During an exploration phase
(more generally, any phase of the agent’s interaction with its
environment), the agent must avoid reaching unacceptable states
(e.g., states that may for example endanger its own integrity). By
associating rewards of −∞ to those states, exploring safely can be
assimilated to a problem of exploration-exploitation.
• Adversarial environment. The environment may be adversarial. In
such a context, one or several other players seek to adopt strategies
that oppose the interests of the RL agent.
8

Diﬀerent characterizations of RL problems
• Stochastic (e.g., xt+1 = f(xt, ut, wt) where the random disturbance
wt is drawn according to the conditional probability distribution
Pw(·|xt, ut)) versus deterministic (e.g., xt+1 = f(xt, ut))
• Partial observability versus full observability. The environment is
said to be partially (fully) observable if the signal st describes
partially (fully) the environment’s state xt at time t.
• Time-invariant (e.g., xt+1 = f(xt, ut, wt) with wt = Pw(·|xt, ut))
versus time-variant (e.g., xt+1 = f(xt, ut, wt, t)) dynamics.
• Continuous (e.g., ˙x = f(x, u, w)) versus discrete dynamics (e.g.,
xt+1 = f(xt, ut, wt)).
9

• Multi-agent framework versus single-agent framework. In a
multi-agent framework the environment may be itself composed of
(intelligent) agents. A multi-agent framework can often be
assimilated to a single-agent framework by considering that the
internal states of the other agents are unobservable variables. Game
theory and, more particularly, the theory of learning in games study
situations where various intelligent agents interact with each other.
• Finite time versus inﬁnite time of interaction.
• Single state versus multi-state environment. In single state
environment, computation of an optimal policy for the agent is often
reduced to the computation of the maximum of a stochastic
function (e.g., ﬁnd u∗ ∈ arg max
u∈U
E
w∼Pw(·|u)
[r(u, w)]).
10

• Multi-objective reinforcement learning agent (reinforcement
learning signal can be multi-dimensional) versus single-objective RL
agent.
• Risk-adverse reinforcement learning agent. The goal of the agent
is not anymore to maximize the expected cumulative reward but
maximize the lowest cumulative reward it could possibly obtain.
11

Characterization of the RL problem adopted in this class
• Dynamics of the environment:
xt+1 = f(xt, ut, wt) t = 0, 1, 2 . . .
where for all t, the state xt is an element of the state space X, the
action ut is an element of the action space U and the random
disturbance wt is an element of the disturbance space W.
Disturbance wt generated by the time-invariant conditional
probability distribution Pw(·|x, u).
• Reward signal:
The function r(x, u, w) is the so-called reward function supposed to
be bounded by a constant Br.
To the transition from t to t + 1 is associated a reward signal
γtrt = γtr(xt, ut, wt) where r(x, u, w) is a reward function supposed to
be bounded by a constant Br and γ ∈ [0, 1[ a decay factor.
12

• Cumulative reward signal:
Let ht ∈ H be the trajectory from instant time 0 to t in the
combined state, action, reward spaces:
ht = (x0, u0, r0, x1, u1, r1, . . . , ut−1, rt−1, xt). Let π ∈ Π be a stochastic
policy such that π : H × U → [0, 1] and let us denote by Jπ(x) the
expected return of a policy π (or expected cumulative reward signal)
when the system starts from x0 = x
Jπ(x) = lim
T→∞
E[
∞
t=0
γtr(xt, ut ∼ π(ht, .), wt)|x0 = x]
• Information available:
The agent does not know f, r and Pw. The only information it has
on thes¡e three elements is the information contained in ht.
13

Goal of reinforcement learning
• Le π∗ ∈ Π a policy such that ∀x ∈ X,
Jπ∗
(x) = max
π∈Π
Jπ(x) (I)
Under some mild assumptions∗ on f, r and Pw, such a policy π∗
indeed exists.
• In reinforcement learning, we want to build policies ˆπ∗ such that
Jˆπ∗
is as close as possible (according to specific metrics) to Jπ∗
.
• If f, r and Pw were known, we could, by putting aside the difficulty
of finding in Π the policy π∗, design the optimal agent by solving the
optimal control problem (I). However, Jπ depends on f, r and Pw
which are supposed to be unknown ⇒ How can we solve this
combined inference - optimization problem ?
∗We will suppose that these mild assumptions are always satisifed afterwards.
14

Dynamic Programming (DP) theory remind: optimality
of stationary policies
• A stationary control policy µ : X × U selects at time t the action
ut = µ(xt). Let Πµ denote the set of stationary policies.
• The expected return of a stationary policy when the system starts
from x0 = x is
Jµ(x) = lim
T→∞
E
w0,w1,...,wT
[
∞
t=0
γtr(xt, µ(xt), wt)|x0 = x]
• Le µ∗ be a policy such that Jµ∗
(x) = max
µ∈Πµ
Jµ(x) everywhere on X.
It can be shown that such a policy indeed exists. We name such a
policy an optimal stationary policy.
• From classical dynamic programming theory, we know
Jµ∗
(x) = Jπ∗
(x) everywhere ⇒ considering only stationary policies is
not suboptimal !
15

From Dynamic Programming (DP): How to compute the
return of a stationary policy
• We deﬁne the functions J
µ
N : X → R by the recurrence equation
J
µ
N(x) = E
w∼Pw(·|x,u)
[r(x, µ(x), w) + γJ
µ
N−1(f(x, µ(x), w))], ∀N ≥ 1
(1)
with J
µ
0 (x) ≡ 0.
• We have as result of the DP theory
lim
N→∞
J
µ
N − Jµ → 0 (2)
and
J
µ
N − Jµ
∞ ≤
γN
1 − γ
Br (3)
16

DP theory remind: QN-functions and µ∗
• We deﬁne the functions QN : X × U → R by the recurrence equation
QN(x, u) = E
w∼Pw(·|x,u)
[r(x, u, w) + γmax
u ∈U
QN−1(f(x, u, w), u )], ∀N ≥ 1
(4)
with Q0(x, u) ≡ 0. These QN-functions are also known as
state-action value functions.
• We denote by µ∗
N : X → U the stationary policy:
µ∗
N(x) ∈ arg max
u∈U
QN(x, u). (5)
• We deﬁne the Q-function as being the unique solution of the
Bellman equation:
Q(x, u) = E
w∼Pw(·|x,u)
[r(x, u, w) + γmax
u ∈U
Q(f(x, u, w), u )]. (6)
17

We have the following results:
• Convergence in inﬁnite norm of the sequence of functions QN to
the Q-function, i.e. lim
N→∞
QN − Q ∞ → 0 (see Appendix I for the
proof)
• A control policy µ∗ is optimal if and only if
µ∗(x) ∈ arg max
u∈U
Q(x, u) (7)
We also have Jµ∗
(x) = max
u∈U
Q(x, u), which is also called the value
function.
• The following bound on the suboptimality of µ∗
N with respect to µ∗
holds: Jµ∗
N − Jµ∗
∞
≤ 2γNBr
(1−γ)2
18

A pragmatic approach for designing (hopefully) good
policies ˆπ∗
We focus ﬁrst on to the design of functions ˆπ∗ which realize
sequentially the following three tasks:
1. “System identiﬁcation” phase. Estimation from ht of an
approximate system dynamics ˆf, an approximate probability
distribution ˆPw and an approximate reward function ˆr.
19

2. Resolution of the optimization problem:
Find in Πµ the policy ˆµ∗ such that ∀x ∈ X, Jˆµ∗
(x) = max
µ∈Πµ
ˆJµ(x)
where ˆJˆµ is deﬁned similarly as function Jµ but with ˆf, ˆPw and ˆr
replacing f, Pw and r, respectively.
3. Afterwards, the policy ˆπ selects with a probability 1 − (ht)
actions according to the policy ˆµ∗ and with a probability 1 − (ht) at
random. Step 3 has been introduced to address the dilemma
between exploration and exploitation.∗
∗We won’t address further the design of the ’right function’ : H → [0, 1]. In many
applications, it is chosen equal to a small constant (say, 0.05) everywhere.
20

Some constructive algorithms for designing ˆπ∗ when
dealing with finite state-action spaces
• Until say otherwise, we consider the particular case of finite state
and action spaces (i.e., X × U finite).
• When X and U are finite, there exists a vast panel of
’well-working’ implementable RL algorithms.
• We focus first on approaches which solve separately Step 1. and
Step 2. and then on approaches which solve both steps together.
• The proposed algorithms infer ˆµ∗ from ht. They can be adapted in
a straigthforward way to episode-based reinforcement learning where
a model of µ∗ must be inferred from several trajectories ht1, ht2, . . .,
htm with ti ∈ N0.
21

Remind on Markov Decision Processes
• A Markov Decision Process (MDP) is defined through the
following objects: a state space X, an action space U, transition
probabilities p(x |x, u) ∀x, x ∈ X, u ∈ U and a reward function r(x, u).
• p(x |x, u) gives the probability of reaching state x after taking
action u while being in state x.
• We consider MDPs for which we want to find decision policies that
maximize the reward signal γtr(xt, ut) over an infinite time horizon.
• MDPs can be seen as a particular type of the discrete-time optimal
control problem introduced earlier where the system dynamics is
expressed under the form of transition probabilities and where the
reward function does not depend on the disturbance w anymore.
22

MDP structure definition from the system dynamics and
reward function
• We define∗
r(x, u) = E
w∼Pw(·|x,u)
[r(x, u, w)] ∀x ∈ X, u ∈ U (8)
p(x |x, u) = E
w∼Pw(·|x,u)
[I{x =f(x,u,w)}] ∀x, x ∈ X, u ∈ U (9)
• Equations (8) and (9) define the structure of an equivalent MDP
in the sense that the expected return of any policy applied to the
original optimal control problem is equal to its expected return for
the MDP.
• The recurrence equation defining the functions QN can be
rewritten:
QN(x, u) = r(x, u) + γ x ∈X p(x |x, u)max
u ∈U
QN−1(x , u ), ∀N ≥ 1 with
Q0(x, u) ≡ 0.
∗I{logical expression} = 1 if logical expression is true and 0 if logical expression is false.
23

Remind: Random variable and strong law of large
numbers
• A random variable is not a variable but rather a function that
maps outcomes (of an experiment) to numbers. Mathematically, a
random variable is defined as a measurable function from a
probability space to some measurable space. We consider here
random variables θ defined on the probability space (Ω, P).∗
• E
P
[θ] is the mean value of the random variable θ.
• Let θ1, θ2, . . ., θ2 be n values of the random variable θ which are
drawn independently. Suppose also that E
P
[|θ|] = Ω |θ|dP is smaller
than ∞. In such a case, the strong law of large number states that:
lim
n→∞
θ1 + θ2 + . . . + θn
n
P
→ E
P
[θ] (10)
∗For the sake of simplicity, we have considered here that (Ω, P) indeed defines a
probability space which is not rigorous.
24

Step 1. Identiﬁcation by learning the structure of the
equivalent MPD
• The objective is to infer some ’good approximations’ of p(x |x, u)
and r(x, u) from:
ht = (x0, u0, r0, x1, u1, r1, . . . , ut−1, rt−1, xt)
Estimation of r(x, u):
Let A(x, u) = {k ∈ {0, 1, . . . , t − 1}|(xk, uk) = (x, u)}. Let k1, k2, . . .,
k#A(x,u) denote the elements of the set.∗ The values rk1
, rk2
, . . .,
rk#A(x,u)
are #A(x, u) values of the random variable r(x, u, w) which
are drawn independently. It follows therefore naturally that to
estimate its mean value r(x, u), we can use the following unbiased
estimator:
ˆr(x, u) =
k∈A(x,u) rk
#A(x, u)
(11)
∗If S is a set of elements, #S denote the cardinality of S.
25

Estimation of p(x |x, u):
The values I{x =xk1+1}, I{x =xk2+1}, . . ., I{x =xk#A(x,u)+1} are #A(x, u)
values of the random variable I{x =f(x,u,w)} which are drawn
independently. To estimate its mean value p(x |x, u), we can use the
unbiased estimator:
ˆp(x |x, u) =
k∈A(x,u) I{xk+1=x }
#A(x, u)
(12)
26

Step 2. Computation of ˆµ∗ dentiﬁcation by learning the
structure of the equivalent MPD
• We compute the ˆQN-functions from the knowledge of ˆr and ˆp by
exploiting the recurrence equation:
ˆQN(x, u) = ˆr(x, u) + γ x ∈X ˆp(x |x, u)max
u ∈U
QN−1(x , u ), ∀N ≥ 1 with
ˆQ0(x, u) ≡ 0 and then take
ˆµ∗
N = arg max
u∈U
ˆQN(x, u) ∀x ∈ X (13)
as approximation of the optimal policy, with N ’large enough’ (e.g.,
right hand side of inequality (2) drops below ).
• One can show that if the estimated MDP structure lies in an
’ -neighborhood’ of the true structure, then, Jˆµ∗
is in a
’O( )-neighborhood’ of Jµ∗
where ˆµ∗(x) = lim
N→∞
arg max
u∈U
ˆQN(x, u).
27

The case of limited computational resources
• Number of operations to estimate the MDP structure grows
linearly with t. Memory requirements needed to store ht also grow
linearly with t ⇒ an agent having limited computational resources
will face problems after certain time of interaction.
• We describe an algorithm which requires at time t a number of
operations that does not depend on t to update the MDP structure
and for which the memory requirements do not grow with t:
At time 0, set N(x, u) = 0, N(x, u, x ) = 0, R(x, u) = 0, p(x |x, u) = 0,
∀x, x ∈ X and u ∈ U.
At time t = 0, do
1. N(xt−1, ut−1) ← N(xt−1, ut−1) + 1
2. N(xt−1, ut−1, xt) ← N(xt−1, ut−1, xt) + 1
3. R(xt−1, ut−1) ← R(xt−1, ut−1) + rt
4. r(xt−1, ut−1) ←
R(xt−1,ut−1)
N(xt−1,ut−1)
5. p(x|xt−1, ut−1) ←
N(xt−1,ut−1,x)
N(xt,ut)
∀x ∈ X
28

Merging Step 1. and 2. to learn directly the Q-function:
the Q-learning algorithm
The Q-learning algorithms is an algorithm that infers directly from
ht = (x0, u0, r0, x1, u1, r1, . . . , ut−1, rt−1, xt)
an approximate value of the Q-function, without identifying the
structure of a Markov Decision Process.
The algorithm can be described by the following steps:
1. Initialisation of ˆQ(x, u) to 0 everywhere. Set k = 0.
2. ˆQ(xk, uk) ← (1 − αk) ˆQ(xk, uk) + αk(rk + γmax
u∈U
ˆQk(xk+1, u))
3. k ← k + 1. If k = t, return ˆQ and stop. Otherwise, go back to 2.
29

Q-learning: some remarks
• Iteration 2. can be rewritten as ˆQ(xk, uk) ← ˆQ(xk, uk) + αδ(xk, ul)
where the term:
δ(xk, uk) = rk + γmax
u∈U
ˆQ(xk+1, u) − ˆQ(xk, uk), (14)
called the temporal diﬀerence.
• Learning ratio αk: The learning ratio αk is often chosen constant
with k and equal to a small value (e.g., αk = 0.05, ∀k).
• Consistency of the Q-learning algorithm: Under some particular
conditions on the way αk decreases to zero ( lim
t→∞
t−1
k=0 αk → ∞ and
lim
t→∞
t−1
k=0 α2
k < ∞) and the history ht (when t → ∞, every
state-action pair needs to be visited an inﬁnite number of times),
ˆQ → Q when t → ∞.
30

• Experience replay: At each iteration, the Q-learning algorihtm uses
a sample lk = (xk, uk, rk, xk+1) to update the function ˆQ. If rather
that to use the ﬁnite sequence of sample l0, l2, . . ., lt−1, we use the
inﬁnite size sequence li1
, li2
, . . . to update in a similar way ˆQ, where
the ij are i.i.d. with uniform distribution on {0, 2, . . . , t − 1}, then ˆQ
converges to the approximate Q-function computed from the
estimated equivalent MDP structure.

Inferring ˆµ∗ from ht when dealing with very large or
infinite state-action spaces
• Up to now, we have considered problems having discrete (and not
too large) state and action spaces ⇒ ˆµ∗ and the ˆQN-functions could
be represented in a tabular form.
• We consider now the case of very large or infinite state-action
spaces: functions approximators need to be used to represent ˆµ∗ and
the ˆQN-functions.
• These function approximators need to be used in a way that there
are able to ’well generalize’ over the whole state-action space the
information contained in ht.
• There is a vast literature on function approximators in
reinforcement learning. We focus first on one algorithm named
’fitted Q iteration’ which computes the functions ˆQN from ht by
solving a sequence of batch mode supervised learning problems.
31

Remind: Batch mode supervised learning
• A batch mode Supervised Learning (SL) algorithm infers from a
set of input-output (input = information state); (output = class
label, real number, graph, etc) a model which explains “at best”
these input-output pairs.
• A loose formalisation of the SL problem: Let I be the input space,
O the output space, Ξ the disturbance space. Let g : I × Ξ → O. Let
Pξ(·|i) a conditional probability distribution over the disturbance
space.
We assume that we have a training set T S = {(il, ol)}#T S
l=1 such that
ol has been generated from il by the following mechanism: draw
ξ ∈ Ξ according to Pξ(·|il) and then set ol = g(il, ξ).
From the sole knowledge of T S, supervised learning aims at ﬁnding
a function ˆg : I → O which is a ’good approximation’ of the function
g(i) = E
ξ∼Pξ(·)
[g(i, ξ)]
32

• Typical supervised learning methods are: kernel-based methods,
(deep) neural networks, tree-based methods.
• Supervised learning highly successful: state-of-the art SL
algorithms have been successfully applied to problems where the
input state was composed thousands of components.
33

The ﬁtted Q iteration algorithm
• Fitted Q iteration computes from ht the functions ˆQ1, ˆQ2, . . ., ˆQN,
approximations of Q1, Q2, . . ., QN. At step N > 1, the algorithm
uses the function ˆQN−1 together with ht to compute a new training
set from which a SL algorithm outputs ˆQN. More precisely, this
iterative algorithm works as follows:
First iteration: the algorithm determines a model ˆQ1 of
Q1(x, u) = E
w∼Pw(·|x,u)
[r(x, u, w)] by running a SL algorithms on the
training set:
T S = {((xk, uk), rk)}t−1
k=0 (15)
Motivation: One can assimilate X × U to I, R to O, W to Ξ,
Pw(·|x, u) to Pi(·|x, u), r(x, u, w) to g(i, ξ) and Q1(x, u) to g. From
there, we can observe that a SL algorithm applied to the training set
described by equation (15) will produce a model of Q1.
34

Iteration N > 1: the algorithm outputs a model ˆQN of
QN(x, u) = E
w∼Pw(·|x,u)
[r(x, u, w) + γmax
u ∈U
QN−1(f(x, u, w), u )] by
running a SL algorithms on the training set:
T S = {((xk, uk), rk + γmax
u ∈U
ˆQN−1(xk+1, u )}t−1
k=0
Motivation: One can reasonably suppose that ˆQN−1 is a a
suﬃciently good approximation of QN−1 to be consider to be equal
to this latter function. Assimilate X × U to I, R to O, W to Ξ,
Pw(·|x, u) to Pi(·|x, u), r(x, u, w) to g(i, ξ) and QN(x, u) to g. From
there, we observe that a SL algorithm applied to the training set
described by equation (16) will produce a model of QN.
• The algorithm stops when N is ’large enough’ and
ˆµ∗
N(x) ∈ arg max
u∈U
ˆQN(x, u) is taken as approximation of µ∗(x).
35

The ﬁtted Q iteration algorithm: some remarks
• Performances of the algorithm depends on the supervised learning
(SL) method chosen.
• Excellent performances have been observed when combined with
supervised learning methods based on ensemble of regression trees.
• Fitted Q iteration algorithm can be used with any set of one-step
system transitions (xt, ut, rt, xt+1) where each one-step system
transition gives information about: a state, the action taken while
being in this state, the reward signal observed and the next state
reached.
• Consistency, that is convergence towards an optimal solution when
the number of one-step system transitions tends to inﬁnity, can be
ensured under appropriate assumptions on the SL method, the
sampling process, the system dynamics and the reward function.
36

Computation of ˆµ∗: from an inference problem to a
problem of computational complexity
• When having at one’s disposal only a few one-step system
transitions, the main problem is a problem of inference.
• Computational complexity of the ﬁtted Q iteration algorithm grows
with the number M of one-step system transitions (xk, uk, rk, xk+1)
(e.g., it grows as M log M when coupled with tree-based methods).
• Above a certain number of one-step system transitions, a problem
of computational complexity appears.
• Should we rely on algorithms having less inference capabilities than
the ’ﬁtted Q iteration algorithm’ but which are also less
computationally demanding to mitigate this problem of
computational complexity ⇒ Open research question.
37

• There is a serious problem plaguing every reinforcement learning
algorithm known as the curse of dimensionality∗: whatever the
mechanism behind the generation of the trajectories and without any
restrictive assumptions on f(x, u, w), r(x, u, w), X and U, the number
of computer operations required to determine (close-to-) optimal
policies tends to grow exponentially with the dimensionality of X × U.
• This exponentional growth makes these techniques rapidly
computationally impractical when the size of the state-action space
increases.
• Many researchers in reinforcement learning/dynamic
programming/optimal control theory focus their eﬀort on designing
algorithms able to break this curse of dimensionality. Can deep
neural networks give a hope?
∗A term introduced by Richard Bellman (the founder of the DP theory) in the ﬁfties.
38

Q-learning with parametric function approximators
Let us extend the Q-learning algorithm to the case where a
parametric Q-function of the form ˜Q(x, u, a) is used:
1. Equation (14) provides us with a desired update for ˜Q(xt, ut, a),
here: δ(xt, ut) = rt + γmax
u∈U
ˆQ(xt+1, u, a) − ˆQ(xt, ut, a), after observing
(xt, ut, rt, xt+1).
2. It follows the following change in parameters:
a ← a + αδ(xt, ut)
∂ ˜Q(xt, ut, a)
∂a
. (16)
39

Appendix : Algorithmic models for computing the ﬁxed
point of a contraction mapping and their application to
reinforcement learning.
40

Contraction mapping
Let B(E) be the set of all bounded real-valued functions deﬁned on
an arbitrary set E. With every function R : E → R that belongs to
B(E), we associate the scalar :
R ∞ = sup
e∈E
|R(e)|. (17)
A mapping G : B(E) → B(E) is said to be a contraction mapping if
there exists a scalar ρ < 1 such that :
GR − GR ∞ ≤ ρ R − R ∞ ∀R, R ∈ B(E). (18)
41

Fixed point
R∗ ∈ B(E) is said to be a fixed point of a mapping G : B(E) → B(E)
if :
GR∗ = R∗. (19)
If G : B(E) → B(E) is a contraction mapping then there exists a
unique fixed point of G. Furthermore if R ∈ B(E), then
lim
k→∞
GkR − R∗
∞ = 0. (20)
From now on, we assume that:
1. E is finite and composed of n elements
2. G : B(E) → B(E) is a contraction mapping whose fixed point is
denoted by R∗
3. R ∈ B(E).
42

Algorithmic models for computing a fixed point
All elements of R are refreshed: Suppose have the algorithm that
updates at stage k (k ≥ 0) R as follows :
R ← GR. (21)
The value of R computed by this algorithm converges to the fixed
point R∗ of G. This is an immediate consequence of equation (20).
One element of R is refreshed: Suppose we have the algorithm that
selects at each stage k (k ≥ 0) an element e ∈ E and updates R(e)
as follows :
R(e) ← (GR)(e) (22)
leaving the other components of R unchanged. If each element e of
E is selected an infinite number of times then the value of R
computed by this algorithm converges to the fixed point R∗.
43

One element of R is refreshed and noise introduction: Let η ∈ R be a
noise factor and α ∈ R. Suppose we have the algorithm that selects
at stage k (k ≥ 0) an element e ∈ E and updates R(e) according to :
R(e) ← (1 − α)R(e) + α((GR)(e) + η) (23)
leaving the other components of R unchanged.
We denote by ek the element of E selected at stage k, by ηk the
noise value at stage k and by Rk the value of R at stage k and by αk
the value of α at stage k. In order to ease further notations we set
αk(e) = αk if e = ek and αk(e) = 0 otherwise.
With this notation equation (23) can be rewritten equivalently as
follows :
Rk+1(ek) = (1 − αk)Rk(ek) + αk((GRk)(ek) + ηk). (24)
44

We deﬁne the history Fk of the algorithm at stage k as being :
Fk = {R0, . . . , Rk, e0, . . . , ek, α0, . . . , αk, η0, . . . , ηk−1}. (25)
We assume moreover that the following conditions are satisﬁed:
1. For every k, we have
E[ηk|Fk] = 0. (26)
2. There exist two constants A and B such that ∀k
E[η2
k|Fk] ≤ A + B Rk
2
∞. (27)
3. The αk(e) are nonnegative and satisfy
∞
k=0
αk(e) = ∞,
∞
k=0
α2
k(e) < ∞. (28)
Then the algorithm converges with probability 1 to R∗.
45

The Q-function as a ﬁxed point of a contraction mapping
We deﬁne the mapping H: B(X × U) → B(X × U) such that
(HK)(x, u) = E
w∼Pw(·|x,u)
[r(x, u, w) + γmax
u ∈U
K(f(x, u, w), u )] (29)
∀(x, u) ∈ X × U.
• The recurrence equation (4) for computing the QN-functions can
be rewritten QN = HQN−1 ∀N > 1, with Q0(x, u) ≡ 0.
• We prove afterwards that H is a contraction mapping. As
immediate consequence, we have, by virtue of the properties
algorithmic model (21), that the sequence of QN-functions
converges to the unique solution of the Bellman equation (6) which
can be rewritten: Q = HQ. Afterwards, we proof, by using the
properties of the algorithmic model (24), the convergence of the
Q-learning algorithm.
46

Q-learning convergence proof
The Q-learning algorithm updates Q at stage k in the following way∗
Qk+1(xk, uk) = (1 − αk)Qk(xk, uk) + αk(r(xk, uk, wk) + (30)
γmax
u∈U
Qk(f(xk, uk, wk), u)), (31)
Qk representing the estimate of the Q-function at stage k. wk is
drawn independently according to Pw(·|xk, uk).
By using the H mapping deﬁnition (equation (29)), equation (31)
can be rewritten as follows :
Qk+1(xk, uk) = (1 − αk)Qk(xk, uk) + αk((HQk)(xk, uk) + ηk) (32)
∗The element (xk, uk, rk, xk+1) used to refresh the Q-function at iteration k of the
Q-learning algorithm is “replaced” here by (xk, uk, r(xk, uk, wk), f(xk, uk, wk)).
48

with
ηk = r(xk, uk, wk) + γmax
u∈U
Qk(f(xk, uk, wk), u) − (HQk)(xk, uk)
= r(xk, uk, wk) + γmax
u∈U
Qk(f(xk, uk, wk), u) −
E
w∼Pw(·|x,u)
[r(xk, uk, w) + γmax
u∈U
Qk(f(xk, uk, w), u)]
which has exactly the same form as equation (24) (Qk corresponding
to Rk, H to G, (xk, uk) to ek and X × U to E).

We know that H is a contraction mapping. If the αk(xk, uk) terms
satisfy expression (28), we still have to verify that ηk satisﬁes
expressions (26) and (27), where
Fk = {Q0, . . . , Qk, (x0, u0), . . . , (xk, uk), α0, . . . , αk, η0, . . . , ηk−1}, (33)
in order to ensure the convergence of the Q-learning algorithm.
We have :
E[ηk|Fk] = E
wk∼Pw(·|xk,uk)
[r(xk, uk, wk) + γmax
u∈U
Qk(f(xk, uk, wk), u) −
E
w∼Pw(·|xk,uk)
[r(xk, uk, w) + γmax
u∈U
Qk(f(xk, uk, w), u)]|Fk]
= 0
and expression (26) is indeed satisﬁed.
49

In order to prove that expression (27) is satisfied, one can first note
that :
|ηk| ≤ 2Br + 2γ max
(x,u)∈X×U
Qk(x, u) (34)
where Br is the bound on the rewards. Therefore we have :
η2
k ≤ 4B2
r + 4γ2( max
(x,u)∈X×U
Qk(x, u))2 + 8Brγ max
(x,u)∈X×U
Qk(x, u) (35)
By noting that
8Brγ max
(x,u)∈X×U
Qk(x, u) < 8Brγ + 8Brγ( max
(x,u)∈X×U
Qk(x, u))2 (36)
and by choosing A = 8Brγ + 4B2
r and B = 8Brγ + 4γ2 we can write
η2
k ≤ A + B Qk
2
∞ (37)
and expression (27) is satisfied. QED
50

Additional readings
Lucian Busoniu, Robert Babuska, Bart De Schutter, and Damien Ernst.
Reinforcement learning and dynamic programming using function approximators.
CRC Press, April 2010. (Available at:
https://orbi.ulg.ac.be/bitstream/2268/27963/1/book-FA-RL-DP.pdf
Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction.
Second edition, in progress. Available at:
http://ufal.mff.cuni.cz/ straka/courses/npfl114/2016/sutton-
bookdraft2016sep.pdf)
Dimitri P. Bertsekas. Dynamic Programming and Optimal Control. Volume I (last
edition: 2017) and Volume II (last edition: 2012. Material available at
http://ufal.mff.cuni.cz/ straka/courses/npfl114/2016/sutton-
bookdraft2016sep.pdf
Csaba Szepezvari. Algorithms for Reinforcement Learning. Synthesis Lectures on
Artificial Intelligence and Machine Learning. June 2009. Available at
http://jmlr.csail.mit.edu/papers/v6/ernst05a.html)
51

Reinforcement learning for data-driven optimisation

More Related Content

Similar to Reinforcement learning for data-driven optimisation (20)

More from Université de Liège (ULg) (20)

Recently uploaded (20)

Reinforcement learning for data-driven optimisation