![CHAPTER 1
INTRODUCTION
1.1 Dynamic Control Systems And Reinforcement
Learning
There are an increasing demands for building automatic controllers for systems with
complex dynamics such as aircraft, magnetic levitation trains, and chemical plants, but
also in objects of daily life such as air conditioners and computer drives [51, 18]. These
systems are characterized as having an output or feedback, that can be measured, inputs
that can be manipulated, and internal dynamics. Feedback control involves computing
and applying suitable control inputs to minimize the differences between the observed
and desired behaviour of a dynamic system. Feedback control design approaches include
classical design methods for linear systems, multivariate control, nonlinear control, op-
timal control, robust control, H-infinity control, adaptive control, and others [77, 8, 35].
Designing a controller has the advantage of being general and applicable to various sys-
tems. However, the design of good controllers is a tedious and demanding job, due
to unknown dynamics and underactuation, modelling errors, and various sorts of dis-
turbances, uncertainties, and noise. Furthermore, since there is no learning involved, for
a new control task with relatively similar dynamics a new system model has to be created.
In contrast to the classical design process, reinforcement learning is geared towards
learning appropriate closed-loop controllers by simply interacting with the process and
incrementally improving control behaviour. Reinforcement learning is learning to act by
trial and error. In this paradigm, an agent can perceive its state and perform actions.
After each action, a numerical reward is given. The goal of the agent is to maximize the
total reward it receives over time. Reinforcement learning has been successful in appli-
cations as diverse as autonomous helicopter flight, robot-legged locomotion, cell-phone
network routing, marketing strategy selection, factory control, and efficient web page
indexing. Despite these successes, there are some challenges and problems which make
learning a control task difficult.
In an online control system, the data is sent and received by network nodes of different
types and manufacturers. Sensor nodes measure process values and transmit these over
the communication network. Actuator nodes receive new values for the process inputs
3](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-8-320.jpg)
![CHAPTER 1. INTRODUCTION 1.1. DYNAMIC CONTROL SYSTEMS AND REINFORCEMENT LEARNING
over the communication network and apply these to the process input. Controller nodes
read process values from sensor nodes. A control algorithm calculates the control signals
and sends them to the actuator nodes. Communication networks inevitably introduce
delays, both due to limited bandwidth, but also due to overhead in the communicating
nodes and the network. In many systems, there are various delay types. From the control
perspective, a control system with varied delay types will no longer be time-invariant.
Therefore, the standard optimal control theory can not be used to analyze and design
these systems. We can categorize different types of control delays based on their place of
occurrence.
• Communication delay between the sensor and the controller.
• Computational delay in the controller.
• Communication delay between the controller and the actuator.
The effect of time delay on the stability and performance of control systems has drawn
the attention of many investigators in different engineering disciplines, including optimal
control system [1, 45, 10, 43, 76], reinforcement learning [5, 39]. In general, the time delay
in active control systems causes an unsynchronized application of the control forces, and
this unsynchronized control not only degrades the system performance but also causes
instability in the system response [61]. Dynamic computational models with delays re-
quire the ability to store and access the time history of their inputs and outputs. And
there are various difficulties in learning and controlling such systems. The first problem
is, there is no explicit teacher signal that indicates the correct output at each time step.
In addition, the temporal delay of the reward signal implies that the learning system
must assign temporal reward/cost to each of the states and actions that resulted in the
final outcome of the sequence.
To overcome this restriction, several, more general approaches are considered that re-
tain state information over time. We refer to these as stored-state methods. The simplest
of these approaches is one which augments immediate sensory inputs with a delay line to
achieve a crude form of short term memory [37]. This approach has been successful in
certain speech recognition tasks [69]. The most common dynamic neural architecture is
the time delayed neural networks [70]. The time delayed neural networks couples delay
lines with a nonlinear static architecture where all the parameters (weights) are adapted
with the Backpropagation algorithm. Another alternative is called the method of pre-
dictive distinctions [2, 13, 59]. Following this approach, the system learns a predictive
model of the sensory inputs i.e. environmental observables, and then uses the internal
state of this model to drive action selection.
A third approach uses a recurrent neural network [50, 21, 73, 57, 28, 19] in combina-
tion with existing reinforcement learning methods to learn a recurrent state-dependent
control policy directly [26]. In the sense of keeping the history of the system states the
recurrent neural networks naturally are powerful tools that can be employed to learn
delayed control systems. Formerly, training of recurrent neural networks was performed
using Backpropagation through time [73], which actually means unfolding the network in
time and constructing a much bigger network, then performing Backpropagation on this
new network. However, besides the fact that this process is very slow, it does not always
4](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-9-320.jpg)
![CHAPTER 2
RELATED WORKS
2.1 Non-Markovian Reinforcement Learning
Markov decision process(MDP) is a popular framework for training control problems[22]
in reinforcement learning. In summary, given a certain state of a system, the agent
selects an action that brings the system to a new state and induces a cost, the new state
is observed and the cost is collected, then the decision maker selects a new action, and
so on. However, the basic MDP framework as it is defined in [34], makes a number of
restrictive assumptions that may limit its applicability:
• the system’s current state is always available to the agent.
• the agent’s actions always take effect immediately
• the cost induced by an action is always collected without delay
The Markov property is usually assumed in reinforcement learning and, therefore, it
vanishes when any of the described conditions is violated. It follows that if an agent must
learn to control a system, there will be periods of time when the internal representation
of system states will be inadequate. Therefore, the decision task will be non-Markovian
which challenges the performance of reinforcement learning algorithms. The presence of
delay in measured states or applied actions causes the violation of the Markov property.
In the following, we viewed three major state-of-the-art approaches for reinforcement
learning with delay.
2.1.1 Markov Decision Process With Delays And Asynchronous
Cost Collection
In the first approach, the state space of the learning agent is augmented with the actions
that were executed during the delay interval. In this work [16, 34] authors showed how
(discrete-time, total-expected-cost, infinite horizon) Markov decision process with obser-
vation, action and delayed cost are reduced to a Markov decision process without delays.
They drew connections amongst the three delay types and demonstrated that the cost
structure of the process without delays is the same as that of the original process with
7](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-12-320.jpg)
![CHAPTER 2. RELATED WORKS 2.1. NON-MARKOVIAN REINFORCEMENT LEARNING
constant or random delays. They considered an embedded process that behaves similarly
to a process with constant delays. However, their results are based on the intuition of
asynchronous cost collection. That means the costs may be induced and collected at
different decision stages and policies can still be compared properly as long as costs are
discounted accordingly. While this approach works well, the state space increase can
cause a large increase in learning time and memory requirements.
2.1.2 Learning And Planning In Environments With Delayed
Feedback
The second approach tried to learn a model of the underlying non-delayed process and
used this model to base control actions on the future state after the delay, predicted
by the model[71]. In this work, the authors evaluated algorithms for environments with
constant observation and reward delay. They covered three different approaches for deal-
ing with the constant delay Markov decision process(CDMDP). First, ”the wait agent”,
which waits for k steps, until the current observation comes through, and then acts us-
ing the optimal action in the non-delayed MDPs. Unfortunately, policies derived from
this strategy will not, in general, provide satisfactory solutions to the CDMDP planning
problem. Instead, the agent’s resulting policy will likely be suboptimal as it is essen-
tially losing potential reward on every wait step. Their second solution was a memoryless
policy, which treated CDMSP as MDP and used memoryless policy for the non-delayed
MDP. Their third solution was the traditional augmented approach, which involves ex-
plicitly constructing an MDP equivalent to original CDMDP in a larger state space. They
augmented each state and applied k previous actions to form a new state representation.
Authors formed new transition probability and reward function but such a solution adds
the extra burden of acquiring the model of the system while the added computational
complexity may actually increase the delay itself.
2.1.3 Control Delay in Reinforcement Learning for Real-Time
Dynamic Systems
The third approach for dealing with delays in MDP has been introduced in [60] which is
an improvement on the older approaches. The authors introduced two new memoryless
solutions and the most important one was an online algorithm named dSARSA(λ). Such
methods base the next control action only on the most recent observation. The downside
of memoryless approaches is that they are likely to learn a suboptimal policy because
they have no means of predicting the state in which the control action will take effect.
Furthermore, SARSA(λ) does not take the presence of the delay into account in its
learning updates. While their complexity remains comparable to that of SARSA(λ),
they exploited the knowledge about the length of the delay to improve their performance.
Then, the authors presented an extension to these algorithms which was applicable where
the delay length is not an integer multiple of the time step.
8](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-13-320.jpg)
![CHAPTER 2. RELATED WORKS 2.2. RECURRENT NEURAL NETWORKS IN REINFORCEMENT LEARNING
2.2 Recurrent Neural Networks In Reinforcement Learn-
ing
There are various attempts to combine the dynamic programming approaches with recur-
rent neural networks(RNN) to tackle reinforcement learning problems. They formulate
their solution in adaptive critic designs(ACDs) [52]. The ACDs have their root in dy-
namic programming and are suitable for learning in noisy, nonlinear, and non-stationary
environments. The ACDs in their application first define what to approximate through
the critic, and then how to adapt the actor in response to the information coming from
the critic. Schmidhuber took one of the earliest approaches [59] toward this direction. He
modeled the dynamics and control of a system with two separate networks and trained
them both in parallel or sequential. However, the proposed fully recurrent networks
struggled in their flexibility and adaptation ability. One of the major difficulties he faced
in his work was the disability of Backpropagation through time and the ACDs to deal
with the long time lag between applied action and its execution. In another work, Bakker
combined the long short term memory [4, 3, 15] with the actor-critic method to learn par-
tially observable reinforcement learning problems. He utilized long short term memory
strength in learning long-term temporal dependencies to infer states in partially observ-
able tasks. He developed an integrated method, which learns the system’s underlying
dynamics and the optimal policy at the same time. Compared to our method which is
a data efficient batch learning and designed to learn non-Markovian problems, Bakker’s
approach had problems in dealing with high dimensionality, partial observability, contin-
uous state and action space, and a limited amount of training data. In a recent approach
by Sch¨afer [58], he applied recurrent neural reinforcement learning approaches to identify
and control a high-dimensional dynamic system with continuous state and action spaces
in partially unknown environments like a gas turbine. He introduced a hybrid recurrent
neural network approach that combines system identification and determination of an
optimal policy in one network. Furthermore, in contrast to our reinforcement learning
methods, it determines the optimal policy directly without making use of a value func-
tion. His approach is model-based and by constructing the model of the system it is able
to learn high dimensional and partially observable reinforcement learning problems with
continuous state and action spaces in a data efficient manner.
As it shown by Lin and Mitchell [38] recurrent neural networks are robust tools for
functional approximation in partially observable systems. In their work authors defined
three different architectures, recurrent-model, recurrent-Q, and window-Q architectures.
The first one learns an action model for the history features. The second approach learns
a Q-function approximation using indirectly observable training examples. And the third
method learns the Q-value approximation by taking a known number of state-action
pairs, window size, into its memory. As they have shown, these architectures are all
capable of learning some non-Markovian problems, but they have their own advantages
and disadvantages. Later on in the work by K. Bush [11], the author used echo state
networks(ESNs) to address the problem of non-Markovian reinforcement learning and
showed how they could successfully learn some standard benchmarks problems. The au-
thor experimentally validated the positive performance and dynamic attributes of the
echo state networks on modeling a system in the non-Markovian reinforcement learning
domains. The major difference between this work and our approach is, we tried a non-
linear readout layer to learn complex dynamic systems, but Bush introduced a memory
9](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-14-320.jpg)
![CHAPTER 2. RELATED WORKS 2.2. RECURRENT NEURAL NETWORKS IN REINFORCEMENT LEARNING
consolidation method using the mixture of expert (MoE) and tested it for stationary
dynamic systems. This framework preserves the ability to train the readout layers via
linear regression. The author showed that echo state networks exhibit low-mobility learn-
ing through trajectory-based features. Later on, echo state networks are also successfully
used and tested as Q-function approximation in work done by Oubbati et.al. [47]. Their
work sees the control task as a result of the interaction between brains, bodies, and
environments. There, the authors utilized reservoir computing as an efficient tool to
understand how the system behaviour emerges from an interaction. Inspired by imita-
tion learning designs, they presented reservoir computing models, to extract the essential
components of the system dynamics which are the result of the agent-environment in-
teraction. They validated their learning architectures by experimenting with a mobile
robot in a collision avoidance scenario. Compared to this work, we trained our policy
offline and with non-linearity on readout layer and applied discrete control actions. But
they first did a short pre-training and then used online linear approach to train echo
state networks and applied a continuous action to control an autonomous driving robot
in order to follow a line.
10](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-15-320.jpg)
![CHAPTER 3
REINFORCEMENT LEARNING IN FEEDBACK CONTROL
3.1 Learning In Feedback Control
The classical feedback control loop basically describes a control mechanism that uses
information from measurements of a process to control it. In each time interval, the
process communicates a system state, vector of measured dynamic variables, to the con-
troller that applies the control commands. In feedback control, the control variables are
measured and compared to the target values. Therefore, the feedback control mecha-
nism manipulates the input variables of the system in a continuous loop to minimize the
differences between estimation and desired targets. This allows for the formulation of a
broad range of challenging control applications. The direct approach to have a feedback
controller is by designing a model of the system. However for a nonlinear control sys-
tem, the task of model identification becomes tedious and complicated. An alternative
approach is to learn to control the feedback control systems. Which means, we need an
intelligent controller component that learns to control a subjected dynamic process using
experiences made from an interaction.
Reinforcement learning (RL) is a type of machine learning method which tries to learn
an appropriate closed-loop controller by simply interacting with the process and incre-
mentally improving the control behaviour. The goal of reinforcement learning algorithms
is to maximize a numerical reward signal by discovering which control commands i.e.
actions yield the most reward. Using reinforcement learning algorithms, a controller can
be learned with only a small amount of prior knowledge of the process. Reinforcement
learning aims at learning control policies for a system in situations where the training
information is basically provided in terms of judging success or failure of the observed
system behaviour [62]. Because this is a very general scenario, a wide range of different
application areas can be addressed. Successful applications are known from such differ-
ent areas as game playing [9], dispatching and scheduling [6], robot control [49, 56], and
autonomic computing [64].
11](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-16-320.jpg)
![CHAPTER 3. REINFORCEMENT LEARNING IN FEEDBACK CONTROL 3.2. MARKOV DECISION PROCESS
3.2 Markov Decision Process
The type of control problems we are trying to learn in this work are discrete time control
problems and can be formulated as a Markov decision process(MDP) [62]. An MDP has
four components: a set S of states, a set A of actions, a stochastic transition probability
function p(s, a, s ) describing system behaviour, and an immediate reward or cost function
c : S × A → R. The state of the system at time t, characterizes the current situation
of the agent in the world, denoted by s(t). The chosen action by agent at time step t is
denoted by a(t). The immediate reward or cost is the consequence of the taken action and
function of state and action. Since the rewards for the taken action can be formulated
as cost, the goal of the control agent would be to find an optimal policy π∗
: S → A
that minimizes the cumulated cost for all states. Basically, in reinforcement learning we
try to choose actions over time to minimize/maximize the expected value of the total
cost/reward:
E[R(s0) + R(s1) + R(s2) + ...]
3.3 Q-Learning
There are various reinforcement approaches that can be formulated based on the MDP
[62] e.g. value iteration and policy iteration, where the transition model and the reward
function of the control task are known. However, in many real-world problems the state
transition probabilities and the reward functions are not given explicitly. But, only a set
of states S and a set of actions A are known and we have to learn the dynamic system
behaviours by interacting with it. Methods of temporal differences were invented to
perform learning and optimization in exactly these circumstances. There are two principal
flavors of temporal difference methods. First, an actor-critic scheme [63, 62], which
parallels the policy iteration methods, and has been suggested as being implemented in
biological reinforcement learning. Second a method called Q-learning [72], which parallels
the value iteration methods. In this work, we consider using the Q-leaning as our principal
reinforcement learning algorithms. The basic idea in Q-learning is to iteratively learn the
value function, Q-function, that maps state-action pairs to expected optimal path costs.
The update rule in Q-learning algorithm is given by:
Qk+1(s, a) := (1 − α)Q(s, a) + α(r(s, a) + γ min
a
Qk(s , a ))
where s denotes the system state where the transition starts, a is the action that is
applied, and s is the successor system state. The learning rate α has to be decreased in
the course of learning in order to fulfill the conditions of the stochastic approximation and
the discounting factor denoted by γ [62]. It can be shown that under mild assumptions
Q-learning converges for finite state and action spaces, as long as the Q-value for every
state-action pair is updated infinitely often. Then, in the limit, the optimal Q-function
is reached.
3.3.1 Batch Reinforcement Leaning
As formulated above the standard Q-learning protocol considers an agent operating in
discrete time. At each time point t it observes the environment state st, takes an action
12](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-17-320.jpg)
![CHAPTER 3. REINFORCEMENT LEARNING IN FEEDBACK CONTROL 3.3. Q-LEARNING
at, and receives feedbacks from the environment including next state st+1 and the instan-
taneous reward rt. The sole information that we assume available to learn the problem is
the one obtained from the observation of a certain number of one-step system transitions
(from t to t + 1). The agent interacts with the control system in the environment and
gathers state transitions in a set of four-tuples (st, at, rt, st+1). Except for very special
conditions, it is not possible to exactly determine an optimal control policy from a finite
set of transition samples. In the literature there is a method called batch mode reinforce-
ment learning [17, 46, 36], which aims at computing an approximation of such optimal
policy π∗
, from a set of four-tuples:
D = {(sl
t, al
t, rl
t, sl
t+1), l = 1 · · · , #D}
This set could be generated by gathering samples corresponding to one single trajec-
tory (or episode) as well as by considering several independently generated trajectories or
multi-step episodes. In the work by Lange et al. [36] authors covered various batch-mode
reinforcement learning algorithms. Among them, we chose the growing batch method for
the purpose of training our learning algorithm. Training algorithms with growing batch
have two major benefits. First, from the interaction perspective, it is very similar to
the ’pure’ online approach. Second, from the learning point of view, it is similar to an
offline approach that all the trajectory samples are used for training the algorithm. The
main idea in growing batch is to alternate between phases of exploration, where a set of
training examples is grown by interacting with the system, and phases of learning, where
the whole batch of observations is used. The distribution of the state transitions in the
provided batch must resemble the ’true’ transition probabilities of the system in order
to allow the derivation of good policies. In practice, exploration cultivates the quality of
learned policies by providing more variety in the distribution of the trajectory samples.
Furthermore, it is often necessary to have a rough idea of a good policy in order to explore
interesting regions that are not in the direct vicinity of the starting states. If ’important’
states i.e. states close to the goal state are not covered by any of the trajectory samples,
then it is obviously not possible to learn a good policy from the batch data. This happens
because the system would not know which series of actions lead the to the goal area.
3.3.2 Neural Fitted Q-Iteration (NFQ)
Model-free learning methods like Q-learning are appealing from a conceptual point of
view and have been very successful when applied to problems with small, discrete state
spaces. But when it comes to applying them to the real world systems with larger and
probably continuous state spaces these algorithms facing some limiting are factors. For
the relatively small or finite state and action space, the Q-function can be represented
in tabular form and it is straightforward to approximate. However, when dealing with
continuous or very large discrete state and/or action spaces, the Q-function cannot be
represented by a table with one entry for each state-action pair. In this respect, there
are three problem can be identified :
• the ’exploration overhead’, causing slow learning in practice
• inefficiencies due to the stochastic approximation
• stability issues when using function approximation
13](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-18-320.jpg)
![CHAPTER 3. REINFORCEMENT LEARNING IN FEEDBACK CONTROL 3.3. Q-LEARNING
A common factor in modern batch reinforcement learning algorithms is that these
algorithms typically address all three issues and offer specific solutions to each of them.
The Fitted-Q iteration algorithm[17] is an efficient batch mode reinforcement learning al-
gorithm that can learn from a sufficiently rich trajectory generated. In this algorithm, the
Q-function approximation is done on an infinite or finite horizon optimal control problem
with discounted rewards. In each step, this algorithm uses batch data together with the
Q-values computed at the previous step to determine a new training set. Then it applies
a regression method on the training data to compute the next Q-value of the sequence.
Among several approaches to approximate Q-function, neural networks are considered
suitable tools [23, 67] because they provide a nonlinear mapping from input to output
data. The neural Q-function needs to define an error function which aims to measure the
difference between the target Q-value and the estimated Q-value. For example, a squared
error measure like
error = (Q(s, a) − (c(s, a) + γ min
a
Q(s , a )))2
.
To train the neural Q-function and minimize the estimation error common gradient
descent learning rules like Backpropagation [20] can be applied. In general, fitted-Q
iteration algorithms are trained online which requires thousands of samples and long
training time to learn a control task. Riedmiller [53] proposes an alternative approach,
neural fitted-Q iteration (NFQ), which performs an offline update step considering the
entire set of transitions. The standard NFQ benefits from the growing batch method
to collect the trajectory samples for training a multilayer perceptron Q-function. It has
been proven [54] that 2-layer of neural networks has sufficient approximation capacity to
generalize well for a closed loop control. The pseudo code for the NFQ algorithm shown
Algorithm 1:
Algorithm 1 Main loop of NFQ . k counts the number of iterations, kmax denotes the maxi-
mum number of iterations. init MLP() returns a multilayer perceptron with randomly initialized
weights. Backpropagation training ( P) takes pattern set P and returns a mulitlayer perceptron
that has been trained on P using backpropagation as a supervised training method.
procedure NFQ Main()
input: a set of transition samples D; output:Q value function QN ;
k = 0;
init MLP() → Q0;
while k < kmax do
generate pattern set P = {(inputl
, targetl
), l = 1, ..., #D} where:
inputl
= sl
, al
targetl
= c(sl
, al
) + γmin
a
Qk(s l
, a l)
Backpropagation training(P) → Qk+1
k := k + 1
3.3.3 Resilient Propagation (Rprop)
The original NFQ uses the Backpropagation [20] algorithm combined with an optimiza-
tion method i.e. gradient decent for training its multilayer perceptron neural network.
14](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-19-320.jpg)
![CHAPTER 3. REINFORCEMENT LEARNING IN FEEDBACK CONTROL 3.3. Q-LEARNING
As the equation below shows, the loss function computes the gradient with respect to
all the weights in the network. Then updates of the weight values are performed with
respect to the computed gradient:
wij(t + 1) = wij(t) −
∂E
∂wij
(t)
The partial derivative of the error function E with respect to the neural network
weights is computed based on the chain rule. The learning rate parameter affects the
convergence of the algorithm. If it’s too small, a system will take so many steps to
converge, and with a large learning rate, the system will oscillate around local minima
and fail to fall into the desired value range. Regular Backpropagation is a slow and
inefficient process. In order to accelerate this supervised learning process, it is possible to
use advanced techniques like Rprop [55], which has a more reliable and faster convergence
than the regular gradient descent method. The pseudocode in Algorithm 2 shows the
core part of the adaptation and learning the rule for Rprop. Using Rprop to train the
NFQ algorithm also reduces its overall complexity by removing the necessity of parameter
tuning for the neural Q-function. Rprop stands for Resilient Propagation, and it is an
efficient learning scheme, that in every iteration updates network’s weights based on the
changes in sign of the error function E. Here each network weight is updated individually
by ∆ij based on its local gradient information. So if the last weight update was so large
that the algorithm jumped over the local minima, then it needs to change it direction
accordingly. In this adaptive method, the new value for ∆ij is computed by multiplying it
with η−
or η+
for the negative or positive sign of the E. The values for 0 < η−
< 1 < η+
are parameters that can be set to some constant. Using the update value ∆ij, Rprop
algorithm increases or decreases individual weight values given the sign of the derivative.
Algorithm 2 The core part of the Rprop algorithm. The minimum (maximum) operator is
supposed to deliver the minimum (maximum) of two numbers; the sign operator returns +1 if
the argument is positive, -1, if the argument is negative, and 0 otherwise.∆ij determines the
size of the individual update value for each weight.
For all weights and biases{
if ( ∂E
∂wij
(t − 1) ∗ ∂E
∂wij
(t) > 0) then
∆ij(t) = minimum(∆ij(t − 1) ∗ η+
, ∆max)
∆wij(t) = −sign( ∂E
∂wij
)∆ij(t)
wij(t + 1) = wij(t) + ∆wij(t)
else if ( ∂E
∂wij
(t − 1) ∗ ∂E
∂wij
(t) < 0) then
∆ij(t) = maximum (∆ij(t − 1) ∗ η−
, ∆min)
wij(t + 1) = wij(t) − ∆wij(t − 1)
∂E
∂wij
(t) = 0
else if ( ∂E
∂wij
(t − 1) ∗ ∂E
∂wij
(t) = 0) then
∆wij(t) = −sign( ∂E
∂∆ij
)∆ij(t)
wij(t + 1) = wij(t) + ∆wij(t)
15](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-20-320.jpg)
![CHAPTER 4
ECHO STATE NETWORKS FOR FUNCTION
APPROXIMATION
4.1 Recurrent Neural Networks
Our goal in this work is to learn a feedback control system with delays without knowing
its dynamic model. As we described in the previous chapter, we chose Neural Fitted-Q
Iteration (NFQ) algorithm, a specific Q-learning algorithm, to learn our model-free task.
In its core, the NFQ algorithm uses multi-later perceptrons as a function approximator
to learn a model of the system dynamics and estimate the proper Q-value for each ac-
tion. In the control task with delays the Markov property will be violated, the ordinary
multilayer perceptron will fail to approximate the desired targets. Therefore, we need a
Q-function that is capable of holding the history of the system states in its memory for
robust approximation and preserving the Markov property. One possible option is to use
recurrent neural networks(RNNs) for the function approximation.
Recurrent neural networks (RNNs) have a structure similar to biological neural net-
works and they are able to hold the history information of the dynamic system. A
recurrent neural network contains (at least) one cycle path of synaptic connections. This
feature makes RNNs an excellent tool for function approximation in delayed control sys-
tems. Mathematically recurrent neural networks implement and approximate dynamic
systems and they are applied to the variety of tasks, for example, system identification
and inverse system identification, pattern classification, stochastic sequence modeling
[58, 47, 30]. However, it is not an easy straightforward task to train RNNs. Gener-
ally speaking, for training RNNs an extended version of the Backpropagation algorithm,
Backpropagation through time has been used [73, 74], but only with partial successes.
One of the conceptual limitations of the Backpropagation methods for the recurrent neu-
ral networks is that bifurcations can make the training non-converging [14]. Even when
they do converge, this convergence is slow, computationally expensive, and can lead to
poor local minima.
In more recent attempt to train recurrent neural networks the reservoir computing
approach [41] was introduced. These networks are in fact dynamic systems driven by the
input signal, or from another point of view they are nonlinear filters of the input signal.
16](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-21-320.jpg)
![CHAPTER 4. ESN FOR FUNCTION APPROXIMATION 4.2. A BRIEF OVERVIEW ON ECHO STATE NETWORKS
The idea of reservoir computing has been discovered and investigated independently under
the name of the echo state networks (ESNs) [28, 28] in machine learning, and liquid
state machines (LSMs) [42] in computational neuroscience. The work on the liquid state
machines is rooted in the biological setting of continuous time, spiking networks while
the ideas of the echo state networks were first conceived in the framework of discrete-
time, non-spiking networks in engineering applications. It was shown that the reservoir
computing approaches often work well enough even without full adaptation of all the
network weights. The reservoir computing methods usually use a supervised learning
scheme for training. In this work, we employ echo state networks as a Q-function for
their simplicity in training and capability in approximating the targets in the presence of
delay in dynamic systems [29]. Perhaps surprisingly, this approach yielded an excellent
performance in many benchmark tasks, e.g. [27, 32, 33, 68].
4.2 A Brief Overview On Echo State Networks
As we mentioned in the previous section, we employed discrete time echo state networks
for the function approximation. The echo state network has M input units, N hidden
units(reservoir neurons), and L output units. The data at each time point t, fed to the
network through the input units in the form of a vector U(n) = [u1(t), u2(t), · · · , uM (t)]T
.
The internal values for the reservoir are declared by X(n) = [x1(t), x2(t), · · · , xN (t)]T
, and
the output of the network in the time point t is showed by Y(t) = [y1(t), y2(t), · · · , yL(t)]T
.
The Figure 4.1 depicts the three-layered topology of the echo state network, the input
layer, the hidden units or reservoir, the and readout layer. There are four types of
weight matrices which connect different echo state network layers together. Three of
these weight matrices are initialized once and will stay constant during the training
time. These constant weights are described in the following. The input weight matrix
Win
with N × M dimensions which connects the input data to the hidden units. The
reservoir weight matrix W with N × N dimensions which connects the hidden units to
each other and also indicates the recurrent connections. The feedback weight matrix
Wfb
with N × L dimensions which connects the outputs back to the reservoir. The
initial values for these constant weight matrices have to be chosen carefully to increase
the performance of the echo state networks. There are various initialization and hyper-
parameter optimization techniques that can be used to initialize the Wfb
, Win
, and W
matrices more sophisticatedly according to the dynamic task. We will cover some of the
important methods along this chapter to initialize and optimize these weight values. In
the echo state networks only the output weights Wout
with L × (M + N) dimensions
will be trained. They map the reservoir states to the estimated targets. Later in this
chapter, we will describe various methods for training the weights Wout
. The update rule
for changing the state of the reservoir is:
X(t + 1) = f(Win
U(t + 1) + WX(t) + Wfb
Y(t))
X(t + 1) = (1 − α)X(t) + αX(t + 1)
Here α is the leaky integrator factor of each hidden unit. As it is shown in the equation
above echo state networks (ESNs) are composed of simple additive units with a nonlinear
activation function f. And the reservoir with a leaky integrator unit type has individual
state dynamics, which can be exploited in various ways to accommodate the network
to store temporal characteristics of a dynamic system. Leaking rate α is the amount of
17](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-22-320.jpg)
![CHAPTER 4. ESN FOR FUNCTION APPROXIMATION 4.2. A BRIEF OVERVIEW ON ECHO STATE NETWORKS
4.2.1 Learning In Echo State Networks
In this work we want to use echo state networks for the function approximation in the
Q-learning algorithm because they could hold the history of the dynamic systems in their
state. Therefore, the ESN output Y(n) ∈ RL
will be the Q-value of the function approx-
imation and we try to minimize the error measure E(Y, Ytarget
) between the estimated
output Y(n) and target output Ytarget
(n). More importantly, we need to make sure that
our approximation generalizes well to unseen data. The error measure E can typically
be a Mean-Square Error (MSE), for example a root mean square error (RMSE):
E(Y, Ytarget
) =
1
L
L
i=1
1
T
T
i=1
(yi(n) − ytarget
i (n))2
which is here averaged over the L dimensions of the output data. The RMSE can
be dimension-wise normalized (divided) by the variance of the target data Ytarget
(n),
producing a normalized root mean square error (NRMSE). The NRMSE has an absolute
interpretation: it does not depend on the arbitrary scaling of the target data Ytarget
(n)
and the value of 1 can be achieved with a simple constant output data Y(n) set to the
mean value of Ytarget
(n). This suggests that a reasonable model of a stationary process
should achieve a NRMSE accuracy between zero and one. As we mentioned earlier, the
only adaptable weight matrix in echo state networks is the output weights matrix Wout
and it is possible to train these weights with different algorithms. In order to train linear
readout weights Wout
there are different methods [48, 28, 30, 66, 31], and using least
square method is a standard offline procedure and formulated as follows:
• Generate the reservoir with specified set of parameters.
• Feed the input data U(t) and compute and collect the reservoir states X(t).
• Compute Wout
from the reservoir using Linear Regression, minimizing the Mean
Square Error between Y(n) and Ytarget
(n).
• Compute Y(n) again from trained network by employing Wout
and feeding the
input data U(t).
4.2.2 Spectral Radius and Echo State Property
Echo state networks have two major functionalities that make them feasible tools for
function approximation in the presence of delays. First acting as a nonlinear filter in
order to expand the input signal U(t) to a higher dimensional space X(t). Second acting
as a memory to gather the temporal features of the input signal and this ability in the
literature called is short term capacity of the echo state network. The Combination of
these two capabilities enriches the computed reservoir state matrix X, containing the
history information about the dynamic behaviour of a control system. However, the
computed state of the reservoir is influenced by various parameters that have to be set-
ting judiciously. Those parameters or hyper-parameters are: the size of reservoir N, the
sparsity of the weight matrices, the distribution of the nonzero elements of weight matri-
ces, the spectral radius of reservoir weight matrix ρ(W), the scaling of the input weight
matrix Win
, and the leaking rate α. M. Lukosevicius [40] describes informative review
on how to set these parameters and explains their affects on the system performance. In
19](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-24-320.jpg)
![CHAPTER 4. ESN FOR FUNCTION APPROXIMATION 4.3. ECHO STATE FITTED-Q ITERATION
this work we mainly focus on the memory aspect of the echo state networks and try to
choose these parameters according to the memory size needed for the delayed control task.
According to Herbert Jaeger [28] a network has echo state property if the network
state X(t) is uniquely determined by any left-infinite input sequence U−∞
. After the
burn-in phase, removing the initial transition sates, the reservoir becomes synchronised
to the input signal U(t) and the current reservoir state X(t), and exhibit an involved
nonlinear response to the new input signal. This phenomenon can be explained by the
echo state property: The network will forget its random initial state after the burn-in
phase and the reservoir state becomes a function of the input signal. This is an essential
property for successful training of a reservoir. Izzet B. Yildiz and et.al. [75] defined a
simple recipe to preserve the echo state property. That procedure is explained in the
following steps:
• Generate a weight matrix W0 from uniform or Gaussian distribution.
• Normalize W0 with its spectral radius λmax, W1 = 1
λmax
W0
• Scale W1 with a factor 0 < α < 1 and use it as the reservoir matrix W = αW1.
The spectral radius of the reservoir weight matrix W is one of the most important
global parameters of the ESN and it has to be chosen according to, in our case, the needs
of the dynamic control task. Usually the spectral radius is related to the input signal,
in the sense that if lower time-scale dynamics is expected (fast oscillating signals) then
a lower spectral radius might be sufficient. However, if longer memory is necessary then
a higher spectral radius will be required. The downside of a larger spectral radius is
the longer time for the settling down of the network oscillations. From an experimental
point of view it means having a smaller region of optimality when searching for a good
echo state network with respect to some dataset. Basically, if the spectral radius value
is larger than one, it can lead to reservoirs hosting multiple fixed points that violate the
echo state property.
4.3 Echo State Fitted-Q Iteration
As we described earlier, an important aspect of echo state networks is their ability to
serve as a memory to capture and preserve the temporal behaviours of their input sig-
nal. For that reason, we substituted the multilayer perceptron Q-function in the NFQ
algorithm with echo state networks and introduced the Echo State Fitted-Q Iteration
(ESFQ) algorithm. Compared to Algorithm 1, we changed the MLP Q-function to the
ESN Q-function and the Backpropagation training is replaced by a particular training
method for echo state networks. The advantage of the ESFQ algorithm is that the cu-
mulative history of the states-action pairs in the reservoir helps the Q-function to deal
with the unknown delays in the control task. In order to train the echo state Q-function
for a dynamic control task first, the weight matrices need to be initialized. Then the
accumulated state-action pairs in the growing batch, are used to feed to the reservoir at
the end of each episode. The reservoir dynamics at each point in time presented in the
activation values of its hidden units. These activation values from the reservoir state ma-
trix, reservoir design matrix, which its rows correspond to the system state-action pairs
20](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-25-320.jpg)
![CHAPTER 4. ESN FOR FUNCTION APPROXIMATION 4.4. TYPES OF READOUT LAYERS FOR ECHO STATE Q-FUNCTION
on each cycle in each trajectory sample and its columns correspond to the computed
features. In the burn-in phase, to remove the early noisy and unknown dynamics of the
reservoir for each trajectory sample, we remove a few beginning states of the reservoir
matrix. Then we compute the readout weights Wout
using the teacher values. These
teacher values for every state and action sample in a trajectory are generated from the
sum of the immediate cost and the Q-value of the successor state. Finally having the
new readout weights Wout
we compute the estimated targets.
4.4 Types Of Readout Layers For Echo State Q-Function
In the Echo State Fitted-Q Iteration algorithm, which is an offline policy learning algo-
rithm, the Q-Function plays the role of a supervised regression method. Compared to the
original Neural Fitted-Q Iteration algorithm described in section 3.3.2, we use an echo
state Q-function with linear least squares on the readout layer. Although, the linear least
squares makes the learning fast, sometimes it struggles in learning more complex nonlin-
ear problems or higher input dimensions. However, because of the echo state networks
topology and the fixed values of the input and reservoir weight matrices, it is possible
to train the echo state Q-function with different types of readout layers. In the follow-
ing, we describe two methods for training the output layer of the echo state Q-function
that help it to estimate more accurate targets for the purpose of reinforcement learning.
In this regard, we investigated the possibility of using ridge regression, and multilayer
Perceptrons.
4.4.1 Ridge Regression
In a standard formulation, the reservoir state matrix or the design matrix, is overdeter-
mined matrix, because the number of trajectory samples is way larger than the hidden
unit numbers N T. Therefore, the solution for this equation Wout
= X−1
Ytarget
needs
to be approximated. To compute stable solution we apply ridge regression, also known
as Regression with Tikhonov Regularization:
Wout
= (XT
X + βI)−1
XT
Ytarget
The regularization parameter β will prevent output weights from growing arbitrarily
large, which causes overfitting and instability in the system. This approach penalizes the
squared length of the weight vector and transforms the error minimization task to a convex
optimization problem that can be solved in analytical closed form. The optimal values of
β can vary by many magnitudes of size, depending on the exact instance of the reservoir
and length of the training data. To choose the value of the regularization parameter β
by doing a simple exhaustive search, it is recommended to search on a logarithmic grid.
Furthermore, β can be found using cross-validation or Gaussian-Processes. It is preferable
to set the value of β not to zero to avoid numerical instability in computing the invert of
the matrix (XT
X)−1
. Interpretation of the linear readout gives an alternative criterion
for setting β directly [12]. Because determining the readout weights is a linear task, the
standard recursive algorithms, from adaptive filtering, for minimizing mean square root
error can be used for online readout weight estimation [30]. The recursive least squares
algorithm is widely used in signal processing when a fast convergence is of prime priority.
21](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-26-320.jpg)
![CHAPTER 4. ESN FOR FUNCTION APPROXIMATION 4.4. TYPES OF READOUT LAYERS FOR ECHO STATE Q-FUNCTION
4.4.2 Multilayer Perceptron Readout Layer
The first and major reason we employed echo state networks as a Q-function approxima-
tion were their ability to hold the history of the changes in the dynamic system behaviours.
We need this history to deal with delays in executed action in the control problem. How-
ever, the estimated Q-value by linear readout layers of the echo state Q-function might
not help with complex nonlinear dynamics. The linear least square readout layer returns
a global minimum which means the estimated targets have the minimum possible mean
squared error to the original target. But a small error value does not always mean we
have a successful policy for reinforcement learning. Also, it has been reported [29] that a
larger reservoir size, even with high randomness in its weight matrices, combined with a
linear readout layer might not be able to learn nonlinear dynamic systems. Therefore, we
need to utilize more advanced readout layers to learn the policy for nonlinear dynamic
systems. To do so, we use a multilayer perceptron on the readout layer of the echo state
Q-function and trained it via the Backpropagation algorithm. The novelty of this ap-
proach is that the reservoir expands the systems states into higher dimensional features
space and also preserves the history of the states. Then the Backpropagation algorithm
trains the multilayer perceptron to learn the value function which estimates the proper
Q-values for controlling the delayed dynamic system. The Figure 4.2 illustrates a topo-
logical example for the Echo State Fitted-Q Iteration algorithm.
Figure 4.2: The topological illustration of the Echo State Fitted-Q Iteration algorithm with
multilayer perceptron readout layer. It consists of two major parts. First, the memory part
containing echo state input weight and reservoir weight matrices and their corresponding neu-
rons. In the second part the readout layer containing one input layer, two hidden layers of
feedforward neural networks and one output layer. Notice that the applied action goes only
through the readout layer but not the reservoir itself.
22](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-27-320.jpg)
![CHAPTER 4. ESN FOR FUNCTION APPROXIMATION 4.5. SMAC FOR HYPER PARAMETER OPTIMIZATION
quential Bayesian optimization allows for global optimization of such black-box functions
by using as few trials as possible. MOCKUS et al. [44] applied Bayesian methods to cases
with linear and nonlinear constraints and to multi-objective optimization. They stud-
ied interactive procedures and reduction of multidimensional data in connection with
global optimization using Bayesian methods. Various implementations of this sequen-
tial optimization match or surpass manually tuned performance for the tasks such as
satisfiability problem [25] or object classification [7]. In this work we employed SMAC
hyper-parameter optimization software which is implemented in [25] and used sequential
Bayesian optimization methods. In the extended version of the SMAC software [24] it is
possible to optimize various types of parameters e.g. categorical parameters and sets of
instances.
24](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-29-320.jpg)
![CHAPTER 5
EXPERIMENTS AND RESULTS
5.1 Experimental Setup
In this section, we demonstrated results of our experiments in order to evaluate the
performance of the Echo State Fitted-Q Iteration (ESFQ) algorithm on the learning of
delayed control tasks. The presence of a delay, in particular, the action delay, violates
the Markov property of a control system. The main goal of this work was to show that
echo state networks with a nonlinear readout layer are practical memory based function
approximation in the non-Markovian reinforcement learning domain. Therefore, we tried
to compare the performance of our method to tapped delay-line memory base algorithms,
as a baseline method and discuss the different advantages and drawbacks of it. We choose
two standard simulated benchmarks, mountain car, and inverted pendulum, for evaluat-
ing our model-free learning algorithms. We tested our algorithms with plants containing
delays in their applied actions with different lengths in the range of 1 to 30. We showed
that the solution we proposed within the Echo State Fitted-Q Iteration algorithm can
be a very stable and reliable learning tool for a system with unknown delay lengths and
complex dynamics.
In the rest of this chapter, we introduce our selected benchmarks and their parameter
settings for each experiment. After that, we describe the setting for our algorithms,
the general scheme of our experiments, and the hyper-parameter optimization for the
ESFQ algorithm. Then we compare and analyze computed the results based on their
accuracy, efficiency, and ability to deal with the complex dynamic system with delays.
We implemented our core methods and algorithms in C + + using the Eigen framework
and integrated them to the CLSquare simulation system. For the hyper-parameter tuning,
we employed SMAC [24], a versatile tool for optimizing algorithm parameters.
5.2 Simulation Benchmarks
In the realm of reinforcement learning, there are various standard simulated benchmarks
[62] used to evaluated algorithm’s performances. In this work, we tried two of these bench-
marks for evaluating the performance of our algorithms and collected measurements to
compare their abilities in dealing with delay in control systems. The benchmarks were
25](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-30-320.jpg)
![CHAPTER 5. EXPERIMENTS AND RESULTS 5.2. SIMULATION BENCHMARKS
mountain car and inverted pendulum [62]. The dynamics of balancing a pendulum at an
unstable position can be employed in the applications of controlling walking robots or
rocket thrusters. This highly nonlinear dynamics make the inverted pendulum a suitable
and difficult task for our algorithm to solve. Our second chosen benchmark, the moun-
tain car problem, is a second-order, nonlinear dynamic system with low-dimensional and
continuous-valued state and action spaces. The simulated plants are provided by the
CLSquare tool, developed and implemented by the Machine Learning Lab of the Uni-
versity of Freiburg. In the following, we briefly introduce each benchmark and their
parameter settings for our experiments.
5.2.1 Mountain Car
Mountain car is a standard testing domain in reinforcement learning. The problem is for-
mulated such an under-actuated car must drive up a parabolic shape steep hill. Starting
from any initial state in the middle of the valley the goal of the system is to drive the
car up the valley’s side and escape in as few steps as possible. Since gravity is stronger
than the car engine, even at full throttle, the car cannot simply accelerate up the steep
slope. The car must rock back and forth along the bottom of the valley to build enough
momentum to escape. The allowed actions are {−4, 4} Newton to drive the system in
left and right directions. The state vector has two dimensions, car position along x-Axis
in meters, and car velocity in m/s. The allowed area for the car is between [−1.2, 0.7]
and the system will receive a transition cost of 0.01 in this range. The terminal area is in
the range of [∞, −1.2] and the agent will receive a terminal cost of 1.01 and episode cycle
will be ended. The target area is between [0.7, ∞] and agent receive no cost in the range
and the episode will be ended. The Figure 5.1 depicts the mountain-car visualization in
the CLSquare simulation.
Figure 5.1: The simulated mountain car visualization. The allowed position for the car is in
the range of [−1.2, 0.7] meters and the goal area in the range of [0.7, ∞] meters. The terminal
area for the car is in the range of [−∞, 0.7] meters.
5.2.2 Inverted Pendulum
The inverted pendulum problem benchmark requires balancing a pendulum of unknown
length and mass at the upright position by applying forces to the cart it is mounted
26](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-31-320.jpg)
![CHAPTER 5. EXPERIMENTS AND RESULTS 5.3. EXPERIMENTAL PROCEDURE AND PARAMETRISATION
The discount factor γ parameter for our Q-learning algorithm was set to 0.98 and no
exploration strategy was used during the training.
5.3.1 Echo State Q-Function
As we described in the previous section the reservoir in echo state networks works as
memory for the Q-function and the multilayer perceptron as a readout layer in the Echo
State Fitted-Q Iteration algorithm. For our memory based approach it was essential
to have sufficient memory capacity for holding the history of delayed states. Here we
benefited from the short-term capacity of the echo state Q-function. However, the Q-
function parameters needed to be specified according to the length of the delay and the
dynamics of the control task at hand.
Table 5.1: The list of parameters for initializing echo state networks.
Name Description And Value Range Optimized
Spectral Radius Real value [0, 1], adjusts memory size True
Leaking Rate Real value [0, 1], adjusts dynamics of each neuron True
Reservoir Size Integer value in 10, 20, 30, 40, 50, 100, 200 True
Transition steps Integer set to {5% or 10%} of cycle length False
Regularization
coefficient
Real value in [10e-3,10e3] True
Activation function Logistic, Tanh, Rectifier True
Input weight sparsity Real value [0.2, 1.0] True
Reservoir weight spar-
sity
Real value [0.2, 1.0] True
A list of echo state networks parameters is available in Table 5.1. The spectral radius
and leaking rate together affect the short term memory length of the network. Basically,
for a system with fast oscillation like the inverted pendulum, we would need a shorter
memory length compared to slower rate systems like the mountain car. A high spectral
radius and or a very small leaking rate increase the short memory length of echo state
network. The transition steps are the number of the time steps in each trajectory samples
that are used for burning the reservoir and these state will be removed in the learning
time. The burning phase helps the reservoir to remove its noisy dynamics and become
input driven rather than affected by the arbitrary initial states of itself. The length of
the transition steps has a linear dependency on the sample trajectory length, usually
5 to 10 percent of the average trajectory length. The reservoir size depends on the
dimension of the input signal and the complexity of the system dynamics, for some
tasks it needs to be set to a very large number. The regularization coefficient and the
activation function need to be searched and found regarding the control system at hand.
The standard choice for activation function is the hyperbolic tangent function. A higher
value for the regularization coefficient is used to prevent linear readout weights from
growing exponentially. Typically it is a real number in logarithmic scale and in the range
28](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-33-320.jpg)
![CHAPTER 5. EXPERIMENTS AND RESULTS 5.3. EXPERIMENTAL PROCEDURE AND PARAMETRISATION
of [0.00001, 1]. In the following we talk about the initialization of the echo state network
weight matrices which are playing a major role in capturing system dynamics.
Figure 5.3: On the left, the reservoir weight matrix is initialized from a normal distribution
with zero mean and unit variance. Then it is normalized with the spectral radius of 0.54. On the
right, the reservoir weight matrix is initialized from a normal distribution with zero mean and
unit variance. Then orthogonal matrix is computed using Schur decomposition implemented in
the Eigen library and it is normalized with the spectral radius of 0.54. Finally 40 percent of its
coefficient set to zero.
The sparsity of the echo state weight matrices boosts its performance by extracting
more distinguishable features from input signals. In the Figure 5.3 reservoir weight matri-
ces for 20 hidden units are illustrated. The matrix in the left is initialized using a normal
distribution with zero mean and unit variance and then normalized with the spectral
radius of 0.54 and fully connected. In fully connected reservoirs, all reservoir neurons
were directly connected to each other with all connection weights being nonzero. The
more sparsely connected matrix on the right graph was generated in two steps. First, a
fully connected reservoir was generated in the same way as the first one. Then from that,
an orthogonal matrix computed using Schur decomposition. Finally, given the sparsity
percentage, 40% in this example, randomly chosen coefficients were set to 0. We need
to consider the fact that if the sparsity percentage is set very high, then the reservoir
does not exist as a group of mutually connected neurons. Usually, larger reservoirs (more
hidden units) tolerate a lower connectivity and still fulfill this requirement. The advan-
tage of a sparse orthogonal reservoir weight matrix is that it has linearly independent
columns and echo states were computed by this matrix could better catch the dynam-
ics of the system. The weights of the reservoir W functions as a dynamic basis for the
original input signal. Therefore, these bases decompose the input signals into pertinent
sub-components to maximize their differences.
29](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-34-320.jpg)
![CHAPTER 5. EXPERIMENTS AND RESULTS 5.3. EXPERIMENTAL PROCEDURE AND PARAMETRISATION
In summary, we suggest that the performance echo state Q-function highly depends
on a proper balance of the internal dynamics and the influence of external activities,
mediated via the input weights, and the strengths of the input and target activities. See-
ing that the external activities can not be directly manipulated, the initialization of the
reservoir and input weight matrices are important. If the input weight values were set
too high, it will dominate the dynamics unfolding in the reservoir and quickly overrule
any other useful internal dynamics. A proper choice of the weight initialization intervals
is necessary to generate a well-performing echo state Q-function.
5.3.2 Hyper-parameter Optimization
The choice of parameters depended on the delay length which was unknown for the
Echo State Fitted-Q Iteration algorithm. Therefore, we had to search over the parameter
space to find out which configuration provides sufficient memory length for our algorithm.
Hence, we used the hyper-parameter optimization tool SMAC to search over given pa-
rameters: spectral radius, leaking rate, reservoir size, non-linear function for reservoir
neurons, input weight matrix sparsity, and reservoir matrix sparsity. These parameters
are inter-depended and affect one another. The parameter optimization algorithms try
to minimize the loss function over sets of discrete and continue values. To compute the
loss value we trained our algorithm for 120 episodes, 200 cycles each on the inverted
pendulum task with an unknown action delay given a particular parameter set. The
algorithm was tested 100 times with different initial positions at the end of each episode.
A successful test was one that ended up in the goal area in the finite horizon task. The
averaged success rate subtracted after 120 episodes were returned as a loss value. SMAC
searched over this parameter space using Bayesian methods of global optimization to find
settings that provide minimum expected deviation from the global minimum. Parameter
tuning over reinforcement learning algorithm is a computationally expensive process, and
for each delay length has to be done separately. Therefore, in the scope of this work, we
only ran a few optimization iterations, roughly 60 to 80 per delay length and in total
2500 experiments, to find different good settings.
The Figure 5.4 shows the Pearson correlation coefficient for 6 hyper-parameters opti-
mized via SMAC. Each cell shows the correlation coefficient between parameters on its
corresponding column and row. Labels are parameter names in both Axis. Coefficient
values range between [−1, 1] with a higher absolute value for higher correlation and the
sign for direct or inverse correlation. And they are computed from samples with at least
80% goal reach. The value range for each parameter is presented in Table 5.1. Our re-
sults showed that the most frequently chosen activation function was logistic activation
therefore we exclude it from correlation computation. The goal of this optimization was
to find the configuration which gives echo state Q-function enough short-term memory
capacity to learn the delayed control task. Therefore, we include corresponding delay
lengths for each experiment to observe its correlation with other parameters.
30](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-35-320.jpg)
![CHAPTER 5. EXPERIMENTS AND RESULTS 5.3. EXPERIMENTAL PROCEDURE AND PARAMETRISATION
Figure 5.4: This plot shows the pairwise correlation matrix between for 6 optimized hyperpa-
rameters using Pearson product-moments method. The labels on the figure assigned to their
corresponding parameter. The delay lengths, third column, was kept constant during each op-
timization iteration and its value is ranged between [1, 30]. The dataset which the correlations
are computed is from samples which have 80% or more goal reach, 1561 experiments, for the
simulated inverted pendulum plant.
There are few observations we could make by looking at the bivariate correlation ma-
trix on Figure 5.4. First, the reservoir matrix and input matrices sparsity were positively
correlated. This means for experiments with different delay length their value increases
and decreases simultaneously. However, this correlation was not strong. Contrary to
that, the leaking rate and spectral radius are inversely correlated. Although this was
a weak correlation it implied that a reservoir with high spectral radius needs a lower
leaking rate for holding the history of delayed actions. According to the update equation
of the reservoir, a high leaking rate means its new state will be influenced less by its
previous state and high spectral radius means the reservoir previous state will have more
influence on the next computed state. Another observation is that the delay length in
this matrix had a weak negative correlation with the spectral radius, the leaking rate,
and the input weight matrix sparsity. Our bivariate correlation matrix also shows the
delay length and the reservoir size had almost zero correlation. Furthermore, we can see
that the reservoir size and the spectral radius have a positive correlation. Both of these
two observations are counter-intuitive, because as we described before for a reservoir with
a large number of hidden units we would expect a relatively small spectral radius. In
addition, for a longer delay length, we need a higher spectral radius to provide enough
short-term memory capacity. Therefore, we can safely conclude that these parameters
are interdependent and they have a multivariate dependency among them which can not
be understood only using a bivariate correlation computation. To analyze the effects of
these parameters on the ESFQ performance in detail we need to run thoroughgoing ex-
periments of hyper-parameter optimization with an extensive number of iteration for each
delay length. Then by knowing their multivariate correlations we can find an efficient
way of selecting them. However, in the scope of this work for selecting these parameters,
31](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-36-320.jpg)
![CHAPTER 5. EXPERIMENTS AND RESULTS 5.3. EXPERIMENTAL PROCEDURE AND PARAMETRISATION
we need to rely on computationally expensive hyper-parameter optimisation done over
our reinforcement learning algorithm.
Figure 5.5: The figure shows a histogram plot of the goal reaching percentage of the spectral
radius, leaking rate, reservoir weight matrix sparsity, input weight matrix sparsity. Each bin on
the Y-Axis shows how frequently and with what percentage the system reaches the goal given
the corresponding subject parameter on X-Axis. There are in total 2500 experiments on the
simulated inverted pendulum containing action delays in the range of [1, 30].
In Figure 5.5 we see a histogram plot for the spectral radius, leaking rate, reservoir
weight matrix sparsity, input weight matrix sparsity parameter relative to reaching the
goal area. In total, there were 1561 configuration samples with more than 80% success
rate, out of 5600 different configurations that we tried and in average 50 samples per
delay length. As we can see on the top left plot system spectral radius values in the
range of [0.0001, 0.3] have dominantly reached the goal area with high success rate. For
the leaking rate in the plot top right, the successful experiments have a broader range of
[0.0001, 0.9]. And for both weight matrices sparsity, the successful experiments frequently
occur in the range of [0.4, 0.8]. This illustration will help us to elicit some general rule
for initializing our hyper-parameters and putting some constraints on their value range
and make the search more efficient. Figure 5.6 depicts mean and variance of the four
echo state hyper-parameters relative to goal reach percentage for experiments above 80%
success rate. In the top rows, we compared spectral radius and leaking rate against
goal reaching percentage. The bottom row compares reservoir and input weight matrices
sparsity. The frequency of successful experiments for each of these values can be seen
in the corresponding histogram plots in Figure 5.5. As we can see for a wide range of
these subject parameters there are successful experiments. This is another support for
32](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-37-320.jpg)
![CHAPTER 5. EXPERIMENTS AND RESULTS 5.3. EXPERIMENTAL PROCEDURE AND PARAMETRISATION
the presence of multivariate interdependencies between these parameters.
Figure 5.6: The figure shows the mean and variance of the goal reach percentage on the Y-Axis,
given value of the particular subject parameter on the X-Axis. The subject parameters are the
spectral radius, leaking rate, reservoir weight matrix sparsity, and input weight matrix sparsity.
The samples selected for experiments with more than 80% goal reach. There are in total 2500
experiments on the simulated inverted pendulum containing action delays in the range of [1, 30].
Our goal in this section was to suggest that there are rules for efficiently choosing
some of the hyper-parameters of the ESFQ algorithm. And according to our parameter
optimization results in the previous section deducing such rules is not a straightforward
task due to multivariate dependencies between these parameters. It will need an extensive
study of the parameter space which is beyond the scope of the work. But in order to
drive some general constraints for setting up and computing a reservoir given a particular
task, our results suggests the following:
• Reservoir size, the number of hidden units, up to 10 times larger than the size of
the input vector. This is usually a large enough number to grasp and reflect the
system dynamics.
• Input weight matrix and reservoir weight matrix should be sparse as we saw in
Figure 5.5, they better to have the sparsity values between [40, 60] percent.
• Our experimental results suggest that the logistic activation works well, 1514 out
of 1561 successful experiments, for the reservoir activation function.
• leaking rate and spectral radius have a weak inverse correlation but it is not in the
scope of this work to comment on the choice of their values.
33](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-38-320.jpg)
![CHAPTER 5. EXPERIMENTS AND RESULTS 5.4. COMPARISONS TO DELAY-LINE NFQ ALGORITHM
Basically, these results suggest that there are possibilities to chose the algorithm
parameters according to the characteristics of a given dynamic control task. But to do
so we would need to do more experiments and collect enough samples per delay lengths.
We may run 300 SMAC iterations for each delay length and then study the change in the
parameters relative to how well the controller learns a task. Then, it is possible to fix
the value of some of these parameters and only search over particular ones. In the last
chapter of this work, we will mention some of the probable solutions for optimizing these
parameters and better understanding their dependencies.
5.4 Comparisons To Delay-Line NFQ Algorithm
In this section, we compare the results of our method to the standard delay-line Neural
Fitted-Q Iteration algorithm. Here we used a Q-function with two layers of hidden
units, 20 neurons each, proven[54] to be feasible structure to learn dynamic control tasks.
In the delay-line method, the action delays lengths are known to the algorithm. The
system history is a collection of observed states from the most recent state-action pair
and previous states, equal to the delay length. This means if there a 3 steps action
delays in the simulated plant, then the input vector contain the current state-action pair
and the past three states. The NFQ algorithm was trained on the simulated inverted
pendulum benchmark for 200 episodes, 200 cycles each, using Rprop and one update
per episode. For the sake of comparison, we added the same nonlinear structure as the
NFQ algorithm on top of our reservoir in Echo State Fitted-Q Iteration algorithm. As
we described in chapter 4, the difference between the NFQ and our method is that we
preserve the memory for our method by using the reservoir of the echo state networks.
And we employed the hyper-parameter optimization tool SMAC to find a suitable memory
capacity for the echo state Q-function. In simple terms, we compared the performance
of the reservoir as a memory to the classical tapped delay-line memories [65]. Although
the Echo State Fitted-Q Iteration algorithm is not aware of the delay length but with
the aid of the hyper-parameter optimization tool SMAC it manages to find the best
configuration. This approach could be considered the same as giving the delay length to
the system.
34](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-39-320.jpg)
![CHAPTER 5. EXPERIMENTS AND RESULTS 5.4. COMPARISONS TO DELAY-LINE NFQ ALGORITHM
Figure 5.7: Maximum percentages of goal reach achieved by delay-line NFQ and ESFQ algo-
rithms given different delay lengths. The results achieved from the simulated inverted pendulum
with action delays in the range of [1, 30]. The X-Axes shows the delay lengths and the Y-Axis
show the maximum percentage of goal reach.
In Figures 5.7 and 5.8 we present a comparison between the delay-line Neural Fitted-
Q Iteration and the Echo State Fitted-Q Iteration algorithms on the test phase with
different initial values and different delay lengths. In the first graph, we can observe
that the ESFQ algorithm has the advantage of achieving the maximum value even with
the long delay length. On the other hand, we can observe that delay-line NFQ faces
difficulties after 10 steps action delays and its peak performance decays down to 40%.
Such observations suggest that reservoir is a very suitable memory based approach for
the control system with longer delay lengths. For every delay length it reaches the goal
area, from different initial positions in the test phase, on average more than 90% of the
times, and in some cases 100% of the times. Furthermore, in the second plot 5.8 we can
observe that the ESFQ algorithm shows fairly robust performance to randomness in the
initialization of its fixed weight matrices. Here for every length of delay, we trained both
algorithms, initialized with 10 different random seeds, and plot the mean and standard
deviation of goal reach percentage over delay lengths. As we see the NFQ algorithm
performance is prone to decay when it is trained with different random seeds especially
for longer delay lengths. On the contrary, the ESFQ algorithm performs more robustly
in the face of such random initialization even for longer delay lengths. In this regard,
we need to consider the fact that the echo state performance presumably should decay
because it has extra random parameters for its weight matrices. However, given the
same configuration for initializing the reservoir over various random seeds, our algorithm
shows much better overall performance. Furthermore, the closed range of performance
variances in both algorithms is another evidence for the robustness of the ESFQ algorithm
toward randomness in its parameter initializations. In our experiments, we are trying to
support the idea that the echo state Q-function is a fairly reliable memory based function
approximation for the dynamic system with delays. And, the facts about the randomness
support the idea that the memory capacity of the echo state Q-function is less likely to
be affected by random initialization of its parameters.
35](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-40-320.jpg)
![CHAPTER 5. EXPERIMENTS AND RESULTS 5.4. COMPARISONS TO DELAY-LINE NFQ ALGORITHM
Figure 5.8: Mean and standard deviation of the goal reaches percentage achieved by delay-line
NFQ and ESFQ algorithms given different delay lengths. The results achieved from experi-
menting with simulated inverted pendulum containing action delays in the range of [1, 30]. The
X-Axis shows the delay lengths and the Y-Axis show the mean and standard deviation of the
goal reach percentage average over experiments with 10 different random seeds. The mean
values are computed from the maximum percentage of goal reach for each experiment.
In Figure 5.9 we compared the learning curve for the ESFQ and NFQ algorithm on
the experiments with 27 delay length. The value is averaged over 10 different random
seeds. Here we can see that although the echo state Q-function has more random pa-
rameters, overall the ESFQ algorithm performs better than the NFQ algorithm. In the
early episodes the delay-line NFQ shows some minor improvement relative to the ESFQ
but after 60 episodes the average performance of the NFQ stays the same and the ESFQ
performance increases. Furthermore, we can see that the ESFQ has higher standard de-
viation compared to delay-line NFQ.
36](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-41-320.jpg)
![CHAPTER 5. EXPERIMENTS AND RESULTS 5.5. WHY NOT LINEAR READOUT LAYERS
Figure 5.12: The readout weighs Wout for the echo state Q-function with 8 hidden units after
training for 19 episodes on the simulated mountain car plant with 2 steps action delay. The
X-Axis shows the episode index. The Y-Axis shows the readout weight values.
In Figure 5.13 we illustrate some of the successful trajectories that lead to the goal
area. The X-Axis indicates the car position and the Y-Axis indicate the car velocity. The
initial position for the car is in the range of [−0.75, 0.25] and the initial velocity is in the
range of [−0.5, 0.5]. All the 5 tests reached to goal area where the car stopped on top of
the hill with the position value equal or larger than 0.7. By looking at each trajectory
we see the policy first drives car backward uphill with a negative velocity and accelerates
with applied positive velocity until it reaches the goal area.
40](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-45-320.jpg)
![CHAPTER 5. EXPERIMENTS AND RESULTS 5.5. WHY NOT LINEAR READOUT LAYERS
Figure 5.16: The figure illustrates 6 trajectory samples of the simulated inverted pendulum
with 2 steps action delays. The trajectories are generated by the interval of 0.05s from training
the ESFQ algorithm with a ridge regression readout layer. From the top, reservoir states with
20 hidden, two the steps delayed action and the genuine action, the cart position, and the
pole angle. Each trajectory sample ended when the car position or the pole angle went to the
terminal area and then the next trajectory sample began. The cycle index on the X-Axis is
corresponding to the sample indexes in the batch dataset.
5.5.3 Very Large Reservoir
In the previous section, we observed that the ESFQ algorithm with a linear readout
layer would fail to learn complex dynamic task like an inverted pendulum. However,
there are arguments that an echo state network with a very large number of hidden units
proportional to its input units could capture the complex dynamics of a control system.
Therefore, we chose a reservoir with 400 hidden units which are 100 times larger than the
state vector of the inverted pendulum. The idea here was that because of randomness
in the reservoir weight matrices with a larger number of neurons the system would be
able to linearise the higher order nonlinearity in dynamic systems. But according to the
literature [29] it is not necessarily true that a reservoir with a larger number of neurons
could provide a better estimation of the target values. Furthermore, our experimental
results proved that even such a large reservoir would not be able to estimate desired target
for reinforcement learning algorithm. In Figure 5.17 we can see the learning curves for
the ESFQ algorithm with a nonlinear readout layer and a linear readout layer with a
very large number of hidden units in its reservoir. We simply can observe that the ESFQ
algorithm with a nonlinear readout layer promises a better performance in the learning of
the inverted pendulum containing delays in its executed action. In this experiment, the
activation function for hidden units of the reservoir was a logistic regression. We trained
our algorithm using hyper-parameter optimization tool SMAC, ridge regression for the
44](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-49-320.jpg)
![CHAPTER 6. CONCLUSION 6.2. FUTURE WORKS
confused the multilayer perceptron so then its performance decayed on the test phase
for the larger delay length. Furthermore, the delay-line Q-function performed relatively
worse on the tests phase for the tasks with an odd number of delay length. Contrary to
that, the ESFQ algorithm stayed stable during the test phase with various delay length.
Our algorithm stated stable given odd or even action delay length. In addition to that,
our method reached the goal area for the different initial position, above 98 percent, in the
test phase. The delay-line Neural Fitted-Q Iteration algorithm faced a performance drop
when we tried out different random seeds, in the range of [1, 10], for the same delay task.
In spite of that, the ESFQ algorithm showed higher performance although it contained
more random variables in its weight matrices. Furthermore, this randomness caused rel-
atively high variances in performances for both of the NFQ and ESFQ algorithm.
We need to consider that the readout layer for both algorithms was similar and the
difference was in keeping the history of the delayed states, reservoir or delay-line. Fur-
thermore, the echo state approach learned a proper short term memory length via costly
parameter tuning i.e. by training it over our reinforcement learning algorithm. But, the
delay-line approach had to know about the delay length in advance to adjust the history
length. It was possible to optimize the delay-line approach over delay parameter using
hyper-parameter tools but it would come with cost of equally high computation time.
Last but not least, we showed that classical linear readout layers for reservoirs would
not work for the task with complex dynamics. In this respect, we trained echo state
Q-Function with 2 different readout layers. The ridged regression performs well on the
mountain car benchmark but failed to achieve mentionable results of inverted pendulum
task. Therefore, We chose multilayer perceptron as final readout layer and trained it
with Rprop algorithm, fast version of Backpropagation. To support our argument for
a nonlinear readout layer, we also tried a reservoir with a relatively larger number of
neurons but it didn’t learn well on inverted pendulum benchmark. The idea there was,
given a very large reservoir and randomness of its weight matrices, it would be possible
to linearize any complex dynamics and learn them with linear readout layer. In fact, we
tried to expand the input state vector to higher dimensions which were linearly separable
using regression methods. But, estimating suitable targets with such large reservoirs was
not possible.
6.2 Future Works
The major focus of this work was to show that echo state networks are the feasible tool for
function approximation in reinforcement learning with delays. But it might be possible
to improve our approach. Currently, we are able to learn control task with the unknown
delay but to the cost of hyperparameter optimization over training reinforcement learn-
ing. It might be possible to train echo state networks in advance such that they learn
to predict the next state of the system given the applied action. Generally, recurrent
neural networks have shown great ability in predicting temporal sequences and they also
have been used for adaptive filtering. Therefore, if we could exploit this potential and
train our recurrent network to predict a noise/delay free successor state of the system
from its current state and action, we would be able to maintain the Markov property.
Such a state prediction could be employed as input for different types of reinforcement
47](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-52-320.jpg)
![BIBLIOGRAPHY
[1] E. Altman and P. Nain. Closed-loop control with delayed information, volume 20.
ACM, 1992.
[2] J. R. Bachrach and M. C. Mozer. Connectionist modeling and control of finite state
systems given partial state information. 1991.
[3] B. Bakker. Reinforcement learning with long short-term memory. In NIPS, pages
1475–1482, 2001.
[4] B. Bakker. Reinforcement learning by backpropagation through an lstm model/critic.
In Proceedings of the IEEE International Symposium on Approximate Dynamic Pro-
gramming and Reinforcement Learning (ADPRL), Honolulu, HI, pages 127–134.
Citeseer, 2007.
[5] J. Bander, I. White, et al. Markov decision processes with noise-corrupted and
delayed state observations. Journal of the Operational Research Society, 50(6):660–
668, 1999.
[6] A. Barto and R. Crites. Improving elevator performance using reinforcement learn-
ing. Advances in neural information processing systems, 8:1017–1023, 1996.
[7] J. Bergstra, D. Yamins, and D. Cox. Making a science of model search: Hyperparam-
eter optimization in hundreds of dimensions for vision architectures. In Proceedings
of The 30th International Conference on Machine Learning, pages 115–123, 2013.
[8] D. P. Bertsekas, D. P. Bertsekas, D. P. Bertsekas, and D. P. Bertsekas. Dynamic
programming and optimal control, volume 1. Athena Scientific Belmont, MA, 1995.
[9] J. A. Boyan and M. L. Littman. Packet routing in dynamically changing networks: A
reinforcement learning approach. Advances in neural information processing systems,
pages 671–671, 1994.
[10] D. Brooks and C. Leondes. Technical note-markov decision processes with state-
information lag. Operations Research, 20(4):904–907, 1972.
[11] K. A. Bush. An echo state model of non-markovian reinforcement learning. Colorado
State University, 2008.
49](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-54-320.jpg)
![BIBLIOGRAPHY BIBLIOGRAPHY
[12] S. P. Chatzis and Y. Demiris. Echo state gaussian process. Neural Networks, IEEE
Transactions on, 22(9):1435–1445, 2011.
[13] L. Chrisman. Reinforcement learning with perceptual aliasing: The perceptual dis-
tinctions approach. In AAAI, pages 183–188. Citeseer, 1992.
[14] K. Doya. Bifurcations in the learning of recurrent neural networks 3. learning
(RTRL), 3:17, 1992.
[15] N. en die der Geneeskunde and P. B. Bakker. The state of mind reinforcement
learning with recurrent neural networks.
[16] S. Engelbrecht and K. Katsikopoulos. Planning with delayed state information title2.
1999.
[17] D. Ernst, P. Geurts, and L. Wehenkel. Tree-based batch mode reinforcement learn-
ing. In Journal of Machine Learning Research, pages 503–556, 2005.
[18] R. A. Freeman and P. V. Kokotovic. Robust nonlinear control design: state-space
and Lyapunov techniques. Springer Science & Business Media, 2008.
[19] B. Hammer and J. J. Steil. Tutorial: Perspectives on learning with rnns. In Proc.
ESANN, pages 357–368, 2002.
[20] R. Hecht-Nielsen. Theory of the backpropagation neural network. In Neural Net-
works, 1989. IJCNN., International Joint Conference on, pages 593–605. IEEE,
1989.
[21] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation,
9(8):1735–1780, 1997.
[22] R. A. Howard. Dynamic programming. Management Science, 12(5):317–348, 1966.
[23] G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew. Extreme learning machine: a new learning
scheme of feedforward neural networks. In Neural Networks, 2004. Proceedings. 2004
IEEE International Joint Conference on, volume 2, pages 985–990. IEEE, 2004.
[24] F. Hutter, H. H. Hoos, and K. Leyton-Brown. Sequential model-based optimization
for general algorithm configuration (extended version). Technical report, Technical
Report TR-2010-10, University of British Columbia, Computer Science, 2010.
[25] F. Hutter, H. H. Hoos, and K. Leyton-Brown. Sequential model-based optimization
for general algorithm configuration. In Learning and Intelligent Optimization, pages
507–523. Springer, 2011.
[26] L.-J. i Lin. Reinforcement learning with hidden states. In From animals to ani-
mats 2: Proceedings of the second international conference on simulation of adaptive
behavior, volume 2, page 271. MIT Press, 1993.
[27] I. Ilies, H. Jaeger, O. Kosuchinas, M. Rincon, V. Sakenas, and N. Vaskevicius. Step-
ping forward through echoes of the past: forecasting with echo state networks.
Avaible: http://www. neural-forecastingcompetition. com/downloads/methods/27-
NN3 Herbert Jaeger report. pdf, 2007.
50](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-55-320.jpg)
![BIBLIOGRAPHY BIBLIOGRAPHY
[28] H. Jaeger. The ”echo state” approach to analysing and training recurrent neural
networks-with an erratum note. Bonn, Germany: German National Research Center
for Information Technology GMD Technical Report, 148:34, 2001.
[29] H. Jaeger. Short term memory in echo state networks. GMD-Forschungszentrum
Informationstechnik, 2001.
[30] H. Jaeger. Adaptive nonlinear system identification with echo state networks. In
Advances in neural information processing systems, pages 593–600, 2002.
[31] H. Jaeger. Reservoir riddles: Suggestions for echo state network research. In Neural
Networks, 2005. IJCNN’05. Proceedings. 2005 IEEE International Joint Conference
on, volume 3, pages 1460–1462. IEEE, 2005.
[32] H. Jaeger and H. Haas. Harnessing nonlinearity: Predicting chaotic systems and
saving energy in wireless communication. Science, 304(5667):78–80, 2004.
[33] H. Jaeger, M. Lukoˇseviˇcius, D. Popovici, and U. Siewert. Optimization and applica-
tions of echo state networks with leaky-integrator neurons. Neural Networks, 20(3):
335–352, 2007.
[34] K. V. Katsikopoulos and S. E. Engelbrecht. Markov decision processes with delays
and asynchronous cost collection. Automatic Control, IEEE Transactions on, 48(4):
568–574, 2003.
[35] H. Kwakernaak and R. Sivan. Linear optimal control systems, volume 1. Wiley-
interscience New York, 1972.
[36] S. Lange, T. Gabel, and M. Riedmiller. Batch reinforcement learning. In Reinforce-
ment Learning, pages 45–73. Springer, 2012.
[37] L.-J. Lin. Self-improving reactive agents based on reinforcement learning, planning
and teaching. Machine learning, 8(3-4):293–321, 1992.
[38] L.-J. Lin and T. M. Mitchell. Memory approaches to reinforcement learning in non-
Markovian domains. Citeseer, 1992.
[39] J. Loch and S. P. Singh. Using eligibility traces to find the best memoryless policy
in partially observable markov decision processes. In ICML, pages 323–331, 1998.
[40] M. Lukoˇseviˇcius. A practical guide to applying echo state networks. In Neural
Networks: Tricks of the Trade, pages 659–686. Springer, 2012.
[41] M. LukoˇsEviˇcIus and H. Jaeger. Reservoir computing approaches to recurrent neural
network training. Computer Science Review, 3(3):127–149, 2009.
[42] W. Maass, T. Natschl¨ager, and H. Markram. Real-time computing without stable
states: A new framework for neural computation based on perturbations. Neural
computation, 14(11):2531–2560, 2002.
[43] M. S. Mahmoud. Dynamic control of systems with variable state-delay. International
Journal of Robust and Nonlinear Control, 6(2):123–146, 1996.
51](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-56-320.jpg)
![BIBLIOGRAPHY BIBLIOGRAPHY
[44] J. Mockus and L. Mockus. Bayesian approach to global optimization and application
to multiobjective and constrained problems. Journal of Optimization Theory and
Applications, 70(1):157–172, 1991.
[45] R. M. Munos and A. W. Moore. Rates of convergence for variable resolution schemes
in optimal control. 2000.
[46] D. Ormoneit and P. Glynn. Kernel-based reinforcement learning in average-cost
problems. Automatic Control, IEEE Transactions on, 47(10):1624–1636, 2002.
[47] M. Oubbati, B. Kord, P. Koprinkova-Hristova, and G. Palm. Learning of embodied
interaction dynamics with recurrent neural networks: some exploratory experiments.
Journal of neural engineering, 11(2):026019, 2014.
[48] H. Palangi, L. Deng, and R. K. Ward. Learning input and recurrent weight matrices
in echo state networks. arXiv preprint arXiv:1311.2987, 2013.
[49] J. Peters and S. Schaal. Policy gradient methods for robotics. In Intelligent Robots
and Systems, 2006 IEEE/RSJ International Conference on, pages 2219–2225. IEEE,
2006.
[50] F. J. Pineda. Generalization of back-propagation to recurrent neural networks. Phys-
ical review letters, 59(19):2229, 1987.
[51] M. M. Polycarpou, P. Ioannou, et al. A robust adaptive nonlinear control design. In
American Control Conference, 1993, pages 1365–1369. IEEE, 1993.
[52] D. V. Prokhorov, D. C. Wunsch, et al. Adaptive critic designs. Neural Networks,
IEEE Transactions on, 8(5):997–1007, 1997.
[53] M. Riedmiller. Neural fitted q iteration–first experiences with a data efficient neural
reinforcement learning method. In Machine Learning: ECML 2005, pages 317–328.
Springer, 2005.
[54] M. Riedmiller. 10 steps and some tricks to set up neural reinforcement controllers.
In Neural Networks: Tricks of the Trade (2nd ed.), pages 735–757, 2012.
[55] M. Riedmiller and H. Braun. Rprop-a fast adaptive learning algorithm. In Proc. of
ISCIS VII), Universitat. Citeseer, 1992.
[56] M. Riedmiller, T. Gabel, R. Hafner, and S. Lange. Reinforcement learning for robot
soccer. Autonomous Robots, 27(1):55–73, 2009.
[57] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by
back-propagating errors. Cognitive modeling, 5:3, 1988.
[58] A. M. Sch¨afer. Reinforcement learning with recurrent neural networks. 2008.
[59] J. Schmidhuber. Reinforcement learning in markovian and non-markovian environ-
ments. 1991.
52](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-57-320.jpg)
![BIBLIOGRAPHY BIBLIOGRAPHY
[60] E. Schuitema, L. Bu¸soniu, R. Babuˇska, and P. Jonker. Control delay in reinforcement
learning for real-time dynamic systems: a memoryless approach. In Intelligent Robots
and Systems (IROS), 2010 IEEE/RSJ International Conference on, pages 3226–
3231. IEEE, 2010.
[61] B. Sinopoli, L. Schenato, M. Franceschetti, K. Poolla, M. Jordan, S. S. Sastry,
et al. Kalman filtering with intermittent observations. Automatic Control, IEEE
Transactions on, 49(9):1453–1464, 2004.
[62] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction, volume 1.
MIT press Cambridge, 1998.
[63] G. Tesauro. Practical issues in temporal difference learning. Springer, 1992.
[64] G. Tesauro. Reinforcement learning in autonomic computing: A manifesto and case
studies. Internet Computing, IEEE, 11(1):22–30, 2007.
[65] S. Timmer and M. Riedmiller. Safe q-learning on complete history spaces. In Machine
Learning: ECML 2007, pages 394–405. Springer, 2007.
[66] M. H. Tong, A. D. Bickett, E. M. Christiansen, and G. W. Cottrell. Learning
grammatical structure with echo state networks. Neural Networks, 20(3):424–432,
2007.
[67] C. F. Touzet. Neural reinforcement learning for behaviour synthesis. Robotics and
Autonomous Systems, 22(3):251–281, 1997.
[68] F. Triefenbach, A. Jalalvand, B. Schrauwen, and J.-P. Martens. Phoneme recogni-
tion with large hierarchical reservoirs. In Advances in neural information processing
systems, pages 2307–2315, 2010.
[69] A. Waibel. Modular construction of time-delay neural networks for speech recogni-
tion. Neural computation, 1(1):39–46, 1989.
[70] A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. J. Lang. Phoneme recog-
nition using time-delay neural networks. Acoustics, Speech and Signal Processing,
IEEE Transactions on, 37(3):328–339, 1989.
[71] T. J. Walsh, A. Nouri, L. Li, and M. L. Littman. Learning and planning in environ-
ments with delayed feedback. Autonomous Agents and Multi-Agent Systems, 18(1):
83–105, 2009.
[72] C. J. Watkins and P. Dayan. Q-learning. Machine learning, 8(3-4):279–292, 1992.
[73] P. J. Werbos. Backpropagation through time: what it does and how to do it. Pro-
ceedings of the IEEE, 78(10):1550–1560, 1990.
[74] R. J. Williams and D. Zipser. A learning algorithm for continually running fully
recurrent neural networks. Neural computation, 1(2):270–280, 1989.
[75] I. B. Yildiz, H. Jaeger, and S. J. Kiebel. Re-visiting the echo state property. Neural
networks, 35:1–9, 2012.
53](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-58-320.jpg)
![BIBLIOGRAPHY BIBLIOGRAPHY
[76] X.-M. Zhang, M. Wu, J.-H. She, and Y. He. Delay-dependent stabilization of linear
systems with time-varying state and input delays. Automatica, 41(8):1405–1412,
2005.
[77] K. Zhou, J. C. Doyle, K. Glover, et al. Robust and optimal control, volume 40.
Prentice hall New Jersey, 1996.
54](https://image.slidesharecdn.com/c37b7651-6b6f-484d-9303-add9314e7784-160522163558/85/esnq_control-59-320.jpg)
esnq_control
- 1. University Of Freiburg Department Of Computer Science Machine Learning Lab Master’s Thesis Echo State Fitted-Q Iteration Approach To Learn Delayed Control Systems Ramin Zohouri 16. December 2015 First Reviewer: Dr. Joschka Boedecker Second Reviewer: Prof. Dr. Moritz Diehl Supervisor: Thomas Lampe
- 3. Machine Learning Lab Department Of Computer Science Systems Control and Optimization Laboratory Department of Microsystem Engineering University Of Freiburg Author : Ramin Zohouri Master’s Degree Program in Computer Science Master’s Thesis Echo State Fitted-Q Iteration Approach To Learn Delayed Control Systems First Reviewer: Dr. Joschka Boedecker Second Reviewer: Prof. Dr. Moritz Diehl Supervisor: Thomas Lampe Submitted : 16.12.2015 II
- 4. ABSTRACT Stability and performance of dynamically controlled systems tend to decay by the pres- ence of delays in their applied actions or measured system states. The presence of such delays magnifies the difficulties of learning control systems, particularly when their dy- namic models are not known to us. In this work, we introduce an effective and efficient Q-learning algorithm, Echo State Fitted-Q Iteration (ESFQ) to learn delayed control systems. Our method employs reservoir computing to hold the history of the raw system states measurement and estimate a proper Q-value for the applied actions. To config- ure our algorithm and achieve the high performance we use hyper-parameter optimiza- tion tools. Experimental results, on different simulated benchmarks with various delay lengths, report improvement in the performance in comparison to the standard tapped delay-line algorithm. Furthermore, our results illustrate the benefits of a nonlinear read- out layer for the echo state Q-function on learning delayed control tasks with complex dynamics. 07.2015 - 12.2015 Ramin Zohouri III
- 5. ACKNOWLEDGEMENT I have taken efforts in this project. However, it would not have been possible without the kind support and help of many individuals and organizations. I would like to extend my sincere thanks to all of them. I would like to express my appreciation and special thanks to my advisers, Dr. Joschka Boedecker and Mr. Thomas Lampe, you have helped me a tremendous amount ! I would like to thank you for encouraging my research ideas and guiding me to grow as a researcher. Your advice on my research and the extended discussions we had in the past few months were priceless for me and I am truly grateful for everything; to you in the first place, but also the University of Freiburg for providing such a fruitful and motivating environment. I would like to express my gratitude towards Prof. Dr. Moritz Diehl for his kind co-operation and encouragement which helped me in the completion of this project. I would like also to thank my friends and members of the machine learning lab Manuel Blum, Jost Tobias Springenberg, Manuel Watter, and Jan W¨ulfing who also helped me a lot with giving me fresh ideas and discussing my problems when I needed it the most. Last but not least, I want to give my special thanks to Thorsten Engesser, Robin Tibor Schirrmeister, and Martin Goth who did the proofreading of my thesis and kindly helped me with translating the abstract of this report to The German language. IV
- 6. CONTENTS 1 Introduction 3 1.1 Dynamic Control Systems And Reinforcement Learning . . . . . . . . . . 3 1.2 Goals and Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2 Related Works 7 2.1 Non-Markovian Reinforcement Learning . . . . . . . . . . . . . . . . . . 7 2.1.1 Markov Decision Process With Delays And Asynchronous Cost Col- lection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.2 Learning And Planning In Environments With Delayed Feedback 8 2.1.3 Control Delay in Reinforcement Learning for Real-Time Dynamic Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2 Recurrent Neural Networks In Reinforcement Learning . . . . . . . . . . 9 3 Reinforcement Learning In Feedback Control 11 3.1 Learning In Feedback Control . . . . . . . . . . . . . . . . . . . . . . . . 11 3.2 Markov Decision Process . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.3 Q-Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.3.1 Batch Reinforcement Leaning . . . . . . . . . . . . . . . . . . . . 12 3.3.2 Neural Fitted Q-Iteration (NFQ) . . . . . . . . . . . . . . . . . . 13 3.3.3 Resilient Propagation (Rprop) . . . . . . . . . . . . . . . . . . . . 14 4 ESN For Function Approximation 16 4.1 Recurrent Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.2 A Brief Overview On Echo State Networks . . . . . . . . . . . . . . . . . 17 4.2.1 Learning In Echo State Networks . . . . . . . . . . . . . . . . . . 19 4.2.2 Spectral Radius and Echo State Property . . . . . . . . . . . . . . 19 4.3 Echo State Fitted-Q Iteration . . . . . . . . . . . . . . . . . . . . . . . . 20 4.4 Types Of Readout Layers For Echo State Q-Function . . . . . . . . . . . 21 4.4.1 Ridge Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4.4.2 Multilayer Perceptron Readout Layer . . . . . . . . . . . . . . . . 22 4.5 SMAC For Hyper Parameter Optimization . . . . . . . . . . . . . . . . . 23 1
- 7. CONTENTS CONTENTS 5 Experiments And Results 25 5.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 5.2 Simulation Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 5.2.1 Mountain Car . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 5.2.2 Inverted Pendulum . . . . . . . . . . . . . . . . . . . . . . . . . . 26 5.3 Experimental Procedure And Parametrisation . . . . . . . . . . . . . . . 27 5.3.1 Echo State Q-Function . . . . . . . . . . . . . . . . . . . . . . . . 28 5.3.2 Hyper-parameter Optimization . . . . . . . . . . . . . . . . . . . 30 5.4 Comparisons To Delay-Line NFQ Algorithm . . . . . . . . . . . . . . . . 34 5.5 Why Not Linear Readout Layers . . . . . . . . . . . . . . . . . . . . . . 38 5.5.1 Ridge Regression On Mountain Car . . . . . . . . . . . . . . . . . 38 5.5.2 Ridge Regression On Inverted Pendulum . . . . . . . . . . . . . . 41 5.5.3 Very Large Reservoir . . . . . . . . . . . . . . . . . . . . . . . . . 44 6 Conclusion 46 6.1 Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 6.2 Future Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 2
- 8. CHAPTER 1 INTRODUCTION 1.1 Dynamic Control Systems And Reinforcement Learning There are an increasing demands for building automatic controllers for systems with complex dynamics such as aircraft, magnetic levitation trains, and chemical plants, but also in objects of daily life such as air conditioners and computer drives [51, 18]. These systems are characterized as having an output or feedback, that can be measured, inputs that can be manipulated, and internal dynamics. Feedback control involves computing and applying suitable control inputs to minimize the differences between the observed and desired behaviour of a dynamic system. Feedback control design approaches include classical design methods for linear systems, multivariate control, nonlinear control, op- timal control, robust control, H-infinity control, adaptive control, and others [77, 8, 35]. Designing a controller has the advantage of being general and applicable to various sys- tems. However, the design of good controllers is a tedious and demanding job, due to unknown dynamics and underactuation, modelling errors, and various sorts of dis- turbances, uncertainties, and noise. Furthermore, since there is no learning involved, for a new control task with relatively similar dynamics a new system model has to be created. In contrast to the classical design process, reinforcement learning is geared towards learning appropriate closed-loop controllers by simply interacting with the process and incrementally improving control behaviour. Reinforcement learning is learning to act by trial and error. In this paradigm, an agent can perceive its state and perform actions. After each action, a numerical reward is given. The goal of the agent is to maximize the total reward it receives over time. Reinforcement learning has been successful in appli- cations as diverse as autonomous helicopter flight, robot-legged locomotion, cell-phone network routing, marketing strategy selection, factory control, and efficient web page indexing. Despite these successes, there are some challenges and problems which make learning a control task difficult. In an online control system, the data is sent and received by network nodes of different types and manufacturers. Sensor nodes measure process values and transmit these over the communication network. Actuator nodes receive new values for the process inputs 3
- 9. CHAPTER 1. INTRODUCTION 1.1. DYNAMIC CONTROL SYSTEMS AND REINFORCEMENT LEARNING over the communication network and apply these to the process input. Controller nodes read process values from sensor nodes. A control algorithm calculates the control signals and sends them to the actuator nodes. Communication networks inevitably introduce delays, both due to limited bandwidth, but also due to overhead in the communicating nodes and the network. In many systems, there are various delay types. From the control perspective, a control system with varied delay types will no longer be time-invariant. Therefore, the standard optimal control theory can not be used to analyze and design these systems. We can categorize different types of control delays based on their place of occurrence. • Communication delay between the sensor and the controller. • Computational delay in the controller. • Communication delay between the controller and the actuator. The effect of time delay on the stability and performance of control systems has drawn the attention of many investigators in different engineering disciplines, including optimal control system [1, 45, 10, 43, 76], reinforcement learning [5, 39]. In general, the time delay in active control systems causes an unsynchronized application of the control forces, and this unsynchronized control not only degrades the system performance but also causes instability in the system response [61]. Dynamic computational models with delays re- quire the ability to store and access the time history of their inputs and outputs. And there are various difficulties in learning and controlling such systems. The first problem is, there is no explicit teacher signal that indicates the correct output at each time step. In addition, the temporal delay of the reward signal implies that the learning system must assign temporal reward/cost to each of the states and actions that resulted in the final outcome of the sequence. To overcome this restriction, several, more general approaches are considered that re- tain state information over time. We refer to these as stored-state methods. The simplest of these approaches is one which augments immediate sensory inputs with a delay line to achieve a crude form of short term memory [37]. This approach has been successful in certain speech recognition tasks [69]. The most common dynamic neural architecture is the time delayed neural networks [70]. The time delayed neural networks couples delay lines with a nonlinear static architecture where all the parameters (weights) are adapted with the Backpropagation algorithm. Another alternative is called the method of pre- dictive distinctions [2, 13, 59]. Following this approach, the system learns a predictive model of the sensory inputs i.e. environmental observables, and then uses the internal state of this model to drive action selection. A third approach uses a recurrent neural network [50, 21, 73, 57, 28, 19] in combina- tion with existing reinforcement learning methods to learn a recurrent state-dependent control policy directly [26]. In the sense of keeping the history of the system states the recurrent neural networks naturally are powerful tools that can be employed to learn delayed control systems. Formerly, training of recurrent neural networks was performed using Backpropagation through time [73], which actually means unfolding the network in time and constructing a much bigger network, then performing Backpropagation on this new network. However, besides the fact that this process is very slow, it does not always 4
- 10. CHAPTER 1. INTRODUCTION 1.2. GOALS AND CONTRIBUTIONS guarantee a good solution because of the fading gradient issue. 1.2 Goals and Contributions As we described in the previous section, in this work we aimed to learn the behaviour of control systems which have unknown delays without knowing their underlying dy- namics. For the purpose of this task Q-learning, a particular model-free reinforcement learning algorithm would be a feasible solution. Basically, The Q-learning algorithm uses function approximation to choose the best control action given the current system state and minimizes the overall cost of the reaching the goal area. The presence of delay in control systems violates the Markov property which is a fundamental method for formu- lating a successful reinforcement learning algorithm. The Markov property simply says that a conditional probability distribution of future states of the process only depends on its present state. Hence, for a delayed task the proper function approximation i.e. Q-function needs to be able to preserve the Markov property by holding the history of a system state. Furthermore, the learning algorithm for a control system, in general, should be able to deal with problems of a long learning time and convergence. Last but not least, the learning algorithm we described above will have various parameters which are interdependent and must be selected carefully to boost the learning performance. To achieve the objectives we mentioned above, we designed and implemented the Echo State Fitted-Q Iteration (ESFQ) algorithm. It is a type of offline batch reinforcement learning algorithm. The ESFQ algorithm has several advantages which make it a suit- able tool to learn control tasks with delays. First, it is a batch reinforcement learning algorithm, which is able to use all the previously seen control trajectories which acceler- ate the learning procedure. Second it employs echo state networks (ESN) as a function approximation to estimate delayed targets. The ESN is a specific type of recurrent neu- ral networks which its short-term capacity enables it to hold the history of the system states. Therefore, it is able to preserve the Markov property for a reinforcement learning algorithm. The third contribution of our method is the ability to have different readout layers for training the Echo State Q-function. This ability helps function approximation to learn complex dynamic systems. Our algorithm benefits from its unconventional way of preserving the history of system states and the ability to train different readout layers which make it suitable for a particular control task. However, it has several parameters which need to be carefully tuned in order to maximize its performance. To achieve this goal we employed hyper-parameter optimization tools to automatically determine good choices and take out a manual step of the configuration. 1.3 Outline In the following, we first cover the major related works to our method and briefly compare and contrast each of them with our method in chapter 2. Then we describe fundamental features of our algorithms, reinforcement learning with feedback control in chapter 3, and echo state networks for function approximation in chapter 4. After that, we proceed by presenting the results of our method on different standard benchmarks in chapter 5. There also we compare our results to the standard tapped delay-line reinforcement 5
- 11. CHAPTER 1. INTRODUCTION 1.3. OUTLINE learning algorithm. At the end, in chapter 6 we summarize this work by mentioning our major contributions and addressing the future works that could be done. 6
- 12. CHAPTER 2 RELATED WORKS 2.1 Non-Markovian Reinforcement Learning Markov decision process(MDP) is a popular framework for training control problems[22] in reinforcement learning. In summary, given a certain state of a system, the agent selects an action that brings the system to a new state and induces a cost, the new state is observed and the cost is collected, then the decision maker selects a new action, and so on. However, the basic MDP framework as it is defined in [34], makes a number of restrictive assumptions that may limit its applicability: • the system’s current state is always available to the agent. • the agent’s actions always take effect immediately • the cost induced by an action is always collected without delay The Markov property is usually assumed in reinforcement learning and, therefore, it vanishes when any of the described conditions is violated. It follows that if an agent must learn to control a system, there will be periods of time when the internal representation of system states will be inadequate. Therefore, the decision task will be non-Markovian which challenges the performance of reinforcement learning algorithms. The presence of delay in measured states or applied actions causes the violation of the Markov property. In the following, we viewed three major state-of-the-art approaches for reinforcement learning with delay. 2.1.1 Markov Decision Process With Delays And Asynchronous Cost Collection In the first approach, the state space of the learning agent is augmented with the actions that were executed during the delay interval. In this work [16, 34] authors showed how (discrete-time, total-expected-cost, infinite horizon) Markov decision process with obser- vation, action and delayed cost are reduced to a Markov decision process without delays. They drew connections amongst the three delay types and demonstrated that the cost structure of the process without delays is the same as that of the original process with 7
- 13. CHAPTER 2. RELATED WORKS 2.1. NON-MARKOVIAN REINFORCEMENT LEARNING constant or random delays. They considered an embedded process that behaves similarly to a process with constant delays. However, their results are based on the intuition of asynchronous cost collection. That means the costs may be induced and collected at different decision stages and policies can still be compared properly as long as costs are discounted accordingly. While this approach works well, the state space increase can cause a large increase in learning time and memory requirements. 2.1.2 Learning And Planning In Environments With Delayed Feedback The second approach tried to learn a model of the underlying non-delayed process and used this model to base control actions on the future state after the delay, predicted by the model[71]. In this work, the authors evaluated algorithms for environments with constant observation and reward delay. They covered three different approaches for deal- ing with the constant delay Markov decision process(CDMDP). First, ”the wait agent”, which waits for k steps, until the current observation comes through, and then acts us- ing the optimal action in the non-delayed MDPs. Unfortunately, policies derived from this strategy will not, in general, provide satisfactory solutions to the CDMDP planning problem. Instead, the agent’s resulting policy will likely be suboptimal as it is essen- tially losing potential reward on every wait step. Their second solution was a memoryless policy, which treated CDMSP as MDP and used memoryless policy for the non-delayed MDP. Their third solution was the traditional augmented approach, which involves ex- plicitly constructing an MDP equivalent to original CDMDP in a larger state space. They augmented each state and applied k previous actions to form a new state representation. Authors formed new transition probability and reward function but such a solution adds the extra burden of acquiring the model of the system while the added computational complexity may actually increase the delay itself. 2.1.3 Control Delay in Reinforcement Learning for Real-Time Dynamic Systems The third approach for dealing with delays in MDP has been introduced in [60] which is an improvement on the older approaches. The authors introduced two new memoryless solutions and the most important one was an online algorithm named dSARSA(λ). Such methods base the next control action only on the most recent observation. The downside of memoryless approaches is that they are likely to learn a suboptimal policy because they have no means of predicting the state in which the control action will take effect. Furthermore, SARSA(λ) does not take the presence of the delay into account in its learning updates. While their complexity remains comparable to that of SARSA(λ), they exploited the knowledge about the length of the delay to improve their performance. Then, the authors presented an extension to these algorithms which was applicable where the delay length is not an integer multiple of the time step. 8
- 14. CHAPTER 2. RELATED WORKS 2.2. RECURRENT NEURAL NETWORKS IN REINFORCEMENT LEARNING 2.2 Recurrent Neural Networks In Reinforcement Learn- ing There are various attempts to combine the dynamic programming approaches with recur- rent neural networks(RNN) to tackle reinforcement learning problems. They formulate their solution in adaptive critic designs(ACDs) [52]. The ACDs have their root in dy- namic programming and are suitable for learning in noisy, nonlinear, and non-stationary environments. The ACDs in their application first define what to approximate through the critic, and then how to adapt the actor in response to the information coming from the critic. Schmidhuber took one of the earliest approaches [59] toward this direction. He modeled the dynamics and control of a system with two separate networks and trained them both in parallel or sequential. However, the proposed fully recurrent networks struggled in their flexibility and adaptation ability. One of the major difficulties he faced in his work was the disability of Backpropagation through time and the ACDs to deal with the long time lag between applied action and its execution. In another work, Bakker combined the long short term memory [4, 3, 15] with the actor-critic method to learn par- tially observable reinforcement learning problems. He utilized long short term memory strength in learning long-term temporal dependencies to infer states in partially observ- able tasks. He developed an integrated method, which learns the system’s underlying dynamics and the optimal policy at the same time. Compared to our method which is a data efficient batch learning and designed to learn non-Markovian problems, Bakker’s approach had problems in dealing with high dimensionality, partial observability, contin- uous state and action space, and a limited amount of training data. In a recent approach by Sch¨afer [58], he applied recurrent neural reinforcement learning approaches to identify and control a high-dimensional dynamic system with continuous state and action spaces in partially unknown environments like a gas turbine. He introduced a hybrid recurrent neural network approach that combines system identification and determination of an optimal policy in one network. Furthermore, in contrast to our reinforcement learning methods, it determines the optimal policy directly without making use of a value func- tion. His approach is model-based and by constructing the model of the system it is able to learn high dimensional and partially observable reinforcement learning problems with continuous state and action spaces in a data efficient manner. As it shown by Lin and Mitchell [38] recurrent neural networks are robust tools for functional approximation in partially observable systems. In their work authors defined three different architectures, recurrent-model, recurrent-Q, and window-Q architectures. The first one learns an action model for the history features. The second approach learns a Q-function approximation using indirectly observable training examples. And the third method learns the Q-value approximation by taking a known number of state-action pairs, window size, into its memory. As they have shown, these architectures are all capable of learning some non-Markovian problems, but they have their own advantages and disadvantages. Later on in the work by K. Bush [11], the author used echo state networks(ESNs) to address the problem of non-Markovian reinforcement learning and showed how they could successfully learn some standard benchmarks problems. The au- thor experimentally validated the positive performance and dynamic attributes of the echo state networks on modeling a system in the non-Markovian reinforcement learning domains. The major difference between this work and our approach is, we tried a non- linear readout layer to learn complex dynamic systems, but Bush introduced a memory 9
- 15. CHAPTER 2. RELATED WORKS 2.2. RECURRENT NEURAL NETWORKS IN REINFORCEMENT LEARNING consolidation method using the mixture of expert (MoE) and tested it for stationary dynamic systems. This framework preserves the ability to train the readout layers via linear regression. The author showed that echo state networks exhibit low-mobility learn- ing through trajectory-based features. Later on, echo state networks are also successfully used and tested as Q-function approximation in work done by Oubbati et.al. [47]. Their work sees the control task as a result of the interaction between brains, bodies, and environments. There, the authors utilized reservoir computing as an efficient tool to understand how the system behaviour emerges from an interaction. Inspired by imita- tion learning designs, they presented reservoir computing models, to extract the essential components of the system dynamics which are the result of the agent-environment in- teraction. They validated their learning architectures by experimenting with a mobile robot in a collision avoidance scenario. Compared to this work, we trained our policy offline and with non-linearity on readout layer and applied discrete control actions. But they first did a short pre-training and then used online linear approach to train echo state networks and applied a continuous action to control an autonomous driving robot in order to follow a line. 10
- 16. CHAPTER 3 REINFORCEMENT LEARNING IN FEEDBACK CONTROL 3.1 Learning In Feedback Control The classical feedback control loop basically describes a control mechanism that uses information from measurements of a process to control it. In each time interval, the process communicates a system state, vector of measured dynamic variables, to the con- troller that applies the control commands. In feedback control, the control variables are measured and compared to the target values. Therefore, the feedback control mecha- nism manipulates the input variables of the system in a continuous loop to minimize the differences between estimation and desired targets. This allows for the formulation of a broad range of challenging control applications. The direct approach to have a feedback controller is by designing a model of the system. However for a nonlinear control sys- tem, the task of model identification becomes tedious and complicated. An alternative approach is to learn to control the feedback control systems. Which means, we need an intelligent controller component that learns to control a subjected dynamic process using experiences made from an interaction. Reinforcement learning (RL) is a type of machine learning method which tries to learn an appropriate closed-loop controller by simply interacting with the process and incre- mentally improving the control behaviour. The goal of reinforcement learning algorithms is to maximize a numerical reward signal by discovering which control commands i.e. actions yield the most reward. Using reinforcement learning algorithms, a controller can be learned with only a small amount of prior knowledge of the process. Reinforcement learning aims at learning control policies for a system in situations where the training information is basically provided in terms of judging success or failure of the observed system behaviour [62]. Because this is a very general scenario, a wide range of different application areas can be addressed. Successful applications are known from such differ- ent areas as game playing [9], dispatching and scheduling [6], robot control [49, 56], and autonomic computing [64]. 11
- 17. CHAPTER 3. REINFORCEMENT LEARNING IN FEEDBACK CONTROL 3.2. MARKOV DECISION PROCESS 3.2 Markov Decision Process The type of control problems we are trying to learn in this work are discrete time control problems and can be formulated as a Markov decision process(MDP) [62]. An MDP has four components: a set S of states, a set A of actions, a stochastic transition probability function p(s, a, s ) describing system behaviour, and an immediate reward or cost function c : S × A → R. The state of the system at time t, characterizes the current situation of the agent in the world, denoted by s(t). The chosen action by agent at time step t is denoted by a(t). The immediate reward or cost is the consequence of the taken action and function of state and action. Since the rewards for the taken action can be formulated as cost, the goal of the control agent would be to find an optimal policy π∗ : S → A that minimizes the cumulated cost for all states. Basically, in reinforcement learning we try to choose actions over time to minimize/maximize the expected value of the total cost/reward: E[R(s0) + R(s1) + R(s2) + ...] 3.3 Q-Learning There are various reinforcement approaches that can be formulated based on the MDP [62] e.g. value iteration and policy iteration, where the transition model and the reward function of the control task are known. However, in many real-world problems the state transition probabilities and the reward functions are not given explicitly. But, only a set of states S and a set of actions A are known and we have to learn the dynamic system behaviours by interacting with it. Methods of temporal differences were invented to perform learning and optimization in exactly these circumstances. There are two principal flavors of temporal difference methods. First, an actor-critic scheme [63, 62], which parallels the policy iteration methods, and has been suggested as being implemented in biological reinforcement learning. Second a method called Q-learning [72], which parallels the value iteration methods. In this work, we consider using the Q-leaning as our principal reinforcement learning algorithms. The basic idea in Q-learning is to iteratively learn the value function, Q-function, that maps state-action pairs to expected optimal path costs. The update rule in Q-learning algorithm is given by: Qk+1(s, a) := (1 − α)Q(s, a) + α(r(s, a) + γ min a Qk(s , a )) where s denotes the system state where the transition starts, a is the action that is applied, and s is the successor system state. The learning rate α has to be decreased in the course of learning in order to fulfill the conditions of the stochastic approximation and the discounting factor denoted by γ [62]. It can be shown that under mild assumptions Q-learning converges for finite state and action spaces, as long as the Q-value for every state-action pair is updated infinitely often. Then, in the limit, the optimal Q-function is reached. 3.3.1 Batch Reinforcement Leaning As formulated above the standard Q-learning protocol considers an agent operating in discrete time. At each time point t it observes the environment state st, takes an action 12
- 18. CHAPTER 3. REINFORCEMENT LEARNING IN FEEDBACK CONTROL 3.3. Q-LEARNING at, and receives feedbacks from the environment including next state st+1 and the instan- taneous reward rt. The sole information that we assume available to learn the problem is the one obtained from the observation of a certain number of one-step system transitions (from t to t + 1). The agent interacts with the control system in the environment and gathers state transitions in a set of four-tuples (st, at, rt, st+1). Except for very special conditions, it is not possible to exactly determine an optimal control policy from a finite set of transition samples. In the literature there is a method called batch mode reinforce- ment learning [17, 46, 36], which aims at computing an approximation of such optimal policy π∗ , from a set of four-tuples: D = {(sl t, al t, rl t, sl t+1), l = 1 · · · , #D} This set could be generated by gathering samples corresponding to one single trajec- tory (or episode) as well as by considering several independently generated trajectories or multi-step episodes. In the work by Lange et al. [36] authors covered various batch-mode reinforcement learning algorithms. Among them, we chose the growing batch method for the purpose of training our learning algorithm. Training algorithms with growing batch have two major benefits. First, from the interaction perspective, it is very similar to the ’pure’ online approach. Second, from the learning point of view, it is similar to an offline approach that all the trajectory samples are used for training the algorithm. The main idea in growing batch is to alternate between phases of exploration, where a set of training examples is grown by interacting with the system, and phases of learning, where the whole batch of observations is used. The distribution of the state transitions in the provided batch must resemble the ’true’ transition probabilities of the system in order to allow the derivation of good policies. In practice, exploration cultivates the quality of learned policies by providing more variety in the distribution of the trajectory samples. Furthermore, it is often necessary to have a rough idea of a good policy in order to explore interesting regions that are not in the direct vicinity of the starting states. If ’important’ states i.e. states close to the goal state are not covered by any of the trajectory samples, then it is obviously not possible to learn a good policy from the batch data. This happens because the system would not know which series of actions lead the to the goal area. 3.3.2 Neural Fitted Q-Iteration (NFQ) Model-free learning methods like Q-learning are appealing from a conceptual point of view and have been very successful when applied to problems with small, discrete state spaces. But when it comes to applying them to the real world systems with larger and probably continuous state spaces these algorithms facing some limiting are factors. For the relatively small or finite state and action space, the Q-function can be represented in tabular form and it is straightforward to approximate. However, when dealing with continuous or very large discrete state and/or action spaces, the Q-function cannot be represented by a table with one entry for each state-action pair. In this respect, there are three problem can be identified : • the ’exploration overhead’, causing slow learning in practice • inefficiencies due to the stochastic approximation • stability issues when using function approximation 13
- 19. CHAPTER 3. REINFORCEMENT LEARNING IN FEEDBACK CONTROL 3.3. Q-LEARNING A common factor in modern batch reinforcement learning algorithms is that these algorithms typically address all three issues and offer specific solutions to each of them. The Fitted-Q iteration algorithm[17] is an efficient batch mode reinforcement learning al- gorithm that can learn from a sufficiently rich trajectory generated. In this algorithm, the Q-function approximation is done on an infinite or finite horizon optimal control problem with discounted rewards. In each step, this algorithm uses batch data together with the Q-values computed at the previous step to determine a new training set. Then it applies a regression method on the training data to compute the next Q-value of the sequence. Among several approaches to approximate Q-function, neural networks are considered suitable tools [23, 67] because they provide a nonlinear mapping from input to output data. The neural Q-function needs to define an error function which aims to measure the difference between the target Q-value and the estimated Q-value. For example, a squared error measure like error = (Q(s, a) − (c(s, a) + γ min a Q(s , a )))2 . To train the neural Q-function and minimize the estimation error common gradient descent learning rules like Backpropagation [20] can be applied. In general, fitted-Q iteration algorithms are trained online which requires thousands of samples and long training time to learn a control task. Riedmiller [53] proposes an alternative approach, neural fitted-Q iteration (NFQ), which performs an offline update step considering the entire set of transitions. The standard NFQ benefits from the growing batch method to collect the trajectory samples for training a multilayer perceptron Q-function. It has been proven [54] that 2-layer of neural networks has sufficient approximation capacity to generalize well for a closed loop control. The pseudo code for the NFQ algorithm shown Algorithm 1: Algorithm 1 Main loop of NFQ . k counts the number of iterations, kmax denotes the maxi- mum number of iterations. init MLP() returns a multilayer perceptron with randomly initialized weights. Backpropagation training ( P) takes pattern set P and returns a mulitlayer perceptron that has been trained on P using backpropagation as a supervised training method. procedure NFQ Main() input: a set of transition samples D; output:Q value function QN ; k = 0; init MLP() → Q0; while k < kmax do generate pattern set P = {(inputl , targetl ), l = 1, ..., #D} where: inputl = sl , al targetl = c(sl , al ) + γmin a Qk(s l , a l) Backpropagation training(P) → Qk+1 k := k + 1 3.3.3 Resilient Propagation (Rprop) The original NFQ uses the Backpropagation [20] algorithm combined with an optimiza- tion method i.e. gradient decent for training its multilayer perceptron neural network. 14
- 20. CHAPTER 3. REINFORCEMENT LEARNING IN FEEDBACK CONTROL 3.3. Q-LEARNING As the equation below shows, the loss function computes the gradient with respect to all the weights in the network. Then updates of the weight values are performed with respect to the computed gradient: wij(t + 1) = wij(t) − ∂E ∂wij (t) The partial derivative of the error function E with respect to the neural network weights is computed based on the chain rule. The learning rate parameter affects the convergence of the algorithm. If it’s too small, a system will take so many steps to converge, and with a large learning rate, the system will oscillate around local minima and fail to fall into the desired value range. Regular Backpropagation is a slow and inefficient process. In order to accelerate this supervised learning process, it is possible to use advanced techniques like Rprop [55], which has a more reliable and faster convergence than the regular gradient descent method. The pseudocode in Algorithm 2 shows the core part of the adaptation and learning the rule for Rprop. Using Rprop to train the NFQ algorithm also reduces its overall complexity by removing the necessity of parameter tuning for the neural Q-function. Rprop stands for Resilient Propagation, and it is an efficient learning scheme, that in every iteration updates network’s weights based on the changes in sign of the error function E. Here each network weight is updated individually by ∆ij based on its local gradient information. So if the last weight update was so large that the algorithm jumped over the local minima, then it needs to change it direction accordingly. In this adaptive method, the new value for ∆ij is computed by multiplying it with η− or η+ for the negative or positive sign of the E. The values for 0 < η− < 1 < η+ are parameters that can be set to some constant. Using the update value ∆ij, Rprop algorithm increases or decreases individual weight values given the sign of the derivative. Algorithm 2 The core part of the Rprop algorithm. The minimum (maximum) operator is supposed to deliver the minimum (maximum) of two numbers; the sign operator returns +1 if the argument is positive, -1, if the argument is negative, and 0 otherwise.∆ij determines the size of the individual update value for each weight. For all weights and biases{ if ( ∂E ∂wij (t − 1) ∗ ∂E ∂wij (t) > 0) then ∆ij(t) = minimum(∆ij(t − 1) ∗ η+ , ∆max) ∆wij(t) = −sign( ∂E ∂wij )∆ij(t) wij(t + 1) = wij(t) + ∆wij(t) else if ( ∂E ∂wij (t − 1) ∗ ∂E ∂wij (t) < 0) then ∆ij(t) = maximum (∆ij(t − 1) ∗ η− , ∆min) wij(t + 1) = wij(t) − ∆wij(t − 1) ∂E ∂wij (t) = 0 else if ( ∂E ∂wij (t − 1) ∗ ∂E ∂wij (t) = 0) then ∆wij(t) = −sign( ∂E ∂∆ij )∆ij(t) wij(t + 1) = wij(t) + ∆wij(t) 15
- 21. CHAPTER 4 ECHO STATE NETWORKS FOR FUNCTION APPROXIMATION 4.1 Recurrent Neural Networks Our goal in this work is to learn a feedback control system with delays without knowing its dynamic model. As we described in the previous chapter, we chose Neural Fitted-Q Iteration (NFQ) algorithm, a specific Q-learning algorithm, to learn our model-free task. In its core, the NFQ algorithm uses multi-later perceptrons as a function approximator to learn a model of the system dynamics and estimate the proper Q-value for each ac- tion. In the control task with delays the Markov property will be violated, the ordinary multilayer perceptron will fail to approximate the desired targets. Therefore, we need a Q-function that is capable of holding the history of the system states in its memory for robust approximation and preserving the Markov property. One possible option is to use recurrent neural networks(RNNs) for the function approximation. Recurrent neural networks (RNNs) have a structure similar to biological neural net- works and they are able to hold the history information of the dynamic system. A recurrent neural network contains (at least) one cycle path of synaptic connections. This feature makes RNNs an excellent tool for function approximation in delayed control sys- tems. Mathematically recurrent neural networks implement and approximate dynamic systems and they are applied to the variety of tasks, for example, system identification and inverse system identification, pattern classification, stochastic sequence modeling [58, 47, 30]. However, it is not an easy straightforward task to train RNNs. Gener- ally speaking, for training RNNs an extended version of the Backpropagation algorithm, Backpropagation through time has been used [73, 74], but only with partial successes. One of the conceptual limitations of the Backpropagation methods for the recurrent neu- ral networks is that bifurcations can make the training non-converging [14]. Even when they do converge, this convergence is slow, computationally expensive, and can lead to poor local minima. In more recent attempt to train recurrent neural networks the reservoir computing approach [41] was introduced. These networks are in fact dynamic systems driven by the input signal, or from another point of view they are nonlinear filters of the input signal. 16
- 22. CHAPTER 4. ESN FOR FUNCTION APPROXIMATION 4.2. A BRIEF OVERVIEW ON ECHO STATE NETWORKS The idea of reservoir computing has been discovered and investigated independently under the name of the echo state networks (ESNs) [28, 28] in machine learning, and liquid state machines (LSMs) [42] in computational neuroscience. The work on the liquid state machines is rooted in the biological setting of continuous time, spiking networks while the ideas of the echo state networks were first conceived in the framework of discrete- time, non-spiking networks in engineering applications. It was shown that the reservoir computing approaches often work well enough even without full adaptation of all the network weights. The reservoir computing methods usually use a supervised learning scheme for training. In this work, we employ echo state networks as a Q-function for their simplicity in training and capability in approximating the targets in the presence of delay in dynamic systems [29]. Perhaps surprisingly, this approach yielded an excellent performance in many benchmark tasks, e.g. [27, 32, 33, 68]. 4.2 A Brief Overview On Echo State Networks As we mentioned in the previous section, we employed discrete time echo state networks for the function approximation. The echo state network has M input units, N hidden units(reservoir neurons), and L output units. The data at each time point t, fed to the network through the input units in the form of a vector U(n) = [u1(t), u2(t), · · · , uM (t)]T . The internal values for the reservoir are declared by X(n) = [x1(t), x2(t), · · · , xN (t)]T , and the output of the network in the time point t is showed by Y(t) = [y1(t), y2(t), · · · , yL(t)]T . The Figure 4.1 depicts the three-layered topology of the echo state network, the input layer, the hidden units or reservoir, the and readout layer. There are four types of weight matrices which connect different echo state network layers together. Three of these weight matrices are initialized once and will stay constant during the training time. These constant weights are described in the following. The input weight matrix Win with N × M dimensions which connects the input data to the hidden units. The reservoir weight matrix W with N × N dimensions which connects the hidden units to each other and also indicates the recurrent connections. The feedback weight matrix Wfb with N × L dimensions which connects the outputs back to the reservoir. The initial values for these constant weight matrices have to be chosen carefully to increase the performance of the echo state networks. There are various initialization and hyper- parameter optimization techniques that can be used to initialize the Wfb , Win , and W matrices more sophisticatedly according to the dynamic task. We will cover some of the important methods along this chapter to initialize and optimize these weight values. In the echo state networks only the output weights Wout with L × (M + N) dimensions will be trained. They map the reservoir states to the estimated targets. Later in this chapter, we will describe various methods for training the weights Wout . The update rule for changing the state of the reservoir is: X(t + 1) = f(Win U(t + 1) + WX(t) + Wfb Y(t)) X(t + 1) = (1 − α)X(t) + αX(t + 1) Here α is the leaky integrator factor of each hidden unit. As it is shown in the equation above echo state networks (ESNs) are composed of simple additive units with a nonlinear activation function f. And the reservoir with a leaky integrator unit type has individual state dynamics, which can be exploited in various ways to accommodate the network to store temporal characteristics of a dynamic system. Leaking rate α is the amount of 17
- 23. CHAPTER 4. ESN FOR FUNCTION APPROXIMATION 4.2. A BRIEF OVERVIEW ON ECHO STATE NETWORKS the excitation (signal) that a neuron discards, basically it implements the concept of the leakage. This has an effect of smoothing the network dynamics, yielding an increased modelling capacity of the network, for example in dealing with a low frequency sine wave. The activation function f = (f1, f2, · · · , ft) for each hidden unit in the reservoir is usually a sigmoid function e.g. logistic activation f(x) = 1 1+e−x or hyperbolic tangent f(x) = ex−e−x ex+e−x . It also can be a linear rectifier unit: f(x) = x if x > 0 0 otherwise Figure 4.1: The topological illustration of the echo state network. Its structure consists of three major parts. First an input layer containing input weight matrix Win . Second the middle layer containning the reservoir state matrix and its weight matrix W. And third the output layer which contains readout weight matrix Wout and readout value Ytarget . The initial state of the reservoir is usually set to random values, in most case X(0) = 0, and this introduces an unnatural starting state which is not normally visited once the network has ”warmed up” to the task. Therefore, when we input the sequence of long trajectory samples to the reservoir we exclude few computed states of each trajectory from the reservoir state. The number of discarded states depends on the memory of the reservoir and usually it is an integer number equal or less that 10% of the input data. The output values for the echo state network are computed according to: Y(t + 1) = fout (Wout (U(t + 1), X(t + 1))) Here fout = (fout 1 , fout 2 · · · , fout t ) is usually a linear function. Depending on the prob- lem, it is also possible to connect the input data onto computed reservoir state at each time point, using direct shortcut connections, and then compute the ESN outputs. Shortcut connections are useful when the echo state networks are trained in the generative mode which they try to output signal values same as the input data. But for the purpose of this work we will not use them. 18
- 24. CHAPTER 4. ESN FOR FUNCTION APPROXIMATION 4.2. A BRIEF OVERVIEW ON ECHO STATE NETWORKS 4.2.1 Learning In Echo State Networks In this work we want to use echo state networks for the function approximation in the Q-learning algorithm because they could hold the history of the dynamic systems in their state. Therefore, the ESN output Y(n) ∈ RL will be the Q-value of the function approx- imation and we try to minimize the error measure E(Y, Ytarget ) between the estimated output Y(n) and target output Ytarget (n). More importantly, we need to make sure that our approximation generalizes well to unseen data. The error measure E can typically be a Mean-Square Error (MSE), for example a root mean square error (RMSE): E(Y, Ytarget ) = 1 L L i=1 1 T T i=1 (yi(n) − ytarget i (n))2 which is here averaged over the L dimensions of the output data. The RMSE can be dimension-wise normalized (divided) by the variance of the target data Ytarget (n), producing a normalized root mean square error (NRMSE). The NRMSE has an absolute interpretation: it does not depend on the arbitrary scaling of the target data Ytarget (n) and the value of 1 can be achieved with a simple constant output data Y(n) set to the mean value of Ytarget (n). This suggests that a reasonable model of a stationary process should achieve a NRMSE accuracy between zero and one. As we mentioned earlier, the only adaptable weight matrix in echo state networks is the output weights matrix Wout and it is possible to train these weights with different algorithms. In order to train linear readout weights Wout there are different methods [48, 28, 30, 66, 31], and using least square method is a standard offline procedure and formulated as follows: • Generate the reservoir with specified set of parameters. • Feed the input data U(t) and compute and collect the reservoir states X(t). • Compute Wout from the reservoir using Linear Regression, minimizing the Mean Square Error between Y(n) and Ytarget (n). • Compute Y(n) again from trained network by employing Wout and feeding the input data U(t). 4.2.2 Spectral Radius and Echo State Property Echo state networks have two major functionalities that make them feasible tools for function approximation in the presence of delays. First acting as a nonlinear filter in order to expand the input signal U(t) to a higher dimensional space X(t). Second acting as a memory to gather the temporal features of the input signal and this ability in the literature called is short term capacity of the echo state network. The Combination of these two capabilities enriches the computed reservoir state matrix X, containing the history information about the dynamic behaviour of a control system. However, the computed state of the reservoir is influenced by various parameters that have to be set- ting judiciously. Those parameters or hyper-parameters are: the size of reservoir N, the sparsity of the weight matrices, the distribution of the nonzero elements of weight matri- ces, the spectral radius of reservoir weight matrix ρ(W), the scaling of the input weight matrix Win , and the leaking rate α. M. Lukosevicius [40] describes informative review on how to set these parameters and explains their affects on the system performance. In 19
- 25. CHAPTER 4. ESN FOR FUNCTION APPROXIMATION 4.3. ECHO STATE FITTED-Q ITERATION this work we mainly focus on the memory aspect of the echo state networks and try to choose these parameters according to the memory size needed for the delayed control task. According to Herbert Jaeger [28] a network has echo state property if the network state X(t) is uniquely determined by any left-infinite input sequence U−∞ . After the burn-in phase, removing the initial transition sates, the reservoir becomes synchronised to the input signal U(t) and the current reservoir state X(t), and exhibit an involved nonlinear response to the new input signal. This phenomenon can be explained by the echo state property: The network will forget its random initial state after the burn-in phase and the reservoir state becomes a function of the input signal. This is an essential property for successful training of a reservoir. Izzet B. Yildiz and et.al. [75] defined a simple recipe to preserve the echo state property. That procedure is explained in the following steps: • Generate a weight matrix W0 from uniform or Gaussian distribution. • Normalize W0 with its spectral radius λmax, W1 = 1 λmax W0 • Scale W1 with a factor 0 < α < 1 and use it as the reservoir matrix W = αW1. The spectral radius of the reservoir weight matrix W is one of the most important global parameters of the ESN and it has to be chosen according to, in our case, the needs of the dynamic control task. Usually the spectral radius is related to the input signal, in the sense that if lower time-scale dynamics is expected (fast oscillating signals) then a lower spectral radius might be sufficient. However, if longer memory is necessary then a higher spectral radius will be required. The downside of a larger spectral radius is the longer time for the settling down of the network oscillations. From an experimental point of view it means having a smaller region of optimality when searching for a good echo state network with respect to some dataset. Basically, if the spectral radius value is larger than one, it can lead to reservoirs hosting multiple fixed points that violate the echo state property. 4.3 Echo State Fitted-Q Iteration As we described earlier, an important aspect of echo state networks is their ability to serve as a memory to capture and preserve the temporal behaviours of their input sig- nal. For that reason, we substituted the multilayer perceptron Q-function in the NFQ algorithm with echo state networks and introduced the Echo State Fitted-Q Iteration (ESFQ) algorithm. Compared to Algorithm 1, we changed the MLP Q-function to the ESN Q-function and the Backpropagation training is replaced by a particular training method for echo state networks. The advantage of the ESFQ algorithm is that the cu- mulative history of the states-action pairs in the reservoir helps the Q-function to deal with the unknown delays in the control task. In order to train the echo state Q-function for a dynamic control task first, the weight matrices need to be initialized. Then the accumulated state-action pairs in the growing batch, are used to feed to the reservoir at the end of each episode. The reservoir dynamics at each point in time presented in the activation values of its hidden units. These activation values from the reservoir state ma- trix, reservoir design matrix, which its rows correspond to the system state-action pairs 20
- 26. CHAPTER 4. ESN FOR FUNCTION APPROXIMATION 4.4. TYPES OF READOUT LAYERS FOR ECHO STATE Q-FUNCTION on each cycle in each trajectory sample and its columns correspond to the computed features. In the burn-in phase, to remove the early noisy and unknown dynamics of the reservoir for each trajectory sample, we remove a few beginning states of the reservoir matrix. Then we compute the readout weights Wout using the teacher values. These teacher values for every state and action sample in a trajectory are generated from the sum of the immediate cost and the Q-value of the successor state. Finally having the new readout weights Wout we compute the estimated targets. 4.4 Types Of Readout Layers For Echo State Q-Function In the Echo State Fitted-Q Iteration algorithm, which is an offline policy learning algo- rithm, the Q-Function plays the role of a supervised regression method. Compared to the original Neural Fitted-Q Iteration algorithm described in section 3.3.2, we use an echo state Q-function with linear least squares on the readout layer. Although, the linear least squares makes the learning fast, sometimes it struggles in learning more complex nonlin- ear problems or higher input dimensions. However, because of the echo state networks topology and the fixed values of the input and reservoir weight matrices, it is possible to train the echo state Q-function with different types of readout layers. In the follow- ing, we describe two methods for training the output layer of the echo state Q-function that help it to estimate more accurate targets for the purpose of reinforcement learning. In this regard, we investigated the possibility of using ridge regression, and multilayer Perceptrons. 4.4.1 Ridge Regression In a standard formulation, the reservoir state matrix or the design matrix, is overdeter- mined matrix, because the number of trajectory samples is way larger than the hidden unit numbers N T. Therefore, the solution for this equation Wout = X−1 Ytarget needs to be approximated. To compute stable solution we apply ridge regression, also known as Regression with Tikhonov Regularization: Wout = (XT X + βI)−1 XT Ytarget The regularization parameter β will prevent output weights from growing arbitrarily large, which causes overfitting and instability in the system. This approach penalizes the squared length of the weight vector and transforms the error minimization task to a convex optimization problem that can be solved in analytical closed form. The optimal values of β can vary by many magnitudes of size, depending on the exact instance of the reservoir and length of the training data. To choose the value of the regularization parameter β by doing a simple exhaustive search, it is recommended to search on a logarithmic grid. Furthermore, β can be found using cross-validation or Gaussian-Processes. It is preferable to set the value of β not to zero to avoid numerical instability in computing the invert of the matrix (XT X)−1 . Interpretation of the linear readout gives an alternative criterion for setting β directly [12]. Because determining the readout weights is a linear task, the standard recursive algorithms, from adaptive filtering, for minimizing mean square root error can be used for online readout weight estimation [30]. The recursive least squares algorithm is widely used in signal processing when a fast convergence is of prime priority. 21
- 27. CHAPTER 4. ESN FOR FUNCTION APPROXIMATION 4.4. TYPES OF READOUT LAYERS FOR ECHO STATE Q-FUNCTION 4.4.2 Multilayer Perceptron Readout Layer The first and major reason we employed echo state networks as a Q-function approxima- tion were their ability to hold the history of the changes in the dynamic system behaviours. We need this history to deal with delays in executed action in the control problem. How- ever, the estimated Q-value by linear readout layers of the echo state Q-function might not help with complex nonlinear dynamics. The linear least square readout layer returns a global minimum which means the estimated targets have the minimum possible mean squared error to the original target. But a small error value does not always mean we have a successful policy for reinforcement learning. Also, it has been reported [29] that a larger reservoir size, even with high randomness in its weight matrices, combined with a linear readout layer might not be able to learn nonlinear dynamic systems. Therefore, we need to utilize more advanced readout layers to learn the policy for nonlinear dynamic systems. To do so, we use a multilayer perceptron on the readout layer of the echo state Q-function and trained it via the Backpropagation algorithm. The novelty of this ap- proach is that the reservoir expands the systems states into higher dimensional features space and also preserves the history of the states. Then the Backpropagation algorithm trains the multilayer perceptron to learn the value function which estimates the proper Q-values for controlling the delayed dynamic system. The Figure 4.2 illustrates a topo- logical example for the Echo State Fitted-Q Iteration algorithm. Figure 4.2: The topological illustration of the Echo State Fitted-Q Iteration algorithm with multilayer perceptron readout layer. It consists of two major parts. First, the memory part containing echo state input weight and reservoir weight matrices and their corresponding neu- rons. In the second part the readout layer containing one input layer, two hidden layers of feedforward neural networks and one output layer. Notice that the applied action goes only through the readout layer but not the reservoir itself. 22
- 28. CHAPTER 4. ESN FOR FUNCTION APPROXIMATION 4.5. SMAC FOR HYPER PARAMETER OPTIMIZATION When we compare the Echo State Fitted-Q Iteration (ESFQ) algorithm to the NFQ algorithm, it can be seen that our method is a specific version of the NFQ algorithm. As it is shown in Algorithm 3 in more detail, after collecting trajectory samples first we fit them into a reservoir with pre-initialized weight matrices. The reservoir collects the expanded trajectory samples in its state matrix and then jointly with their corresponded actions fits them to a multilayer perceptron algorithm i.e. the Q-function in the Algorithm 3. The Rprop algorithm trains the multilayer perceptron and tries to minimize the estimation error with respect to the target values. This novel structure is computationally efficient, as long as the reservoir has fairly small size. The immediate costs/reward for each action is based no system state not the reservoir state. Algorithm 3 The main loop of Echo State Fitted-Q Iteration.k counts the number of it- erations, kmax denotes the maximum number of iterations. init MLP() returns a multilayer perceptron with randomly initialized weights. Rprop training(P) takes pattern set P and re- turns a multilayer perceptron that has been trained on P using Rprop as the supervised training method procedure ESFQ Main() input: a set of transition samples D; output:Q value function QN ; k = 0; init MLP() → Q0; init ESN(); while k < kmax do generate pattern set P = {(inputl , targetl ), l = 1, ..., #D} where: inputl = ESN(sl ), al targetl = c(sl , al ) + γmin a Qk(input l , a l ) Rprop training(P) → Qk+1 k := k + 1 4.5 SMAC For Hyper Parameter Optimization There are a number of parameters that need to be carefully selected in order to boost the performance of the Echo State Fitted-Q Iteration algorithm. These are parameters related to the short-term memory capacity of the echo state Q-function. In the following, we named those parameters whose their effect need to be studied. First is the spectral radius of the reservoir which directly affect the short term memory capacity of the reser- voir. Second, the leaking rate which determines how fast neurons in the hidden units will shed off their dynamics. Third, the reservoir size or number hidden units, usually chosen relatively larger than the input signals dimension. Fourth, the sparsity percentage of the input and reservoir weight matrices which affect the computation time and to some degree performance. Fifth, the reservoir activation function which is usually logistic, hyperbolic-tangent, or rectifier units. As we see the echo state Q-function is a black-box function with various parameters. Most of these parameters have continuous real values and using combinatorial space of parameter settings for tuning them is inefficient and leads to unsatisfactory outcomes. Recently, automated approaches for solving this algorithm configuration problem have led to substantial improvements in the state of the art for solving various problems. Se- 23
- 29. CHAPTER 4. ESN FOR FUNCTION APPROXIMATION 4.5. SMAC FOR HYPER PARAMETER OPTIMIZATION quential Bayesian optimization allows for global optimization of such black-box functions by using as few trials as possible. MOCKUS et al. [44] applied Bayesian methods to cases with linear and nonlinear constraints and to multi-objective optimization. They stud- ied interactive procedures and reduction of multidimensional data in connection with global optimization using Bayesian methods. Various implementations of this sequen- tial optimization match or surpass manually tuned performance for the tasks such as satisfiability problem [25] or object classification [7]. In this work we employed SMAC hyper-parameter optimization software which is implemented in [25] and used sequential Bayesian optimization methods. In the extended version of the SMAC software [24] it is possible to optimize various types of parameters e.g. categorical parameters and sets of instances. 24
- 30. CHAPTER 5 EXPERIMENTS AND RESULTS 5.1 Experimental Setup In this section, we demonstrated results of our experiments in order to evaluate the performance of the Echo State Fitted-Q Iteration (ESFQ) algorithm on the learning of delayed control tasks. The presence of a delay, in particular, the action delay, violates the Markov property of a control system. The main goal of this work was to show that echo state networks with a nonlinear readout layer are practical memory based function approximation in the non-Markovian reinforcement learning domain. Therefore, we tried to compare the performance of our method to tapped delay-line memory base algorithms, as a baseline method and discuss the different advantages and drawbacks of it. We choose two standard simulated benchmarks, mountain car, and inverted pendulum, for evaluat- ing our model-free learning algorithms. We tested our algorithms with plants containing delays in their applied actions with different lengths in the range of 1 to 30. We showed that the solution we proposed within the Echo State Fitted-Q Iteration algorithm can be a very stable and reliable learning tool for a system with unknown delay lengths and complex dynamics. In the rest of this chapter, we introduce our selected benchmarks and their parameter settings for each experiment. After that, we describe the setting for our algorithms, the general scheme of our experiments, and the hyper-parameter optimization for the ESFQ algorithm. Then we compare and analyze computed the results based on their accuracy, efficiency, and ability to deal with the complex dynamic system with delays. We implemented our core methods and algorithms in C + + using the Eigen framework and integrated them to the CLSquare simulation system. For the hyper-parameter tuning, we employed SMAC [24], a versatile tool for optimizing algorithm parameters. 5.2 Simulation Benchmarks In the realm of reinforcement learning, there are various standard simulated benchmarks [62] used to evaluated algorithm’s performances. In this work, we tried two of these bench- marks for evaluating the performance of our algorithms and collected measurements to compare their abilities in dealing with delay in control systems. The benchmarks were 25
- 31. CHAPTER 5. EXPERIMENTS AND RESULTS 5.2. SIMULATION BENCHMARKS mountain car and inverted pendulum [62]. The dynamics of balancing a pendulum at an unstable position can be employed in the applications of controlling walking robots or rocket thrusters. This highly nonlinear dynamics make the inverted pendulum a suitable and difficult task for our algorithm to solve. Our second chosen benchmark, the moun- tain car problem, is a second-order, nonlinear dynamic system with low-dimensional and continuous-valued state and action spaces. The simulated plants are provided by the CLSquare tool, developed and implemented by the Machine Learning Lab of the Uni- versity of Freiburg. In the following, we briefly introduce each benchmark and their parameter settings for our experiments. 5.2.1 Mountain Car Mountain car is a standard testing domain in reinforcement learning. The problem is for- mulated such an under-actuated car must drive up a parabolic shape steep hill. Starting from any initial state in the middle of the valley the goal of the system is to drive the car up the valley’s side and escape in as few steps as possible. Since gravity is stronger than the car engine, even at full throttle, the car cannot simply accelerate up the steep slope. The car must rock back and forth along the bottom of the valley to build enough momentum to escape. The allowed actions are {−4, 4} Newton to drive the system in left and right directions. The state vector has two dimensions, car position along x-Axis in meters, and car velocity in m/s. The allowed area for the car is between [−1.2, 0.7] and the system will receive a transition cost of 0.01 in this range. The terminal area is in the range of [∞, −1.2] and the agent will receive a terminal cost of 1.01 and episode cycle will be ended. The target area is between [0.7, ∞] and agent receive no cost in the range and the episode will be ended. The Figure 5.1 depicts the mountain-car visualization in the CLSquare simulation. Figure 5.1: The simulated mountain car visualization. The allowed position for the car is in the range of [−1.2, 0.7] meters and the goal area in the range of [0.7, ∞] meters. The terminal area for the car is in the range of [−∞, 0.7] meters. 5.2.2 Inverted Pendulum The inverted pendulum problem benchmark requires balancing a pendulum of unknown length and mass at the upright position by applying forces to the cart it is mounted 26
- 32. CHAPTER 5. EXPERIMENTS AND RESULTS 5.3. EXPERIMENTAL PROCEDURE AND PARAMETRISATION on. The allowed actions are {−10, 10} Newton to move the cart to the left or right directions. Actions can be given at intervals of 0.05s. The system state is described by a four dimensions vector, pole angle θ, pole angular velocity ˙θ, cart position P, and cart transitional velocity ˙P. The cart can move in the range of {−2.4, 2.4} meters and anything outside this ranges is considered terminal area and the episode will be ended. The pole is allowed to rotate in between {−0.7, 0.7} radians. When the inverted pendulum is moving in the allowed range will receive transition cost of 0.01 and cost for entering the terminal area is 1.01. The target area for the pole is defined as the range {−0.05, 0.05} radiant, and for the cart as the range {−0.05, 0.05} meters. There, the system will receive no transition cost and the episode will end. The Figure 5.2 depicts the inverted pendulum visualization in the CLSquare simulation. Figure 5.2: The inverted pendulum visualization in the CLSquare simulation. The goal areas for the pole and cart is indicated. The allowed area for the cart is in the range of {−2.4, 2.4} meters and for the pole is in the range of {−0.7, 0.7} radians. Outside the allowed area is considered the terminal area. The goal area for the cart is in the range of {−0.05, 0.05} meters and the goal area for the pole is in the range of {−0.05, 0.05} radians. 5.3 Experimental Procedure And Parametrisation The process that takes the Echo State Fitted-Q Iteration algorithm to learn a control task has the following steps: first, we initialize the echo state networks input and reservoir weights matrices, Win , W. Then generated trajectory samples in each episode, containing the plant state variable in a vector form, are passed through the reservoir at the end of each episode and expanded to a higher dimensions feature space, the same dimensionality as the reservoir state. Then these expanded samples and their corresponding actions are accumulated in a growing batch format. The trajectory samples are generated by a simulated plant containing unknown delay lengths in their executed actions in the time intervals of 0.05s. At the end of each episode, the batch is fed to a multilayer perceptron which contains a number of input units equal to the size of the reservoir hidden units. The data flows through two extra hidden layers with 20 neurons each, and one output layer for the Q-value. The activation function for the multilayer perceptrons is logistic. Finally, the Rprop algorithm updated the weights of the multilayer perceptron. We designed our finite horizon experiments on the introduced benchmarks such that the reinforcement learning algorithm tries to minimize the time the agent needs to reach the goal area. 27
- 33. CHAPTER 5. EXPERIMENTS AND RESULTS 5.3. EXPERIMENTAL PROCEDURE AND PARAMETRISATION The discount factor γ parameter for our Q-learning algorithm was set to 0.98 and no exploration strategy was used during the training. 5.3.1 Echo State Q-Function As we described in the previous section the reservoir in echo state networks works as memory for the Q-function and the multilayer perceptron as a readout layer in the Echo State Fitted-Q Iteration algorithm. For our memory based approach it was essential to have sufficient memory capacity for holding the history of delayed states. Here we benefited from the short-term capacity of the echo state Q-function. However, the Q- function parameters needed to be specified according to the length of the delay and the dynamics of the control task at hand. Table 5.1: The list of parameters for initializing echo state networks. Name Description And Value Range Optimized Spectral Radius Real value [0, 1], adjusts memory size True Leaking Rate Real value [0, 1], adjusts dynamics of each neuron True Reservoir Size Integer value in 10, 20, 30, 40, 50, 100, 200 True Transition steps Integer set to {5% or 10%} of cycle length False Regularization coefficient Real value in [10e-3,10e3] True Activation function Logistic, Tanh, Rectifier True Input weight sparsity Real value [0.2, 1.0] True Reservoir weight spar- sity Real value [0.2, 1.0] True A list of echo state networks parameters is available in Table 5.1. The spectral radius and leaking rate together affect the short term memory length of the network. Basically, for a system with fast oscillation like the inverted pendulum, we would need a shorter memory length compared to slower rate systems like the mountain car. A high spectral radius and or a very small leaking rate increase the short memory length of echo state network. The transition steps are the number of the time steps in each trajectory samples that are used for burning the reservoir and these state will be removed in the learning time. The burning phase helps the reservoir to remove its noisy dynamics and become input driven rather than affected by the arbitrary initial states of itself. The length of the transition steps has a linear dependency on the sample trajectory length, usually 5 to 10 percent of the average trajectory length. The reservoir size depends on the dimension of the input signal and the complexity of the system dynamics, for some tasks it needs to be set to a very large number. The regularization coefficient and the activation function need to be searched and found regarding the control system at hand. The standard choice for activation function is the hyperbolic tangent function. A higher value for the regularization coefficient is used to prevent linear readout weights from growing exponentially. Typically it is a real number in logarithmic scale and in the range 28
- 34. CHAPTER 5. EXPERIMENTS AND RESULTS 5.3. EXPERIMENTAL PROCEDURE AND PARAMETRISATION of [0.00001, 1]. In the following we talk about the initialization of the echo state network weight matrices which are playing a major role in capturing system dynamics. Figure 5.3: On the left, the reservoir weight matrix is initialized from a normal distribution with zero mean and unit variance. Then it is normalized with the spectral radius of 0.54. On the right, the reservoir weight matrix is initialized from a normal distribution with zero mean and unit variance. Then orthogonal matrix is computed using Schur decomposition implemented in the Eigen library and it is normalized with the spectral radius of 0.54. Finally 40 percent of its coefficient set to zero. The sparsity of the echo state weight matrices boosts its performance by extracting more distinguishable features from input signals. In the Figure 5.3 reservoir weight matri- ces for 20 hidden units are illustrated. The matrix in the left is initialized using a normal distribution with zero mean and unit variance and then normalized with the spectral radius of 0.54 and fully connected. In fully connected reservoirs, all reservoir neurons were directly connected to each other with all connection weights being nonzero. The more sparsely connected matrix on the right graph was generated in two steps. First, a fully connected reservoir was generated in the same way as the first one. Then from that, an orthogonal matrix computed using Schur decomposition. Finally, given the sparsity percentage, 40% in this example, randomly chosen coefficients were set to 0. We need to consider the fact that if the sparsity percentage is set very high, then the reservoir does not exist as a group of mutually connected neurons. Usually, larger reservoirs (more hidden units) tolerate a lower connectivity and still fulfill this requirement. The advan- tage of a sparse orthogonal reservoir weight matrix is that it has linearly independent columns and echo states were computed by this matrix could better catch the dynam- ics of the system. The weights of the reservoir W functions as a dynamic basis for the original input signal. Therefore, these bases decompose the input signals into pertinent sub-components to maximize their differences. 29
- 35. CHAPTER 5. EXPERIMENTS AND RESULTS 5.3. EXPERIMENTAL PROCEDURE AND PARAMETRISATION In summary, we suggest that the performance echo state Q-function highly depends on a proper balance of the internal dynamics and the influence of external activities, mediated via the input weights, and the strengths of the input and target activities. See- ing that the external activities can not be directly manipulated, the initialization of the reservoir and input weight matrices are important. If the input weight values were set too high, it will dominate the dynamics unfolding in the reservoir and quickly overrule any other useful internal dynamics. A proper choice of the weight initialization intervals is necessary to generate a well-performing echo state Q-function. 5.3.2 Hyper-parameter Optimization The choice of parameters depended on the delay length which was unknown for the Echo State Fitted-Q Iteration algorithm. Therefore, we had to search over the parameter space to find out which configuration provides sufficient memory length for our algorithm. Hence, we used the hyper-parameter optimization tool SMAC to search over given pa- rameters: spectral radius, leaking rate, reservoir size, non-linear function for reservoir neurons, input weight matrix sparsity, and reservoir matrix sparsity. These parameters are inter-depended and affect one another. The parameter optimization algorithms try to minimize the loss function over sets of discrete and continue values. To compute the loss value we trained our algorithm for 120 episodes, 200 cycles each on the inverted pendulum task with an unknown action delay given a particular parameter set. The algorithm was tested 100 times with different initial positions at the end of each episode. A successful test was one that ended up in the goal area in the finite horizon task. The averaged success rate subtracted after 120 episodes were returned as a loss value. SMAC searched over this parameter space using Bayesian methods of global optimization to find settings that provide minimum expected deviation from the global minimum. Parameter tuning over reinforcement learning algorithm is a computationally expensive process, and for each delay length has to be done separately. Therefore, in the scope of this work, we only ran a few optimization iterations, roughly 60 to 80 per delay length and in total 2500 experiments, to find different good settings. The Figure 5.4 shows the Pearson correlation coefficient for 6 hyper-parameters opti- mized via SMAC. Each cell shows the correlation coefficient between parameters on its corresponding column and row. Labels are parameter names in both Axis. Coefficient values range between [−1, 1] with a higher absolute value for higher correlation and the sign for direct or inverse correlation. And they are computed from samples with at least 80% goal reach. The value range for each parameter is presented in Table 5.1. Our re- sults showed that the most frequently chosen activation function was logistic activation therefore we exclude it from correlation computation. The goal of this optimization was to find the configuration which gives echo state Q-function enough short-term memory capacity to learn the delayed control task. Therefore, we include corresponding delay lengths for each experiment to observe its correlation with other parameters. 30
- 36. CHAPTER 5. EXPERIMENTS AND RESULTS 5.3. EXPERIMENTAL PROCEDURE AND PARAMETRISATION Figure 5.4: This plot shows the pairwise correlation matrix between for 6 optimized hyperpa- rameters using Pearson product-moments method. The labels on the figure assigned to their corresponding parameter. The delay lengths, third column, was kept constant during each op- timization iteration and its value is ranged between [1, 30]. The dataset which the correlations are computed is from samples which have 80% or more goal reach, 1561 experiments, for the simulated inverted pendulum plant. There are few observations we could make by looking at the bivariate correlation ma- trix on Figure 5.4. First, the reservoir matrix and input matrices sparsity were positively correlated. This means for experiments with different delay length their value increases and decreases simultaneously. However, this correlation was not strong. Contrary to that, the leaking rate and spectral radius are inversely correlated. Although this was a weak correlation it implied that a reservoir with high spectral radius needs a lower leaking rate for holding the history of delayed actions. According to the update equation of the reservoir, a high leaking rate means its new state will be influenced less by its previous state and high spectral radius means the reservoir previous state will have more influence on the next computed state. Another observation is that the delay length in this matrix had a weak negative correlation with the spectral radius, the leaking rate, and the input weight matrix sparsity. Our bivariate correlation matrix also shows the delay length and the reservoir size had almost zero correlation. Furthermore, we can see that the reservoir size and the spectral radius have a positive correlation. Both of these two observations are counter-intuitive, because as we described before for a reservoir with a large number of hidden units we would expect a relatively small spectral radius. In addition, for a longer delay length, we need a higher spectral radius to provide enough short-term memory capacity. Therefore, we can safely conclude that these parameters are interdependent and they have a multivariate dependency among them which can not be understood only using a bivariate correlation computation. To analyze the effects of these parameters on the ESFQ performance in detail we need to run thoroughgoing ex- periments of hyper-parameter optimization with an extensive number of iteration for each delay length. Then by knowing their multivariate correlations we can find an efficient way of selecting them. However, in the scope of this work for selecting these parameters, 31
- 37. CHAPTER 5. EXPERIMENTS AND RESULTS 5.3. EXPERIMENTAL PROCEDURE AND PARAMETRISATION we need to rely on computationally expensive hyper-parameter optimisation done over our reinforcement learning algorithm. Figure 5.5: The figure shows a histogram plot of the goal reaching percentage of the spectral radius, leaking rate, reservoir weight matrix sparsity, input weight matrix sparsity. Each bin on the Y-Axis shows how frequently and with what percentage the system reaches the goal given the corresponding subject parameter on X-Axis. There are in total 2500 experiments on the simulated inverted pendulum containing action delays in the range of [1, 30]. In Figure 5.5 we see a histogram plot for the spectral radius, leaking rate, reservoir weight matrix sparsity, input weight matrix sparsity parameter relative to reaching the goal area. In total, there were 1561 configuration samples with more than 80% success rate, out of 5600 different configurations that we tried and in average 50 samples per delay length. As we can see on the top left plot system spectral radius values in the range of [0.0001, 0.3] have dominantly reached the goal area with high success rate. For the leaking rate in the plot top right, the successful experiments have a broader range of [0.0001, 0.9]. And for both weight matrices sparsity, the successful experiments frequently occur in the range of [0.4, 0.8]. This illustration will help us to elicit some general rule for initializing our hyper-parameters and putting some constraints on their value range and make the search more efficient. Figure 5.6 depicts mean and variance of the four echo state hyper-parameters relative to goal reach percentage for experiments above 80% success rate. In the top rows, we compared spectral radius and leaking rate against goal reaching percentage. The bottom row compares reservoir and input weight matrices sparsity. The frequency of successful experiments for each of these values can be seen in the corresponding histogram plots in Figure 5.5. As we can see for a wide range of these subject parameters there are successful experiments. This is another support for 32
- 38. CHAPTER 5. EXPERIMENTS AND RESULTS 5.3. EXPERIMENTAL PROCEDURE AND PARAMETRISATION the presence of multivariate interdependencies between these parameters. Figure 5.6: The figure shows the mean and variance of the goal reach percentage on the Y-Axis, given value of the particular subject parameter on the X-Axis. The subject parameters are the spectral radius, leaking rate, reservoir weight matrix sparsity, and input weight matrix sparsity. The samples selected for experiments with more than 80% goal reach. There are in total 2500 experiments on the simulated inverted pendulum containing action delays in the range of [1, 30]. Our goal in this section was to suggest that there are rules for efficiently choosing some of the hyper-parameters of the ESFQ algorithm. And according to our parameter optimization results in the previous section deducing such rules is not a straightforward task due to multivariate dependencies between these parameters. It will need an extensive study of the parameter space which is beyond the scope of the work. But in order to drive some general constraints for setting up and computing a reservoir given a particular task, our results suggests the following: • Reservoir size, the number of hidden units, up to 10 times larger than the size of the input vector. This is usually a large enough number to grasp and reflect the system dynamics. • Input weight matrix and reservoir weight matrix should be sparse as we saw in Figure 5.5, they better to have the sparsity values between [40, 60] percent. • Our experimental results suggest that the logistic activation works well, 1514 out of 1561 successful experiments, for the reservoir activation function. • leaking rate and spectral radius have a weak inverse correlation but it is not in the scope of this work to comment on the choice of their values. 33
- 39. CHAPTER 5. EXPERIMENTS AND RESULTS 5.4. COMPARISONS TO DELAY-LINE NFQ ALGORITHM Basically, these results suggest that there are possibilities to chose the algorithm parameters according to the characteristics of a given dynamic control task. But to do so we would need to do more experiments and collect enough samples per delay lengths. We may run 300 SMAC iterations for each delay length and then study the change in the parameters relative to how well the controller learns a task. Then, it is possible to fix the value of some of these parameters and only search over particular ones. In the last chapter of this work, we will mention some of the probable solutions for optimizing these parameters and better understanding their dependencies. 5.4 Comparisons To Delay-Line NFQ Algorithm In this section, we compare the results of our method to the standard delay-line Neural Fitted-Q Iteration algorithm. Here we used a Q-function with two layers of hidden units, 20 neurons each, proven[54] to be feasible structure to learn dynamic control tasks. In the delay-line method, the action delays lengths are known to the algorithm. The system history is a collection of observed states from the most recent state-action pair and previous states, equal to the delay length. This means if there a 3 steps action delays in the simulated plant, then the input vector contain the current state-action pair and the past three states. The NFQ algorithm was trained on the simulated inverted pendulum benchmark for 200 episodes, 200 cycles each, using Rprop and one update per episode. For the sake of comparison, we added the same nonlinear structure as the NFQ algorithm on top of our reservoir in Echo State Fitted-Q Iteration algorithm. As we described in chapter 4, the difference between the NFQ and our method is that we preserve the memory for our method by using the reservoir of the echo state networks. And we employed the hyper-parameter optimization tool SMAC to find a suitable memory capacity for the echo state Q-function. In simple terms, we compared the performance of the reservoir as a memory to the classical tapped delay-line memories [65]. Although the Echo State Fitted-Q Iteration algorithm is not aware of the delay length but with the aid of the hyper-parameter optimization tool SMAC it manages to find the best configuration. This approach could be considered the same as giving the delay length to the system. 34
- 40. CHAPTER 5. EXPERIMENTS AND RESULTS 5.4. COMPARISONS TO DELAY-LINE NFQ ALGORITHM Figure 5.7: Maximum percentages of goal reach achieved by delay-line NFQ and ESFQ algo- rithms given different delay lengths. The results achieved from the simulated inverted pendulum with action delays in the range of [1, 30]. The X-Axes shows the delay lengths and the Y-Axis show the maximum percentage of goal reach. In Figures 5.7 and 5.8 we present a comparison between the delay-line Neural Fitted- Q Iteration and the Echo State Fitted-Q Iteration algorithms on the test phase with different initial values and different delay lengths. In the first graph, we can observe that the ESFQ algorithm has the advantage of achieving the maximum value even with the long delay length. On the other hand, we can observe that delay-line NFQ faces difficulties after 10 steps action delays and its peak performance decays down to 40%. Such observations suggest that reservoir is a very suitable memory based approach for the control system with longer delay lengths. For every delay length it reaches the goal area, from different initial positions in the test phase, on average more than 90% of the times, and in some cases 100% of the times. Furthermore, in the second plot 5.8 we can observe that the ESFQ algorithm shows fairly robust performance to randomness in the initialization of its fixed weight matrices. Here for every length of delay, we trained both algorithms, initialized with 10 different random seeds, and plot the mean and standard deviation of goal reach percentage over delay lengths. As we see the NFQ algorithm performance is prone to decay when it is trained with different random seeds especially for longer delay lengths. On the contrary, the ESFQ algorithm performs more robustly in the face of such random initialization even for longer delay lengths. In this regard, we need to consider the fact that the echo state performance presumably should decay because it has extra random parameters for its weight matrices. However, given the same configuration for initializing the reservoir over various random seeds, our algorithm shows much better overall performance. Furthermore, the closed range of performance variances in both algorithms is another evidence for the robustness of the ESFQ algorithm toward randomness in its parameter initializations. In our experiments, we are trying to support the idea that the echo state Q-function is a fairly reliable memory based function approximation for the dynamic system with delays. And, the facts about the randomness support the idea that the memory capacity of the echo state Q-function is less likely to be affected by random initialization of its parameters. 35
- 41. CHAPTER 5. EXPERIMENTS AND RESULTS 5.4. COMPARISONS TO DELAY-LINE NFQ ALGORITHM Figure 5.8: Mean and standard deviation of the goal reaches percentage achieved by delay-line NFQ and ESFQ algorithms given different delay lengths. The results achieved from experi- menting with simulated inverted pendulum containing action delays in the range of [1, 30]. The X-Axis shows the delay lengths and the Y-Axis show the mean and standard deviation of the goal reach percentage average over experiments with 10 different random seeds. The mean values are computed from the maximum percentage of goal reach for each experiment. In Figure 5.9 we compared the learning curve for the ESFQ and NFQ algorithm on the experiments with 27 delay length. The value is averaged over 10 different random seeds. Here we can see that although the echo state Q-function has more random pa- rameters, overall the ESFQ algorithm performs better than the NFQ algorithm. In the early episodes the delay-line NFQ shows some minor improvement relative to the ESFQ but after 60 episodes the average performance of the NFQ stays the same and the ESFQ performance increases. Furthermore, we can see that the ESFQ has higher standard de- viation compared to delay-line NFQ. 36
- 42. CHAPTER 5. EXPERIMENTS AND RESULTS 5.4. COMPARISONS TO DELAY-LINE NFQ ALGORITHM Figure 5.9: The learning curve of the delay-line NFQ and the ESFQ algorithms. The results achieved from experimenting with the simulated inverted pendulum containing 27 steps action delay. The X-Axis shows the episodes index and the Y-Axis shows the mean and standard deviation of the goal reach percentage average over experiments with 10 different random seeds. The mean values computed from the percentage of goal reach for corresponding episodes over all experiments. In Figure 5.10 we compared how fast each algorithm reached the 80% success rate or more. As we could see the Echo State Fitted-Q Iteration algorithm average needs 100 episodes to reach the minimum 80% success rate. Although the delay-line NFQ algorithm for some short delay lengths reaches the maximum success rates faster it fails for most of the delay lengths larger than 15. This fact indicates that the memory provided by the reservoir contains information that helps the Echo State Fitted-Q Iteration algorithm to perform more successfully and reach higher performances quicker. However, we should remember that in order to find a suitable parameter setting for the ESFQ algorithm we still need to use hyper-parameter optimization tools, explained in the previous section. From all these observations we can conclude that the Echo State Fitted-Q Iteration algorithm with nonlinear readout layer is a reliable algorithm for learning a dynamic control task with delays. And, its performance can stay relatively stable regardless of the randomness in the parameter initializations. 37
- 43. CHAPTER 5. EXPERIMENTS AND RESULTS 5.5. WHY NOT LINEAR READOUT LAYERS Figure 5.10: In this figure, we see how many episodes for the ESFQ and the NFQ algorithms take to reach success rate above 80%. The X-Axis corresponds to delay length and the Y-Axis corresponds to episode number. The experiments are done on the simulated inverted pendulum for a given delay length, for 120 episodes and 200 cycle each. For some delay lengths the delay-line NFQ did not manage to reach the minimum of 80% success rate. 5.5 Why Not Linear Readout Layers In the previous section, we showed the advantages of the Echo State Fitted-Q Iteration algorithm with a nonlinear readout layer and compared its performance to the tapped delay-line NFQ algorithm. As we described in section 4.4 it is possible to train different readout layers for the ESFQ algorithm and in the following we present results for training the algorithm with various readout layers. Here we show the performance evaluation for ridge regression and very large reservoirs. Our aim is to illustrate that a linear method for training the echo state Q-function, in offline growing batch mode, performs well for the control task with simpler dynamics such as mountain car but fails to learn a proper policy on more complex dynamic tasks such as the inverted pendulum. 5.5.1 Ridge Regression On Mountain Car The first choice for training the echo state Q-function is the ridge regression algorithm. We designed an experiment with the simulated mountain car containing 2 steps action delay. In Figure 5.11 we can see how the dynamics of the mountain car are captured by the reservoir and how they changed over the time given the input signal i.e. the car position and velocity and the applied action. The ESFQ algorithm with a linear readout layer achieves 100% test success in reaching the goal area starting from 100 different initial positions after 17 episodes running with 300 cycles each. The effects of the delayed actions are reflected in the reservoir states in the top plot. Hidden units adapted their states according to the changes in the car position and velocity. It took 290 cycles for the controller to reach the goal area. Other parameters of this experiment were set as following. The spectral radius is 0.40, the reservoir size is 20, the regularization coefficient is 10, 30 steps of the initial transitions, γ is 0.98, with 10% greedy exploration rate, and 38
- 44. CHAPTER 5. EXPERIMENTS AND RESULTS 5.5. WHY NOT LINEAR READOUT LAYERS a hyperbolic tangent activation function. Figure 5.11: The graph depicts from the top, the reservoir states with 20 hidden units, 2 steps delayed action and the genuine action, the car velocity and position on the parabolic hill. The trajectory sample belongs to a successful episode of the simulated mountain car plant. The X-Axis shows the cycle indexes in one episode, generated with 0.05s interval. The values on the Y-Axis from bottom to top, car position, car velocity, applied actions, and the activation values of the reservoir units. In Figure 5.12 the learning curve in the first row and the readout weights Wout on the second row for the same experiment are depicted. The controller learned the task in 17 episodes and achieved 100 percent success in reaching the goal area starting from different initial positions. The experiment stopped after we reached the 100% success rate. As we see in the early episodes the echo state Q-function output weight values are infinitesimal but after the 13th episode they grow and converge to be a successful policy. The initial readout weights Wout are set to zero and each episode a new value is computed regardless of previous ones. Readout weights Wout could grow higher with a larger number of hidden units or after training for longer episodes. A proper choice of the regularization coefficient and adding a bias value of one to the reservoir matrix before solving the least square problem could penalize these effects. With the high readout weight values, the system may also fit well to the training data but would be unstable in the test phase. By looking at the learning curve of this experiments we can see that after episode 13, when the readout weights become larger, it takes the system 4 more episodes to collect enough trajectory samples in the batch file to reach a 100% success rate. 39
- 45. CHAPTER 5. EXPERIMENTS AND RESULTS 5.5. WHY NOT LINEAR READOUT LAYERS Figure 5.12: The readout weighs Wout for the echo state Q-function with 8 hidden units after training for 19 episodes on the simulated mountain car plant with 2 steps action delay. The X-Axis shows the episode index. The Y-Axis shows the readout weight values. In Figure 5.13 we illustrate some of the successful trajectories that lead to the goal area. The X-Axis indicates the car position and the Y-Axis indicate the car velocity. The initial position for the car is in the range of [−0.75, 0.25] and the initial velocity is in the range of [−0.5, 0.5]. All the 5 tests reached to goal area where the car stopped on top of the hill with the position value equal or larger than 0.7. By looking at each trajectory we see the policy first drives car backward uphill with a negative velocity and accelerates with applied positive velocity until it reaches the goal area. 40
- 46. CHAPTER 5. EXPERIMENTS AND RESULTS 5.5. WHY NOT LINEAR READOUT LAYERS Figure 5.13: The figure depicts 5 successful trajectory samples for the mountain car plant controlled by the ESFQ algorithm with a linear readout layer. The X,Y Axis indicate the car position and velocity for the simulated plant with 2 steps action delay. Each marker indicates one specific trajectory sample. 5.5.2 Ridge Regression On Inverted Pendulum In the standard echo state Q-function the readout weight matrix Wout stores the policy which is trained by the least squares regression algorithm. The ridge regression readout layer returns a globally optimal estimation of the Q-value by minimizing its differences to the target value. As we showed in the previous section this linear readout layer can perform well in the learning dynamic tasks like mountain car. However, when it comes to the control tasks with more complex dynamics like the inverted pendulum it fails to perform well. In Figure 5.17 we see the averaged learning curve for the ESFQ algorithm with ridge regression readout layer trained on the simulated inverted pendulum with an action delay length of 2. To generate these results we used SMAC, a hyper-parameter optimization tool, and optimized the ESFQ algorithm over the following parameters for 120 iterations. The reservoir size, the spectral radius, the leaking rate, the input weight matrix sparsity, the reservoir weight matrix sparsity, the regularization coefficient, and the type of activation function. As illustrated, the algorithm with a linear readout layer in comparison with a nonlinear readout layer fails to learn the model of the system. 41
- 47. CHAPTER 5. EXPERIMENTS AND RESULTS 5.5. WHY NOT LINEAR READOUT LAYERS Figure 5.14: The learning curve comparison of the Echo State Fitted-Q Iteration algorithm with a ridge regression readout layer and the multilayer perceptron readout layer. The results are averaged over 122 experiments for the simulated inverted pendulum with a delay length of 2. The X-Axis shows the episode index and the Y-Axis show the mean and standard deviation of the goal reach percentage. The mean values are computed from the percentage of goal reach for corresponding episodes over 100 separate experiments for each algorithm. In order to analyze our results in more detail, we tried to have a look at some tra- jectory samples and their corresponding reservoir states. Figure 5.15 shows 6 different trajectory samples for the simulated inverted pendulum with 2 steps unknown action de- lays. The Q-values estimated via the Echo State Fitted-Q Iteration algorithm with linear readout layer is given the state-action vector as input. One important observation we can make from this figure is, in the bottom plot, those actions which have led the system to the terminal area receive a lower estimated Q-value. This phenomena is caused by the inherent property of least squares which tries to minimize the normalized root mean square error linearly and maps these states to the wrong Q-value. 42
- 48. CHAPTER 5. EXPERIMENTS AND RESULTS 5.5. WHY NOT LINEAR READOUT LAYERS Figure 5.15: The figure illustrates 6 trajectory samples of the inverted pendulum with 2 steps action delays. The trajectories are generated from training the ESFQ algorithm with a ridge regression readout layer. In the first row, we see the cart position and pole angle. The second row depicts the estimated Q-values for applied positive action and negative action. The cycle index on the X-Axis indicates the corresponding to the sample indexes in the batch dataset generated by 0.05s interval. The algorithm is trained for 200 episodes and 200 cycles each. In Figure 5.16 we can have a closer look at what is going on in the echo state Q- function in the experiment above when it fails to control the inverted pendulum. The policy in the first episode has chosen to oscillate between positive and negative actions, but the system failed to balance the pendulum. In the next episode, the learned policy tried to apply the same action to prevent the pole from falling in the negative direction but that also failed. The same pattern repeated in the next three trajectory samples. The last two rows in the graph show the cart position and the pole angle. Each trajectory sample ended when the car position or the pole angle went to the terminal area and then the next trajectory sample began. In the top row, we can observe that the dynamics of the reservoir reflect the changes on the system state and action. However, it is not possible for a linear readout layer to learn from them. The wrong target estimations are the problem with a linear echo state Q-function. It is possible to clip these wrong estimations, setting them to zero or some other constant value, hoping that after a long training time the system eventually will learn the desired policy. But our experiments showed this approach would not compensate for the problem of avoiding a bad policy. Basically by clipping the Q-value we still would have the same policy that estimates wrong targets. 43
- 49. CHAPTER 5. EXPERIMENTS AND RESULTS 5.5. WHY NOT LINEAR READOUT LAYERS Figure 5.16: The figure illustrates 6 trajectory samples of the simulated inverted pendulum with 2 steps action delays. The trajectories are generated by the interval of 0.05s from training the ESFQ algorithm with a ridge regression readout layer. From the top, reservoir states with 20 hidden, two the steps delayed action and the genuine action, the cart position, and the pole angle. Each trajectory sample ended when the car position or the pole angle went to the terminal area and then the next trajectory sample began. The cycle index on the X-Axis is corresponding to the sample indexes in the batch dataset. 5.5.3 Very Large Reservoir In the previous section, we observed that the ESFQ algorithm with a linear readout layer would fail to learn complex dynamic task like an inverted pendulum. However, there are arguments that an echo state network with a very large number of hidden units proportional to its input units could capture the complex dynamics of a control system. Therefore, we chose a reservoir with 400 hidden units which are 100 times larger than the state vector of the inverted pendulum. The idea here was that because of randomness in the reservoir weight matrices with a larger number of neurons the system would be able to linearise the higher order nonlinearity in dynamic systems. But according to the literature [29] it is not necessarily true that a reservoir with a larger number of neurons could provide a better estimation of the target values. Furthermore, our experimental results proved that even such a large reservoir would not be able to estimate desired target for reinforcement learning algorithm. In Figure 5.17 we can see the learning curves for the ESFQ algorithm with a nonlinear readout layer and a linear readout layer with a very large number of hidden units in its reservoir. We simply can observe that the ESFQ algorithm with a nonlinear readout layer promises a better performance in the learning of the inverted pendulum containing delays in its executed action. In this experiment, the activation function for hidden units of the reservoir was a logistic regression. We trained our algorithm using hyper-parameter optimization tool SMAC, ridge regression for the 44
- 50. CHAPTER 5. EXPERIMENTS AND RESULTS 5.5. WHY NOT LINEAR READOUT LAYERS readout layer, and 2 steps of action delay in the simulated plant. Figure 5.17: This figure shows the comparison between the learning curve of the Echo State Fitted-Q Iteration algorithms with linear and nonlinear readout layers. The results are averaged over 122 experiments for the simulated inverted pendulum with a delay length of 2. The X-Axis shows the episode index and the Y-Axis show the mean and standard deviation of the goal reach percentage. The mean values are computed from the percentage of goal reach for the corresponding episode over 100 separate experiments for each algorithm. 45
- 51. CHAPTER 6 CONCLUSION 6.1 Summary and Discussion In this work we introduced and experimented with a new memory based Q-learning al- gorithm, Echo State Fitted-Q Iteration (ESFQ), to tackle non-Markovian reinforcement learning problems. We learned to control a system in the presence of unknown delay length in their applied actions which violate the Markov property of a dynamic control system. The ESFQ algorithm employs recurrent neural networks as a Q-function to ap- proximate the utility of the taken action given measured system states. In order to deal with delays, we exploited the short term memory capacity of the echo state networks to keep a history of the raw system states in the reservoir design matrix. We trained and tested our new fitted-Q iteration algorithm using growing batch reinforcement learn- ing on different simulated benchmarks. We showed that it is possible to train the echo state Q-function with linear and nonlinear readout layers i.e. ridge regression and mul- tilayer perceptron. Our experimental results showed that the echo state Q-function with a nonlinear readout layer was more suitable for the task with complex dynamics. Such a function could estimate targets for plants containing action delays up to 30 steps. Further- more, the combination of linear readout layer and very large reservoir in the Q-function did not show a high performance in learning to control complex dynamical systems. The Echo State Fitted-Q Iteration algorithm has various parameters which needed to be carefully tuned to first capture nonlinear dynamics of a control system and also hold long enough memory of a system state in order to deal with delays. Therefore, we em- ployed the sequential Bayesian parameter optimization algorithm implement pySMAC to configure them. The following parameters were optimized: spectral radius, leaking rate, reservoir size, the nonlinear function for reservoir neurons, input weight matrix sparsity, and reservoir matrix sparsity. For each configuration chosen by pySMAC, we trained a particular plant with a constant unknown delay. We compared performances of the best- learned models with standard tapped delay-line Neural Fitted-Q Iteration algorithm on the simulated inverted pendulum benchmark. The major achievement of our novel method was that it outperformed delay-line Neu- ral Fitted-Q Iteration for longer delay length. The delay-line kept the history of state equal to the length of delay up to 30 steps. However these high dimensional input data 46
- 52. CHAPTER 6. CONCLUSION 6.2. FUTURE WORKS confused the multilayer perceptron so then its performance decayed on the test phase for the larger delay length. Furthermore, the delay-line Q-function performed relatively worse on the tests phase for the tasks with an odd number of delay length. Contrary to that, the ESFQ algorithm stayed stable during the test phase with various delay length. Our algorithm stated stable given odd or even action delay length. In addition to that, our method reached the goal area for the different initial position, above 98 percent, in the test phase. The delay-line Neural Fitted-Q Iteration algorithm faced a performance drop when we tried out different random seeds, in the range of [1, 10], for the same delay task. In spite of that, the ESFQ algorithm showed higher performance although it contained more random variables in its weight matrices. Furthermore, this randomness caused rel- atively high variances in performances for both of the NFQ and ESFQ algorithm. We need to consider that the readout layer for both algorithms was similar and the difference was in keeping the history of the delayed states, reservoir or delay-line. Fur- thermore, the echo state approach learned a proper short term memory length via costly parameter tuning i.e. by training it over our reinforcement learning algorithm. But, the delay-line approach had to know about the delay length in advance to adjust the history length. It was possible to optimize the delay-line approach over delay parameter using hyper-parameter tools but it would come with cost of equally high computation time. Last but not least, we showed that classical linear readout layers for reservoirs would not work for the task with complex dynamics. In this respect, we trained echo state Q-Function with 2 different readout layers. The ridged regression performs well on the mountain car benchmark but failed to achieve mentionable results of inverted pendulum task. Therefore, We chose multilayer perceptron as final readout layer and trained it with Rprop algorithm, fast version of Backpropagation. To support our argument for a nonlinear readout layer, we also tried a reservoir with a relatively larger number of neurons but it didn’t learn well on inverted pendulum benchmark. The idea there was, given a very large reservoir and randomness of its weight matrices, it would be possible to linearize any complex dynamics and learn them with linear readout layer. In fact, we tried to expand the input state vector to higher dimensions which were linearly separable using regression methods. But, estimating suitable targets with such large reservoirs was not possible. 6.2 Future Works The major focus of this work was to show that echo state networks are the feasible tool for function approximation in reinforcement learning with delays. But it might be possible to improve our approach. Currently, we are able to learn control task with the unknown delay but to the cost of hyperparameter optimization over training reinforcement learn- ing. It might be possible to train echo state networks in advance such that they learn to predict the next state of the system given the applied action. Generally, recurrent neural networks have shown great ability in predicting temporal sequences and they also have been used for adaptive filtering. Therefore, if we could exploit this potential and train our recurrent network to predict a noise/delay free successor state of the system from its current state and action, we would be able to maintain the Markov property. Such a state prediction could be employed as input for different types of reinforcement 47
- 53. CHAPTER 6. CONCLUSION 6.2. FUTURE WORKS learning algorithms. The idea would be first to collect trajectory samples from the real world system and feed them to the reservoir and then use the simulated teacher state values and let system adapt to those noisy inputs at its best to the teacher values. Then the trained state predictor could be used in controlling dynamic systems with unknown measurement noise. Such idea was not in the scope of this research project but it is one way to improve our method. In addition, it could be possible to circumvent the expensive hyper-parameter opti- mization over reinforcement algorithm in ESFQ algorithm for learning control tasks with delays. The effect of a constant delay length in applied action of a dynamic system can be observed in the difference between consecutive states. It would be possible to exploit this knowledge as a teacher value for training echo state networks. We need to find a configuration for echo state networks that learns these differences optimally. Basically we could use hyper-parameter optimization tools to first find a configuration that has enough short term memory capacity to hold a history for the system with unknown delay. Then having this configuration we could train our Echo State Fitted-Q Iteration algorithm to learn the control task. By doing so we will be able to avoid the costly hyper-parameter optimization over ESFQ algorithm. The possibility of the proposed algorithms remains to be studied. 48
- 54. BIBLIOGRAPHY [1] E. Altman and P. Nain. Closed-loop control with delayed information, volume 20. ACM, 1992. [2] J. R. Bachrach and M. C. Mozer. Connectionist modeling and control of finite state systems given partial state information. 1991. [3] B. Bakker. Reinforcement learning with long short-term memory. In NIPS, pages 1475–1482, 2001. [4] B. Bakker. Reinforcement learning by backpropagation through an lstm model/critic. In Proceedings of the IEEE International Symposium on Approximate Dynamic Pro- gramming and Reinforcement Learning (ADPRL), Honolulu, HI, pages 127–134. Citeseer, 2007. [5] J. Bander, I. White, et al. Markov decision processes with noise-corrupted and delayed state observations. Journal of the Operational Research Society, 50(6):660– 668, 1999. [6] A. Barto and R. Crites. Improving elevator performance using reinforcement learn- ing. Advances in neural information processing systems, 8:1017–1023, 1996. [7] J. Bergstra, D. Yamins, and D. Cox. Making a science of model search: Hyperparam- eter optimization in hundreds of dimensions for vision architectures. In Proceedings of The 30th International Conference on Machine Learning, pages 115–123, 2013. [8] D. P. Bertsekas, D. P. Bertsekas, D. P. Bertsekas, and D. P. Bertsekas. Dynamic programming and optimal control, volume 1. Athena Scientific Belmont, MA, 1995. [9] J. A. Boyan and M. L. Littman. Packet routing in dynamically changing networks: A reinforcement learning approach. Advances in neural information processing systems, pages 671–671, 1994. [10] D. Brooks and C. Leondes. Technical note-markov decision processes with state- information lag. Operations Research, 20(4):904–907, 1972. [11] K. A. Bush. An echo state model of non-markovian reinforcement learning. Colorado State University, 2008. 49
- 55. BIBLIOGRAPHY BIBLIOGRAPHY [12] S. P. Chatzis and Y. Demiris. Echo state gaussian process. Neural Networks, IEEE Transactions on, 22(9):1435–1445, 2011. [13] L. Chrisman. Reinforcement learning with perceptual aliasing: The perceptual dis- tinctions approach. In AAAI, pages 183–188. Citeseer, 1992. [14] K. Doya. Bifurcations in the learning of recurrent neural networks 3. learning (RTRL), 3:17, 1992. [15] N. en die der Geneeskunde and P. B. Bakker. The state of mind reinforcement learning with recurrent neural networks. [16] S. Engelbrecht and K. Katsikopoulos. Planning with delayed state information title2. 1999. [17] D. Ernst, P. Geurts, and L. Wehenkel. Tree-based batch mode reinforcement learn- ing. In Journal of Machine Learning Research, pages 503–556, 2005. [18] R. A. Freeman and P. V. Kokotovic. Robust nonlinear control design: state-space and Lyapunov techniques. Springer Science & Business Media, 2008. [19] B. Hammer and J. J. Steil. Tutorial: Perspectives on learning with rnns. In Proc. ESANN, pages 357–368, 2002. [20] R. Hecht-Nielsen. Theory of the backpropagation neural network. In Neural Net- works, 1989. IJCNN., International Joint Conference on, pages 593–605. IEEE, 1989. [21] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. [22] R. A. Howard. Dynamic programming. Management Science, 12(5):317–348, 1966. [23] G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew. Extreme learning machine: a new learning scheme of feedforward neural networks. In Neural Networks, 2004. Proceedings. 2004 IEEE International Joint Conference on, volume 2, pages 985–990. IEEE, 2004. [24] F. Hutter, H. H. Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration (extended version). Technical report, Technical Report TR-2010-10, University of British Columbia, Computer Science, 2010. [25] F. Hutter, H. H. Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization, pages 507–523. Springer, 2011. [26] L.-J. i Lin. Reinforcement learning with hidden states. In From animals to ani- mats 2: Proceedings of the second international conference on simulation of adaptive behavior, volume 2, page 271. MIT Press, 1993. [27] I. Ilies, H. Jaeger, O. Kosuchinas, M. Rincon, V. Sakenas, and N. Vaskevicius. Step- ping forward through echoes of the past: forecasting with echo state networks. Avaible: http://www. neural-forecastingcompetition. com/downloads/methods/27- NN3 Herbert Jaeger report. pdf, 2007. 50
- 56. BIBLIOGRAPHY BIBLIOGRAPHY [28] H. Jaeger. The ”echo state” approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 148:34, 2001. [29] H. Jaeger. Short term memory in echo state networks. GMD-Forschungszentrum Informationstechnik, 2001. [30] H. Jaeger. Adaptive nonlinear system identification with echo state networks. In Advances in neural information processing systems, pages 593–600, 2002. [31] H. Jaeger. Reservoir riddles: Suggestions for echo state network research. In Neural Networks, 2005. IJCNN’05. Proceedings. 2005 IEEE International Joint Conference on, volume 3, pages 1460–1462. IEEE, 2005. [32] H. Jaeger and H. Haas. Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science, 304(5667):78–80, 2004. [33] H. Jaeger, M. Lukoˇseviˇcius, D. Popovici, and U. Siewert. Optimization and applica- tions of echo state networks with leaky-integrator neurons. Neural Networks, 20(3): 335–352, 2007. [34] K. V. Katsikopoulos and S. E. Engelbrecht. Markov decision processes with delays and asynchronous cost collection. Automatic Control, IEEE Transactions on, 48(4): 568–574, 2003. [35] H. Kwakernaak and R. Sivan. Linear optimal control systems, volume 1. Wiley- interscience New York, 1972. [36] S. Lange, T. Gabel, and M. Riedmiller. Batch reinforcement learning. In Reinforce- ment Learning, pages 45–73. Springer, 2012. [37] L.-J. Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293–321, 1992. [38] L.-J. Lin and T. M. Mitchell. Memory approaches to reinforcement learning in non- Markovian domains. Citeseer, 1992. [39] J. Loch and S. P. Singh. Using eligibility traces to find the best memoryless policy in partially observable markov decision processes. In ICML, pages 323–331, 1998. [40] M. Lukoˇseviˇcius. A practical guide to applying echo state networks. In Neural Networks: Tricks of the Trade, pages 659–686. Springer, 2012. [41] M. LukoˇsEviˇcIus and H. Jaeger. Reservoir computing approaches to recurrent neural network training. Computer Science Review, 3(3):127–149, 2009. [42] W. Maass, T. Natschl¨ager, and H. Markram. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural computation, 14(11):2531–2560, 2002. [43] M. S. Mahmoud. Dynamic control of systems with variable state-delay. International Journal of Robust and Nonlinear Control, 6(2):123–146, 1996. 51
- 57. BIBLIOGRAPHY BIBLIOGRAPHY [44] J. Mockus and L. Mockus. Bayesian approach to global optimization and application to multiobjective and constrained problems. Journal of Optimization Theory and Applications, 70(1):157–172, 1991. [45] R. M. Munos and A. W. Moore. Rates of convergence for variable resolution schemes in optimal control. 2000. [46] D. Ormoneit and P. Glynn. Kernel-based reinforcement learning in average-cost problems. Automatic Control, IEEE Transactions on, 47(10):1624–1636, 2002. [47] M. Oubbati, B. Kord, P. Koprinkova-Hristova, and G. Palm. Learning of embodied interaction dynamics with recurrent neural networks: some exploratory experiments. Journal of neural engineering, 11(2):026019, 2014. [48] H. Palangi, L. Deng, and R. K. Ward. Learning input and recurrent weight matrices in echo state networks. arXiv preprint arXiv:1311.2987, 2013. [49] J. Peters and S. Schaal. Policy gradient methods for robotics. In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on, pages 2219–2225. IEEE, 2006. [50] F. J. Pineda. Generalization of back-propagation to recurrent neural networks. Phys- ical review letters, 59(19):2229, 1987. [51] M. M. Polycarpou, P. Ioannou, et al. A robust adaptive nonlinear control design. In American Control Conference, 1993, pages 1365–1369. IEEE, 1993. [52] D. V. Prokhorov, D. C. Wunsch, et al. Adaptive critic designs. Neural Networks, IEEE Transactions on, 8(5):997–1007, 1997. [53] M. Riedmiller. Neural fitted q iteration–first experiences with a data efficient neural reinforcement learning method. In Machine Learning: ECML 2005, pages 317–328. Springer, 2005. [54] M. Riedmiller. 10 steps and some tricks to set up neural reinforcement controllers. In Neural Networks: Tricks of the Trade (2nd ed.), pages 735–757, 2012. [55] M. Riedmiller and H. Braun. Rprop-a fast adaptive learning algorithm. In Proc. of ISCIS VII), Universitat. Citeseer, 1992. [56] M. Riedmiller, T. Gabel, R. Hafner, and S. Lange. Reinforcement learning for robot soccer. Autonomous Robots, 27(1):55–73, 2009. [57] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. Cognitive modeling, 5:3, 1988. [58] A. M. Sch¨afer. Reinforcement learning with recurrent neural networks. 2008. [59] J. Schmidhuber. Reinforcement learning in markovian and non-markovian environ- ments. 1991. 52
- 58. BIBLIOGRAPHY BIBLIOGRAPHY [60] E. Schuitema, L. Bu¸soniu, R. Babuˇska, and P. Jonker. Control delay in reinforcement learning for real-time dynamic systems: a memoryless approach. In Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on, pages 3226– 3231. IEEE, 2010. [61] B. Sinopoli, L. Schenato, M. Franceschetti, K. Poolla, M. Jordan, S. S. Sastry, et al. Kalman filtering with intermittent observations. Automatic Control, IEEE Transactions on, 49(9):1453–1464, 2004. [62] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. [63] G. Tesauro. Practical issues in temporal difference learning. Springer, 1992. [64] G. Tesauro. Reinforcement learning in autonomic computing: A manifesto and case studies. Internet Computing, IEEE, 11(1):22–30, 2007. [65] S. Timmer and M. Riedmiller. Safe q-learning on complete history spaces. In Machine Learning: ECML 2007, pages 394–405. Springer, 2007. [66] M. H. Tong, A. D. Bickett, E. M. Christiansen, and G. W. Cottrell. Learning grammatical structure with echo state networks. Neural Networks, 20(3):424–432, 2007. [67] C. F. Touzet. Neural reinforcement learning for behaviour synthesis. Robotics and Autonomous Systems, 22(3):251–281, 1997. [68] F. Triefenbach, A. Jalalvand, B. Schrauwen, and J.-P. Martens. Phoneme recogni- tion with large hierarchical reservoirs. In Advances in neural information processing systems, pages 2307–2315, 2010. [69] A. Waibel. Modular construction of time-delay neural networks for speech recogni- tion. Neural computation, 1(1):39–46, 1989. [70] A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. J. Lang. Phoneme recog- nition using time-delay neural networks. Acoustics, Speech and Signal Processing, IEEE Transactions on, 37(3):328–339, 1989. [71] T. J. Walsh, A. Nouri, L. Li, and M. L. Littman. Learning and planning in environ- ments with delayed feedback. Autonomous Agents and Multi-Agent Systems, 18(1): 83–105, 2009. [72] C. J. Watkins and P. Dayan. Q-learning. Machine learning, 8(3-4):279–292, 1992. [73] P. J. Werbos. Backpropagation through time: what it does and how to do it. Pro- ceedings of the IEEE, 78(10):1550–1560, 1990. [74] R. J. Williams and D. Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270–280, 1989. [75] I. B. Yildiz, H. Jaeger, and S. J. Kiebel. Re-visiting the echo state property. Neural networks, 35:1–9, 2012. 53
- 59. BIBLIOGRAPHY BIBLIOGRAPHY [76] X.-M. Zhang, M. Wu, J.-H. She, and Y. He. Delay-dependent stabilization of linear systems with time-varying state and input delays. Automatica, 41(8):1405–1412, 2005. [77] K. Zhou, J. C. Doyle, K. Glover, et al. Robust and optimal control, volume 40. Prentice hall New Jersey, 1996. 54
- 60. DECLARATION I hereby declare, that I am the sole author and composer of my Thesis and that no other sources or learning aids, other than those listed, have been used. Furthermore, I declare that I have acknowledged the work of others by providing detailed references of said work. I hereby also declare, that my Thesis has not been prepared for another examination or assignment, either wholly or excerpts thereof. 16. December 2015