Tutorial on Bayesian optimization

Tutorial on Bayesian optimization
Prof. Dr. Loc Nguyen, PhD, Postdoc
Loc Nguyen’s Academic Network, Vietnam
Email: ng_phloc@yahoo.com
Homepage: www.locnguyen.net
Tutorial on Bayesian optimization - Loc Nguyen
20/04/2023 1

Abstract
Machine learning forks into three main branches such as supervised learning, unsupervised learning, and
reinforcement learning where reinforcement learning is much potential to artificial intelligence (AI)
applications because it solves real problems by progressive process in which possible solutions are
improved and finetuned continuously. The progressive approach, which reflects ability of adaptation, is
appropriate to the real world where most events occur and change continuously and unexpectedly.
Moreover, data is getting too huge for supervised learning and unsupervised learning to draw valuable
knowledge from such huge data at one time. Bayesian optimization (BO) models an optimization problem
as a probabilistic form called surrogate model and then directly maximizes an acquisition function created
from such surrogate model in order to maximize implicitly and indirectly the target function for finding
out solution of the optimization problem. A popular surrogate model is Gaussian process regression model.
The process of maximizing acquisition function is based on updating posterior probability of surrogate
model repeatedly, which is improved after every iteration. Taking advantages of acquisition function or
utility function is also common in decision theory but the semantic meaning behind BO is that BO solves
problems by progressive and adaptive approach via updating surrogate model from a small piece of data at
each time, according to ideology of reinforcement learning. Undoubtedly, BO is a reinforcement learning
algorithm with many potential applications and thus it is surveyed in this research with attention to its
mathematical ideas. Moreover, the solution of optimization problem is important to not only applied
mathematics but also AI.
2
20/04/2023

Table of contents
1. Introduction
2. Bayesian optimization
3. Gaussian process regression
4. Acquisition functions
5. Discussions
3
20/04/2023

1. Introduction
Given target function y = f(x), optimization problem is to find out extremizer x* so that f(x*) gets extreme value y*
= f(x*). As a convention, f(x) is scalar-by-vector function whose output (observed value or evaluated value) y is
scalar and whose variable x is n-dimension vector. The extremizer x* can be minimizer or maximizer so that y* =
f(x*) is minimum or maximum, respectively. Optimization problem is specified as follows:
𝒙∗ = argmin
𝒙
𝑓 𝒙
Or
𝒙∗ = argmax
𝒙
𝑓 𝒙
1.1
If the extremizer x* is local minimizer or local maximizer, the optimization problem is local optimization problem
where traditional methods such as gradient descent are perfect solutions, but they require f(x) is totally
convex for minimization (concave for maximization). The problem becomes much more complex if f(x)
is not totally convex (concave) which leads that x* is global extremizer. This is the global optimization
problem which is mentioned in this research. As usual, x* is minimizer by default, but in context of Bayesian
optimization (BO) it is better to consider x* as maximizer because BO mainly relates to probabilistic distributions
whose peaks are concerned much. However, it is not serious because minimization is inverse of maximization, for
example:
argmin
𝒙
𝑓 𝒙 ∼ argmax
𝒙
−𝑓 𝒙
20/04/2023 Tutorial on Bayesian optimization - Loc Nguyen 4

1. Introduction
Moreover, which one among minimization and maximization can be determined by changing
activation function a little bit in BO. There are three approaches to solve global optimization
problem such as analytic approach, probabilistic approach, and heuristic approach. Analytic
approach applies purely mathematical tools into finding out optimizers such as
approximation, cutting plane, branch and bound, and interval method, in which these
methods focus on analytic essence of algebraic target function. Probabilistic approach
considers looking for optimizers as random selection, but such random selection is
guided by some probabilistic model so as to reach an optimizer. Heuristic approach which
is the most flexible one among three approaches tries to apply or imitate heuristic assumptions
into searching for optimizers. It does not concern much mathematical reasonings because
feasibility and effectiveness are most important. As usual, heuristic approach imitates natural
activities, for example, particle swarm optimization (PSO) simulates how a flock of birds search
for food. Evolutional algorithms like PSO and ant bee colony (ABC) which are inspired
from biological activities are popular methods of heuristic algorithms. However, there
are some implicit connections between heuristic approach (concretely, evolutional
algorithms) and probabilistic approach in a research raised by Luo (Luo, 2019) that I
researched in (Nguyen, 2022).

1. Introduction
Bayesian optimization (BO) belongs to the probabilistic approach. It is based on
Bayesian inference which considers parameter as random variable and updates
posterior probability of parameter based on evidence and prior probability. Because
BO does not impact directly on target function f(x), it must model f(x) as a
probabilistic model and then define an acquisition function for such probabilistic
model. BO solves the optimization problem by maximizing the acquisition function
instead of maximizing the target function. Shortly, two main tasks of BO are:
1. Modeling f(x) by the probabilistic model called surrogate model (Shahriari,
Swersky, Wang, Adams, & Freitas, 2016, pp. 149-150).
2. Defining the acquisition function for the surrogate model so that it is possible to
maximize the acquisition function.
Posterior probability of surrogate model in BO is updated continuously along with
maximizing acquisition function continuously until a maximizer of target function is
reached. Next slide is pseudo code of BO.

1. Introduction
Model f(x) by the surrogate model S(f | Dn) where sample Dn is a set of variable
values xi.
Define the acquisition function α(x | f, S) based on both f(x) and S(f | Dn) whose
variable is x.
Initialize n. Initialize randomly Dn = {x1, x2,…, xn}.
While maximizer x* is not reached or the number of iterations is not many enough
Update posterior probability of S(f | Dn) with sample Dn.
Determine acquisition function α(x | f, S) with S(f | Dn).
Find xn+1 as a maximizer of α(x | f, S) with regard to x.
𝒙𝑛+1 = argmax
𝒙
𝛼 𝒙 𝑓, 𝑆
(Checking whether xn+1 is maximizer x*).
Add xn+1 to sample Dn.
𝑫𝑛 = 𝑫𝑛 ∪ 𝒙𝑛+1
Increase n = n + 1.
End while
In the table above, there is a
question about how to check if a
given xn+1 is the global maximizer
x*. The checking task here is
implicitly executed by BO
applications, for example, if two or
more sequential iterations produce
the same value xn+1 = xn (or small
enough deviation |xn+1 – xn|) such
value xn+1 can be the maximizer x*
or be considered as the maximizer
x* because BO algorithm improves
xn+1 to be higher and higher by
maximizing updated acquisition
function after every iteration.
Because the checking task is not
essential task of BO, it is often not
listed in BO literature. Besides,
how to define acquisition function
can decide which one among
maximization or minimization is
optimization problem.
Table 1.1. BO algorithm

1. Introduction
If the surrogate model S(f | Dn) has explicit parameters related directly to f(x), updating posterior
probability of S(f | Dn) is indeed to update posterior probability of its explicit parameters and
then, BO is called parametric BO (PBO). If the surrogate model S(f | Dn) has no explicit
parameters or its parameters are not related directly to f(x) then, BO is called nonparametric BO
(NBO). As seen in BO algorithm, BO does not concern algebraic formulation of target function
f(x) because it only concerns output y=f(x) and hence, f(x) is a black box with regard to BO. In
other words, f(x) is an arbitrary mapping with subject to BO and so BO is a flexible solution for
global optimization, which has many potential applications. For BO, especially NBO, f(x) is a
black box except its output y = f(x) and so BO maximizes the acquisition function α(x | f, S)
instead of maximizing directly f(x) because BO knows α(x | f, S) which is created from S(f | Dn)
that BO built up before. How to define acquisition function depends on which kind of BO is, for
instance, PBO or NBO, and how to define surrogate model. BO and acquisition function will be
mentioned in the next sections.
In general, the essence of BO is continuous improvement via updating posterior probability,
which follows ideology of reinforcement learning (Shahriari, Swersky, Wang, Adams, & Freitas,
2016, p. 151).

As aforementioned, Bayesian optimization (BO) solves the optimization problem by maximizing the acquisition
function α(x | f, S) which is derived from the surrogate model S(f | Dn) where Dn is the current sample. The surrogate
model S(f | Dn), in turn, was a probabilistic representation of target function f(x). The way to define surrogate model
decides the kind of BO where there are two kinds of BO such as parametric BO (PBO) and nonparametric BO (NBO).
PBO implies that parameters of the surrogate model S(f | Dn) were included explicitly in target function f(x).
Otherwise, the surrogate model S(f | Dn) of NBO has no explicit parameters or its parameters were not included
explicitly in target function f(x). Firstly, PBO is surveyed and then NPO is researched.
The simplest PBO inspired by a system consisting of r events Xk is known Bernoulli system, for example, a lottery
machine or bandit system has r arms represented by r random variables Xk and probability of an arm k yielding lottery
prize is P(Xk=1) (Shahriari, Swersky, Wang, Adams, & Freitas, 2016, pp. 152-153). Such PBO is called Bernoulli
PBO where each binary random variable Xk whose outcome is successful (Xk=1) or failed (Xk=0) follows Bernoulli
distribution whose parameter is θk (0 ≤ θi ≤1).
𝑃 𝑋𝑘 𝜃𝑘 =
𝜃𝑘 if 𝑋𝑘 = 1
1 − 𝜃𝑘 if 𝑋𝑘 = 0
This means:
𝜃𝑘 = 𝑃 𝑋𝑘 = 1 𝜃𝑘 2.1
Note, the general notation P(.) denotes probability function in both discrete case and continuous case.

In Bayesian inference, parameter is considered as random variable which also follows distribution and so suppose each θk
has prior probability P(θk) and hence, the probability P(Xk=1) of event Xk=1 is expectation of Bernoulli distribution P(Xk=1 |
θk) within the prior probability P(θk).
𝐸 𝜃𝑘 = 𝑃 𝑋𝑘 = 1 =
𝜃𝑘
𝑃 𝑋𝑘 = 1 𝜃𝑘 𝑃 𝜃𝑘 𝑑𝜃𝑘 =
𝜃𝑘
𝜃𝑘𝑃 𝜃𝑘 𝑑𝜃𝑘 2.2
Note, notation E(.) denotes expectation of random variable. Going back to the example of lottery machine, P(Xk=1) is the
probability that an arm k yields lottery prize, which is called winning probability of arm k. A gambler does not know exactly
which arm is the best one to win a prize and so he tries to choose an arm k having largest winning probability P(Xk=1).
Consequently, if he wins (Xk=1) then the winning probability P(Xk=1) is increased and otherwise if he loses then P(Xk=1) is
decreased. In the next time, the gambler will continue to choose the arm whose winning probability is largest and so on.
Suppose the gambler chooses arms through n times are considered as sample Dn = {Xv1, Xv2,…, Xvn} where each observation
Xvi indicates that the gambler selects arm vi at the ith time with result Xvi (which equals to 0 or 1). Let Dk denote a sub-sample
of Dn in which Xk is selected and let sk and tk be the numbers of Xk=1 and Xk=0, respectively, which are extracted from Dk.
Suppose Dk obeys identically independent distribution (iid) criterion, likelihood function of Dk is:
𝑃 𝐷𝑘 𝜃𝑘 = 𝜃𝑘
𝑠𝑘 1 − 𝜃𝑘
𝑡𝑘
Sample Dk here is called binomial sample when Xk follows Bernoulli distribution which leads to the event that the likelihood
function of Dk above follows binomial distribution.

Marginal probability of Dk within the prior probability P(θk) is:
𝑃 𝐷𝑘 =
𝜃𝑘
𝑃 𝐷𝑘 𝜃𝑘 𝑃 𝜃𝑘 𝑑𝜃𝑘 = 𝐸 𝜃𝑘
𝑠𝑘 1 − 𝜃𝑘
𝑡𝑘
Posterior probability of θk given sample Dk according to Bayes’ rule is:
𝑃 𝜃𝑘 𝐷𝑘 =
𝑃 𝐷𝑘 𝜃𝑘 𝑃 𝜃𝑘
𝑃 𝐷𝑘
2.3
The probability P(Xk=1 | Dk) of Xk=1 given Dk is.
𝐸 𝜃𝑘 𝐷𝑘 = 𝑃 𝑋𝑘 = 1 𝐷𝑘 =
𝜃𝑘
𝑃 𝑋𝑘 = 1 𝜃𝑘 𝑃 𝜃𝑘 𝐷𝑘 𝑑𝜃𝑘 =
𝜃𝑘
𝜃𝑘𝑃 𝜃𝑘 𝐷𝑘 𝑑𝜃𝑘 2.4
Obviously, for the example of lottery machine, the winning probability of arm k is expectation of parameter θk
within its posterior probability given sample Dk. Therefore, given r parameters as parameter vector Θ = (θ1, θ2,…,
θr)T, the target function of Bernoulli PBO is essentially an index function specified as follows:
𝑓 𝒙 = 𝑘 = 𝑓𝑘 = 𝐸 𝜃𝑘 𝐷𝑘 where 𝑘 = 1, 𝑟 2.5
The optimization problem is specified as follows:
𝑘∗ = argmax
𝑘
𝑓𝑘 = argmax
𝑘
𝐸 𝜃𝑘 𝐷𝑘

Obviously, the target function is defined based on the posterior probability of θk which is now considered as
surrogate model of Bernoulli PBO.
𝑆 𝑓 𝑫𝑛 ≝ 𝑃 𝜃𝑘 𝐷𝑘 2.6
Bernoulli PBO is a PBO because surrogate model and target function share the same parameter θk. Acquisition
function of Bernoulli PBO is the same to target function:
𝛼 𝑘 𝑓, 𝑆 = 𝑓𝑘 = 𝐸 𝜃𝑘 𝐷𝑘 2.7
The problem now is how to specify the posterior probability of θk. Suppose the prior probability of each θk
follows beta distribution:
𝑃 𝜃𝑘 = beta 𝜃𝑘 𝑎𝑘, 𝑏𝑘 =
Γ 𝑎𝑘 + 𝑏𝑘
Γ 𝑎𝑘 Γ 𝑏𝑘
𝜃𝑘
𝑎𝑘−1 1 − 𝜃𝑘
𝑏𝑘−1 2.8
There are two hyper parameters of beta distribution in which ak indicates the number of successful outcomes and
bk indicates the number of failed outcomes in ak + bk trials. Note, Γ(.) denotes gamma function (Neapolitan, 2003,
p. 298) which is continuous form of factorial function, Γ 𝑥 = 0
+∞
𝑡𝑥−1
𝑒−𝑡
d𝑡. Given prior beta distribution, the
probability of Xi=1 is:
𝐸 𝜃𝑘 = 𝑃 𝑋𝑘 = 1 =
𝜃𝑘
𝜃𝑘beta 𝜃𝑘 𝑎𝑘, 𝑏𝑘 𝑑𝜃𝑘 =
𝑎𝑘
𝑎𝑘 + 𝑏𝑘
2.9

Given sub-sample Dk the posterior probability of each θk is:
𝑃 𝜃𝑘 𝐷𝑘 = beta 𝜃𝑘 𝑎𝑘 + 𝑠𝑘, 𝑏𝑘 + 𝑡𝑘 =
Γ 𝑎𝑘 + 𝑠𝑘 + 𝑏𝑘 + 𝑡𝑘
Γ 𝑎𝑘 + 𝑠𝑘 Γ 𝑏𝑘 + 𝑡𝑘
∗ 𝜃𝑘
𝑎𝑘+𝑠𝑘−1
1 − 𝜃𝑘
𝑏𝑘+𝑡𝑘−1
2.10
Where sk and tk be the numbers of Xk=1 and Xk=0, respectively. Recall that such posterior probability is also
called Bernoulli PBO. Because the prior probability P(θk) and the posterior probability P(θk|Dk) have the
same form, they are called conjugate distributions and such conjugation is very important to Bayesian
inference. The probability of Xk=1 given Dk which is also target function is:
𝐸 𝜃𝑘 𝐷𝑘 = 𝑃 𝑋𝑘 = 1 𝐷𝑘 =
𝜃𝑘
𝜃𝑘𝑃 𝜃𝑘 𝐷𝑘 𝑑𝜃𝑘 =
𝑎𝑘 + 𝑠𝑘
𝑎𝑘 + 𝑠𝑘 + 𝑏𝑘 + 𝑡𝑘
2.11
Formulation of surrogate model becomes neat because of the assumption of beta distribution with binomial
sample. When prior probability P(θk) follows beta distribution and Xk follows Bernoulli distribution (so that
likelihood function P(Dk | θk) follows binomial distribution with binomial sample Dk) then we obtain
probabilistic conjugation in which the posterior probability P(θk|Dk) follows beta distribution. Going back
to the example of lottery machine, it is easy to derive Bernoulli PBO with beta distribution shown in the
next slide (Shahriari, Swersky, Wang, Adams, & Freitas, 2016, p. 153):

Initialize hyper parameters ak and bk of r distributions beta(θk | ak, bk).
While the number of iterations is not many enough
Find kn+1 as an index maximizer among r distributions beta(θk | ak, bk) so that E(θk | Dk) is maximum.
𝑘𝑛+1 = argmax
𝑘
𝐸 𝜃𝑘 𝐷𝑘 = argmax
𝑘
𝑎𝑘 + 𝑠𝑘
𝑎𝑘 + 𝑠𝑘 + 𝑏𝑘 + 𝑡𝑘
If 𝑋𝑘𝑛+1
= 1 then increasing 𝑎𝑘𝑛+1
= 𝑎𝑘𝑛+1
+ 1.
Otherwise, if 𝑋𝑘𝑛+1
= 0 then increasing 𝑏𝑘𝑛+1
= 𝑏𝑘𝑛+1
+ 1.
Increase n = n + 1.
End while
Because the expectation E(θk | Dk) is essentially an estimate of θk given binomial sample according Bayesian
statistics, the target function of Bernoulli PBO can be rewritten: 𝑓𝑘 = 𝜃𝑘 where 𝜃𝑘 = 𝐸 𝜃𝑘 𝐷𝑘 . This implies that
target function in Bernoulli PBO is “identified” with probabilistic parameter and hence, for example, given lottery
machine, each arm has no associated properties, which cannot be applied into more complex system.
Table 2.1. Bernoulli PBO algorithm

Therefore, suppose each arm k is associated with a real number vector Xk so that the target function is linear function of
p-dimension Xk and vector parameter W as follows (Shahriari, Swersky, Wang, Adams, & Freitas, 2016, p. 154):
𝑓 𝑘 = 𝑓𝑘 = 𝑊𝑇
𝑋𝑘 where 𝑘 = 1, 𝑟 2.12
Note, the superscript “T” denotes transposition operator of vector and matrix. There are r property vectors {X1, X2,…,
Xr} when the number of arms is r. Suppose variable Y = WTXk distributes normally with mean WTXk and variance σ2 as
follows:
𝑌~𝒩 𝑌 𝑊𝑇
𝑋𝑘, 𝜎2
=
1
2𝜋𝜎2
exp −
𝑌 − 𝑊𝑇
𝑋𝑘
2
2𝜎2
Note, the tilde sign “~” indicates probabilistic distribution of a random variable and notation 𝒩 . . denotes normal
distribution. Essentially, this normal distribution is likelihood function:
𝑃 𝑌 Θ = 𝒩 𝑌 𝑊𝑇
𝑋𝑘, 𝜎2
Normal distribution is also called Gaussian distribution. According to Bayesian inference, suppose parameter W
distributes normally in prior with mean μ0 and covariance matrix Σ0 as follows:
𝑊~𝒩 𝑊 𝜇0, Σ0 = 2𝜋 −
𝑝
2 Σ0
−
1
2exp −
1
2
𝑊 − 𝜇0
𝑇
Σ0
−1
(𝑊 − 𝜇0)
The distribution above is multinormal distribution because W is random vector variable. Multinormal distribution is also
called multivariate normal distribution or multivariate Gaussian distribution.

Suppose parameter σ2 follows inverse gamma distribution in prior with shape parameter α0 and scale
parameter β0 as follows:
𝜎2
~IG 𝜎2
𝛼0, 𝛽0 =
𝛽0
𝛼0
Γ 𝛼0
𝜎2 −𝛼0−1
exp −
𝛽0
𝜎2
Therefore, the parameter of entire system like lottery machine is Θ = θ = {μ, Σ, α, β} and the NBO for
this parameter is called normal – inverse gamma NBO (NIG-NBO) because it combines multinormal
distribution and inverse gamma distribution by product operator. The prior probability of Θ follows
NIG distribution (Shahriari, Swersky, Wang, Adams, & Freitas, 2016, p. 154):
𝑃 Θ = 𝒩 𝑊 𝜇0, Σ0 IG 𝜎2 𝛼0, 𝛽0 2.13
The prior parameter is Θ0 = {μ0, Σ0, α0, β0}. The sample Dn now is Dn = X = (x1
T, x2
T,…, xn
T)T of size n
where xi is some Xk that the gambler selects at the ith time, as follows:
𝑫𝑛 = 𝑿 =
𝒙1
𝑇
𝒙2
𝑇
⋮
𝒙𝑛
𝑇
=
𝑥11 𝑥12 ⋯ 𝑥1𝑝
𝑥21 𝑥22 ⋯ 𝑥2𝑝
⋮ ⋮ ⋱ ⋮
𝑥𝑛1 𝑥𝑛2 ⋯ 𝑥𝑛𝑝
, 𝒙𝑖 =
𝑥𝑖1
𝑥𝑖2
⋮
𝑥𝑖𝑝

Let y = (y1, y2,…, yn)T be the output vector of X with target function where yi = WTxi where 𝒚 =
𝑦1 = 𝑊𝑇
𝒙1
𝑦2 = 𝑊𝑇
𝒙2
⋮
𝑦𝑛 = 𝑊𝑇
𝒙𝑛
.
Fortunately, the posterior probability of Θ given sample Dn is also a NIG distribution which is the probabilistic
conjugation with the prior probability. Of course, such posterior probability is NIG-PBO surrogate model.
𝑆 𝑓 𝑫𝑛 ≝ 𝑃 Θ 𝑫𝑛 = 𝒩 𝑊 𝜇𝑛, Σ𝑛 IG 𝜎2
𝛼𝑛, 𝛽𝑛 2.14
The posterior parameter is Θn = {μn, Σn, αn, βn}. Fortunately, Θn was calculated from Dn (Shahriari, Swersky, Wang,
Adams, & Freitas, 2016, p. 154).
𝜇𝑛 = Σ𝑛 Σ0
−1
𝜇0 + 𝑿𝑇
𝒚
Σ𝑛 = Σ0
−1
+ 𝑿𝑇
𝑿
𝛼𝑛 = 𝛼0 + 𝑛 2
𝛽 = 𝛽0 +
1
2
𝜇0
𝑇
Σ0
−1
𝜇0 + 𝑦𝑇
𝑦 − 𝜇𝑛
𝑇
Σ𝑛
−1
𝜇𝑛
2.15
Acquisition function of NIG-PBO is defined from target function and the posterior mean μn:
𝛼 𝑘 𝑓, 𝑆 = 𝛼 𝑘 𝜇𝑛 = 𝜇𝑛
𝑇
𝑋𝑘 2.16
Note, α(k|μn) is a function of index k. When Θn is determined, it is easy to derive NIG-NBO algorithm shown in the next
slide.

Initialize prior parameters Θ0 = {μ0, Σ0, α0, β0}.
While the number of iterations is not many enough
Update posterior parameter Θn = {μn, Σn, αn, βn} from X=Dn.
𝜇𝑛 = Σ𝑛 Σ0
−1
𝜇0 + 𝑿𝑇𝒚
Σ𝑛 = Σ0
−1
+ 𝑿𝑇𝑿
𝛼𝑛 = 𝛼0 + 𝑛 2
𝛽 = 𝛽0 +
1
2
𝜇0
𝑇
Σ0
−1
𝜇0 + 𝑦𝑇𝑦 − 𝜇𝑛
𝑇Σ𝑛
−1𝜇𝑛
Find kn+1 as an index maximizer of the acquisition function α(k|μn) among r property vectors Xk.
𝑘𝑛+1 = argmax
𝑘
𝜇𝑛
𝑇𝑋𝑘
Add 𝑋𝑘𝑛+1
to sample Dn.
𝑫𝑛 = 𝑫𝑛 ∪ 𝑋𝑘𝑛+1
Increase n = n + 1.
End while
Table 2.2. NIG-PBO algorithm

When f(x) does not have specific (explicit) aspect or property which becomes an explicit parameter of the surrogate
model S(f | Dn), the corresponding BO becomes nonparametric BO (NBO). The most popular technique to establish S(f |
Dn) for NBO is to use Gaussian process regression (GPR) for modeling S(f | Dn). In other words, GPR is a surrogate
model of NBO. This research focuses on NBO with GPR. Because kernel function is very important to GPR when it is
used to not only build up GPR but also make GPR line smoother. Kernel function measures similarity between two
variables (two points), according to that, the closer the two variables are, the larger their kernel function is. Let Σ(xi, xj)
denote kernel function of two variables xi and xj, for example, a popular kernel function is simple squared exponential
function.
Σ𝑆𝐸 𝒙𝑖, 𝒙𝑗 = exp −
𝒙𝑖 − 𝒙𝑗
2
2
In this research the notation |.| can denote absolute value of scalar, length (module) of vector, determinant of matrix, and
cardinality of set. Concretely, kernel function Σ(xi, xj) is used to define covariance function in GPR. As a convention, let
xi:j where i≤j denote a sequential subset of X such that xi:j = {xi, xi+1,…, xj}. Of course, we have xi:i = xi. Given two
subsets xi:j and xk:l, their covariance function is:
Σ 𝒙𝑖:𝑗, 𝒙𝑘:𝑙 =
Σ 𝒙𝑖, 𝒙𝑘 Σ 𝒙𝑖, 𝒙𝑘+1 ⋯ Σ 𝒙𝑖, 𝒙𝑙
Σ 𝒙𝑖+1, 𝒙𝑘 Σ 𝒙𝑖+1, 𝑋𝑘+1 ⋯ Σ 𝒙𝑖+1, 𝒙𝑙
⋮ ⋮ ⋱ ⋮
Σ 𝒙𝑗, 𝒙𝑘 Σ 𝒙𝑗, 𝒙𝑘+1 ⋯ Σ 𝒙𝑗, 𝒙𝑙
2.17

Output of a covariance function is a covariance matrix if such output is invertible and symmetric. For NBO, given sample Dn
= x1:n = {x1, x2,…, xn}, the consequential sequence y1:n = f(x1:n) = f(Dn) = {y1 = f(x1), y2 = f(x2),…, yn = f(xn)} is established to
distribute normally with prior probability density function (prior PDF) as follows:
𝑦1:𝑛 𝒙1:𝑛 ∼ 𝑓 𝑦1:𝑛 𝒙1:𝑛, 𝜇 . , Σ . = 𝒩 𝑦1:𝑛 𝜇 𝑦1:𝑛 , Σ 𝒙1:𝑛, 𝒙1:𝑛
Where, in this research, notation f(.|.) often denotes probability density function (PDF), which is not concrete function and
notation 𝒩 . . denotes normal distribution, with note that the general notation P(.) denotes probability function in both
discrete case and continuous case whereas the term “PDF” focuses on probability function of continuous variable. Note, μ(.)
and Σ(.) are mean function and covariance function, respectively. In this special case, covariance function Σ(.) is matrix
function defined based on kernel function; exactly Σ(.) is matrix-by-vector function if each xi is vector.
Σ 𝒙1:𝑛, 𝒙1:𝑛 =
Σ 𝒙1, 𝒙1 Σ 𝒙1, 𝒙2 ⋯ Σ 𝒙1, 𝒙𝑛
Σ 𝒙2, 𝒙1 Σ 𝒙2, 𝒙2 ⋯ Σ 𝒙2, 𝒙𝑛
⋮ ⋮ ⋱ ⋮
Σ 𝒙𝑛, 𝒙1 Σ 𝒙𝑛, 𝒙2 ⋯ Σ 𝒙𝑛, 𝒙𝑛
Of course, each element Σ(xi, xj) of covariance function Σ(x1:n, x1:n) is kernel function. Recall that
Σ 𝒙, 𝒙1:𝑛 =
Σ 𝒙, 𝒙1
Σ 𝒙, 𝒙2
⋮
Σ 𝒙, 𝒙𝑛
𝑇
, Σ 𝒙1:𝑛, 𝒙 =
Σ 𝒙1, 𝒙
Σ 𝒙2, 𝒙
⋮
Σ 𝒙𝑛, 𝒙

It is popular to define μ(y1:n) based on μ(y) separately: 𝜇 𝑦1:𝑛 =
𝜇 𝑦1
𝜇 𝑦2
⋮
𝜇 𝑦𝑛
. Moreover, for BO, mean function μ(.)
is often set to be zero as follows: 𝜇 𝑦 = 0 so that 𝜇 𝑦1:𝑛 =
𝜇 𝑦1 = 0
𝜇 𝑦2 = 0
⋮
𝜇 𝑦𝑛 = 0
= 𝟎𝑇. Given variable x, GPR
surrogate model is represented by the posterior PDF of y = f(x) given y1:n, x1:n, and x as follows:
𝑦 𝑦1:𝑛, 𝒙1:𝑛, 𝒙 ∼ 𝑓 𝑦 𝑦1:𝑛, 𝒙1:𝑛, 𝒙, 𝜇 . , Σ . = 𝒩 𝑦 𝜇 𝑦 𝑦1:𝑛, 𝒙1:𝑛, 𝒙 , 𝜎2
𝑦 𝒙1:𝑛, 𝒙
This posterior PDF is derived from interesting properties of normal distribution which will be mentioned in the
next section. Note that μ(y | y1:n, x1:n, x) and σ2(y | x1:n, x) are mean and variance of the multinormal posterior PDF
of y given y1:n, x1:n, and x, respectively.
𝜇 𝑦 𝑦1:𝑛, 𝒙1:𝑛, 𝒙 = 𝜇 𝑦 + Σ 𝒙, 𝒙1:𝑛 Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
𝑦1:𝑛 − 𝑚 𝑦1:𝑛
𝜎2 𝑦 𝒙1:𝑛, 𝒙 = Σ 𝒙, 𝒙 − Σ 𝒙, 𝒙1:𝑛 Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
Σ 𝒙1:𝑛, 𝒙
Note, (Σ(x1:n, x1:n))–1 denotes inverse of covariance matrix (output of covariance function) Σ(x1:n, x1:n).

Please pay attention that m(y1:n) is a realistic mean which is recurrently determined by previously known μ(y | y1:n–1,
x1:n–1, x):
𝑚 𝑦𝑛 = 𝜇 𝑦 𝑦1:𝑛−1, 𝒙1:𝑛−1, 𝒙𝑛
The variance σ2(y | x1:n, x) is function of only x and so, in practice, mean function μ(y) is set to be zero so that the
mean function μ(y | y1:n, x1:n, x) is also function of only x too, as follows (Frazier, 2018, p. 4):
𝜇𝑛 𝒙 = 𝜇 𝑦 𝑦1:𝑛, 𝒙1:𝑛, 𝒙 = Σ 𝒙, 𝒙1:𝑛 Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
𝑦1:𝑛 − 𝑚 𝑦1:𝑛
𝜎𝑛
2
𝒙 = 𝜎2
𝑦 𝒙1:𝑛, 𝒙 = Σ 𝒙, 𝒙 − Σ 𝒙, 𝒙1:𝑛 Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
Σ 𝒙1:𝑛, 𝒙
Please pay attention that m(y1:n) is a realistic mean which is recurrently determined by previously known μn–1(x):
𝑚 𝑦𝑛 = 𝜇𝑛−1 𝒙𝑛 2.18
The event that both μn(x) and σn
2(x) are functions of only x is necessary to determine acquisition function of BO
later with note that x1:n and y1:n were known. GPR surrogate model is rewritten:
𝑦 𝒙1:𝑛, 𝒙 ∼ 𝑓 𝑦 𝒙1:𝑛, 𝜇 . , Σ . = 𝒩 𝑦 𝜇𝑛 𝒙 , 𝜎𝑛
2
𝒙 = 𝑆 𝑓 𝑫𝑛 2.19
Where,
𝜇𝑛 𝒙 = Σ 𝒙, 𝒙1:𝑛 Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
𝑦1:𝑛 − 𝑚 𝑦1:𝑛
𝜎𝑛
2
𝒙 = Σ 𝒙, 𝒙 − Σ 𝒙, 𝒙1:𝑛 Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
Σ 𝒙1:𝑛, 𝒙
2.20

Therefore, the acquisition function α(x | f, S) which will be based on μn(x) and
σn
2(x) is denoted as follows:
𝛼 𝒙 𝑓, 𝑆 = 𝛼 𝒙 𝜇𝑛 𝒙 , 𝜎𝑛
2
𝒙
Indeed, GPR is represented by the two parameters μn(x) and σn
2(x) but such
parameters are not included in the target function f(x) and so this is a NBO.
Given acquisition function α(x | μn(x), σn
2(x)) based on μn(x) and σn
2(x), and
also known sample Dn = x1:n = {x1, x2,…, xn}, let xn+1 be a maximizer of α(x |
μn(x), σn
2(x)) with regard to x and hence xn+1 will be updated continuously
after every iteration until it reaches the entire maximizer x* of f(x).
𝒙𝑛+1 = argmax
𝒙
𝛼 𝒙 𝜇𝑛 𝒙 , 𝜎𝑛
2
𝒙 2.21
As a result, the pseudo code of NBO with GPR is finetuned as shown in the
next slide.

Initialize randomly Dn = x1:n = {x1, x2,…, xn}.
Initialize m(y1:n) = μ(y1:n).
While maximizer x* is not reached or the number of iterations is not many enough
Update posterior mean μn(x) and variance σn
2(x) of GPR with sample Dn as follows:
−1
𝑦1:𝑛 − 𝑚 𝑦1:𝑛
𝜎𝑛
2 𝒙 = Σ 𝒙, 𝒙 − Σ 𝒙, 𝒙1:𝑛 Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
Σ 𝒙1:𝑛, 𝒙
Determine acquisition function α(x | μn(x), σn
2(x) of GPR.
Find xn+1 as a maximizer of α(x | μn(x), σn
2(x)) with regard to x.
𝒙𝑛+1 = argmax
𝒙
2
𝒙
Set
𝑚 𝑦𝑛+1 = 𝜇𝑛 𝒙𝑛+1
Add xn+1 to sample Dn.
𝑫𝑛 = 𝑫𝑛 ∪ 𝒙𝑛+1 = 𝒙𝑖: 𝑛+1
Increase n = n + 1.
End while
Table
2.3.
NBO
algorithm
with
GPR

In general, there are two important tasks of NBO in which the first one is to determine the
posterior mean μn(x) and variance σn
2(x) of GPR. The second one is to specify the
acquisition function α(x | μn(x), σn
2(x)) which is described shortly here. Detailed description
of GPR and acquisition function will be mentioned in the next section. Note that α(x | μn(x),
σn
2(x)) is function of x. Recall that BO maximizes the acquisition function α(x | f, S) so as to
search for maximizer x* because target function f(x) is assumed to be a black box for BO
and BO creates previously surrogate model from which acquisition function is derived later.
Acquisition function is especially important to NBO because NBO does not know f(x) and
parameters of NBO surrogate model are not relevant to f(x). Acquisition function may not
be strict with PBO but it is very strict with NBO. Moreover, finding maximizer of
acquisition function should be cheaper than finding maximizer of target function f(x) so that
researchers in optimization domain will pay more attention to BO. There are some
acquisition functions, for example, probability of improvement, expected improvement,
entropy search, and upper confidence bound but expected improvement (EI) is the most
popular one. EI is mentioned here and other ones are described in the next section.

Given sample Dn = x1:n = {x1, x2,…, xn} and its evaluations y1:n = {y1, y2,…, yn}, let yn
* = max{y1, y2,…, yn } be
the temporal maximum at current iteration of NBO algorithm. EI acquisition function is determined as follows:
𝛼EI 𝒙 = 𝛼 𝒙 𝜇𝑛 𝒙 , 𝜎𝑛
2
𝒙 = 𝜇𝑛 𝒙 − 𝑦𝑛
∗
Φ
𝜇𝑛 𝒙 − 𝑦𝑛
∗
𝜎𝑛 𝒙
+ 𝜎𝑛 𝒙 𝜙
∗
𝜎𝑛 𝒙
Where σn(x) is called standard deviation of y given y1:n, x1:n, and x.
𝜎𝑛 𝒙 = 𝜎𝑛
2
𝒙
The most important here is that ϕ(.) and Φ(.) are standard normal PDF and standard normal cumulative
distribution function (standard normal CDF). Standard normal distribution is also called standard Gaussian
distribution.
𝜙 𝑧 =
1
2𝜋
exp −
𝑧2
2
Φ 𝑧0 = 𝜙 𝑧 < 𝑧0 = 𝜙 𝑧 ≤ 𝑧0 =
−∞
𝑧0
𝜙 𝑧 𝑑𝑧
Mean and variance of standard normal distribution are 0 and 1, respectively. Moreover, values of standard normal
CDF were evaluated in practice.

As a result, the maximizer xn+1 of EI acquisition function in NBO algorithm at a
current iteration is determined as follows:
𝒙𝑛+1 = argmax
𝒙
𝛼EI 𝒙
= argmax
𝒙
∗
Φ
∗
𝜎𝑛 𝒙
∗
𝜎𝑛 𝒙
2.22
Recall that defining and maximizing acquisition function is one of two most
important tasks in BO whereas the other one is to build up surrogate model. The
essential criterion is that maximizing acquisition function like EI should be
cheaper than maximizing analytic target function f(x) by analytic methods.
Anyhow, BO is always significant because f(x) can be arbitrary or have no
analytic formulation for BO.

Nonparametric BO is based on Gaussian process regression (GPR) which, in turn, is based on Gaussian
process. Therefore, we start the description of GPR with a concept of Gaussian process. Given a random
process X = (X1, X2,…, Xn) over n timepoints in which each Xt where t belongs to {1, 2,…, n} is random
variable then, X is called Gaussian random process or Gaussian process (GP) in brief if and only if for any
finite index set {t1, t2,…, tk} of {1, 2,…, n} where tj belongs to {1, 2,…, n}, the subset 𝑋𝑡1
, 𝑋𝑡2
, … , 𝑋𝑡𝑘
considered as tk-dimension random vector 𝑋𝑡1
, 𝑋𝑡2
, … , 𝑋𝑡𝑘
𝑇
follows multinormal distribution known as
multivariate Gaussian distribution. Note, each 𝑋𝑡𝑗
represents one dimension of the k-dimension random vector
variable 𝑋𝑡1
, 𝑋𝑡2
, … , 𝑋𝑡𝑘
𝑇
. Moreover, please pay attention that any combination of 𝑋𝑡1
, 𝑋𝑡2
, … , 𝑋𝑡𝑘
follows
multinormal distribution too. Without loss of generality, we denote the random process as random variable X =
(X1, X2,…, Xn)T where {t1, t2,…, tk} = {1, 2,…, n} obeying multinormal distribution also called multivariate
normal distribution or multivariate Gaussian distribution as follows:
𝑿 = 𝑋1, 𝑋2, … , 𝑋𝑛
𝑇 ∼ 𝒩 𝑿 𝜇 𝑿 , Σ 𝑿 3.1
Where μ(X) and Σ(X) are mean function and covariance function of X, respectively. Note, the superscript “T”
denotes transposition operator of vector and matrix whereas the tilde sign “~” indicates probabilistic
distribution of a random variable.

GP is known as infinite multinormal distribution which is the generalized form of n-dimension multinormal distribution
because n can approach positive infinity n = +∞. Let f(X | μ(X), Σ(X) be probability density function (PDF) of X when X is
continuous, we have:
𝑓 𝑿 𝜇 𝑿 , Σ 𝑿 = 𝒩 𝑿 𝜇 𝑿 , Σ 𝑿
Where mean μ(X) and covariance Σ(X) are functions of X, respectively. Note, in this research, notation f(.|.) often denotes
PDF, which is not concrete function, and notation 𝒩 . . denotes normal distribution. Probability function, distribution
function, cumulative distribution function (CDF), and PDF are terms which can be exchangeable in some cases, but the term
“PDF” focuses on probability function of continuous variable whereas the general notation P(.) denotes probability function
in both discrete case and continuous case. In literature, μ(X) is assumed to be zero as μ(X) = 0T for convenience but it can be
defined as the random process X itself, μ(X) = X. Besides, μ(X) can be customized according to concrete applications, for
example it can be constant as μ(X) = μ. However, it is better to set μ(X) = 0T for my opinion. As a convention in Gaussian
process, output of a mean function is a mean and so μ(X) also denotes the theoretical mean of X. In general case, μ(X) is
vector function of combination of X1, X3,…, and Xn (s) belong to X but, as a convention, each μ(Xi) depends only Xi and
moreover, μ(Xi) has the same formulation of every Xi. Therefore, μ(X) should be identified with μ(Xi) or μ(X) where X
denotes any Xi belonging to X.
𝜇 𝑿 should be
𝜇 𝑋1
𝜇 𝑋2
⋮
𝜇 𝑋𝑛

Covariance function Σ(X) measures the correlation between random variables when the random process “moves”
them, in which the closer the given two random variables are, the larger their covariance. The two most important
properties based on covariance function Σ(X) of random process are stationarity and isotropy among four basic
properties (Wikipedia, 2003): stationarity, isotropy, smoothness, and periodicity. Stationarity implies that the PDF
f(X | μ(X) of random process X will not be changed when the process is moved in time, for example, if new
random variable Xn+1 raises to be added then means and covariances of old (previous) variables Xi where 1 ≤ i ≤ n
in X will not be changed. It is proved that if GP X satisfies stationarity, Σ(X) will depend only on the deviation Xi–
Xj but the inversed statement is not asserted. However, if Σ(X) depends only on the Euclidean distance |Xi–Xj| then,
GP X will satisfy isotropy. If X satisfies both stationarity and isotropy, X is called homogeneous process. In cases
where each element of matrix function Σ(X) depends on only Xi and Xj like stationarity case and isotropy, it will
result out a following matrix.
Σ 𝑿 = Σ 𝑿, 𝑿 =
Σ 𝑋1, 𝑋1 Σ 𝑋1, 𝑋2 ⋯ Σ 𝑋1, 𝑋𝑛
Σ 𝑋2, 𝑋1 Σ 𝑋2, 𝑋2 ⋯ Σ 𝑋2, 𝑋𝑛
⋮ ⋮ ⋱ ⋮
Σ 𝑋𝑛, 𝑋1 Σ 𝑋𝑛, 𝑋2 ⋯ Σ 𝑋𝑛, 𝑋𝑛
3.2
In these cases, covariance function Σ(X) can be “identified” with its element function Σ(Xi, Xj) when the
formulation of Σ(Xi, Xj) is not changed formally and hence, Σ(Xi, Xj) is called kernel function.

Please pay attention that the term “covariance function” is slightly different from the term “kernel function”. When
referring to only two variables Xi and Xj, they are the same. When referring to one or two sets of variables such as X and
X*, the notations Σ(X, X*), Σ(X, X), or Σ(X) mentions covariance function as matrix function whose elements are defined
by kernel function if each element depends on only two variables like Xi and Xj. Exactly Σ(.) is matrix-by-vector function
if each Xi is vector. Output of a covariance function is a covariance matrix if such output is invertible and symmetric; in
this case, covariance function is “identified” with covariance matrix. For explanation, suppose each Xi belonging to X is
scalar (so that X is vector) and the output of covariance function Σ(X) is covariance matrix, the multinormal PDF of X if
formulated as follows:
𝑓 𝑿 𝜇 𝑿 , Σ 𝑿 = 𝒩 𝑿 𝜇 𝑿 , Σ 𝑿 = 2𝜋 −
𝑛
2 Σ 𝑿 −
1
2exp −
1
2
𝑿 − 𝜇 𝑿
𝑇
Σ 𝑿 −1
(𝑿 − 𝜇 𝑿 )
However, each Xi in GPR is arbitrary, such as scalar, vector, and matrix. Note, Σ(X)–1 denotes inverse of covariance
matrix Σ(X). Kernel function is not only essential to define covariance function of Gaussian process but also used to
make GPR line smoother. The following are some kernel functions (Wikipedia, 2003).
Linear: Σ𝐿 𝑋𝑖, 𝑋𝑗 = 𝑋𝑖
𝑇
𝑋𝑗
Squared exponential: Σ𝑆𝐸 𝑋𝑖, 𝑋𝑗 = exp −
𝑋𝑖 − 𝑋𝑗
2
2𝑙2
Where l is the characteristic length-scale of the process which reinforces similarity of Xi and Xj. In this research the
notation |.| denotes absolute value of scalar, length (module) of vector, determinant of matrix, and cardinality of set.

As a convention, let Xi:j where i≤j denote a sequential subset of X such that Xi:j = {Xi, Xi+1,…,
Xj}. Of course, we have Xi:i = Xi. In general case, Xi:j is arbitrary subset of distinguish
variables. Let Σ(Xi, Xj:k) denote a row-vector covariance function of Xi and Xj:k as follows:
Σ 𝑋𝑖, 𝑋𝑗:𝑘 = Σ 𝑋𝑖, 𝑋𝑗 , Σ 𝑋𝑖, 𝑋𝑗+1 , … , Σ 𝑋𝑖, 𝑋𝑘 =
Σ 𝑋𝑖, 𝑋𝑗
Σ 𝑋𝑖, 𝑋𝑗+1
⋮
Σ 𝑋𝑖, 𝑋𝑘
𝑇
Let Σ(Xj:k, Xi) denote a column-vector covariance function of Xj:k and Xi as follows:
Σ 𝑋𝑗:𝑘, 𝑋𝑖 =
Σ 𝑋𝑗, 𝑋𝑖
Σ 𝑋𝑗+1, 𝑋𝑖
⋮
Σ 𝑋𝑘, 𝑋𝑖

As a convention, let Σ(Xi:j, Xk:l) denote a covariance function of Xi:j and Xk:l as follows:
Σ 𝑋𝑖:𝑗, 𝑋𝑘:𝑙 =
Σ 𝑋𝑖, 𝑋𝑘 Σ 𝑋𝑖, 𝑋𝑘+1 ⋯ Σ 𝑋𝑖, 𝑋𝑙
Σ 𝑋𝑖+1, 𝑋𝑘 Σ 𝑋𝑖+1, 𝑋𝑘+1 ⋯ Σ 𝑋𝑖+1, 𝑋𝑙
⋮ ⋮ ⋱ ⋮
Σ 𝑋𝑗, 𝑋𝑘 Σ 𝑋𝑗, 𝑋𝑘+1 ⋯ Σ 𝑋𝑗, 𝑋𝑙
3.3
Obviously, the output of Σ(Xi:j, Xk:l) cannot be a covariance matrix if it is not squared. Note, Σ(Xi:j, Xk:l) is a partition
of Σ(X) and we have Σ(X) = Σ(Xi:n, Xi:n). If denoting X1 = (Xi, Xi+1,…, Xj)T and X2 = (Xk, Xk+1,…, Xl)T, we denote:
Σ 𝑿1, 𝑿2 = Σ 𝑋𝑖:𝑗, 𝑋𝑘:𝑙
Σ 𝑿1, 𝑿1 = Σ 𝑿1 = Σ 𝑋𝑖:𝑗, 𝑋𝑖:𝑗 = Σ 𝑋𝑖:𝑗
3.4
Gaussian process repression (GPR) is based on Gaussian process (GP) when there is a target function which
attaches to each X where X represents any Xi in X such that Y = f(X). Please distinguish the target function f from
the formal notations f(.|.) of probability density function (PDF). Of course, Y or Yi is also random variable and we
also have Yi = f(Xi). Besides, in context of regression model, the target function f(X) is not a formal function with
arithmetic operators and exactly, it is a mapping between X and Y. For example, sample {X, Y} has two paired
datasets X and Y in which for every Xi belonging to X there is a Yi belong to Y and hence, the equation Y = f(X) only
indicates such mapping.

GPR model tries to represent or draw a regressive PDF of Y from its previous ones (which will be
explained later) and X. Suppose we had a GP X = (X1, X2,…, Xn)T and target function Y = f(X),
assuming that the prior PDF of Y = (Y1, Y2,…, Yn)T = f(X) given X is also derived from the
multinormal PDF of X.
𝒀 𝑿 ← 𝑓 𝑿 𝜇 𝑿 , Σ 𝑿 = 𝒩 𝑿 𝜇 𝑿 , Σ 𝑿
Therefore, we have:
𝒀 𝑿 ∼ 𝑓 𝒀 𝑿, 𝜇 . , Σ . = 𝒩 𝒀 𝜇 𝒀 , Σ 𝑿, 𝑿 3.5
Where μ(.) and Σ(.) denote mean function and covariance function in formality, respectively, and Σ(X,
X) = Σ(X). The equation above implies that the prior PDF of GP Y is initialized with the PDF of GP X.
Mean vector and covariance matrix of Y are μ(Y) and Σ(X, X) within the PDF f(Y | X, μ(.), Σ(.)),
respectively. The mean function μ(Y) is redefined here with variable Y but it should be the same to the
mean function μ(X) in formulation if μ(X) can be extended with the lower/higher dimensional space.
For instance, if μ(X) is defined as identity function μ(X) = X then, μ(Y) will be defined as identity
function μ(Y) = Y = (Y1, Y2,…, Yn)T. For simplicity, μ(Y) is often set to be zero as μ(Y) = 0T. It is also
better to set μ(Y) = 0T.

Now suppose we randomize a new set of variables X* = (Xn+1, Xn+2,…, Xn+k)T and obtain their
evaluated values Y* = f(X*) = (Yn+1, Yn+2,…, Yn+k)T. Similarly, mean vector and covariance matrix of
Y* are μ(Y*) and Σ(X*, X*) within the PDF f(Y* | X*, μ(.), Σ(.)), respectively. Of course, the mean
function μ(.) for both Y and Y* is the same. Consequently, the joint PDF of Y and Y* which is of
course multinormal distribution is denoted as follows (Wang, 2022, p. 11):
𝑓 𝒀, 𝒀∗ 𝑿, 𝑿∗, 𝜇 . , Σ . = 𝒩 𝒀
𝜇 𝒀
𝜇 𝒀∗ ,
Σ 𝑿, 𝑿 Σ 𝑿, 𝑿∗
Σ 𝑿∗
, 𝑿 Σ 𝑿∗
, 𝑿∗ 3.6
Where mean vector and covariance matrix of the joint PDF of Y and Y* are
𝜇 𝒀
𝜇 𝒀∗ and
Σ 𝑿, 𝑿 Σ 𝑿, 𝑿∗
Σ 𝑿∗, 𝑿 Σ 𝑿∗, 𝑿∗ , respectively. The equation above is one of interesting properties of
multinormal distribution (Hardle & Simar, 2013, pp. 156-158). As a result, Gaussian process
regression (GPR) model of dependent variable Y is conditional PDF of Y* given Y, X, and X* such
as f(Y* | Y, X, X*, μ(.), Σ(.)) which is also predictive PDF of Y* because it can be used to predict
next occurrence of Y.

From interesting properties of multinormal distribution (Hardle & Simar, 2013, pp. 156-158), it is
easy to drawn GPR from the joint PDF f(Y, Y* | Y, X, X*, μ(.), Σ(.)) as follows:
𝒀∗
𝒀, 𝑿, 𝑿∗
∼ 𝑓 𝒀∗
𝒀, 𝑿, 𝑿∗
, 𝜇 . , Σ . = 𝒩 𝒀 𝜇 𝒀∗
𝒀, 𝑿, 𝑿∗
, Σ 𝒀∗
𝑿, 𝑿∗
3.7
Where mean vector μ(Y* | Y, X, X*) and covariance matrix Σ(Y* | X, X*) are (Wang, 2022, p. 11):
𝜇 𝒀∗
𝒀, 𝑿, 𝑿∗
= 𝜇 𝒀∗
+ Σ 𝑿∗
, 𝑿 Σ 𝑿, 𝑿
−1
𝒀 − 𝜇 𝒀
Σ 𝒀∗
𝑿, 𝑿∗
= Σ 𝑿∗
, 𝑿∗
− Σ 𝑿∗
−1
Σ 𝑿, 𝑿∗
When Y* is variable then, μ(Y*) is still mean function but μ(Y) was determined and hence, for
making clear this vagueness, let m(Y) replace μ(Y) so that mean vector μ(Y* | Y, X, X*) and
covariance matrix Σ(Y* | X, X*) are rewritten as follows:
𝜇 𝒀∗
𝒀, 𝑿, 𝑿∗
= 𝜇 𝒀∗
+ Σ 𝑿∗
−1
𝒀 − 𝑚 𝒀
Σ 𝒀∗
𝑿, 𝑿∗
= Σ 𝑿∗
, 𝑿∗
− Σ 𝑿∗
−1
Σ 𝑿, 𝑿∗
3.8

The quantity m(Y) is a realistic mean. At the beginning of any GPR algorithm, it is defined as the same to the mean
function μ(.), for example, m(Y) = μ(Y) = 0T, m(Y) = μ(Y) = Y, etc., but, at the later phase of any GPR algorithm, it is
recurrently determined by previously known μ(Y | Y1:n0, X1:n0, X) as follows:
𝑚 𝒀 = 𝜇 𝒀 𝑌1:𝑛0
, 𝑋1:𝑛0
, 𝑿 =
𝜇 𝑌1 𝑌1:𝑛0
, 𝑋1:𝑛0
, 𝑋1
𝜇 𝑌2 𝑌1:𝑛0
, 𝑋1:𝑛0
, 𝑋2
⋮
𝜇 𝑌𝑛 𝑌1:𝑛0
, 𝑋1:𝑛0
, 𝑋𝑛
3.9
Obviously, the GPR of Y* distributes normally with mean vector μ(Y* | Y, X, X*) and covariance matrix Σ(Y* | X, X*).
Indeed, the GPR f(Y* | Y, X, X*, μ(.), Σ(.)) is posterior PDF of Y whose prior PDF is f(Y | X, μ(.), Σ(.)). Initializing such
prior PDF by the PDF of X as f(Y | X, μ(.), Σ(.)) = f(X | μ(.), Σ(.)) is not totally correct but implicit biases will be decreased
after the posterior PDF f(Y* | Y, X, X*, μ(.), Σ(.)) is updated. We have following summary:
Prior: 𝒀 𝑿 ∼ 𝒩 𝒀 𝜇 𝑿 , Σ 𝑿
Posterior GPR : 𝒀∗
𝒀, 𝑿, 𝑿∗
∼ 𝒩 𝒀 𝜇 𝒀∗
𝒀, 𝑿, 𝑿∗
, Σ 𝒀∗
𝑿, 𝑿∗ 3.10
Although the GPR of Y* depends on Y, X, and X*, the main semantic meaning of regression model here is mentioned
mainly that Y* is determined based on its previous one Y when both of them are assumed to be based on X via the PDF of
Y as f(Y | X, μ(.), Σ(.)) and the PDF of Y* as f(Y* | X*, μ(.), Σ(.)). This implies that X is the intermediate point for the
probabilistic dependence of Y* on Y.

In the most general case when random variable (random process) Y is associated with a random error whose
mean and variance are 0 and δ2, respectively with assumption that all Yi belonging to Y issue noises (Wang, 2022,
p. 11) which obey the same error distribution. Therefore, the prior PDF of Y = (Y1, Y2,…, Yn)T = f(X) given X is
elaborated as follows (Bai, et al., 2023, p. 6):
𝒀 𝑿 ∼ 𝑓 𝒀 𝑿, 𝜇 . , Σ . = 𝒩 𝒀 𝜇 𝒀 , Σ 𝑿, 𝑿 + 𝛿2
𝐼
Where I is identity matrix. So, mean vector and covariance matrix of Y are μ(Y) and Σ(X, X) + δ2I within the
PDF f(Y | X, μ(.), Σ(.)), respectively. Consequently, the joint PDF of Y and Y* is elaborated as follows (Wang,
2022, p. 11):
𝑓 𝒀, 𝒀∗ 𝑿, 𝑿∗, 𝜇 . , Σ . = 𝒩 𝒀
𝜇 𝒀
𝜇 𝒀∗ ,
Σ 𝑿, 𝑿 + 𝛿2
𝐼 Σ 𝑿, 𝑿∗
Σ 𝑿∗, 𝑿 Σ 𝑿∗, 𝑿∗
As a result, mean vector μ(Y* | Y, X, X*) and covariance matrix Σ(Y* | X, X*) of GPR f(Y* | Y, X, X*, μ(.), Σ(.))
are:
𝜇 𝒀∗ 𝒀, 𝑿, 𝑿∗ = 𝜇 𝒀∗ + Σ 𝑿∗, 𝑿 Σ 𝑿, 𝑿 + 𝛿2𝐼 −1 𝒀 − 𝑚 𝒀
Σ 𝒀∗ 𝑿, 𝑿∗ = Σ 𝑿∗, 𝑿∗ − Σ 𝑿∗, 𝑿 Σ 𝑿, 𝑿 + 𝛿2𝐼 −1Σ 𝑿, 𝑿∗
3.11
The equation above is the most general formulation of GPR with occurrence of noises or errors. However, we
can ignore the error variance δ2I without loss of generality.

If X* and Y* are 1-element vectors such that X* = Xn+1 and Y* = Yn+1 and let x1:n = X, x = X*, y1:n = Y, and y = Y*, the
GPR of Y* which is now GPR of y becomes:
𝑦 𝑦1:𝑛, 𝒙1:𝑛, 𝒙 ∼ 𝒩 𝑦 𝜇 𝑦 𝑦1:𝑛, 𝒙1:𝑛, 𝒙 , 𝜎2
𝑦 𝒙1:𝑛, 𝒙 3.12
Where μ(y | y1:n, x1:n, x) and σ2(y | x1:n, x) are mean and variance of the posterior PDF of y given y1:n, x1:n, and x.
𝜇 𝑦 𝑦1:𝑛, 𝒙1:𝑛, 𝒙 = 𝜇 𝑦 + Σ 𝒙, 𝒙1:𝑛 Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
𝑦1:𝑛 − 𝑚 𝑦1:𝑛
𝜎2
𝑦 𝒙1:𝑛, 𝒙 = Σ 𝒙, 𝒙 − Σ 𝒙, 𝒙1:𝑛 Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
Σ 𝒙1:𝑛, 𝒙
3.13
The equation above with single variables x and y (single posterior processes) is popular in BO, especially x is vector
and y is scalar although x and y can be arbitrary such as scalar, vector, and matrix with note that high dimensional
spaces require tensor products for example. Please pay attention that m(y1:n) in GPR algorithm is the realistic mean
which is recurrently determined by previously known μ(y | y1:n–1 , x1:n–1, x):
𝑚 𝑦𝑛 = 𝜇 𝑦 𝑦1:𝑛−1, 𝒙1:𝑛−1, 𝒙𝑛 3.14
Note,
Σ 𝒙1:𝑛, 𝒙1:𝑛 =
Σ 𝒙1, 𝒙1 Σ 𝒙1, 𝒙2 ⋯ Σ 𝒙1, 𝒙𝑛
Σ 𝒙2, 𝒙1 Σ 𝒙2, 𝒙2 ⋯ Σ 𝒙2, 𝒙𝑛
⋮ ⋮ ⋱ ⋮
Σ 𝒙𝑛, 𝒙1 Σ 𝒙𝑛, 𝒙2 ⋯ Σ 𝒙𝑛, 𝒙𝑛
, Σ 𝒙, 𝒙1:𝑛 =
Σ 𝒙, 𝒙1
Σ 𝒙, 𝒙2
⋮
Σ 𝒙, 𝒙𝑛
𝑇
, Σ 𝒙1:𝑛, 𝒙 =
Σ 𝒙1, 𝒙
Σ 𝒙2, 𝒙
⋮
Σ 𝒙𝑛, 𝒙

Suppose only x is considered variable whereas y, y1:n, and x1:n are known then, mean
function μ(.) is set to be zero as μ(y) = 0 so that both the mean μ(y | y1:n, x1:n, x) and
the variance σ2(y | x1:n, x) are functions of only x as follows:
𝜇𝑛 𝒙 = 𝜇 𝑦 𝑦1:𝑛, 𝒙1:𝑛, 𝒙 = Σ 𝒙, 𝒙1:𝑛 Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
𝑦1:𝑛 − 𝑚 𝑦1:𝑛
𝜎𝑛
2 𝒙 = 𝜎2 𝑦 𝒙1:𝑛, 𝒙 = Σ 𝒙, 𝒙 − Σ 𝒙, 𝒙1:𝑛 Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
Σ 𝒙1:𝑛, 𝒙
Recall that m(y1:n) is the realistic mean which is recurrently determined by
previously known μn–1(x):
𝑚 𝑦𝑛 = 𝜇𝑛−1 𝒙𝑛
The event that μn(x) and σn
2(x) are functions of only x is necessary to define
acquisition function for optimization task in BO.
GPR can be executed continuously with new X* while the old X* is incorporated
into larger GP X. In practice, such continuous execution is implemented as iterative
algorithm whose each iteration has two following steps in the next slide:

Step 1:
Take new X*
𝜇 𝒀∗
𝒀, 𝑿, 𝑿∗
= 𝜇 𝒀∗
+ Σ 𝑿∗
−1
𝒀 − 𝑚 𝒀
Σ 𝒀∗ 𝑿, 𝑿∗ = Σ 𝑿∗, 𝑿∗ − Σ 𝑿∗, 𝑿 Σ 𝑿, 𝑿
−1
Σ 𝑿, 𝑿∗
Step 2:
𝑚 𝒀 = 𝜇 𝒀∗ 𝒀, 𝑿, 𝑿∗
𝑿 =
𝑿
𝑿∗
In step 1, the covariance matrix Σ(X, X) is not recomputed entirely because some of its elements Σ(Xi,
Xj) were determined before.
Because GP X will get huge when the iterative algorithm repeats many iterations and X* is
incorporated into X over and over again, it is possible to apply first-order Markov property so that each
iteration remembers only one previous X. Therefore, we can assign X* to X as X = X* in step 2 so that
the iterative algorithm runs faster and saves more computational resources as in the next slide:
Table 3.1. GPR algorithm

Step 1:
Take new X*
𝜇 𝒀∗
𝒀, 𝑿, 𝑿∗
= 𝜇 𝒀∗
+ Σ 𝑿∗
−1
𝒀 − 𝑚 𝒀
Σ 𝒀∗ 𝑿, 𝑿∗ = Σ 𝑿∗, 𝑿∗ − Σ 𝑿∗, 𝑿 Σ 𝑿, 𝑿
−1
Σ 𝑿, 𝑿∗
Step 2:
𝑚 𝒀 = 𝜇 𝒀∗ 𝒀, 𝑿, 𝑿∗
𝑿 = 𝑿∗
However, this advantage of first-order Markov property cannot be applied into BO (by defining how
to take new X* with acquisition functions) because it is necessary to look entire X domain forward
and backward to find out potential global optimizer. The use of Markov property will be appropriate
to tasks of prediction and estimation in which entire time series are split into many smaller time
windows where it is necessary to remember w windows with w-order Markov property. Shortly, the
assignment X = X* is suitable to forward modeling.
Table 3.2. GPR algorithm with first-order Markov property

GPR focuses mainly on the regressive (posterior) covariance matrix Σ(Y* | X, X*) via kernel function Σ(Xi,
Xj) because adjusting the regressive (posterior) mean vector μ(Y* | Y, X, X*) via adjusting the mean function
μ(Y) is unnecessary due to:
𝜇 𝒀∗ 𝒀, 𝑿, 𝑿∗ = 𝜇 𝒀∗ + Σ 𝑿∗, 𝑿 Σ 𝑿, 𝑿
−1
𝒀 − 𝑚 𝒀
As usual, mean function μ(Y*) are set to be zero as μ(Y*) = 0T so that
𝜇 𝒀∗ 𝒀, 𝑿, 𝑿∗ = Σ 𝑿∗, 𝑿 Σ 𝑿, 𝑿
−1
𝒀 − 𝑚 𝒀
Which is indeed an arithmetic regression of Y* on Y, X, and X* in addition to the main semantic meaning
that the posterior PDF of Y* given Y is the main regression. However, it is still better to define exactly the
mean function μ(Y) in cases of predicting more precisely confident intervals of Y based on X because the
equation above:
𝜇 𝒀∗
𝒀, 𝑿, 𝑿∗
= Σ 𝑿∗
−1
𝒀 − 𝑚 𝒀
Which implies variation of Y according to variation of X from the origin. It is not a real value of Y. By
another way, if setting μ(Y*) = Y* (also deriving μ(Y) = Y) so that
𝜇 𝒀∗
𝒀, 𝑿, 𝑿∗
= 𝒀∗
+ Σ 𝑿∗
−1
𝒀 − 𝑚 𝒀
That is not a regression function, which is impossible to predict Y* based on X* if Y* is unknown.

Therefore, I propose a technique based on linear regression model to define μ(Y*) with constraint that each element
μ(Y*) of μ(Y*) depend only on Y*, for instance:
𝜇 𝒀∗
=
𝜇 𝑌𝑛+1
𝜇 𝑌𝑛+2
⋮
𝜇 𝑌𝑛+𝑘
Note, Y* represents any Yi belonging to the subprocess Y* = (Yn+1, Yn+2,…, Yn+k)T and X* represents any Xi belonging to
the subprocess X* = (Xn+1, Xn+2,…, Xn+k)T. Let φ(X) be the transformation function which transforms X space into Y
space such that φ(.) is invertible.
𝑌 = 𝜑 𝑋 , 𝑌∗
= 𝜑 𝑋∗
𝑋 = 𝜑−1
𝑌 , 𝑋∗
= 𝜑−1
𝑌∗ 3.15
Therefore, φ(.) should be defined with constraint that X space and Y space have the same dimension for information
preservation. The simplest form of φ(.) is identity function.
𝜑 𝑋 = 𝑋
Let Z* be a regressive estimate of X*:
𝑍∗
= 𝐴∗
𝑋∗
3.16
Where A* is regressive coefficient. How to calculate A* from sample {X*, φ–1(Y*)} will be described later. Please pay
attention that this linear regression is totally different from the regression meaning of GPR via posterior PDF.

Let 𝑌∗ be an estimate of Y* from X* with association of linear regression model and transformation function φ(.).
𝑌∗ = 𝜑 𝑍∗ = 𝜑 𝐴∗𝑋∗ 3.17
Obviously, we have:
𝑍∗ = 𝜑−1 𝑌∗
Let |X|, |X*|, |Y|, and |Y*| be cardinalities (also dimensions) of X, X*, Y, and Y*, respectively. Of course, we have
|X| = |Y| and |X*| = |Y*|, for example, we also have |X| = |Y| = n and |X*| = |Y*| = k. Concentration ratio (CR) of a
subprocess which is point density of such subprocess is defined by the ratio of cardinality of such subprocess to
the cardinality of entire process, for example, CR of X* over {X, X*} is:
𝑐𝑟 𝑿∗
𝑿, 𝑿∗
=
𝑿∗
𝑿 + 𝑿∗
3.18
Suppose A also being the regressive coefficient estimated from sample {X, φ–1(Y)} was determined before, the
mean function μ*(X*) is proposed as a weighted estimation with concentration ratios of X and X* as follows:
𝜇∗
𝑋∗
= 𝜑
𝑿
𝑿 + 𝑿∗
𝐴𝑋∗
+
𝑿∗
𝑿 + 𝑿∗
𝐴∗
𝑋∗
3.19
As a result, the two steps at each iteration of the iterative algorithm for estimating GPR are finetuned shown in
next slide:

Step 1:
Take new X*
Calculate A* from sample {X*, φ–1(Y*)}
𝜇 𝒀∗
𝑿, 𝑿∗
= 𝜇∗
𝑿∗
+ Σ 𝑿∗
−1
𝒀 − 𝑚 𝒀
Σ 𝒀∗
𝑿, 𝑿∗
= Σ 𝑿∗
, 𝑿∗
− Σ 𝑿∗
−1
Σ 𝑿, 𝑿∗
Step 2:
𝑚 𝒀 = 𝜇 𝒀∗ 𝑿, 𝑿∗
𝐴 = 𝐴∗
𝑿 =
𝑿
𝑿∗ or 𝑿 = 𝑿∗
Where μ*(X*) is determined based on
μ*(X*) as follows: 𝜇∗ 𝑿∗ =
𝜇∗ 𝑋𝑛+1
𝜇∗ 𝑋𝑛+2
⋮
𝜇∗
𝑋𝑛+𝑘
.
Note, the vector φ–1(Y*) is determined:
𝜑−1 𝒀∗ =
𝜑−1 𝑌𝑛+1
𝜑−1
𝑌𝑛+2
⋮
𝜑−1
𝑌𝑛+𝑘
Both mean vector μ(Y* | X, X*) and
covariance matrix Σ(Y* | X, X*) of GPR
with linear regression are free from Y
because they are totally based on only X
and X*.
Table 3.3. GPR algorithm with linear regression

For interval estimation of Y* with given X*, suppose GPR algorithm finished obtaining
regression coefficient A after some iterations on X, the mean function μ(Y*) is reduced with X*
as follows:
𝜇∗
𝑋∗
= 𝜑 𝐴𝑋∗
As a result, the confident interval of Y* which is the pair {μ(Y* | X, X*), Σ(Y* | X, X*)} is
determined as follows:
𝜇 𝑌∗
𝑿, 𝑋∗
= 𝜑 𝐴𝑋∗
+ Σ 𝑋∗
−1
𝒀 − 𝑚 𝒀
Σ 𝑌∗
𝑿, 𝑋∗
= Σ 𝑋∗
, 𝑋∗
− Σ 𝑋∗
−1
Σ 𝑿, 𝑋∗
3.20
Recall that m(Y) is a realistic mean vector which is recurrently determined by previously known
μ(Y | X1:n0, X):
𝑚 𝒀 = 𝜇 𝒀 𝑋1:𝑛0
, 𝑿 =
𝜇 𝑌1 𝑋1:𝑛0
, 𝑋1
𝜇 𝑌2 𝑋1:𝑛0
, 𝑋2
⋮
𝜇 𝑌𝑛 𝑋1:𝑛0
, 𝑋𝑛

Because the GPR algorithm here defines the mean function μ(Y*) based on multiple linear regression (MLR) model, it is
necessary to describe MLR in short. Given a dependent random vector variable Z = (z1, z2,…, zm)T and an independent
random variable X = (1, x1, x2,…, xn)T, MLR tries to establish linear relationship between Z = (z1, z2,…, zm)T and X so that
Z is sum of a linear combination of X and an random error vector ε.
𝑍 = 𝐴𝑋 + 𝜺 3.21
X is called regressor and Z is called responsor whereas A = (α0, α1, α2,…, αq)T is mx(n+1) regressive coefficient matrix.
𝐴 =
𝑎00 𝑎11 𝑎12 ⋯ 𝑎1𝑛
𝑎10 𝑎21 𝑎22 ⋯ 𝑎2𝑛
⋮ ⋮ ⋮ ⋱ ⋮
𝑎𝑚0 𝑎𝑚1 𝑎𝑚2 ⋯ 𝑎𝑚𝑛
3.22
Suppose ε distributes normally with mean vector 0T and covariance matrix Σ then Z distributes normally with mean
vector AX and covariance matrix Σ due to:
𝐸 𝑍 = 𝐸 𝐴𝑋 + 𝜺 = 𝐴𝑋
𝑉 𝑍 = 𝑉 𝐴𝑋 + 𝜺 = Σ
3.23
Note, E(.) and V(.) denote theoretical expectation and variance, respectively and Σ is mxm invertible matrix. This implies
that the PDF of random variable Z is:
𝑍 𝑋 ∼ 𝑓 𝑍 𝑋, 𝐴, Σ = 𝒩 𝑍 𝐴𝑋, Σ = 2𝜋 −
𝑚
2 Σ −
1
2exp −
1
2
𝑍 − 𝐴𝑋 𝑇
Σ−1
(𝑍 − 𝐴𝑋)

MLR of Z given X is built from sample {X, Z} of size N in form of data matrix as follows:
𝑿 =
𝒙1
𝑇
𝒙2
𝑇
⋮
𝒙𝑁
𝑇
=
1 𝑥11 𝑥12 ⋯ 𝑥1𝑛
1 𝑥21 𝑥22 ⋯ 𝑥2𝑛
⋮ ⋮ ⋮ ⋱ ⋮
1 𝑥𝑁1 𝑥𝑁2 ⋯ 𝑥𝑁𝑛
, 𝒙𝑖 =
1
𝑥𝑖1
𝑥𝑖2
⋮
𝑥𝑖𝑛
𝒁 =
𝒛1
𝑇
𝒛2
𝑇
⋮
𝒛𝑁
𝑇
=
𝑧11 𝑧12 ⋯ 𝑧1𝑚
𝑧21 𝑧22 ⋯ 𝑧2𝑚
⋮ ⋮ ⋱ ⋮
𝑧𝑁1 𝑧𝑁2 ⋯ 𝑧𝑁𝑚
, 𝒛𝑖 =
𝑧𝑖1
𝑧𝑖2
⋮
𝑧𝑖𝑚
3.24
Therefore, xi and zi is the ith instances of regressor X and responsor Z at the ith row of matrix (X, Z). As a convention, we can
connect datasets X and Z of MLR here with the GP X* and the set φ–1(Y*) aforementioned, respectively if removing the first
column of values 1 from datasets X.
𝑁 ∼ 𝑘
𝒙𝑖 ∼ 𝑋𝑛+𝑖
𝑿 ∼ 𝑿∗
𝒛𝑖 ∼ 𝜑−1
𝑌𝑛+𝑖
𝒁 ∼ 𝜑−1 𝒀∗
3.25

The essence of MLR is to estimate the regressive
coefficient matrix A and the covariance matrix Σ. By
applying maximum likelihood estimation (MLE) method,
we obtain estimates 𝐴 and Σ of A and Σ.
𝐴 = 𝒁𝑇
𝑿 𝑿𝑇
𝑿 −1
Σ =
1
𝑁
𝑖=1
𝑁
𝒛𝑖 − 𝐴𝒙𝑖 𝒛𝑖 − 𝐴𝒙𝑖
𝑇 3.26

Recall that Bayesian optimization (BO) maximizes the acquisition function α(x | f, S) so as to search for
maximizer x*. Especially, nonparametric BO (NBO) based on Gaussian process regression (GPR) requires
support of acquisition function to guide movement of x in the search space so as to reach maximizer x*.
Moreover, acquisition function of NBO is created from GPR surrogate model; concretely, it is defined with
two essential parameters such as posterior mean μn(x) and variance σn
2(x) of GPR.
−1
𝑦1:𝑛 − 𝑚 𝑦1:𝑛 𝜎𝑛
2
𝒙
= Σ 𝒙, 𝒙 − Σ 𝒙, 𝒙1:𝑛 Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
Σ 𝒙1:𝑛, 𝒙
Note that m(y1:n) is a realistic mean which is recurrently determined by previously known μn–1(x):
𝑚 𝑦𝑛 = 𝜇𝑛−1 𝒙𝑛
This is the reason that acquisition function of NBO is denoted as of α(x | μn(x), σn
2(x)) in NBO. Therefore, this
section focuses on NBO acquisition function. GPR surrogate model of NBO is represented by the posterior
PDF of y = f(x) given y1:n, x1:n, and x as follows:
𝑦 𝑦1:𝑛, 𝒙1:𝑛, 𝒙 ∼ 𝑓 𝑦 𝑦1:𝑛, 𝒙1:𝑛, 𝒙, 𝜇 . , Σ . = 𝒩 𝑦 𝜇𝑛 𝒙 , 𝜎𝑛
2
𝒙
Both mean μn(x) and variance σn
2(x) are functions of only x because y1:n and x1:n were known, which is
necessary to determine acquisition function which is function of x too. Note, in this research, notation f(.|.)
often denotes PDF, which is not concrete function and notation 𝒩 . . denotes normal distribution.

The normal PDF of y = f(x) given y1:n, x1:n, and x is formulated as follows:
𝑦 𝑦1:𝑛, 𝒙1:𝑛, 𝒙 ∼ 𝒩 𝑦 𝜇𝑛 𝒙 , 𝜎𝑛
2
𝒙 =
1
2𝜋𝜎𝑛
2
𝒙
exp −
𝑦 − 𝜇𝑛 𝒙
2
2𝜎𝑛
2
𝒙
This PDF should be standardized into standard normal PDF ϕ(z) because values of standard normal cumulative
distribution function (standard normal CDF) Φ(z) were evaluated in practice. Standard normal distribution is also
called standard Gaussian distribution. If variable y is standardized into variable z such that 𝑧 =
𝑦−𝜇𝑛 𝒙
𝜎𝑛 𝒙
then,
distribution of y is equivalent to distribution of z via its PDF ϕ(z) also its CDF Φ(z).
𝑧 =
𝜎𝑛 𝒙
𝜙 𝑧 = 𝒩 𝑧 0,1 =
1
2𝜋
exp −
𝑧2
2
Φ 𝑧0 = 𝜙 𝑧 < 𝑧0 =
−∞
𝑧0
𝜙 𝑧 𝑑𝑧
4.1

Where the quantity 𝜎𝑛 𝒙 = 𝜎𝑛
2
𝒙 is called standard deviation of y given y1:n, x1:n, and x. Note, standard
normal PDF has mean 0 and variance 1. We have some important properties of standard normal distribution.
𝜙 𝑧 < 𝑧0 = 𝜙 𝑧 ≤ 𝑧0
𝜙 𝑧 > 𝑧0 = 𝜙 𝑧 ≥ 𝑧0
Φ′ 𝑧 = 𝜙 𝑧
Φ −𝑧0 = 1 − Φ 𝑧0 = 𝜙 𝑧 > 𝑧0 =
𝑧0
+∞
𝜙 𝑧 𝑑𝑧
If the PDF of y is standardized, the statement that y distributes normally given y1:n, x1:n, and x with mean
μn(x) and variance σn
2(x) will be equivalent to the statement that the distribution of z given y1:n, x1:n, and x is
standard normal distribution where:
𝑧 =
𝜎𝑛 𝒙
Of course, we have:
𝑦 = 𝜇𝑛 𝒙 + 𝜎𝑛 𝒙 𝑧 4.2

Recall that the acquisition function α(x | f, S) based on mean μn(x) and variance σn
2(x) of GPR surrogate model is denoted
as follows, which implies that it is function of x.
𝛼 𝒙 𝑓, 𝑆 = 𝛼 𝒙 𝜇𝑛 𝒙 , 𝜎𝑛
2
𝒙
Recall, given acquisition function α(x | μn(x), σn
2(x), and also known sample Dn = x1:n = {x1,
x2,…, xn}, let xn+1 be a maximizer of α(x | μn(x), σn
2(x)) in NBO algorithm at a current iteration.
𝒙𝑛+1 = argmax
𝒙
2
𝒙
There are some acquisition functions, for example, probability of improvement, expected improvement, entropy search,
and upper confidence bound. Let x* and y* = f(x*) be the maximizer and its respective maximum value of target function
to which all acquisitions aim. Both probability of improvement (PI) technique and expected improvement (EI) technique
belong to improvement approach in which their acquisition functions, which are also utility functions in decision theory,
are expectations of a predefined respective improvement function but their definitions of improvement function are
different. For instance, the improvement function denoted I(x) sets the deviation between the next value y = f(x) and the
current and temporal maximum yn
* = f(xn
*) with expectation that y will be larger than yn
* as much as possible for
searching realistic maximizer x*; otherwise, I(x) is 0.
𝐼 𝒙 ≝ max 𝑦 − 𝑦∗
, 0 = max 𝑓 𝒙 − 𝑦𝑛
∗
, 0
Note, given sample Dn = x1:n = {x1, x2,…, xn} and its evaluations y1:n = {y1, y2,…, yn}, we have yn
* = max{y1, y2,…, yn }
which is the temporal maximum at current iteration of NBO algorithm.

When the PDF of y is standardized, y = μn(x) + σn(x)z then the improvement function is rewritten:
𝐼 𝒙 = max 𝜇𝑛 𝒙 + 𝜎𝑛 𝒙 𝑧 − 𝑦𝑛
∗, 0 4.3
Probability of improvement (PI) technique defines acquisition function as the probability of I(x) > 0, which means
that the probability of the event that next candidate of target maximizer is larger than the old one must be
maximized (Shahriari, Swersky, Wang, Adams, & Freitas, 2016, p. 160).
𝛼PI 𝒙 = 𝛼 𝒙 𝜇𝑛 𝒙 , 𝜎𝑛
2
𝒙 = 𝑃 𝐼 𝒙 > 0
Note, notation P(.) denotes probability. As a result, PI acquisition function is determined as follows:
𝛼PI 𝒙 = Φ
∗
𝜎𝑛 𝒙
4.4
Proof, 𝛼PI 𝒙 = 𝑃 𝐼 𝒙 > 0 = 𝑃 𝜇𝑛 𝒙 + 𝜎𝑛 𝒙 𝑧 − 𝑦𝑛
∗
> 0 = 𝑃 𝑧 >
𝑦𝑛
∗ −𝜇𝑛 𝒙
𝜎𝑛 𝒙
= 𝜙 𝑧 >
𝑦𝑛
∗ −𝜇𝑛 𝒙
𝜎𝑛 𝒙
= Φ
𝜇𝑛 𝒙 −𝑦𝑛
∗
𝜎𝑛 𝒙
Due to 𝜙 𝑧 > 𝑧0 = Φ −𝑧0 ∎
Therefore, the maximizer xn+1 of PI acquisition function in NBO algorithm at current iteration is determined as
follows:
𝒙𝑛+1 = argmax
𝒙
Φ
∗
𝜎𝑛 𝒙

Expected improvement (EI) technique defines acquisition function as expectation of I(x),
which means that the mean of the event that next candidate of target maximizer is larger than
the old one must be maximized.
𝛼EI 𝒙 = 𝛼 𝒙 𝜇𝑛 𝒙 , 𝜎𝑛
2 𝒙 =
−∞
+∞
𝐼 𝒙 𝜙 𝑧 𝑑𝑧
=
−∞
+∞
max 𝜇𝑛 𝒙 + 𝜎𝑛 𝒙 𝑧 − 𝑦𝑛
∗, 0 𝜙 𝑧 𝑑𝑧
Note, such expectation of I(x) is determined with the standard PDF ϕ(z) given y1:n, x1:n, and x.
As a result, EI acquisition function is determined as follows (Frazier, 2018, p. 7):
𝛼EI 𝒙 = 𝜇𝑛 𝒙 − 𝑦𝑛
∗
Φ
∗
𝜎𝑛 𝒙
∗
𝜎𝑛 𝒙
4.5

Proof (Kamperis, 2021),
𝛼EI 𝒙 =
−∞
+∞
∗
, 0 𝜙 𝑧 𝑑𝑧
=
−∞
𝑧𝑛 𝒙
∗
, 0 𝜙 𝑧 𝑑𝑧 +
𝑧𝑛 𝒙
+∞
∗
, 0 𝜙 𝑧 𝑑𝑧
Let 𝑧𝑛 𝒙 =
𝑦𝑛
∗
− 𝜇𝑛 𝒙
𝜎𝑛 𝒙
=
𝑧𝑛 𝒙
+∞
𝜇𝑛 𝒙 + 𝜎𝑛 𝒙 𝑧 − 𝑦𝑛
∗ 𝜙 𝑧 𝑑𝑧
Due to max 𝜇𝑛 𝒙 + 𝜎𝑛 𝒙 𝑧 − 𝑦𝑛
∗, 0
= 𝜇𝑛 𝒙 − 𝑦𝑛
∗
𝑧𝑛 𝒙
+∞
𝜙 𝑧 𝑑𝑧 + 𝜎𝑛 𝒙
𝑧𝑛 𝒙
+∞
𝑧𝜙 𝑧 𝑑𝑧

∗
Φ −𝑧𝑛 𝒙 + 𝜎𝑛 𝒙
𝑧𝑛 𝒙
+∞
𝑧𝜙 𝑧 𝑑𝑧
Due to
𝑧𝑛 𝒙
+∞
𝜙 𝑧 𝑑𝑧 = Φ −𝑧𝑛 𝒙 = Φ
∗
𝜎𝑛 𝒙
∗
Φ −𝑧𝑛 𝒙 +
𝜎𝑛 𝒙
2𝜋
𝑧𝑛 𝒙
+∞
𝑧exp −
𝑧2
2
𝑑𝑧
Due to 𝜙 𝑧 =
1
2𝜋
exp −
𝑧2
2
∗
Φ −𝑧𝑛 𝒙 −
𝜎𝑛 𝒙
2𝜋
𝑧𝑛 𝒙
+∞
𝑑 exp −
𝑧2
2

∗ Φ −𝑧𝑛 𝒙 −
𝜎𝑛 𝒙
2𝜋
exp −
𝑧2
2
+∞
𝑧𝑛 𝒙
∗ Φ −𝑧𝑛 𝒙 +
𝜎𝑛 𝒙
2𝜋
exp −
𝑧𝑛 𝒙
2
2
∗ Φ −𝑧𝑛 𝒙 + 𝜎𝑛 𝒙 𝜙 𝑧𝑛 𝒙
Due to 𝜙 𝑧𝑛 𝒙 = 𝜙 −𝑧𝑛 𝒙 =
1
2𝜋
exp −
𝑧𝑛 𝒙
2
2
∎
Therefore, the maximizer xn+1 of EI acquisition function in NBO algorithm at current iteration is
determined as follows:
𝒙𝑛+1 = argmax
𝒙
∗ Φ
∗
𝜎𝑛 𝒙
∗
𝜎𝑛 𝒙

An important clue of NBO is that maximizing acquisition function should be cheaper than maximizing target function
f(x) if f(x) is analytic function which is appropriate to analytic approach. Fortunately, it is not prerequisite to find out
global maximizer of acquisition function because essentially NBO is a reinforcement algorithm whose solution candidate
for x* is improved progressively and moreover, acquisition function itself makes tradeoff between exploitation
(convergence in searching maximizer) and exploration (searching global maximizer) for NBO. Therefore, it is possible to
apply traditional mathematical methods such as gradient descent for obtaining the local maximizer xn+1 of acquisition
function. It is only necessary to check first-order derivative known as gradient of acquisition function when traditional
optimization methods like gradient descent require existence of gradient. Because PI is simpler than EI, how to
maximize PI is mentioned here as an example of acquisition maximization.
𝒙𝑛+1 = argmax
𝒙
𝛼PI 𝒙 = argmax
𝒙
Φ
∗
𝜎𝑛 𝒙
Now we only need to determine the gradient of αPI(x) with regard to variable x. Let Σ’(xi, x) be gradient (first-order
derivative) of Σ(xi, x) with regard to x given known xi = (xi1, xi2,…, xin)T and unknown x = (x1, x2,…, xn)T. As a
convention, Σ’(xi, x) is row vector. If covariance function produces symmetric matrix then Σ’(xi, x) = Σ’(x, xi). For
example, given simple squared exponential kernel function Σ𝑆𝐸 𝒙, 𝒙𝑖 = exp −
𝒙−𝒙𝑖
2
2
, its gradient with regard to x is:
Σ′
𝒙, 𝒙𝑖 =
𝑑
𝑑𝒙
exp −
𝒙 − 𝒙𝑖
2
2
=
𝑑
𝑑𝒙
exp −
𝒙 − 𝒙𝑖
𝑇
𝒙 − 𝒙𝑖
2
= − 𝒙 − 𝒙𝑖
𝑇
exp −
𝒙 − 𝒙𝑖
𝑇
𝒙 − 𝒙𝑖
2

Let Σ’(x1:n, x) be gradient of Σ(x1:n, x), essentially it is a matrix but written as column vector whose each element is
row vector:
Σ′ 𝒙1:𝑛, 𝒙 =
Σ′ 𝒙1, 𝒙
Σ′ 𝒙2, 𝒙
⋮
Σ′ 𝒙𝑛, 𝒙
If covariance function produces symmetric matrix, we have:
Σ′ 𝒙, 𝒙1:𝑛 = Σ′ 𝒙1:𝑛, 𝒙
𝑇
Gradients of μn(x) and σn
2(x) with regard to x are:
𝜇𝑛
′
𝒙 = 𝑦1:𝑛 − 𝑚 𝑦1:𝑛
𝑇
Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
Σ′
𝒙1:𝑛, 𝒙
𝜎𝑛
2 𝒙
′
= Σ′ 𝒙, 𝒙 − 2Σ 𝒙, 𝒙1:𝑛 Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
Σ′ 𝒙1:𝑛, 𝒙
4.6
Note, Σ(x1:n, x1:n) is invertible and symmetric. Because of 𝜎𝑛 𝒙 = 𝜎𝑛
2
𝒙 , we have gradient of σn(x) as follows:
𝜎𝑛
′ 𝒙 =
𝜎𝑛
2 𝒙
′
2 𝜎𝑛
2
𝒙

Let zn(x) = (yn
* – μn(x)) / σn(x) as function of x, its gradient with regard to x is:
𝑧𝑛
′
𝒙 = −
𝜇𝑛
′ 𝒙 𝜎𝑛 𝒙 + 𝑦𝑛
∗ − 𝜇𝑛 𝒙 𝜎𝑛
′ 𝒙
𝜎𝑛
2
𝒙
4.7
Gradient of PI is totally determined as follows:
𝛼PI
′
𝒙 = Φ′
∗
𝜎𝑛 𝒙
= −𝜙 𝑧𝑛 𝒙 𝑧𝑛
′
𝒙 4.8
Where,
𝑧𝑛 𝒙 =
𝑦𝑛
∗ − 𝜇𝑛 𝒙
𝜎𝑛 𝒙
It is possible to apply traditional methods like gradient descent to find out local maximizer xn+1 of PI
because PI gradient is determined given x. Similarly, it is easy to obtain gradient of EI acquisition
function because gradients of μn(x) and σn
2(x) are most important, which are the base to calculate gradients of
PI and EI.
𝛼EI
′
𝒙 = 𝜙 𝑧𝑛 𝒙 𝜎𝑛
′ 𝒙 − Φ −𝑧𝑛 𝒙 𝜎𝑛 𝒙 𝑧𝑛 𝒙
′
4.9
In turn, gradients of μn(x) and σn
2(x) are based on derivative Σ’(x, xi) of kernel function and so,
differentiability of kernel function always asserts feasibility of improvement approach.

As aforementioned, optimization problem with BO is maximization problem by default. However, which one
between minimization and maximization can be determined by changing activation function a little bit in BO.
Exactly, BO minimization can be obtained by changing the definition of improvement function I(x) a little bit.
Concretely, given next value y = f(x) and temporal minimum yn
* = f(xn
*), the improvement function for
minimization is specified with expectation that y will be smaller than yn
* as much as possible, as follows:
𝐼 𝒙 ≝ max 𝑦∗
− 𝑦, 0 = max 𝑦𝑛
∗
− 𝑓 𝒙 , 0
As a result, PI acquisition function and EI acquisition function for minimization are:
𝛼PI 𝒙 = Φ
𝑦𝑛
∗
− 𝜇𝑛 𝒙
𝜎𝑛 𝒙
𝛼EI 𝒙 = 𝑦𝑛
∗
− 𝜇𝑛 𝒙 Φ
𝑦𝑛
∗
− 𝜇𝑛 𝒙
𝜎𝑛 𝒙
𝑦𝑛
∗
− 𝜇𝑛 𝒙
𝜎𝑛 𝒙
These acquisition functions are maximized locally by traditional methods like gradient descent at current
iteration as usual.
𝒙𝑛+1 = argmax
𝒙
𝛼PI 𝒙 , for minimization
𝒙𝑛+1 = argmax
𝒙
𝛼EI 𝒙 , for minimization

Another simple acquisition function called upper confidence bound (UCB) function is based confidence
interval of GPR which is determined by posterior mean μn(x) and variance σn
2(x) of GPR.
−1
𝑦1:𝑛 − 𝑚 𝑦1:𝑛
𝜎𝑛
2
𝒙 = Σ 𝒙, 𝒙 − Σ 𝒙, 𝒙1:𝑛 Σ 𝒙1:𝑛, 𝒙1:𝑛
−1
Σ 𝒙1:𝑛, 𝒙
Therefore, the confidence interval of GPR is [μn(x) – σn(x), μn(x) + σn(x)] where the upper bound is μn(x) +
σn(x) and the lower bound is μn(x) – σn(x), obviously. UCB acquisition function is composed of such upper
bound and a parameter λ as follows (Kamperis, 2021):
𝛼UCB 𝒙 = 𝜇𝑛 𝒙 + 𝜆𝜎𝑛 𝒙 4.10
The maximizer xn+1 of UCB acquisition function in NBO algorithm at current iteration is determined as follows:
𝒙𝑛+1 = argmax
𝒙
𝛼UCB 𝒙 = argmax
𝒙
𝜇𝑛 𝒙 + 𝜆𝜎𝑛 𝒙
The parameter λ controls tradeoff of exploitation and exploration in which the exploitation aspect indicates that
NBO leans forwards fast seeking maximizer whereas the exploitation aspect indicates that NBO leans forwards
seeking global maximizer. If λ is small, exploitation is focused so that NBO can converge fast but the maximizer x*
can be local. If λ is large, exploration is focused so that NBO can find out the global maximizer, but its convergence
can be slow with note that a terminated condition of NBO iterative algorithm can be the maximum number of
iterations.

It is easy to apply traditional optimization methods like gradient descent to find out local
maximizer xn+1 of UCB because its gradient is determined simply as follows:
𝛼UCB
′
𝒙 = 𝜇𝑛
′
𝒙 + 𝜆𝜎𝑛
′
𝒙
For minimization, UCB acquisition function is changed into lower confidence bound (LCB)
function as follows (Bai, et al., 2023, p. 8):
𝛼LCB 𝒙 = 𝜇𝑛 𝒙 − 𝜆𝜎𝑛 𝒙
For minimization, LCB is minimized locally by traditional methods like gradient descent as
usual at current iteration when its gradient is determined simply too.
𝒙𝑛+1 = argmin
𝒙
𝛼LCB 𝒙 = argmin
𝒙
𝜇𝑛 𝒙 − 𝜆𝜎𝑛 𝒙 , for minimization
In general, UCB acquisition function is the simplest acquisition but for my opinion it is not as
good as EI because essentially UCB is a rough box containing the target function f(x) and so it
cannot simulate exactly peaks of f(x) whereas EI takes advantages of heuristic information about
peaks of f(x) via the probabilistic distribution of GPR and EI balances exploitation and exploration
via its inside mean under the GPR distribution.

5. Discussions
The main purpose of Bayesian optimization (BO) is to solve the global optimization problem where
the target function f(x) is a black box. The event that f(x) is not necessary to be defined analytically
broadens applications of BO in which reinforcement learning and robotics learning are suitable
applications of BO (Shahriari, Swersky, Wang, Adams, & Freitas, 2016, p. 3). Especially, in deep
learning, whereas stochastic gradient descent (SGD) method is appropriate and successful to train
main parameters of neural network such as weights and biases, BO which will consider the entire
neural network as a black box can be applied to train other augmented parameters such as learning
rate of SGD because SGD cannot learn such augmented parameters. In the situation of data stream
and real-world indetermination, it is impossible to set the learning rate of SGD to be inversely
proportional to iterations because such iterations cannot be counted exactly in such situation, but it is
also not good to fix such learning rate. A possible solution is to use BO to find out the optimal
learning rate in the background process in parallel with SGD process. Recently, because of black box
aspect of BO, transfer learning (Bai, et al., 2023) is applied into improving BO to select appropriate
surrogate model and appropriate acquisition function. In this research, Gaussian process regression
(GPR) is surrogate model for nonparametric Bayesian optimization (NBO) but there are other
surrogate models such as Bayesian neural network and neural process (Bai, et al., 2023, pp. 20-23).

5. Discussions
In general, this research focuses on nonparametric Bayesian optimization (NBO) although there are
interesting aspects of parametric Bayesian optimization (PBO). Strictness is emphasized in some
researches but other ones need only reasonableness, especially applied researches or heuristic
researches. Although NBO belongs to applied mathematics, its mathematical base is strict even its
outlook of surrogate model and acquisition function maximization is seemly tricky. Therefore, its
costing price is that NBO is, in turn, based on traditional convex optimization methods like gradient
descent to maximize acquisition function, which means that NBO requires other knowledge
outside its circle while heuristic methods like particle swarm optimization (PSO) and ant bee colony
(ABC) do not require mathematical constraints. Anyhow, NBO is a significant solution of global
optimization problem because, within requisite assumption of too strict mathematical constraints, the
progress of applied mathematics will be hesitative. Therefore, the main problem here is effectiveness
of NBO in balancing exploration and exploitation for search global optimizer. Heuristic methods like
PSO are proved their simplicity and feasibility when they do not focus on complicated mathematical
tools although there are some researches trying to apply mathematical theories into explaining them.
For human research, whether thinking via theory is prior to natural imitation or vice versa is a big
question whose answers can be dependent on individuals. This question is equivalent to the other
question that which one among logicality and imagination is important.

References
1. Bai, T., Li, Y., Shen, Y., Zhang, X., Zhang, W., & Cui, B. (2023, February 12). Transfer Learning for Bayesian
Optimization: A Survey. arXiv Preprint. doi:10.48550/arXiv.2302.05927
2. Frazier, P. I. (2018, July 8). A Tutorial on Bayesian Optimization. arXiv. doi:10.48550/arXiv.1807.02811
3. Hardle, W., & Simar, L. (2013). Applied Multivariate Statistical Analysis. Berlin, Germany: Research Data Center,
School of Business and Economics, Humboldt University.
4. Kamperis, S. (2021, June 11). Acquisition functions in Bayesian Optimization. Retrieved from Stathis Kamperis's
GitHub page: https://ekamperi.github.io/machine%20learning/2021/06/11/acquisition-functions.html
5. Luo, X. (2019, May 24). Minima distribution for global optimization. arXiv preprint. doi:10.48550/arXiv.1812.03457
6. Neapolitan, R. E. (2003). Learning Bayesian Networks. Upper Saddle River, New Jersey, USA: Prentice Hall.
7. Nguyen, L. (2022, June 27). A short study on minima distribution. Preprints. doi:10.20944/preprints202206.0361.v1
8. Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & Freitas, N. d. (2016, January). Taking the Human Out of the
Loop: A Review of Bayesian Optimization. (J. H. Trussell, Ed.) Proceedings of the IEEE, 104(1), 148 - 175.
doi:10.1109/JPROC.2015.2494218
9. Wang, J. (2022, April 18). An Intuitive Tutorial to Gaussian Processes Regression. arXiv.
doi:10.48550/arXiv.2009.10862
10. Wikipedia. (2003, August 23). Gaussian process. (Wikimedia Foundation) Retrieved from Wikipedia website:
https://en.wikipedia.org/wiki/Gaussian_process

Thank you for listening
69
20/04/2023

Tutorial on Bayesian optimization

More Related Content

Similar to Tutorial on Bayesian optimization (20)

More from Loc Nguyen (20)

Recently uploaded (20)

Tutorial on Bayesian optimization