Talk in BayesComp 2018

Controlled Sequential Monte Carlo
Jeremy Heng
Department of Statistics, Harvard University
Joint work with Adrian Bishop (UTS & CSIRO), George Deligiannidis and Arnaud Doucet
(Oxford)
BayesComp 2018
Jeremy Heng Controlled SMC 1 / 25

Intractable likelihoods
This work is about eﬃciently estimating
p(y|θ) = p(y|x, θ)p(dx|θ)

In state space models, law of latent Markov chain
p(dx|θ) = µθ(dx0)
T
t=1
Mt,θ(xt−1, dxt )
i.e. X0 ∼ µθ and Xt |Xt−1 ∼ Mt,θ(Xt−1, ·)

T
t=1
Mt,θ(xt−1, dxt )
Observations are conditionally independent given (Xt )t∈[0:T]
p(y|x, θ) =
T
t=0
gθ(xt , yt )

T
t=1
Mt,θ(xt−1, dxt )
Observations are conditionally independent given (Xt )t∈[0:T]
p(y|x, θ) =
T
t=0
gθ(xt , yt )
Want to also approximate smoothing distribution
p(x|y, θ) =
T
t=0
gθ(xt , yt )p(dx|θ)p(y|θ)−1

Motivating example
Measurements (y0, . . . , yT ) ∈ {0, . . . , 50}3000
collected from a
neuroscience experiment (Temereanca et al., 2008)
500 1000 1500 2000 2500 3000
0
2
4
6
8
10
12
14

Motivating example
collected from a
500 1000 1500 2000 2500 3000
0
2
4
6
8
10
12
14
Observation model:
Yt|Xt = xt ∼ Binomial (50, κ(xt)) , κ(u) := (1 + exp(−u))−1

Motivating example
collected from a
500 1000 1500 2000 2500 3000
0
2
4
6
8
10
12
14
Observation model:
Yt|Xt = xt ∼ Binomial (50, κ(xt)) , κ(u) := (1 + exp(−u))−1
Latent Markov chain:
X0 ∼ N(0, 1), Xt |Xt−1 = xt−1 ∼ N(αxt−1, σ2
), t ∈ [1 : 2999],
θ = (α, σ2
) ∈ [0, 1] × R+ are the unknown parameters

Feynman-Kac formulation
Consider proposal Markov chain (Xt )t∈[0:T] with law
Q(dx0:T ) = µ(dx0)
T
t=1
Mt(xt−1, dxt )

Q(dx0:T ) = µ(dx0)
T
t=1
Mt(xt−1, dxt )
Consider estimating
Z =
XT+1
G0(x0)
T
t=1
Gt(xt−1, xt)Q(dx0:T )
given positive bounded potential functions (Gt )t∈[0:T]

Q(dx0:T ) = µ(dx0)
T
t=1
Mt(xt−1, dxt )
Consider estimating
Z =
XT+1
G0(x0)
T
t=1
Deﬁne target path measure
P(dx0:T ) = G0(x0)
T
t=1
Gt (xt−1, xt )Q(dx0:T )Z−1

Q(dx0:T ) = µ(dx0)
T
t=1
Mt(xt−1, dxt )
Consider estimating
Z =
XT+1
G0(x0)
T
t=1
Deﬁne target path measure
P(dx0:T ) = G0(x0)
T
t=1
Gt (xt−1, xt )Q(dx0:T )Z−1
The quantities µ, (Mt )t∈[1:T], (Gt )t∈[0:T] depend on the speciﬁc
application

Sequential Monte Carlo methods
SMC methods simulate an interacting particle system of size
N ∈ N

N ∈ N
At time t = 0 and particle n ∈ [1 : N]

N ∈ N
sample Xn
0 ∼ µ and weight W n
0 ∝ G0(Xn
0 );

N ∈ N
sample Xn
0 ∝ G0(Xn
0 );
sample ancestor index An
0 ∼ R W 1
0 , . . . , W N
0

N ∈ N
sample Xn
0 ∝ G0(Xn
0 );
0 ∼ R W 1
0 , . . . , W N
0
For time t ∈ [1 : T] and particle n ∈ [1 : N]

N ∈ N
sample Xn
0 ∝ G0(Xn
0 );
0 ∼ R W 1
0 , . . . , W N
0
sample Xn
t ∼ Mt (X
An
t−1
t−1 , ·) and weight W n
t ∝ Gt (X
An
t−1
t−1 , Xn
t );

N ∈ N
sample Xn
0 ∝ G0(Xn
0 );
0 ∼ R W 1
0 , . . . , W N
0
sample Xn
t ∼ Mt (X
An
t−1
t−1 , ·) and weight W n
t ∝ Gt (X
An
t−1
t−1 , Xn
t );
t ∼ R W 1
t , . . . , W N
t

Unbiased estimator of Z
ZN
=
1
N
N
n=1
G0(Xn
0 )
T
t=1
1
N
N
n=1
Gt (X
An
t−1
t−1 , Xn
t )

ZN
=
1
N
N
n=1
G0(Xn
0 )
T
t=1
1
N
N
n=1
Gt (X
An
t−1
t−1 , Xn
t )
Particle approximation of P
PN
=
1
N
N
n=1
δXn
0:T
where Xn
0:T is obtained by tracing ancestral lineage of particle Xn
T

ZN
=
1
N
N
n=1
G0(Xn
0 )
T
t=1
1
N
N
n=1
Gt (X
An
t−1
t−1 , Xn
t )
PN
=
1
N
N
n=1
δXn
0:T
where Xn
T
Convergence properties of ZN
and PN
as N → ∞ are now
well-understood

ZN
=
1
N
N
n=1
G0(Xn
0 )
T
t=1
1
N
N
n=1
Gt (X
An
t−1
t−1 , Xn
t )
PN
=
1
N
N
n=1
δXn
0:T
where Xn
T
and PN
well-understood
However quality of approximation can be inadequate for practical
choices of N

ZN
=
1
N
N
n=1
G0(Xn
0 )
T
t=1
1
N
N
n=1
Gt (X
An
t−1
t−1 , Xn
t )
PN
=
1
N
N
n=1
δXn
0:T
where Xn
T
and PN
well-understood
However quality of approximation can be inadequate for practical
choices of N
Performance crucially depends on discrepancy between P and Q

Neuroscience model continued
Using bootstrap particle ﬁlter (BPF), i.e. deﬁne
Q(dx0:T ) = µ(dx0) T
t=1 Mt(xt−1, dxt ) by
X0 ∼ N(0, 1), Xt |Xt−1 = xt−1 ∼ N(αxt−1, σ2
)

X0 ∼ N(0, 1), Xt |Xt−1 = xt−1 ∼ N(αxt−1, σ2
)
BPF struggles whenever the observations change abruptly
500 1000 1500 2000 2500 3000
0
2
4
6
8
10
12
14
0 500 1000 1500 2000 2500
0
10
20
30
40
50
60
70
80
90
100

X0 ∼ N(0, 1), Xt |Xt−1 = xt−1 ∼ N(αxt−1, σ2
)
BPF struggles whenever the observations change abruptly
500 1000 1500 2000 2500 3000
0
2
4
6
8
10
12
14
0 500 1000 1500 2000 2500
0
10
20
30
40
50
60
70
80
90
100
Better performance obtained with observations dependent
dynamics

T
P(dx0|y0:T ) =
µ(dx0)ψ∗
0 (x0)
µ(ψ∗
0 )
, P(dxt |xt−1, yt:T ) =
t (xt )
Mt(ψ∗
t )(xt−1)
where ψ∗
Given a policy ψ = (ψt )t∈[0:T], i.e. positive and bounded functions

T
P(dx0|y0:T ) =
µ(dx0)ψ∗
0 (x0)
µ(ψ∗
0 )
, P(dxt |xt−1, yt:T ) =
t (xt )
Mt(ψ∗
t )(xt−1)
where ψ∗
Given a policy ψ = (ψt )t∈[0:T], i.e. positive and bounded functions
Deﬁne ψ-twisted path measure of Q as
Qψ
(dx0:T ) = µψ
(dx0)
T
t=1
Mψ
t (xt−1, dxt )
where
µψ
(dx0) :=
µ(dx0)ψ0(x0)
µ(ψ0)
, Mψ
t (xt−1, dxt) :=
Mt(xt−1, dxt)ψt (xt−1, xt )
Mt(ψt )(xt−1)

Now deﬁne twisted potentials (Gψ
t )t∈[0;T] such that
P(dx0:T ) = Gψ
0 (x0)
T
t=1
Gψ
t (xt−1, xt ) Qψ
(dx0:T )Z−1
hence Z = EQψ Gψ
0 (X0) T
t=1 Gψ
t (Xt−1, Xt )

Now deﬁne twisted potentials (Gψ
t )t∈[0;T] such that
P(dx0:T ) = Gψ
0 (x0)
T
t=1
Gψ
t (xt−1, xt ) Qψ
(dx0:T )Z−1
hence Z = EQψ Gψ
0 (X0) T
t=1 Gψ
t (Xt−1, Xt )
Achieved with
Gψ
0 (x0) :=
µ(ψ0)G0(x0)M1(ψ1)(x0)
ψ0(x0)
,
Gψ
t (xt−1, xt ) :=
Gt(xt−1, xt)Mt+1(ψt+1)(xt )
ψt (xt−1, xt )
, t ∈ [1 : T − 1],
Gψ
T (xT−1, xT ) :=
GT (xT−1, xT )
ψT (xT−1, xT )

Assume policy ψ is such that:

sampling µψ
and (Mψ
t )t∈[1:T] feasible

sampling µψ
and (Mψ
evaluating (Gψ
t )t∈[0:T] tractable

sampling µψ
and (Mψ
evaluating (Gψ
For neuroscience model, we consider Gaussian approximation
ψt(xt ) = exp −atx2
t − bt xt − ct , (at , bt, ct ) ∈ R3
so
µψ
(x0) = N(x0; −k0b0, k0),
Mψ
t (xt−1, xt ) = N xt ; kt(ασ−2
xt−1 − bt), kt
with k0 := (1 + 2a0)−1
and kt := (σ−2
+ 2at)−1

sampling µψ
and (Mψ
evaluating (Gψ
For neuroscience model, we consider Gaussian approximation
ψt(xt ) = exp −atx2
t − bt xt − ct , (at , bt, ct ) ∈ R3
so
µψ
(x0) = N(x0; −k0b0, k0),
Mψ
t (xt−1, xt ) = N xt ; kt(ασ−2
xt−1 − bt), kt
with k0 := (1 + 2a0)−1
and kt := (σ−2
+ 2at)−1
Integrals µ(ψ0) and xt → Mt+1(ψt+1)(xt ) are tractable

Twisted SMC
Construct ψ-twisted SMC as standard SMC applied to
µψ
, (Mψ
t )t∈[1:T], (Gψ
t )t∈[0:T]

Twisted SMC
µψ
, (Mψ
t )t∈[1:T], (Gψ
t )t∈[0:T]
Zψ,N
=
1
N
N
n=1
Gψ
0 (Xn
0 )
T
t=1
1
N
N
n=1
Gψ
t (X
An
t−1
t−1 , Xn
t )

Twisted SMC
µψ
, (Mψ
t )t∈[1:T], (Gψ
t )t∈[0:T]
Zψ,N
=
1
N
N
n=1
Gψ
0 (Xn
0 )
T
t=1
1
N
N
n=1
Gψ
t (X
An
t−1
t−1 , Xn
t )
Pψ,N
=
1
N
N
n=1
δXn
0:T

Optimal policies
Policy with constant functions recover standard SMC

Optimal policies
Optimal policy
ψ∗
T (xT−1xT ) = GT (xT−1, xT ),
ψ∗
t (xt−1, xt) = Gt (xt−1, xt )Mt+1(ψ∗
t+1)(xt ), t ∈ [T − 1 : 1],
ψ∗
0 (x0) = G0(x0)M1(ψ∗
1 )(x0)

Optimal policies
Optimal policy
ψ∗
T (xT−1xT ) = GT (xT−1, xT ),
ψ∗
t (xt−1, xt) = Gt (xt−1, xt )Mt+1(ψ∗
t+1)(xt ), t ∈ [T − 1 : 1],
ψ∗
0 (x0) = G0(x0)M1(ψ∗
1 )(x0)
Under ψ∗
= (ψ∗
t )t∈[0:T]
Qψ∗
= P and Zψ∗
,N
= Z for N ≥ 1

Optimal policies
Optimal policy
ψ∗
T (xT−1xT ) = GT (xT−1, xT ),
ψ∗
t (xt−1, xt) = Gt (xt−1, xt )Mt+1(ψ∗
t+1)(xt ), t ∈ [T − 1 : 1],
ψ∗
0 (x0) = G0(x0)M1(ψ∗
1 )(x0)
Under ψ∗
= (ψ∗
t )t∈[0:T]
Qψ∗
= P and Zψ∗
,N
= Z for N ≥ 1
Backward recursion deﬁning ψ∗
is typically intractable

Optimal policies
Optimal policy
ψ∗
T (xT−1xT ) = GT (xT−1, xT ),
ψ∗
t (xt−1, xt) = Gt (xt−1, xt )Mt+1(ψ∗
t+1)(xt ), t ∈ [T − 1 : 1],
ψ∗
0 (x0) = G0(x0)M1(ψ∗
1 )(x0)
Under ψ∗
= (ψ∗
t )t∈[0:T]
Qψ∗
= P and Zψ∗
,N
= Z for N ≥ 1
If potentials (Gt )t∈[0:T] and transition densities of (Mt )t∈[1:T] are
log-concave, then ψ∗
is log-concave

Optimal policies
Optimal policy
ψ∗
T (xT−1xT ) = GT (xT−1, xT ),
ψ∗
t (xt−1, xt) = Gt (xt−1, xt )Mt+1(ψ∗
t+1)(xt ), t ∈ [T − 1 : 1],
ψ∗
0 (x0) = G0(x0)M1(ψ∗
1 )(x0)
Under ψ∗
= (ψ∗
t )t∈[0:T]
Qψ∗
= P and Zψ∗
,N
= Z for N ≥ 1
If potentials (Gt )t∈[0:T] and transition densities of (Mt )t∈[1:T] are
log-concave, then ψ∗
is log-concave
For neuroscience model, this justiﬁes a Gaussian approximation
F := ξ(x) = exp −ax2
− bx − c : (a, b, c) ∈ R3

Connection to optimal control
V ∗
t := − log ψ∗
t are the optimal value functions of the
Kullback-Leibler control problem
inf
ψ∈Φ
KL Qψ
|P
Φ := {ψ ∈ Ψ : KL(Qψ
|P) < ∞}

V ∗
t := − log ψ∗
inf
ψ∈Φ
KL Qψ
|P
Φ := {ψ ∈ Ψ : KL(Qψ
|P) < ∞}
Methods to approximate backward recursion are known as
approximate dynamic programming (ADP) for ﬁnite horizon
control problems (Bertsekas and Tsitsiklis, 1996)

V ∗
t := − log ψ∗
inf
ψ∈Φ
KL Qψ
|P
Φ := {ψ ∈ Ψ : KL(Qψ
|P) < ∞}
Methods to approximate backward recursion are known as
approximate dynamic programming (ADP) for ﬁnite horizon
control problems (Bertsekas and Tsitsiklis, 1996)
Connection also useful for analysis (Tsitsiklis and Van Roy, 2001)

Approximate dynamic programming
First run standard SMC to get particles (Xn
t ) n ∈ [1 : N], t ∈ [0 : T]

t ) n ∈ [1 : N], t ∈ [0 : T]
Approximate backward recursion

t ) n ∈ [1 : N], t ∈ [0 : T]
For time T: approximate ψ∗
T = GT using
ˆψT = arg min
ξ∈FT
N
n=1
{log ξ − log GT }2
(X
An
T−1
T−1 , Xn
T )

t ) n ∈ [1 : N], t ∈ [0 : T]
T = GT using
ˆψT = arg min
ξ∈FT
N
n=1
(X
An
T−1
T−1 , Xn
T )
For time t ∈ [T − 1 : 0]: approximate ψ∗
t = Gt Mt+1(ψ∗
t+1) using
ˆψt = arg min
ξ∈Ft
N
n=1
log ξ − log Gt − log Mt+1( ˆψt+1)
2
(X
An
t−1
t−1 , Xn
t )

t ) n ∈ [1 : N], t ∈ [0 : T]
T = GT using
ˆψT = arg min
ξ∈FT
N
n=1
(X
An
T−1
T−1 , Xn
T )
t = Gt Mt+1(ψ∗
t+1) using
ˆψt = arg min
ξ∈Ft
N
n=1
2
(X
An
t−1
t−1 , Xn
t )
Output: policy ˆψ = ( ˆψt )t∈[0:T]

t ) n ∈ [1 : N], t ∈ [0 : T]
T = GT using
ˆψT = arg min
ξ∈FT
N
n=1
(X
An
T−1
T−1 , Xn
T )
t = Gt Mt+1(ψ∗
t+1) using
ˆψt = arg min
ξ∈Ft
N
n=1
2
(X
An
t−1
t−1 , Xn
t )
Run ˆψ-twisted SMC

t ) n ∈ [1 : N], t ∈ [0 : T]
T = GT using
ˆψT = arg min
ξ∈FT
N
n=1
(X
An
T−1
T−1 , Xn
T )
t = Gt Mt+1(ψ∗
t+1) using
ˆψt = arg min
ξ∈Ft
N
n=1
2
(X
An
t−1
t−1 , Xn
t )
Run ˆψ-twisted SMC
Algorithm is essentially Guarniero, Johansen & Lee (2017) if
projecting in natural scale

We obtain error bounds like
E ˆψt − ψ∗
t L2 ≤
T
s=t
Ct−1,s−1eN
s
where Ct,s are stability constants and eN
t are approximation
errors

We obtain error bounds like
E ˆψt − ψ∗
t L2 ≤
T
s=t
Ct−1,s−1eN
s
where Ct,s are stability constants and eN
t are approximation
errors
As N → ∞, ˆψ converges (LLN & CLT) to idealized ADP
˜ψT = arg min
ξ∈FT
E {log ξ − log GT }
2
(XT−1, XT ) ,
˜ψt = arg min
ξ∈Ft
E log ξ − log Gt − log Mt+1( ˜ψt+1)
2
(Xt−1, Xt )

ˆψ-twisted SMC improves upon standard SMC
0 500 1000 1500 2000 2500
0
10
20
30
40
50
60
70
80
90
100
Uncontrolled
Iteration 1
0 500 1000 1500 2000 2500
-3000
-2500
-2000
-1500
-1000
-500
Uncontrolled
Iteration 1

0 500 1000 1500 2000 2500
0
10
20
30
40
50
60
70
80
90
100
Uncontrolled
Iteration 1
0 500 1000 1500 2000 2500
-3000
-2500
-2000
-1500
-1000
-500
Uncontrolled
Iteration 1
Uncontrolled SMC approximates P(y0:t ), t ∈ [0 : T]

0 500 1000 1500 2000 2500
0
10
20
30
40
50
60
70
80
90
100
Uncontrolled
Iteration 1
0 500 1000 1500 2000 2500
-3000
-2500
-2000
-1500
-1000
-500
Uncontrolled
Iteration 1
Controlled SMC approximates P(y0:T ), t ∈ [0 : T]

0 500 1000 1500 2000 2500
0
10
20
30
40
50
60
70
80
90
100
Uncontrolled
Iteration 1
0 500 1000 1500 2000 2500
-3000
-2500
-2000
-1500
-1000
-500
Uncontrolled
Iteration 1
Controlled SMC approximates P(y0:T ), t ∈ [0 : T]
Can we do better?

Policy reﬁnement
Want to reﬁne current policy ˆψ

Policy reﬁnement
Twist Q
ˆψ
further with policy φ = (φt )t∈[0:T] gives
Q
ˆψ
φ
= Q
ˆψ·φ
, ˆψ · φ = ( ˆψt · φt )t∈[0:T]

Policy reﬁnement
Twist Q
ˆψ
Q
ˆψ
φ
= Q
ˆψ·φ
, ˆψ · φ = ( ˆψt · φt )t∈[0:T]
Optimal choice of φ (with respect to ˆψ)
φ∗
T (xT−1xT ) = G
ˆψ
T (xT−1, xT ),
φ∗
t (xt−1, xt ) = G
ˆψ
t (xt−1, xt )M
ˆψ
t+1(φ∗
t+1)(xt ), t ∈ [T − 1 : 1],
φ∗
0(x0) = G
ˆψ
0 (x0)M
ˆψ
1 (φ∗
1)(x0)
as ˆψ · φ∗
= ψ∗

Policy reﬁnement
Twist Q
ˆψ
Q
ˆψ
φ
= Q
ˆψ·φ
, ˆψ · φ = ( ˆψt · φt )t∈[0:T]
φ∗
T (xT−1xT ) = G
ˆψ
T (xT−1, xT ),
φ∗
t (xt−1, xt ) = G
ˆψ
t (xt−1, xt )M
ˆψ
t+1(φ∗
t+1)(xt ), t ∈ [T − 1 : 1],
φ∗
0(x0) = G
ˆψ
0 (x0)M
ˆψ
1 (φ∗
1)(x0)
as ˆψ · φ∗
= ψ∗
Approximate this backward recursion to obtain ˆφ = (ˆφt )t∈[0:T]

Policy reﬁnement
Twist Q
ˆψ
Q
ˆψ
φ
= Q
ˆψ·φ
, ˆψ · φ = ( ˆψt · φt )t∈[0:T]
φ∗
T (xT−1xT ) = G
ˆψ
T (xT−1, xT ),
φ∗
t (xt−1, xt ) = G
ˆψ
t (xt−1, xt )M
ˆψ
t+1(φ∗
t+1)(xt ), t ∈ [T − 1 : 1],
φ∗
0(x0) = G
ˆψ
0 (x0)M
ˆψ
1 (φ∗
1)(x0)
as ˆψ · φ∗
= ψ∗
using particles from ˆψ-twisted SMC and same function classes
(Ft )t∈[0:T]

Policy reﬁnement
Twist Q
ˆψ
Q
ˆψ
φ
= Q
ˆψ·φ
, ˆψ · φ = ( ˆψt · φt )t∈[0:T]
φ∗
T (xT−1xT ) = G
ˆψ
T (xT−1, xT ),
φ∗
t (xt−1, xt ) = G
ˆψ
t (xt−1, xt )M
ˆψ
t+1(φ∗
t+1)(xt ), t ∈ [T − 1 : 1],
φ∗
0(x0) = G
ˆψ
0 (x0)M
ˆψ
1 (φ∗
1)(x0)
as ˆψ · φ∗
= ψ∗
(Ft )t∈[0:T]
Construct reﬁned policy with ˆψ · ˆφ

Policy refinement
Twist Q
ˆψ
Q
ˆψ
φ
= Q
ˆψ·φ
, ˆψ · φ = ( ˆψt · φt )t∈[0:T]
φ∗
T (xT−1xT ) = G
ˆψ
T (xT−1, xT ),
φ∗
t (xt−1, xt ) = G
ˆψ
t (xt−1, xt )M
ˆψ
t+1(φ∗
t+1)(xt ), t ∈ [T − 1 : 1],
φ∗
0(x0) = G
ˆψ
0 (x0)M
ˆψ
1 (φ∗
1)(x0)
as ˆψ · φ∗
= ψ∗
(Ft )t∈[0:T]
Construct refined policy with ˆψ · ˆφ
Call this iterative scheme to refine policies controlled SMC

Iteration 2
0 500 1000 1500 2000 2500
0
10
20
30
40
50
60
70
80
90
100
Uncontrolled
Iteration 1
Iteration 2
0 500 1000 1500 2000 2500
-3000
-2500
-2000
-1500
-1000
-500
Uncontrolled
Iteration 1
Iteration 2
0 1000 2000
-3100
-3090
-3080
-3070

Iteration 3
0 500 1000 1500 2000 2500
0
10
20
30
40
50
60
70
80
90
100
Uncontrolled
Iteration 1
Iteration 2
Iteration 3
0 500 1000 1500 2000 2500
-3000
-2500
-2000
-1500
-1000
-500
Uncontrolled
Iteration 1
Iteration 2
Iteration 3
0 1000 2000
-3100
-3090
-3080
-3070

Under Ft := ξ(x) = exp −atx2
− bt x − ct : (at, bt , ct) ∈ R3
,
policy at iteration i ≥ 1
ψ
(i)
t (xt ) = exp −a
(i)
t x2
t − b
(i)
t xt − c
(i)
t
where (a
(i)
t , b
(i)
t , c
(i)
t ) =
i
k=1(ak
t , bk
t , ck
t )

Under Ft := ξ(x) = exp −atx2
− bt x − ct : (at, bt , ct) ∈ R3
,
policy at iteration i ≥ 1
ψ
(i)
t (xt ) = exp −a
(i)
t x2
t − b
(i)
t xt − c
(i)
t
where (a
(i)
t , b
(i)
t , c
(i)
t ) =
i
k=1(ak
t , bk
t , ck
t )
(ak
t , bk
t , ck
t ) are coeﬃcients estimated at iteration k ≥ 1
0 500 1000 1500 2000 2500 3000
-4
-2
0
2
4
6
8
Iteration 1
Iteration 2
Iteration 3
0 500 1000 1500 2000 2500 3000
-20
-10
0
10
20
30
40
Iteration 1
Iteration 2
Iteration 3

Policy refinement
Residual from first ADP when fitting ˆψ is
εt := log ˆψt − log Gt − log Mt+1( ˆψt+1)

Policy refinement
Next ADP refinement re-fits residual (like L2
-boosting)
ˆφt = arg min
ξ∈F
N
n=1
log ξ − εt − log M
ˆψ
t+1(ˆφt+1)
2
(X
An
t−1
t−1 , Xn
t )

Policy refinement
Next ADP refinement re-fits residual (like L2
-boosting)
ˆφt = arg min
ξ∈F
N
n=1
log ξ − εt − log M
ˆψ
t+1(ˆφt+1)
2
(X
An
t−1
t−1 , Xn
t )
Twisted potential of refined policy ˆψ · ˆφ
− log G
ˆψ· ˆφ
t = log ˆφt − log G
ˆψ
t − log M
ˆψ
t+1(ˆφt+1)
new residual from fitting ˆφ

Policy reﬁnement
Iterative scheme generates a Markov chain on policy space
FN ( ˆψ, U) = ˆψ · ˆφ
where ˆψ is current policy and ˆφ is ADP output

Policy reﬁnement
Iterative scheme generates a Markov chain on policy space
FN ( ˆψ, U) = ˆψ · ˆφ
where ˆψ is current policy and ˆφ is ADP output
Converges to a unique invariant distribution that concentrates
around ﬁxed points of
F( ˆψ) = ˆψ · ˜φ
where ˆψ is current policy and ˜φ is idealized ADP output
1.8 2 2.2 2.4 2.6 2.8 3 3.2
0.1
0.2
0.3
0.4
0.5
0.6
0.7
14 16 18 20 22 24
0
1
2
3
4
5
6

(Left) Comparison to bootstrap particle ﬁlter (BPF)

(Left) Comparison to bootstrap particle ﬁlter (BPF)
(Right) Comparison to forward ﬁltering and backward smoother
(FFBS) for functional x0:T → 50κ(x0:T )
0.05 0.1 0.15 0.2
-9.5
-9
-8.5
-8
-7.5
-7
-6.5
-6 BPF
cSMC
0 500 1000 1500 2000 2500 3000
-4
-3.5
-3
-2.5
-2
-1.5
-1
-0.5
FFBS
cSMC
Figure: Relative variance of marginal likelihood estimates (left) and
estimates of smoothing expectation (right).

Bayesian inference for parameters θ = (α, σ2
) within particle
marginal Metropolis-Hastings (PMMH)

) within particle
cSMC and BPF to produce unbiased estimates of marginal likelihood

) within particle
Autocorrelation function (ACF) of each PMMH chain
0 10 20 30 40 50
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
BPF
cSMC
0 10 20 30 40 50
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
BPF
cSMC

) within particle
Autocorrelation function (ACF) of each PMMH chain
0 10 20 30 40 50
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
BPF
cSMC
0 10 20 30 40 50
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
BPF
cSMC
ESS improvement roughly 5 times for both parameters

Concluding remarks
Methodology can be extended to static models or SMC samplers
πt (dx) =
π0(dx)L(x)λt
Zt
where 0 = λ0 < λ1 < · · · < λT = 1

Concluding remarks
πt (dx) =
π0(dx)L(x)λt
Zt
where 0 = λ0 < λ1 < · · · < λT = 1
Draft: Controlled Sequential Monte Carlo, arXiv:1708.08396, 2017.

Concluding remarks
πt (dx) =
π0(dx)L(x)λt
Zt
where 0 = λ0 < λ1 < · · · < λT = 1
Draft: Controlled Sequential Monte Carlo, arXiv:1708.08396, 2017.
MATLAB code: https://github.com/jeremyhengjm/controlledSMC

Talk in BayesComp 2018

More Related Content

What's hot (20)

Similar to Talk in BayesComp 2018 (20)

More from JeremyHeng10 (8)

Recently uploaded (20)

Talk in BayesComp 2018