Program on Quasi-Monte Carlo and High-Dimensional Sampling Methods for Applied Mathematics Opening Workshop, Error Analysis for Quasi-Monte Carlo Methods - Fred Hickernell, Aug 28, 2017

Error Analysis for Quasi-Monte Carlo Methods
Fred J. Hickernell
Department of Applied Mathematics, Illinois Institute of Technology
hickernell@iit.edu mypages.iit.edu/~hickernell
Thanks to the Guaranteed Automatic Integration Library (GAIL) team and friends
Supported by NSF-DMS-1522687
Thanks to SAMSI and the QMC Program organizers
SAMSI-QMC Opening Worskshop, August 28, 2017

Introduction RKHS Dimension Eﬀects Interlude Bayesian Cubature GAIL Summary References
Integration Examples
µ =
Rd
f(x) ν(dx) = option price,
f(x) = discounted payoﬀ determined by market forces x
µ =
X
g(x) dx = P(X ∈ X) = probability
g = probability density function
µj
µ0
= X
bjL(b|data) (b) db
X
L(b|data) (b) db
= Bayesian posterior expectation of βj
µ = E[f(X)] =
X
f(x) ν(dx) ≈ ^µ =
n
i=1
f(xi)wi =
X
f(x) ^ν(dx) µ − ^µ = ?
2/22

µ := E[f(X)] :=
X
f(x) ν(dx) ≈ ^µ :=
n
i=1
f(xi)wi =
X
f(x) ^ν(dx)
µ − ^µ =
X
f(x) (ν − ^ν)(dx) = ?
Reality Error Analysis (Theory)
What is What we think it should be by rigorous argument
Not certain that our assumptions hold
3/22

µ := E[f(X)] :=
X
f(x) ν(dx) ≈ ^µ :=
n
i=1
f(xi)wi =
X
f(x) ^ν(dx)
µ − ^µ =
X
f(x) (ν − ^ν)(dx) = ?
Reality Error Analysis (Theory)
What is What we think it should be by rigorous argument
Not certain that our assumptions hold
Quasi-Monte Carlo Methods
x1, x2, . . . chosen more carefully that IID, dependent
3/22

Error Decomposition in Reproducing Kernel Hilbert Spaces
Let the integrand lie in a Hilbert space, H, with reproducing kernel, K : X × X → R:
K(·, t) ∈ H, f(t) = K(·, t), f ∀t ∈ X, f ∈ H
The cubature error is given by the inner product with the representer, ηerr:
µ − ^µ =
X
f(x) (ν − ^ν)(dx) = ηerr, f , where ηerr(t) = K(·, t), ηerr =
X
K(x, t) (ν − ^ν)(dx)
The Cauchy-Schwartz inequality says that (H., 2000)
µ − ^µ = ηerr, f = cos(f, ηerr) DSC(ν − ^ν) f H discrepancy DSC(ν − ^ν) := ηerr H
DSC2
= ηerr, ηerr =
X×X
K(x, t) (ν − ^ν)(dx)(ν − ^ν)(dt)
=
X2
K(x, t) ν(dx)ν(dt) − 2
n
i=1
wi
X
K(xi, t) ν(dt) +
n
i,j=1
wiwjK(xi, xj)
= k0 − 2kT
w + wT
Kw w = K−1
k is optimal but expensive to compute
4/22

L2
-Discrepancy
ν = uniform on X = [0, 1]d
, ^ν = uniform on {xi}n
i=1, K(x, t) =
d
k=1
[2 − max(xk, tk)] (H., 1998)
f H = ∂u
f L2
u⊆1:d 2
, ∂u
f :=
∂|u|
f
∂xu xsu = 1
, 1:d := {1, . . . , d}, su := 1:d u
DSC2
=
4
3
d
−
2
n
n
i=1
d
k=1
3 − x2
ik
2
+
1
n2
n
i,j=1
d
k=1
[2 − max(xik, xjk)]
5/22

L2
-Discrepancy
i=1, K(x, t) =
d
k=1
[2 − max(xk, tk)] (H., 1998)
f H = ∂u
f L2
u⊆1:d 2
, ∂u
f :=
∂|u|
f
∂xu xsu = 1
, 1:d := {1, . . . , d}, su := 1:d u
DSC2
=
4
3
d
−
2
n
n
i=1
d
k=1
3 − x2
ik
2
+
1
n2
n
i,j=1
d
k=1
= ν([0, ·u]) − ^ν([0, ·u]) L2
∅=u⊆1:d
2
2
geometric interpretation
x = (. . . , 0.6, . . . , 0.4, . . .)
ν([0, x{5,8}]) − ^ν([0, x{5,8}])
= 0.24 − 7/32 = 0.02125
5/22

L2
-Discrepancy
i=1, K(x, t) =
d
k=1
[2 − max(xk, tk)] (H., 1998)
f H = ∂u
f L2
u⊆1:d 2
, ∂u
f :=
∂|u|
f
∂xu xsu = 1
, 1:d := {1, . . . , d}, su := 1:d u
DSC2
=
4
3
d
−
2
n
n
i=1
d
k=1
3 − x2
ik
2
+
1
n2
n
i,j=1
d
k=1
= ν([0, ·u]) − ^ν([0, ·u]) L2
∅=u⊆1:d
2
2
geometric interpretation
µ − ^µ = cos(f, ηerr) DSC(ν − ^ν) f − f(1) H
DSC(ν − ^ν)
requires O(dn2
) operations to evaluate



= O(n−1/2
) on average for IID Monte Carlo
= O(n−1+
) for digital nets, integration lattices, ...
(Niederreiter, 1992; Dick and Pillichshammer, 2010)
O(n−1
) for all ^ν because this K has limited smoothness 5/22

Multivariate Normal Probability
µ =
[a,b]
exp −1
2 tT
Σ−1
t
(2π)d det(Σ)
dt
Genz (1993)
=
[0,1]d−1
f(x) dx, ^ν = uniform on {xi}n
i=1
µ − ^µ = cos(f, ηerr) DSC(ν − ^ν) f H
DSC(ν−^ν) =



O(n−1/2
) for IID MC
O(n−1+
) for Sobol’
6/22

µ =
[a,b]
exp −1
2 tT
Σ−1
t
(2π)d det(Σ)
dt
Genz (1993)
=
[0,1]d−1
i=1
DSC(ν−^ν) =



O(n−1/2
) for IID MC
O(n−1+
) for Sobol’
O(n−1.5+
) for Scr. Sobol’
w/ smoother kernel (Owen, 1997)
(H. and Yue, 2000; Heinrich et al., 2004)
6/22

µ =
[a,b]
exp −1
2 tT
Σ−1
t
(2π)d det(Σ)
dt
Genz (1993)
=
[0,1]d−1
i=1
DSC(ν−^ν) =



O(n−1/2
) for IID MC
O(n−1+
) for Sobol’
O(n−1.5+
) for Scr. Sobol’
w/ smoother kernel (Owen, 1997)
(H. and Yue, 2000; Heinrich et al., 2004)
For some typical choice of a, b, Σ, and d = 3
µ ≈ 0.6763
Low discrepancy sampling reduces cubature error;
randomization can help more
6/22

Mean Square Discrepancy for Randomized Low Discrepancy Sets
Popular low discrepancy sets on [0, 1)d
—digital nets and lattice node sets—can be randomized while
preserving their low discrepancy properties. Let ψ : [0, 1)d
→ [0, 1)d
be a random mapping and
ψ(x) ∼ U[0, 1)d
, Kf(x, t) := E[K(ψ(x), ψ(t)] Kf1 = (1T
Kf,1)1, k0 = 1
where the vector Kf,1 is the ﬁrst column of the matrix Kf, the analog of K with the original sequence but
for the ﬁltered kernel, Kf. Taking equal weights w = 1/n,
E{[DSC(ν − ^ν; K)]2
} = E[k0 − 2kT
w + wT
Kw] = −1 +
1T
Kf,1
n
= [DSC(ν − ^ν; Kf)]2
7/22

Mean Square Discrepancy for Randomized Low Discrepancy Sets
Popular low discrepancy sets on [0, 1)d
—digital nets and lattice node sets—can be randomized while
preserving their low discrepancy properties. Let ψ : [0, 1)d
→ [0, 1)d
be a random mapping and
ψ(x) ∼ U[0, 1)d
, Kf(x, t) := E[K(ψ(x), ψ(t)] Kf1 = (1T
Kf,1)1, k0 = 1
where the vector Kf,1 is the ﬁrst column of the matrix Kf, the analog of K with the original sequence but
for the ﬁltered kernel, Kf. Taking equal weights w = 1/n,
E{[DSC(ν − ^ν; K)]2
} = E[k0 − 2kT
w + wT
Kw] = −1 +
1T
Kf,1
n
= [DSC(ν − ^ν; Kf)]2
DSC(ν − ^ν; Kf) only requires O(n) operations to compute
Suggests using Hilbert spaces with reproducing kernels of the form Kf
Kf(x, t) =
l
λlφl(x)sφl(t),
φl(x) = Walsh functions for digital nets
φl(x) = exp(2πlT
x) for lattice nodesets
with λl chosen carefully so that the series sums in closed form and Kf has the desired smoothness
properties
7/22

The Eﬀect of Dimension on the Decay of the Discrepancy
What happens when dimension, d, is large and but sample size, n, is modest?
L2
-discrepancy and variation:
DSC2
=
4
3
d
−
2
n
n
i=1
d
k=1
3 − x2
ik
2
+
1
n2
n
i,j=1
d
k=1
f H =
∂|u|
f
∂xu xsu=1 L2 u⊆1:d 2
For Scrambled Sobol’ points
The onset of O(n−1+
) convergence of DSC is d dependent.
8/22

Pricing an Asian Option
µ =
Rd
payoﬀ(t)
exp −1
2 tT
Σ−1
t
(2π)d det(Σ)
dt =
[0,1]d
f(x) dx
Asian arithmetic mean option, d = 12, µ ≈ $13.1220
Error converges like O(n−1+
) for (scrambled)
Sobol’ even though discrepancy does not.
Phenomenon ﬁrst seen by Paskov and Traub
(1995) for Collaterlized Mortgage Obligation
with d = 360
Scrambled Sobol’ does not achieve
O(n−3/2+
) convergence because f is not
smooth enough.
f is not even smooth enough for O(n−1+
)
convergence, except by a delicate argument
(Griebel et al., 2010; 2016+)
9/22

Overcoming the Curse of Dimensionality with Weights
Sloan and Woźniakowski (1998) show how inserting decaying coordinate weights yields O(n−1+
)
convergence independent of dimension
Caﬂisch et al. (1997) explain how quasi-Monte Carlo methods work well for low eﬀective dimension
Novak and Woźniakowski (2010) cover tractability comprehensively
K(x, t) =
d
k=1
[1 + γ2
k(1 − max(xk, tk))]
DSC2
=
d
k=1
1 +
γ2
k
3
−
2
n
n
i=1
d
k=1
1 +
γ2
k(1 − x2
ik)
2
+
1
n2
n
i,j=1
d
k=1
[1 + γ2
k(1 − max(xik, xjk))]
f H =
1
γu
∂|u|
f
∂xu xsu=1 L2
u⊆1:d 2
γu =
k∈u
γk
For Scrambled Sobol’ points
γ2
k = k−3
10/22

Comments So Far
µ := E[f(X)] :=
X
f(x) ν(dx) ≈ ^µ :=
n
i=1
f(xi)wi =
X
f(x) ^ν(dx)
Cubature error represented as a trio identity (Meng, 2017+; H., 2017+); without the cos() term it is a
Koksma-Hlawka inequality
There are other versions than the deterministic one shown here
RKHS is an oft used tool—not only for cubature error (Fasshauer and McCourt, 2015)—and the analysis
may be extended to Banach spaces (H., 1998; 2000)
Error analysis can be attacked directly through Fourier Walsh or complex exponential series
If kernel has special form, then DSC(ν − ^ν) and a lower bound on f H can be computed in O(n log n)
operations; otherwise they are expensive
For this error decomposition the adaptive choice of the position of the next data site xn+1 does not
depend on f
11/22

Comments So Far
µ := E[f(X)] :=
X
f(x) ν(dx) ≈ ^µ :=
n
i=1
f(xi)wi =
X
f(x) ^ν(dx)
The discrepancy measures the sample quality, and is independent of the integrand.
Decay rate may be improved by samples more carefully chosen than IID
Clever randomization low discrepancy points can reduce the discrepancy even further
Definition depends on the reproducing kernel (and the corresponding Hilbert space of integrands)
Working with filtered kernels may expedite computation of the discrepancy and of the optimal weights.
What is the appropriate filtered kernel for higher order nets?
Non-uniform low discrepancy points are often transformations of uniform low discrepancy points
There are other quality measures for points than the discrepancy, e.g., covering radius, minimum
distance between two points, the power function in RKHS approximation
11/22

Comments So Far
µ := E[f(X)] :=
X
f(x) ν(dx) ≈ ^µ :=
n
i=1
f(xi)wi =
X
f(x) ^ν(dx)
The discrepancy measures the sample quality, and is independent of the integrand.
The strategy for choosing points that optimize the quality measure depends on whether you want to be
asymptotically optimal or pre-asymptotically optimal. See
WG IV Representative Points for Small-data and Big-data Problems
WG V Sampling and Analysis in High Dimensions When Samples Are Expensive
The size of the discrepancy may be affected by the dimension unless the space of integrands is defined
to de-emphasize higher dimensions. For how to choose the weights and sampling optimally for your
problem, see
WG AO Tuning QMC to the Specific Integrand of Interest
11/22

Comments So Far
µ := E[f(X)] :=
X
f(x) ν(dx) ≈ ^µ :=
n
i=1
f(xi)wi =
X
f(x) ^ν(dx)
The term f H is often called the variation.
It can sometimes be made smaller by using a change of measure (variable transformation) to re-write
the integral with a diﬀerent integrand.
11/22

Comments So Far
µ := E[f(X)] :=
X
f(x) ν(dx) ≈ ^µ :=
n
i=1
f(xi)wi =
X
f(x) ^ν(dx)
The term f H is often called the variation.
It can sometimes be made smaller by using a change of measure (variable transformation) to re-write
the integral with a diﬀerent integrand.
The term cos(f, ηerr) is called the confounding. It cannot exceed one in magnitude. It could be
quite small if your integrand is atypically nice.
11/22

Bayesian Cubature
Random f postulated by Diaconis (1988), O’Hagan (1991), Ritter (2000), Rasmussen and Ghahramani
(2003) and others: f ∼ GP(0, s2
Cθ), a Gaussian process from the sample space F with zero mean and
covariance kernel, s2
Cθ, Cθ : X × X → R. Then
c0 =
X2
C(x, t) ν(dx)ν(dt), c =
X
C(xi, t) ν(dt)
n
i=1
C = C(xi, xj)
n
i,j=1
,
w = wi
n
i=1
= C−1
c is optimal
µ − ^µ = N(0, 1) × c0 − cTC−1c × s
The scale parameter, s, and shape parameter, θ, should be estimated, e.g. by maximum likelihood
estimation
Pr |µ − ^µ| 2.58 c0 − cTC−1c × s = 99% allows probabilistic error bounds
Requires O(n3
) operations to compute C−1
θ , but see Anitescu et al. (2016)
Ill-conditioning for smoother kernels (faster convergence)
DSC(ν − ^ν; K) = c0 − cTC−1c for optimal weights an d K = Cθ
See WG II Probabilistic Numerics 12/22

µ =
[a,b]
exp −1
2 tT
Σ−1
t
(2π)d det(Σ)
dt
Genz (1993)
=
[0,1]d−1
f(x) dx
µ − ^µ = N(0, 1) × c0 − cTC−1c × s
Use a product Matérn kernel with modest smoothness:
Cθ(x, t) =
d
k=1
(1 + θ |xk − tk|)e−θ|xk−tk|
Smaller error using Bayesian cubature with
scrambled Sobol’ data sites
13/22

µ =
[a,b]
exp −1
2 tT
Σ−1
t
(2π)d det(Σ)
dt
Genz (1993)
=
[0,1]d−1
f(x) dx
µ − ^µ = N(0, 1) × c0 − cTC−1c × s
Use a product Matérn kernel with modest smoothness:
Cθ(x, t) =
d
k=1
(1 + θ |xk − tk|)e−θ|xk−tk|
Smaller error using Bayesian cubature with
scrambled Sobol’ data sites
Conﬁdence intervals succeed ≈ 83% of the time
13/22

How Many Samples Are Needed?
We know that
But, given an error tolerance, ε, how do we decide how many samples, n, are needed to make
|µ − ^µ| ε
Bayesian cubature provides data-based confidence intervals for ^µ
For digital net (e.g., Sobol’) and integration lattice sampling
Independent replications present a trade-off between number of replications and number of samples
But rigorous stopping criteria for a single sequence have been constructed in terms of the discrete
Fourier Walsh/complex exponential coefficients of f assuming that the true Fourier coefficients decay
steadily (H. and Jiménez Rugama, 2016; Jiménez Rugama and H., 2016; Li, 2016)
Functions of integrals and relative error criteria may also be handled (H. and Jiménez Rugama, 2016)
These automatic cubature rules are part of the Guaranteed Automatic Integration Library (GAIL) (Choi et
al., 2013–2017)
14/22

Option Pricing
µ = fair price =
Rd
e−rT
max

1
d
d
j=1
Sj − K, 0

 e−zT
z/2
(2π)d/2
dz
≈ $13.12
Sj = S0e(r−σ2
/2)jT/d+σxj
= stock price at time jT/d,
x = Az, AAT
= Σ = min(i, j)T/d
d
i,j=1
, T = 1/4, d = 13 here
Abs. Error Median Worst 10% Worst 10%
Tolerance Method A Error Accuracy n Time (s)
1E−2 IID diﬀ 2E−3 100% 6.1E7 33.000
1E−2 Scr. Sobol’ PCA 1E−3 100% 1.6E4 0.040
1E−2 Scr. Sob. cont. var. PCA 2E−3 100% 4.1E3 0.019
1E−2 Bayes. Latt. PCA 2E−3 99% 1.6E4 0.051
15/22

Gaussian Probability
µ =
[a,b]
exp −1
2 tT
Σ−1
t
(2π)d det(Σ)
dt
Genz (1993)
=
[0,1]d−1
f(x) dx
For some typical choice of a, b, Σ, d = 3; µ ≈ 0.6763
Rel. Error Median Worst 10% Worst 10%
Tolerance Method Error Accuracy n Time (s)
1E−2 IID Genz 5E−4 100% 8.1E4 0.020
1E−2 Scr. Sobol’ Genz 4E−5 100% 1.0E3 0.005
1E−2 Bayes. Latt. Genz 5E−5 100% 4.1E3 0.023
1E−3 IID Genz 9E−5 100% 2.0E6 0.400
1E−4 Bayes. Latt. Smth. Genz 1E−7 100% 3.3E4 0.047
16/22

Bayesian Inference for Logistic Regression
µj
µ0
= X
bjL(b|data) (b) db
X
L(b|data) (b) db
= Bayesian posterior expectation of βj
yi ∼ Ber
exp(β1 + β2xi)
1 + exp(β1 + β2xi)
, (b) =
exp −(b2
1 + b2
2)/2
2π
ε = 0.001, n = 9 000–17 000
17/22

Summary
Reproducing kernel Hilbert spaces are a powerful for characterizing the error of quasi-Monte Carlo
methods
Digital nets and nodesets of lattice have faster decaying discrepancies which suggests faster
decaying cubature errors
How well these bounds ﬁt what is observed depends on whether the integrand is typical within the
space of intgrands
Practical stopping criteria are an area of active research
18/22

Thank you
Slides available at www.slideshare.net/fjhickernell/
hickernell-fred-samsi-tutorial-aug-2017

References I
Anitescu, M., J. Chen, and M. Stein. 2016. An inversion-free estimating equation approach for Gaussian process models, J.
Comput. Graph. Statist.
Caflisch, R. E., W. Morokoff, and A. Owen. 1997. Valuation of mortgage backed securities using Brownian bridges to reduce
effective dimension, J. Comput. Finance 1, 27–46.
Choi, S.-C. T., Y. Ding, F. J. H., L. Jiang, Ll. A. Jiménez Rugama, X. Tong, Y. Zhang, and X. Zhou. 2013–2017. GAIL:
Guaranteed Automatic Integration Library (versions 1.0–2.2).
Diaconis, P. 1988. Bayesian numerical analysis, Statistical decision theory and related topics IV, Papers from the 4th Purdue
symp., West Lafayette, Indiana 1986, pp. 163–175.
Dick, J. and F. Pillichshammer. 2010. Digital nets and sequences: Discrepancy theory and quasi-Monte Carlo integration,
Cambridge University Press, Cambridge.
Fasshauer, G. E. and M. McCourt. 2015. Kernel-based approximation methods using MATLAB, Interdisciplinary Mathematical
Sciences, vol. 19, World Scientific Publishing Co., Singapore.
Genz, A. 1993. Comparison of methods for the computation of multivariate normal probabilities, Computing Science and
Statistics 25, 400–405.
Griebel, M., F. Y. Kuo, and I. H. Sloan. 2010. The smoothing effect of the ANOVA decomposition, J. Complexity 26, 523–551.
. 2016+. The ANOVA decomposition of a non-smooth function of infinitely many variables can have every term smooth,
Math. Comp. in press.
20/22

References II
Heinrich, S., F. J. H., and R. X. Yue. 2004. Optimal quadrature for Haar wavelet spaces, Math. Comp. 73, 259–277.
H., F. J. 1998. A generalized discrepancy and quadrature error bound, Math. Comp. 67, 299–322.
. 2000. What aﬀects the accuracy of quasi-Monte Carlo quadrature?, Monte Carlo and quasi-Monte Carlo methods 1998,
pp. 16–55.
. 2017+. The trio identity for quasi-Monte Carlo error analysis, Monte Carlo and quasi-Monte Carlo methods: MCQMC,
Stanford, usa, August 2016. submitted for publication, arXiv:1702.01487.
H., F. J. and Ll. A. Jiménez Rugama. 2016. Reliable adaptive cubature using digital sequences, Monte Carlo and quasi-Monte
Carlo methods: MCQMC, Leuven, Belgium, April 2014, pp. 367–383. arXiv:1410.8615 [math.NA].
H., F. J. and R. X. Yue. 2000. The mean square discrepancy of scrambled (t, s)-sequences, SIAM J. Numer. Anal. 38,
1089–1112.
Jiménez Rugama, Ll. A. and F. J. H. 2016. Adaptive multidimensional integration based on rank-1 lattices, Monte Carlo and
quasi-Monte Carlo methods: MCQMC, Leuven, Belgium, April 2014, pp. 407–422. arXiv:1411.1966.
Li, D. 2016. Reliable quasi-Monte Carlo with control variates, Master’s Thesis.
Meng, X. 2017+. Statistical paradises and paradoxes in big data. in preparation.
Niederreiter, H. 1992. Random number generation and quasi-Monte Carlo methods, CBMS-NSF Regional Conference Series in
Applied Mathematics, SIAM, Philadelphia.
Novak, E. and H. Woźniakowski. 2010. Tractability of multivariate problems Volume ii: Standard information for functionals, EMS
Tracts in Mathematics, European Mathematical Society, Zürich.
21/22

References III
O’Hagan, A. 1991. Bayes-Hermite quadrature, J. Statist. Plann. Inference 29, 245–260.
Owen, A. B. 1997. Scrambled net variance for integrals of smooth functions, Ann. Stat. 25, 1541–1562.
Paskov, S. and J. Traub. 1995. Faster valuation of ﬁnancial derivatives, J. Portfolio Management 22, 113–120.
Rasmussen, C. E. and Z. Ghahramani. 2003. Bayesian Monte Carlo, Advances in Neural Information Processing Systems,
pp. 489–496.
Ritter, K. 2000. Average-case analysis of numerical problems, Lecture Notes in Mathematics, vol. 1733, Springer-Verlag, Berlin.
Sloan, I. H. and H. Woźniakowski. 1998. When are quasi-Monte Carlo algorithms eﬃcient for high dimensional integrals?, J.
Complexity 14, 1–33.
22/22

Program on Quasi-Monte Carlo and High-Dimensional Sampling Methods for Applied Mathematics Opening Workshop, Error Analysis for Quasi-Monte Carlo Methods - Fred Hickernell, Aug 28, 2017

More Related Content

What's hot (20)

Similar to Program on Quasi-Monte Carlo and High-Dimensional Sampling Methods for Applied Mathematics Opening Workshop, Error Analysis for Quasi-Monte Carlo Methods - Fred Hickernell, Aug 28, 2017 (20)

More from The Statistical and Applied Mathematical Sciences Institute (20)

Recently uploaded (20)

Program on Quasi-Monte Carlo and High-Dimensional Sampling Methods for Applied Mathematics Opening Workshop, Error Analysis for Quasi-Monte Carlo Methods - Fred Hickernell, Aug 28, 2017