The jackknife and bootstrap

BIOL309: The Jackknife & Bootstrap
Paul Gardner
September 25, 2017
Paul Gardner BIOL309: The Jackknife & Bootstrap

What is Resampling?
“I do not believe in any statistical test unless I can prove it with a permutation test.” – R.A. Fisher
Resampling is a statistical technique in which multiple new
samples are drawn from a sample or from the population
Statistics of interest (e.g. sample median) are calculated for
each new sample. The distribution of new statistics can be
analysed to investigate diﬀerent properties (e.g., conﬁdence
intervals, the error, the bias) of the statistics.
Sampling
Inference

First, some definitions & reminders
Mean ¯x = 1
n
n
i=1 xi
Variance s2 = 1
n−1
n
i=1(xi − ¯x)2
Standard deviation s = (xi −¯x)2
n−1
Standard error SE¯x = (xi −¯x)2
n(n−1)
Bias of an estimator is the difference between the estimators
expected value and the true value of the parameter being
estimated
Confidence interval
out
in
0
50
100
150
200
250
300
350
out
in
0
200
400
600
800

Jackknifing
a resampling technique especially useful for finding standard
error, variance and bias of estimators
the jackknife is a small, handy tool
also called leave-one-out (LOO)
This approach tests that some outlier datapoint is not having
a disproportionate influence on the outcome.

Jackkniﬁng
The jackknife deletes each observation and calculates an
estimate based on the remaining n − 1 values
It uses this collection of estimates to do things like estimate
the bias and the standard error

Jackkniﬁng: deﬁnition
Let x1, . . . xn be a dataset
θ is a paramater you want to estimate from the data (e.g.
mean, median, standard deviation, ...)
Let ˆθ be the estimate based upon the entire dataset
Let ˆθi be the estimate of θ obtained by deleting observation
xi
Let ¯θ = 1
n
n
i=1
ˆθi
Sometimes ¯θ is written ¯θ(.)

Jackkniﬁng: estimating bias of a method, and correcting it
This provides an estimated correction of bias due to the
estimation method. The jackknife does not correct for a
biased sample.(Wikipedia/Jackknife resampling)
The jackknife estimate of bias is B = (n − 1)(¯θ − ˆθ)
In other words, is the diﬀerence between the actual and the
average of the delete-one estimates.
We can then correct ˆθ (the estimator on the entire dataset),
using:
ˆθcorrected = ˆθ − B
With the magic of algebra:
ˆθcorrected = nˆθ − (n − 1)¯θ

Jackkniﬁng
The jackknife estimate of the standard error is:
SEJK (ˆθ) =
n − 1
n
n
i=1
(ˆθi − ¯θ)2
This simpliﬁes to the standard error (SE¯x = (xi −¯x)2
n(n−1) ) when
θ is the mean

Example
x1 <- rnorm(1000, mean = 2, sd = 1)
x <- c(x1,-10)
hist(x,breaks=500)
library(bootstrap)
#define theta function
theta <- function(x){sd(x)}
j <- jackknife(x,theta)
mean(j$jack.values)
#check j$jack.values is normal
hist(j$jack.values,breaks=500)
#What is the bias corrected sd?
Histogram of x
x
Frequency
−4 −2 0 2 4
01030
Histogram of j$jack.values
j$jack.values
Frequency
1.02 1.04 1.06 1.08 1.10
050100150200250300

CORRECTION: testing bias corrected values...
The lab example...
for (i in c(10,100, 1000) ){
for (j in c(-100,-10,-1,1, 10,100) ){
x <- c(rnorm(i, mean = 2, sd = 1),j)
jk <- jackknife(x,sd)
corr <- sd(x) - jk$jack.bias
cat(paste(round(corr, digits = 2), "t"))
}
cat("n")
}
Jackknife bias corrected values (i = N, j =outlier, expected= 1)
ij -100 -10 -1 1 10 100
10 44.29 4.76 1.31 0.65 2.96 42.25
100 14.28 1.67 0.94 0.96 1.34 13.64
1000 4.21 1.07 0.99 1.01 1.05 4.03

Example: more bad...
The jackknife estimate of variance is slightly biased upward!
Efron & Stein (1981) The jackknife estimate of variance. The Annals of Statistics, pp. 586-596

Jackkniﬁng issues
When the estimator is not normally distributed jackkniﬁng
may fail
May be unreliable on a small number of datasets
This provides an estimated correction of bias due to the
estimation method. The jackknife does not correct for a
biased sample.(Wikipedia/Jackknife resampling)
Not great when θ is the standard deviation!

What is bootstrapping?
Bootstrapping is a useful means for assessing the reliability of
your data (e.g. confidence intervals, bias, variance, prediction
error, ...).
It refers to any metric that relies on random sampling with
replacement.
Used to estimate SE, confidence intervals, and test for
significance

First, a deﬁnition
Central limit theorem:
the means from a large number of independent random
samples will be approximately normally distributed, regardless
of the underlying distribution
X
Frequency
0 2 4 6 8
04080140
Bootstrap means of X
Frequency
0.90 1.00 1.10
0204060
X
Frequency
0 5 10 15
0100200
Bootstrap means of X
Frequency
2.3 2.5 2.7
0204060
Bootstrap
Bootstrap
N=1,000
N=1,000

Bootstrapping illustrated
(unknown) true distribution
(unknown) true value of θ
empirical distribution of sample
estimate of θ
bootstrap replicate 1
distribution of estimates of θ
Bootstrap sampling from a distribution (a mixture of 3 normal
distributions) to estimate the variance of the mean

Bootstrapping is used a lot in phylogenetics
Yang & Rannala (2012) Molecular phylogenetics: principles and practice. Nature Reviews Genetics.

Application: DNA surveillance
http://dna-surveillance.fos.auckland.ac.nz/

Bootstrap sampling
To infer the error in a quantity, θ, estimated from a dataset
x1, x2, . . . xN we do the following R times (e.g. R = 1, 000):
1. Draw a “bootstrap sample” by sampling n times with
replacement from the sample. Call these X*
1 , X∗
2 , . . . X∗
n . Note
that some points are represented more than once in the
bootstrap samples, some once, some not at all.
2. Estimate θ from the bootstrap sample, call this ˆθ∗
k
(k = 1, 2, . . . R).
3. When all R bootstrap samples have been done, the
distribution of ˆθ∗
k estimates the distribution one would get if
one were able to draw repeated samples of n points from the
unknown true distribution.

Example: conﬁdence intervals for the median
x1=rnorm(500, mean = 2, sd = 1)
x2=rnorm(500, mean = -2, sd = 1)
x=c(x1,x2)
hist(x,breaks=50)
summary(x)
library(bootstrap)
#define theta function
theta = function(x){median(x)}
bs = bootstrap(x,50,theta)
summary(bs$thetastar)
#What is the 50% confidence
#interval for boostrap estimates
#of median?
boott(x,theta,nboott=1000,perc=c(0.025,0.975))
Histogram of x
x
Frequency
−4 −2 0 2 4
01030

Example: regression (I)
#create a simulated dataset, sampling from a normal distribution
x<-runif(1000,-10,10)
#generate a y dataset with a little noise:
#y = m * x + c
y<-rnorm(length(x),1,0.1)*x + rnorm(length(x),mean=0,sd=1)
#plot a regression
reg1<-lm(y ~ x)
plot(x,y,type="p")
abline(reg1,col="red",lwd=3)

Example: regression (II)
library(bootstrap)
#column bind x & y
xdata <- cbind(x,y)
#create functions, theta1 & theta2,
#1 returns the intercept, 2 returns the slope
theta1 <- function(i,xdata){
coef(lm(xdata[i,2] ~ xdata[i,1]))[1]
}
theta2 <- function(i,xdata){
coef(lm(xdata[i,2] ~ xdata[i,1]))[2]
}
#bootstrap!
bs1=bootstrap(1:length(x),1000,theta1,xdata)
bs2=bootstrap(1:length(x),1000,theta2,xdata)
quantile(bs2$thetastar,probs = c(0.025,0.975))

Example: regression (III)
#plot the resulting lines:
for (i in 1:length(bs1$thetastar)){
abline(bs1$thetastar[i],bs2$thetastar[i], lty=2,col="pink")
}
abline(reg1,col="red",lwd=3)
hist(bs1$thetastar,breaks=100,main="Intercepts")
hist(bs2$thetastar,breaks=100,main="Slopes")
q
q
q
q
q
q
qq
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
qq
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
qq
qq
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q q
q
q
qq q
q
q q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q qq
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
qq
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
−10 −5 0 5 10
−10−50510
x
y
Intercepts
bs1$thetastar
Frequency
−0.8 −0.4 0.0
0102030
Slopes
bs2$thetastarFrequency
0.90 0.95 1.00 1.05 1.10
010203040

Bootstrap issues
Need a large number of bootstrap samples (e.g. R ≥ 1000).
The larger the number, the better the estimates.
If θ is hard to calculate (e.g. tree building) then
bootstrapping can be very computationally intensive.

Another example:
−4 −2 0 2
Robust Z−score (F−measure)
EBI−mg
NBC
MetaPhlan
MLTreeMap
Treephyler
RITA
MEGAN
taxator−tk
RAIphy
MetaPhyler
mothur
Kraken
phymmBL
Taxy−Pro
Genometa
Quickr
BMP
QIIME
metaCV
GOTTCHA
LMAT
mOTU
TIPP
CLARK
FOCUS
MG−RAST
MetaBin
CLARK−S
PhyloPythiaS
OneCodex
DUDes
CARMA3
commonkmers
DiScRIBinATE
MetaPhlAn2.0
TACOA
qqqqqq
|| ||
qq qq q qq q qq qqqqq qqq qqq
| qqqq
|| ||
qqqqqq
|| ||
q
|| ||
q
|| ||
qqqqqq
|| ||
qqqqqqq qqqqqqq qqqqqqq qqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
|qqq qqqq qqqq qq q qq qqq qq
| qqqqqq
|q qqqq
||||
qqqqqqq q qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
| qqqqqq
||||
qq qq
|| ||
q qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq q
|q q qq qq
|qqqqqq
|q qqq q
| | ||
q qq qqqqqq qq qqqq qqqqqq qq qqqq qqqqqq qq qq qq qqqqqqqqqq qq qq qq qqqqqq qq qqqq qqqqqq qq q qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqq qqqqqqqqqqqq qq qq qq qqqqqq qqq
||||
qqqqqqq qqqqqqq qqqqqq qqqqqqq qqqqqqq qqqqqqq
|qqqq qq qqqq qq
|qqq qqqq qqq q qqq qq qqq qq
| qqqqqq
|| ||
qqqqq
| q qqq
|| ||
q qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq q
||||
qqqqqq
|| ||
||||
qqqq qqqqqqqqqqqq qqqqqqqqqqqq qqqqqqqqqq qq qq qqqq qqqqqqqq qq qq qqqq qqqq qq qq qqqq qq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqq q qqq q qqqq qqq q qqq q qqqqqqq q qqq qq
||||
qqqqqqq qqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqq qqqqqq qqqqqqqqqqqqqqqqqqqqqqq qqq qqq qq q qq qqq qq q qq qqq q qq qqqqqq q qqq qqq qq q qqqqq qqq qq qq qqq qqq qqq qq qqqqqq qqqqqqq q qqqqqqqqqqqqqqqqqqq qq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqq qq qq q qq qqqqqq q qq qq qq qqqq qq qqq qq qqqqqq qqq qqqqqqqq qq qqq qq qqqqqq qq q qq qqq
|qqqqqq
| |||
qqqqq
|| ||
qqqqqq
|| ||
qq q qq qq q qqq qqq qq qqq qq
| qqqqqq
||||
|qqqqqq
| | ||
|| ||
q qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qqqq qq qq qq qq qqqq qq qqqq qq qqqq qq qqq
|q qqqqqqqqqq qqqqqqqqqqqqqqqqqqqqq
| qqqqqq
| qqq qqqq qqq q qqq qqqqq qq
|qqqqqq
||||
q qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq q
|| ||
qqqqqq
|qq qqq
||||
q qqqq q
|| ||
qqq qqqq qqqq qqq qqqqq qq
|| ||
q qqqqqqqqqqqqq q qqqqqq q qqqqqqq qqqqqq qqqqqqq qqqqqqq
|qqqq
| | ||
qqqqqq
|| ||
q qq qq qq qq qq qq qq qq qq qq qq q
|| ||
q qqqqq
|q qqqq
| |||
q qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq qq q
||||
q qq qqqqqqqqq
||||
||||
q q qqqq
|| ||
q Bazinet.2012q Lindgreen.2016
q McIntyre.2017q Peabody.2015
q Sczyrba.2017q Siegwald.2017

For more information
Chapters 1, 2 and 3:
Manly, B. F. (2006). Randomization, bootstrap and Monte Carlo
methods in biology (Vol. 70). CRC Press.
https://books.google.co.nz/books?id=j2UN5xDMbIsC&redir esc=y

UC Summer undergraduate Research Scholarships
A list of Summer Research Scholarships is now available at
http://www.canterbury.ac.nz/summer-school/summer-scholarships/
Scholarships are for final year undergraduates only
Scholarships are for 10 weeks (Nov-Feb) and valued at $5,000
Students should apply by the 19th
September by completing the
Application form found at the above website

Need to talk things over?
Practical guidance, advice and
support for our domestic and
international students.
Student Care
studentcare@canterbury.ac.nz
Advice, help and
support on campus
Are you Māori
and need
advice, cultural
or academic support?
Māori Student Development Team
maoridevelopment@canterbury.ac.nz
A disability or
medical condition
affecting your
study?
Disability Resource Service
disabilities@canterbury.ac.nz
Upskill your
academic
writing and
study skills.
Academic Skills Centre
academicskills@canterbury.ac.nz
Are you Pasifika
and need advice,
cultural or academic
support?
Pacific Development Team
pasifika@canterbury.ac.nz
Have issues?
Need help?
Students’ Association
(UCSA)
help@ucsa.org.nz
Medical care,
counselling,
travel advice,
or physiotherapy.
UC Health Centre
healthcentre@canterbury.ac.nz
Feel more energised.
Lift. Move. Play.
Compete. Excel.
UC RecCentre
@UC RecCentre
UC Sport
@UC Sport
Develop your employability.
Visit UC Careers: www.canterbury.ac.nz/careers
Feeling unsafe or need emergency
help? UC Security 0800 823 637

Not tested: Visualisation: good vs bad
Image source: https://commons.wikimedia.org/wiki/File:Piecharts.svg

Not tested: Visualisation: good vs bad
Weissgerber et al. (2015) Beyond Bar and Line Graphs: Time for a New Data Presentation Paradigm. PLOS
Biology.

The End

The jackknife and bootstrap

More Related Content

What's hot (20)

Similar to The jackknife and bootstrap (20)

More from Paul Gardner (20)

Recently uploaded (20)

The jackknife and bootstrap