![2.1 What Is Statistical Learning? 19
Consider a given estimate ˆ
f and a set of predictors X, which yields the
prediction Ŷ = ˆ
f(X). Assume for a moment that both ˆ
f and X are fixed,
so that the only variability comes from ϵ. Then, it is easy to show that
E(Y − Ŷ )2
= E[f(X) + ϵ − ˆ
f(X)]2
= [f(X) − ˆ
f(X)]2
, -. /
Reducible
+ Var(ϵ)
, -. /
Irreducible
, (2.3)
where E(Y − Ŷ )2
represents the average, or expected value, of the squared
expected
value
difference between the predicted and actual value of Y , and Var(ϵ) repre-
sents the variance associated with the error term ϵ.
variance
The focus of this book is on techniques for estimating f with the aim of
minimizing the reducible error. It is important to keep in mind that the
irreducible error will always provide an upper bound on the accuracy of
our prediction for Y . This bound is almost always unknown in practice.
Inference
We are often interested in understanding the association between Y and
X1, . . . , Xp. In this situation we wish to estimate f, but our goal is not
necessarily to make predictions for Y . Now ˆ
f cannot be treated as a black
box, because we need to know its exact form. In this setting, one may be
interested in answering the following questions:
• Which predictors are associated with the response? It is often the case
that only a small fraction of the available predictors are substantially
associated with Y . Identifying the few important predictors among a
large set of possible variables can be extremely useful, depending on
the application.
• What is the relationship between the response and each predictor?
Some predictors may have a positive relationship with Y , in the sense
that larger values of the predictor are associated with larger values of
Y . Other predictors may have the opposite relationship. Depending
on the complexity of f, the relationship between the response and a
given predictor may also depend on the values of the other predictors.
• Can the relationship between Y and each predictor be adequately sum-
marized using a linear equation, or is the relationship more compli-
cated? Historically, most methods for estimating f have taken a linear
form. In some situations, such an assumption is reasonable or even de-
sirable. But often the true relationship is more complicated, in which
case a linear model may not provide an accurate representation of
the relationship between the input and output variables.
In this book, we will see a number of examples that fall into the prediction
setting, the inference setting, or a combination of the two.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-30-320.jpg)
![34 2. Statistical Learning
0 20 40 60 80 100
−10
0
10
20
X
Y
2 5 10 20
0
5
10
15
20 Flexibility
Mean
Squared
Error
FIGURE 2.11. Details are as in Figure 2.9, using a different f that is far from
linear. In this setting, linear regression provides a very poor fit to the data.
terms ϵ. That is,
E
1
y0 − ˆ
f(x0)
22
= Var( ˆ
f(x0)) + [Bias( ˆ
f(x0))]2
+ Var(ϵ). (2.7)
Here the notation E
1
y0 − ˆ
f(x0)
22
defines the expected test MSE at x0,
expected
test MSE
and refers to the average test MSE that we would obtain if we repeatedly
estimated f using a large number of training sets, and tested each at x0. The
overall expected test MSE can be computed by averaging E
1
y0 − ˆ
f(x0)
22
over all possible values of x0 in the test set.
Equation 2.7 tells us that in order to minimize the expected test error,
we need to select a statistical learning method that simultaneously achieves
low variance and low bias. Note that variance is inherently a nonnegative
quantity, and squared bias is also nonnegative. Hence, we see that the
expected test MSE can never lie below Var(ϵ), the irreducible error from
(2.3).
What do we mean by the variance and bias of a statistical learning
method? Variance refers to the amount by which ˆ
f would change if we
estimated it using a different training data set. Since the training data
are used to fit the statistical learning method, different training data sets
will result in a different ˆ
f. But ideally the estimate for f should not vary
too much between training sets. However, if a method has high variance
then small changes in the training data can result in large changes in ˆ
f. In
general, more flexible statistical methods have higher variance. Consider the
green and orange curves in Figure 2.9. The flexible green curve is following
the observations very closely. It has high variance because changing any](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-45-320.jpg)
![2.3 Lab: Introduction to R 43
The RStudio website also provides a cloud-based version of R, which does
not require installing any software.
2.3.1 Basic Commands
R uses functions to perform operations. To run a function called funcname,
function
we type funcname(input1, input2), where the inputs (or arguments) input1
argument
and input2 tell R how to run the function. A function can have any number
of inputs. For example, to create a vector of numbers, we use the function
c() (for concatenate). Any numbers inside the parentheses are joined to-
c()
gether. The following command instructs R to join together the numbers
1, 3, 2, and 5, and to save them as a vector named x. When we type x, it vector
gives us back the vector.
> x <- c(1, 3, 2, 5)
> x
[1] 1 3 2 5
Note that the > is not part of the command; rather, it is printed by R to
indicate that it is ready for another command to be entered. We can also
save things using = rather than <-:
> x = c(1, 6, 2)
> x
[1] 1 6 2
> y = c(1, 4, 3)
Hitting the up arrow multiple times will display the previous commands,
which can then be edited. This is useful since one often wishes to repeat
a similar command. In addition, typing ?funcname will always cause R to
open a new help file window with additional information about the function
funcname().
We can tell R to add two sets of numbers together. It will then add the
first number from x to the first number from y, and so on. However, x and
y should be the same length. We can check their length using the length()
length()
function.
> length(x)
[1] 3
> length(y)
[1] 3
> x + y
[1] 2 10 5
The ls() function allows us to look at a list of all of the objects, such
ls()
as data and functions, that we have saved so far. The rm() function can be
rm()
used to delete any that we don’t want.
> ls()
[1] "x" "y"
> rm(x, y)](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-54-320.jpg)
![44 2. Statistical Learning
> ls()
character (0)
It’s also possible to remove all objects at once:
> rm(list = ls())
The matrix() function can be used to create a matrix of numbers. Before
matrix()
we use the matrix() function, we can learn more about it:
> ?matrix
The help file reveals that the matrix() function takes a number of inputs,
but for now we focus on the first three: the data (the entries in the matrix),
the number of rows, and the number of columns. First, we create a simple
matrix.
> x <- matrix(data = c(1, 2, 3, 4), nrow = 2, ncol = 2)
> x
[,1] [,2]
[1,] 1 3
[2,] 2 4
Note that we could just as well omit typing data=, nrow=, and ncol= in the
matrix() command above: that is, we could just type
> x <- matrix(c(1, 2, 3, 4), 2, 2)
and this would have the same effect. However, it can sometimes be useful to
specify the names of the arguments passed in, since otherwise R will assume
that the function arguments are passed into the function in the same order
that is given in the function’s help file. As this example illustrates, by
default R creates matrices by successively filling in columns. Alternatively,
the byrow = TRUE option can be used to populate the matrix in order of the
rows.
> matrix(c(1, 2, 3, 4), 2, 2, byrow = TRUE)
[,1] [,2]
[1,] 1 2
[2,] 3 4
Notice that in the above command we did not assign the matrix to a value
such as x. In this case the matrix is printed to the screen but is not saved
for future calculations. The sqrt() function returns the square root of each
sqrt()
element of a vector or matrix. The command x^2 raises each element of x
to the power 2; any powers are possible, including fractional or negative
powers.
> sqrt(x)
[,1] [,2]
[1,] 1.00 1.73
[2,] 1.41 2.00
> x^2
[,1] [,2]
[1,] 1 9
[2,] 4 16](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-55-320.jpg)
![2.3 Lab: Introduction to R 45
The rnorm() function generates a vector of random normal variables,
rnorm()
with first argument n the sample size. Each time we call this function, we
will get a different answer. Here we create two correlated sets of numbers,
x and y, and use the cor() function to compute the correlation between
cor()
them.
> x <- rnorm (50)
> y <- x + rnorm (50, mean = 50, sd = .1)
> cor(x, y)
[1] 0.995
By default, rnorm() creates standard normal random variables with a mean
of 0 and a standard deviation of 1. However, the mean and standard devi-
ation can be altered using the mean and sd arguments, as illustrated above.
Sometimes we want our code to reproduce the exact same set of random
numbers; we can use the set.seed() function to do this. The set.seed()
set.seed()
function takes an (arbitrary) integer argument.
> set.seed (1303)
> rnorm (50)
[1] -1.1440 1.3421 2.1854 0.5364 0.0632 0.5022 -0.0004
. . .
We use set.seed() throughout the labs whenever we perform calculations
involving random quantities. In general this should allow the user to re-
produce our results. However, as new versions of R become available, small
discrepancies may arise between this book and the output from R.
The mean() and var() functions can be used to compute the mean and
mean()
var()
variance of a vector of numbers. Applying sqrt() to the output of var()
will give the standard deviation. Or we can simply use the sd() function.
sd()
> set.seed (3)
> y <- rnorm (100)
> mean(y)
[1] 0.0110
> var(y)
[1] 0.7329
> sqrt(var(y))
[1] 0.8561
> sd(y)
[1] 0.8561
2.3.2 Graphics
The plot() function is the primary way to plot data in R. For instance,
plot()
plot(x, y) produces a scatterplot of the numbers in x versus the numbers
in y. There are many additional options that can be passed in to the plot()
function. For example, passing in the argument xlab will result in a label
on the x-axis. To find out more information about the plot() function,
type ?plot.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-56-320.jpg)
![46 2. Statistical Learning
> x <- rnorm (100)
> y <- rnorm (100)
> plot(x, y)
> plot(x, y, xlab = "this is the x-axis",
ylab = "this is the y-axis",
main = "Plot of X vs Y")
We will often want to save the output of an R plot. The command that we
use to do this will depend on the file type that we would like to create. For
instance, to create a pdf, we use the pdf() function, and to create a jpeg,
pdf()
we use the jpeg() function.
jpeg()
> pdf("Figure.pdf")
> plot(x, y, col = "green")
> dev.off ()
null device
1
The function dev.off() indicates to R that we are done creating the plot.
dev.off()
Alternatively, we can simply copy the plot window and paste it into an
appropriate file type, such as a Word document.
The function seq() can be used to create a sequence of numbers. For
seq()
instance, seq(a, b) makes a vector of integers between a and b. There are
many other options: for instance, seq(0, 1, length = 10) makes a sequence
of 10 numbers that are equally spaced between 0 and 1. Typing 3:11 is a
shorthand for seq(3, 11) for integer arguments.
> x <- seq(1, 10)
> x
[1] 1 2 3 4 5 6 7 8 9 10
> x <- 1:10
> x
[1] 1 2 3 4 5 6 7 8 9 10
> x <- seq(-pi , pi , length = 50)
We will now create some more sophisticated plots. The contour() func-
contour()
tion produces a contour plot in order to represent three-dimensional data;
contour plot
it is like a topographical map. It takes three arguments:
1. A vector of the x values (the first dimension),
2. A vector of the y values (the second dimension), and
3. A matrix whose elements correspond to the z value (the third dimen-
sion) for each pair of (x, y) coordinates.
As with the plot() function, there are many other inputs that can be used
to fine-tune the output of the contour() function. To learn more about
these, take a look at the help file by typing ?contour.
> y <- x
> f <- outer(x, y, function (x, y) cos(y) / (1 + x^2))
> contour(x, y, f)
> contour(x, y, f, nlevels = 45, add = T)](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-57-320.jpg)
![2.3 Lab: Introduction to R 47
> fa <- (f - t(f)) / 2
> contour(x, y, fa , nlevels = 15)
The image() function works the same way as contour(), except that it
image()
produces a color-coded plot whose colors depend on the z value. This is
known as a heatmap, and is sometimes used to plot temperature in weather
heatmap
forecasts. Alternatively, persp() can be used to produce a three-dimensional
persp()
plot. The arguments theta and phi control the angles at which the plot is
viewed.
> image(x, y, fa)
> persp(x, y, fa)
> persp(x, y, fa , theta = 30)
> persp(x, y, fa , theta = 30, phi = 20)
> persp(x, y, fa , theta = 30, phi = 70)
> persp(x, y, fa , theta = 30, phi = 40)
2.3.3 Indexing Data
We often wish to examine part of a set of data. Suppose that our data is
stored in the matrix A.
> A <- matrix (1:16 , 4, 4)
> A
[,1] [,2] [,3] [,4]
[1,] 1 5 9 13
[2,] 2 6 10 14
[3,] 3 7 11 15
[4,] 4 8 12 16
Then, typing
> A[2, 3]
[1] 10
will select the element corresponding to the second row and the third col-
umn. The first number after the open-bracket symbol [ always refers to
the row, and the second number always refers to the column. We can also
select multiple rows and columns at a time, by providing vectors as the
indices.
> A[c(1, 3), c(2, 4)]
[,1] [,2]
[1,] 5 13
[2,] 7 15
> A[1:3 , 2:4]
[,1] [,2] [,3]
[1,] 5 9 13
[2,] 6 10 14
[3,] 7 11 15
> A[1:2 , ]
[,1] [,2] [,3] [,4]
[1,] 1 5 9 13
[2,] 2 6 10 14](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-58-320.jpg)
![48 2. Statistical Learning
> A[, 1:2]
[,1] [,2]
[1,] 1 5
[2,] 2 6
[3,] 3 7
[4,] 4 8
The last two examples include either no index for the columns or no index
for the rows. These indicate that R should include all columns or all rows,
respectively. R treats a single row or column of a matrix as a vector.
> A[1, ]
[1] 1 5 9 13
The use of a negative sign - in the index tells R to keep all rows or columns
except those indicated in the index.
> A[-c(1, 3), ]
[,1] [,2] [,3] [,4]
[1,] 2 6 10 14
[2,] 4 8 12 16
> A[-c(1, 3), -c(1, 3, 4)]
[1] 6 8
The dim() function outputs the number of rows followed by the number of
dim()
columns of a given matrix.
> dim(A)
[1] 4 4
2.3.4 Loading Data
For most analyses, the first step involves importing a data set into R. The
read.table() function is one of the primary ways to do this. The help file
read.table()
contains details about how to use this function. We can use the function
write.table() to export data.
write.table()
Before attempting to load a data set, we must make sure that R knows
to search for the data in the proper directory. For example, on a Windows
system one could select the directory using the Change dir. . . option under
the File menu. However, the details of how to do this depend on the oper-
ating system (e.g. Windows, Mac, Unix) that is being used, and so we do
not give further details here.
We begin by loading in the Auto data set. This data is part of the ISLR2
library, discussed in Chapter 3. To illustrate the read.table() function, we
load it now from a text file, Auto.data, which you can find on the textbook
website. The following command will load the Auto.data file into R and
store it as an object called Auto, in a format referred to as a data frame.
data frame
Once the data has been loaded, the View() function can be used to view](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-59-320.jpg)
![50 2. Statistical Learning
> dim(Auto)
[1] 397 9
> Auto [1:4 , ]
The dim() function tells us that the data has 397 observations, or rows, and
dim()
nine variables, or columns. There are various ways to deal with the missing
data. In this case, only five of the rows contain missing observations, and
so we choose to use the na.omit() function to simply remove these rows.
na.omit()
> Auto <- na.omit(Auto)
> dim(Auto)
[1] 392 9
Once the data are loaded correctly, we can use names() to check the
names()
variable names.
> names(Auto)
[1] "mpg" "cylinders" " displacement " "horsepower"
[5] "weight" " acceleration " "year" "origin"
[9] "name"
2.3.5 Additional Graphical and Numerical Summaries
We can use the plot() function to produce scatterplots of the quantitative
scatterplot
variables. However, simply typing the variable names will produce an error
message, because R does not know to look in the Auto data set for those
variables.
> plot(cylinders , mpg)
Error in plot(cylinders , mpg) : object ‘cylinders ’ not found
To refer to a variable, we must type the data set and the variable name
joined with a $ symbol. Alternatively, we can use the attach() function in
attach()
order to tell R to make the variables in this data frame available by name.
> plot(Auto$cylinders , Auto$mpg)
> attach(Auto)
> plot(cylinders , mpg)
The cylinders variable is stored as a numeric vector, so R has treated it
as quantitative. However, since there are only a small number of possible
values for cylinders, one may prefer to treat it as a qualitative variable.
The as.factor() function converts quantitative variables into qualitative
as.factor()
variables.
> cylinders <- as.factor(cylinders)
If the variable plotted on the x-axis is qualitative, then boxplots will
boxplot
automatically be produced by the plot() function. As usual, a number
of options can be specified in order to customize the plots.
> plot(cylinders , mpg)
> plot(cylinders , mpg , col = "red")
> plot(cylinders , mpg , col = "red", varwidth = T)](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-61-320.jpg)
![2.4 Exercises 55
• Outstate : Out-of-state tuition
• Room.Board : Room and board costs
• Books : Estimated book costs
• Personal : Estimated personal spending
• PhD : Percent of faculty with Ph.D.’s
• Terminal : Percent of faculty with terminal degree
• S.F.Ratio : Student/faculty ratio
• perc.alumni : Percent of alumni who donate
• Expend : Instructional expenditure per student
• Grad.Rate : Graduation rate
Before reading the data into R, it can be viewed in Excel or a text
editor.
(a) Use the read.csv() function to read the data into R. Call the
loaded data college. Make sure that you have the directory set
to the correct location for the data.
(b) Look at the data using the View() function. You should notice
that the first column is just the name of each university. We don’t
really want R to treat this as data. However, it may be handy to
have these names for later. Try the following commands:
> rownames (college) <- college[, 1]
> View(college)
You should see that there is now a row.names column with the
name of each university recorded. This means that R has given
each row a name corresponding to the appropriate university. R
will not try to perform calculations on the row names. However,
we still need to eliminate the first column in the data where the
names are stored. Try
> college <- college[, -1]
> View(college)
Now you should see that the first data column is Private. Note
that another column labeled row.names now appears before the
Private column. However, this is not a data column but rather
the name that R is giving to each row.
(c) i. Use the summary() function to produce a numerical summary
of the variables in the data set.
ii. Use the pairs() function to produce a scatterplot matrix of
the first ten columns or variables of the data. Recall that
you can reference the first ten columns of a matrix A using
A[,1:10].](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-66-320.jpg)
![56 2. Statistical Learning
iii. Use the plot() function to produce side-by-side boxplots of
Outstate versus Private.
iv. Create a new qualitative variable, called Elite, by binning
the Top10perc variable. We are going to divide universities
into two groups based on whether or not the proportion
of students coming from the top 10 % of their high school
classes exceeds 50 %.
> Elite <- rep("No", nrow(college))
> Elite[college$Top10perc > 50] <- "Yes"
> Elite <- as.factor(Elite)
> college <- data.frame(college , Elite)
Use the summary() function to see how many elite univer-
sities there are. Now use the plot() function to produce
side-by-side boxplots of Outstate versus Elite.
v. Use the hist() function to produce some histograms with
differing numbers of bins for a few of the quantitative vari-
ables. You may find the command par(mfrow = c(2, 2))
useful: it will divide the print window into four regions so
that four plots can be made simultaneously. Modifying the
arguments to this function will divide the screen in other
ways.
vi. Continue exploring the data, and provide a brief summary
of what you discover.
9. This exercise involves the Auto data set studied in the lab. Make sure
that the missing values have been removed from the data.
(a) Which of the predictors are quantitative, and which are quali-
tative?
(b) What is the range of each quantitative predictor? You can an-
swer this using the range() function.
range()
(c) What is the mean and standard deviation of each quantitative
predictor?
(d) Now remove the 10th through 85th observations. What is the
range, mean, and standard deviation of each predictor in the
subset of the data that remains?
(e) Using the full data set, investigate the predictors graphically,
using scatterplots or other tools of your choice. Create some plots
highlighting the relationships among the predictors. Comment
on your findings.
(f) Suppose that we wish to predict gas mileage (mpg) on the basis
of the other variables. Do your plots suggest that any of the
other variables might be useful in predicting mpg? Justify your
answer.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-67-320.jpg)
![3.1 Simple Linear Regression 67
In the case of the advertising data, the 95 % confidence interval for β0
is [6.130, 7.935] and the 95 % confidence interval for β1 is [0.042, 0.053].
Therefore, we can conclude that in the absence of any advertising, sales will,
on average, fall somewhere between 6,130 and 7,935 units. Furthermore,
for each $1,000 increase in television advertising, there will be an average
increase in sales of between 42 and 53 units.
Standard errors can also be used to perform hypothesis tests on the
hypothesis
test
coefficients. The most common hypothesis test involves testing the null
hypothesis of
null
hypothesis
H0 : There is no relationship between X and Y (3.12)
versus the alternative hypothesis
alternative
hypothesis
Ha : There is some relationship between X and Y . (3.13)
Mathematically, this corresponds to testing
H0 : β1 = 0
versus
Ha : β1 ̸= 0,
since if β1 = 0 then the model (3.5) reduces to Y = β0 + ϵ, and X is
not associated with Y . To test the null hypothesis, we need to determine
whether β̂1, our estimate for β1, is sufficiently far from zero that we can
be confident that β1 is non-zero. How far is far enough? This of course
depends on the accuracy of β̂1—that is, it depends on SE(β̂1). If SE(β̂1) is
small, then even relatively small values of β̂1 may provide strong evidence
that β1 ̸= 0, and hence that there is a relationship between X and Y . In
contrast, if SE(β̂1) is large, then β̂1 must be large in absolute value in order
for us to reject the null hypothesis. In practice, we compute a t-statistic,
t-statistic
given by
t =
β̂1 − 0
SE(β̂1)
, (3.14)
which measures the number of standard deviations that β̂1 is away from 0. If
there really is no relationship between X and Y , then we expect that (3.14)
will have a t-distribution with n − 2 degrees of freedom. The t-distribution
has a bell shape and for values of n greater than approximately 30 it is
quite similar to the standard normal distribution. Consequently, it is a
simple matter to compute the probability of observing any number equal
to |t| or larger in absolute value, assuming β1 = 0. We call this probability
the p-value. Roughly speaking, we interpret the p-value as follows: a small
p-value
p-value indicates that it is unlikely to observe such a substantial association
between the predictor and the response due to chance, in the absence of
any real association between the predictor and the response. Hence, if we](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-77-320.jpg)
![82 3. Linear Regression
is only an estimate for the true population regression plane
f(X) = β0 + β1X1 + · · · + βpXp.
The inaccuracy in the coefficient estimates is related to the reducible
error from Chapter 2. We can compute a confidence interval in order
to determine how close Ŷ will be to f(X).
2. Of course, in practice assuming a linear model for f(X) is almost
always an approximation of reality, so there is an additional source of
potentially reducible error which we call model bias. So when we use a
linear model, we are in fact estimating the best linear approximation
to the true surface. However, here we will ignore this discrepancy,
and operate as if the linear model were correct.
3. Even if we knew f(X)—that is, even if we knew the true values
for β0, β1, . . . , βp—the response value cannot be predicted perfectly
because of the random error ϵ in the model (3.20). In Chapter 2, we
referred to this as the irreducible error. How much will Y vary from
Ŷ ? We use prediction intervals to answer this question. Prediction
intervals are always wider than confidence intervals, because they
incorporate both the error in the estimate for f(X) (the reducible
error) and the uncertainty as to how much an individual point will
differ from the population regression plane (the irreducible error).
We use a confidence interval to quantify the uncertainty surrounding
confidence
interval
the average sales over a large number of cities. For example, given that
$100,000 is spent on TV advertising and $20,000 is spent on radio advertising
in each city, the 95 % confidence interval is [10,985, 11,528]. We interpret
this to mean that 95 % of intervals of this form will contain the true value of
f(X).9
On the other hand, a prediction interval can be used to quantify the
prediction
interval
uncertainty surrounding sales for a particular city. Given that $100,000 is
spent on TV advertising and $20,000 is spent on radio advertising in that city
the 95 % prediction interval is [7,930, 14,580]. We interpret this to mean
that 95 % of intervals of this form will contain the true value of Y for this
city. Note that both intervals are centered at 11,256, but that the prediction
interval is substantially wider than the confidence interval, reflecting the
increased uncertainty about sales for a given city in comparison to the
average sales over many locations.
9In other words, if we collect a large number of data sets like the Advertising data
set, and we construct a confidence interval for the average sales on the basis of each
data set (given $100,000 in TV and $20,000 in radio advertising), then 95 % of these
confidence intervals will contain the true value of average sales.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-92-320.jpg)
![3.3 Other Considerations in the Regression Model 85
Coefficient Std. error t-statistic p-value
Intercept 509.80 33.13 15.389 < 0.0001
own[Yes] 19.73 46.05 0.429 0.6690
TABLE 3.7. Least squares coefficient estimates associated with the regression of
balance onto own in the Credit data set. The linear model is given in (3.27).
That is, ownership is encoded as a dummy variable, as in (3.26).
for owners. Alternatively, instead of a 0/1 coding scheme, we could create
a dummy variable
xi =
=
1 if ith person owns a house
−1 if ith person does not own a house
and use this variable in the regression equation. This results in the model
yi = β0 +β1xi +ϵi =
=
β0 + β1 + ϵi if ith person owns a house
β0 − β1 + ϵi if ith person does not own a house.
Now β0 can be interpreted as the overall average credit card balance (ig-
noring the house ownership effect), and β1 is the amount by which house
owners and non-owners have credit card balances that are above and below
the average, respectively. In this example, the estimate for β0 is $519.665,
halfway between the non-owner and owner averages of $509.80 and $529.53.
The estimate for β1 is $9.865, which is half of $19.73, the average difference
between owners and non-owners. It is important to note that the final pre-
dictions for the credit balances of owners and non-owners will be identical
regardless of the coding scheme used. The only difference is in the way that
the coefficients are interpreted.
Qualitative Predictors with More than Two Levels
When a qualitative predictor has more than two levels, a single dummy
variable cannot represent all possible values. In this situation, we can create
additional dummy variables. For example, for the region variable we create
two dummy variables. The first could be
xi1 =
=
1 if ith person is from the South
0 if ith person is not from the South,
(3.28)
and the second could be
xi2 =
=
1 if ith person is from the West
0 if ith person is not from the West.
(3.29)](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-95-320.jpg)
![86 3. Linear Regression
Coefficient Std. error t-statistic p-value
Intercept 531.00 46.32 11.464 < 0.0001
region[South] −18.69 65.02 −0.287 0.7740
region[West] −12.50 56.68 −0.221 0.8260
TABLE 3.8. Least squares coefficient estimates associated with the regression of
balance onto region in the Credit data set. The linear model is given in (3.30).
That is, region is encoded via two dummy variables (3.28) and (3.29).
Then both of these variables can be used in the regression equation, in
order to obtain the model
yi = β0+β1xi1+β2xi2+ϵi =
⎧
⎪
⎨
⎪
⎩
β0+β1+ϵi if ith person is from the South
β0+β2+ϵi if ith person is from the West
β0+ϵi if ith person is from the East.
(3.30)
Now β0 can be interpreted as the average credit card balance for individuals
from the East, β1 can be interpreted as the difference in the average balance
between people from the South versus the East, and β2 can be interpreted
as the difference in the average balance between those from the West versus
the East. There will always be one fewer dummy variable than the number
of levels. The level with no dummy variable—East in this example—is
known as the baseline.
baseline
From Table 3.8, we see that the estimated balance for the baseline, East,
is $531.00. It is estimated that those in the South will have $18.69 less
debt than those in the East, and that those in the West will have $12.50
less debt than those in the East. However, the p-values associated with the
coefficient estimates for the two dummy variables are very large, suggesting
no statistical evidence of a real difference in average credit card balance
between South and East or between West and East.11
Once again, the
level selected as the baseline category is arbitrary, and the final predictions
for each group will be the same regardless of this choice. However, the
coefficients and their p-values do depend on the choice of dummy variable
coding. Rather than rely on the individual coefficients, we can use an F-test
to test H0 : β1 = β2 = 0; this does not depend on the coding. This F-test
has a p-value of 0.96, indicating that we cannot reject the null hypothesis
that there is no relationship between balance and region.
Using this dummy variable approach presents no difficulties when in-
corporating both quantitative and qualitative predictors. For example, to
regress balance on both a quantitative variable such as income and a qual-
itative variable such as student, we must simply create a dummy variable
for student and then fit a multiple regression model using income and the
dummy variable as predictors for credit card balance.
11There could still in theory be a difference between South and West, although the
data here does not suggest any difference.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-96-320.jpg)
![112 3. Linear Regression
If we type lm.fit, some basic information about the model is output.
For more detailed information, we use summary(lm.fit). This gives us p-
values and standard errors for the coefficients, as well as the R2
statistic
and F-statistic for the model.
> lm.fit
Call:
lm(formula = medv ∼ lstat)
Coefficients :
(Intercept) lstat
34.55 -0.95
> summary(lm.fit)
Call:
lm(formula = medv ∼ lstat)
Residuals:
Min 1Q Median 3Q Max
-15.17 -3.99 -1.32 2.03 24.50
Coefficients :
Estimate Std. Error t value Pr(>|t|)
(Intercept) 34.5538 0.5626 61.4 <2e -16 ***
lstat -0.9500 0.0387 -24.5 <2e -16 ***
---
Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1
Residual standard error: 6.22 on 504 degrees of freedom
Multiple R-squared: 0.544 , Adjusted R-squared: 0.543
F-statistic: 602 on 1 and 504 DF , p-value: < 2e-16
We can use the names() function in order to find out what other pieces
names()
of information are stored in lm.fit. Although we can extract these quan-
tities by name—e.g. lm.fit$coefficients—it is safer to use the extractor
functions like coef() to access them.
coef()
> names(lm.fit)
[1] " coefficients " "residuals" "effects"
[4] "rank" "fitted.values" "assign"
[7] "qr" "df.residual" "xlevels"
[10] "call" "terms" "model"
> coef(lm.fit)
(Intercept) lstat
34.55 -0.95
In order to obtain a confidence interval for the coefficient estimates, we can
use the confint() command.
confint()
> confint(lm.fit)
2.5 % 97.5 %
(Intercept) 33.45 35.659](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-122-320.jpg)
![3.7 Exercises 121
are about to be input. Hitting Enter after typing { will cause R to print the
+ symbol. We can then input as many commands as we wish, hitting Enter
after each one. Finally the } symbol informs R that no further commands
will be entered.
> LoadLibraries <- function () {
+ library(ISLR2)
+ library(MASS)
+ print("The libraries have been loaded.")
+ }
Now if we type in LoadLibraries, R will tell us what is in the function.
> LoadLibraries
function () {
library(ISLR2)
library(MASS)
print (" The libraries have been loaded .")
}
If we call the function, the libraries are loaded in and the print statement
is output.
> LoadLibraries ()
[1] "The libraries have been loaded ."
3.7 Exercises
Conceptual
1. Describe the null hypotheses to which the p-values given in Table 3.4
correspond. Explain what conclusions you can draw based on these
p-values. Your explanation should be phrased in terms of sales, TV,
radio, and newspaper, rather than in terms of the coefficients of the
linear model.
2. Carefully explain the differences between the KNN classifier and KNN
regression methods.
3. Suppose we have a data set with five predictors, X1 = GPA, X2 =
IQ, X3 = Level (1 for College and 0 for High School), X4 = Interac-
tion between GPA and IQ, and X5 = Interaction between GPA and
Level. The response is starting salary after graduation (in thousands
of dollars). Suppose we use least squares to fit the model, and get
β̂0 = 50, β̂1 = 20, β̂2 = 0.07, β̂3 = 35, β̂4 = 0.01, β̂5 = −10.
(a) Which answer is correct, and why?
i. For a fixed value of IQ and GPA, high school graduates earn
more, on average, than college graduates.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-131-320.jpg)
![132 4. Classification
equally reasonable coding,
Y =
⎧
⎪
⎨
⎪
⎩
1 if epileptic seizure;
2 if stroke;
3 if drug overdose.
which would imply a totally different relationship among the three condi-
tions. Each of these codings would produce fundamentally different linear
models that would ultimately lead to different sets of predictions on test
observations.
If the response variable’s values did take on a natural ordering, such as
mild, moderate, and severe, and we felt the gap between mild and moderate
was similar to the gap between moderate and severe, then a 1, 2, 3 coding
would be reasonable. Unfortunately, in general there is no natural way to
convert a qualitative response variable with more than two levels into a
quantitative response that is ready for linear regression.
For a binary (two level) qualitative response, the situation is better. For
binary
instance, perhaps there are only two possibilities for the patient’s med-
ical condition: stroke and drug overdose. We could then potentially use
the dummy variable approach from Section 3.3.1 to code the response as
follows:
Y =
=
0 if stroke;
1 if drug overdose.
We could then fit a linear regression to this binary response, and predict
drug overdose if Ŷ > 0.5 and stroke otherwise. In the binary case it is not
hard to show that even if we flip the above coding, linear regression will
produce the same final predictions.
For a binary response with a 0/1 coding as above, regression by least
squares is not completely unreasonable: it can be shown that the Xβ̂ ob-
tained using linear regression is in fact an estimate of Pr(drug overdose|X)
in this special case. However, if we use linear regression, some of our es-
timates might be outside the [0, 1] interval (see Figure 4.2), making them
hard to interpret as probabilities! Nevertheless, the predictions provide an
ordering and can be interpreted as crude probability estimates. Curiously,
it turns out that the classifications that we get if we use linear regression
to predict a binary response will be the same as for the linear discriminant
analysis (LDA) procedure we discuss in Section 4.4.
To summarize, there are at least two reasons not to perform classifica-
tion using a regression method: (a) a regression method cannot accommo-
date a qualitative response with more than two classes; (b) a regression
method will not provide meaningful estimates of Pr(Y |X), even with just
two classes. Thus, it is preferable to use a classification method that is
truly suited for qualitative response values. In the next section, we present
logistic regression, which is well-suited for the case of a binary qualita-](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-142-320.jpg)
![134 4. Classification
In Section 4.2 we considered using a linear regression model to represent
these probabilities:
p(X) = β0 + β1X. (4.1)
If we use this approach to predict default=Yes using balance, then we
obtain the model shown in the left-hand panel of Figure 4.2. Here we see
the problem with this approach: for balances close to zero we predict a
negative probability of default; if we were to predict for very large balances,
we would get values bigger than 1. These predictions are not sensible, since
of course the true probability of default, regardless of credit card balance,
must fall between 0 and 1. This problem is not unique to the credit default
data. Any time a straight line is fit to a binary response that is coded as
0 or 1, in principle we can always predict p(X) < 0 for some values of X
and p(X) > 1 for others (unless the range of X is limited).
To avoid this problem, we must model p(X) using a function that gives
outputs between 0 and 1 for all values of X. Many functions meet this
description. In logistic regression, we use the logistic function,
logistic
function
p(X) =
eβ0+β1X
1 + eβ0+β1X
. (4.2)
To fit the model (4.2), we use a method called maximum likelihood, which
maximum
likelihood
we discuss in the next section. The right-hand panel of Figure 4.2 illustrates
the fit of the logistic regression model to the Default data. Notice that for
low balances we now predict the probability of default as close to, but never
below, zero. Likewise, for high balances we predict a default probability
close to, but never above, one. The logistic function will always produce
an S-shaped curve of this form, and so regardless of the value of X, we
will obtain a sensible prediction. We also see that the logistic model is
better able to capture the range of probabilities than is the linear regression
model in the left-hand plot. The average fitted probability in both cases is
0.0333 (averaged over the training data), which is the same as the overall
proportion of defaulters in the data set.
After a bit of manipulation of (4.2), we find that
p(X)
1 − p(X)
= eβ0+β1X
. (4.3)
The quantity p(X)/[1−p(X)] is called the odds, and can take on any value
odds
between 0 and ∞. Values of the odds close to 0 and ∞ indicate very low
and very high probabilities of default, respectively. For example, on average
1 in 5 people with an odds of 1/4 will default, since p(X) = 0.2 implies an
odds of 0.2
1−0.2 = 1/4. Likewise, on average nine out of every ten people with
an odds of 9 will default, since p(X) = 0.9 implies an odds of 0.9
1−0.9 = 9.
Odds are traditionally used instead of probabilities in horse-racing, since
they relate more naturally to the correct betting strategy.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-144-320.jpg)
![4.3 Logistic Regression 137
Coefficient Std. error z-statistic p-value
Intercept −3.5041 0.0707 −49.55 <0.0001
student[Yes] 0.4049 0.1150 3.52 0.0004
TABLE 4.2. For the Default data, estimated coefficients of the logistic regres-
sion model that predicts the probability of default using student status. Student
status is encoded as a dummy variable, with a value of 1 for a student and a value
of 0 for a non-student, and represented by the variable student[Yes] in the table.
which is below 1 %. In contrast, the predicted probability of default for an
individual with a balance of $2, 000 is much higher, and equals 0.586 or
58.6 %.
One can use qualitative predictors with the logistic regression model us-
ing the dummy variable approach from Section 3.3.1. As an example, the
Default data set contains the qualitative variable student. To fit a model
that uses student status as a predictor variable, we simply create a dummy
variable that takes on a value of 1 for students and 0 for non-students. The
logistic regression model that results from predicting probability of default
from student status can be seen in Table 4.2. The coefficient associated
with the dummy variable is positive, and the associated p-value is statis-
tically significant. This indicates that students tend to have higher default
probabilities than non-students:
6
Pr(default=Yes|student=Yes) =
e−3.5041+0.4049×1
1 + e−3.5041+0.4049×1
= 0.0431,
6
Pr(default=Yes|student=No) =
e−3.5041+0.4049×0
1 + e−3.5041+0.4049×0
= 0.0292.
4.3.4 Multiple Logistic Regression
We now consider the problem of predicting a binary response using multiple
predictors. By analogy with the extension from simple to multiple linear
regression in Chapter 3, we can generalize (4.4) as follows:
log
*
p(X)
1 − p(X)
+
= β0 + β1X1 + · · · + βpXp, (4.6)
where X = (X1, . . . , Xp) are p predictors. Equation 4.6 can be rewritten as
p(X) =
eβ0+β1X1+···+βpXp
1 + eβ0+β1X1+···+βpXp
. (4.7)
Just as in Section 4.3.2, we use the maximum likelihood method to estimate
β0, β1, . . . , βp.
Table 4.3 shows the coefficient estimates for a logistic regression model
that uses balance, income (in thousands of dollars), and student status to
predict probability of default. There is a surprising result here. The p-](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-147-320.jpg)
![138 4. Classification
Coefficient Std. error z-statistic p-value
Intercept −10.8690 0.4923 −22.08 <0.0001
balance 0.0057 0.0002 24.74 <0.0001
income 0.0030 0.0082 0.37 0.7115
student[Yes] −0.6468 0.2362 −2.74 0.0062
TABLE 4.3. For the Default data, estimated coefficients of the logistic regres-
sion model that predicts the probability of default using balance, income, and
student status. Student status is encoded as a dummy variable student[Yes],
with a value of 1 for a student and a value of 0 for a non-student. In fitting this
model, income was measured in thousands of dollars.
values associated with balance and the dummy variable for student status
are very small, indicating that each of these variables is associated with
the probability of default. However, the coefficient for the dummy variable
is negative, indicating that students are less likely to default than non-
students. In contrast, the coefficient for the dummy variable is positive in
Table 4.2. How is it possible for student status to be associated with an
increase in probability of default in Table 4.2 and a decrease in probability
of default in Table 4.3? The left-hand panel of Figure 4.3 provides a graph-
ical illustration of this apparent paradox. The orange and blue solid lines
show the average default rates for students and non-students, respectively,
as a function of credit card balance. The negative coefficient for student in
the multiple logistic regression indicates that for a fixed value of balance
and income, a student is less likely to default than a non-student. Indeed,
we observe from the left-hand panel of Figure 4.3 that the student default
rate is at or below that of the non-student default rate for every value of
balance. But the horizontal broken lines near the base of the plot, which
show the default rates for students and non-students averaged over all val-
ues of balance and income, suggest the opposite effect: the overall student
default rate is higher than the non-student default rate. Consequently, there
is a positive coefficient for student in the single variable logistic regression
output shown in Table 4.2.
The right-hand panel of Figure 4.3 provides an explanation for this dis-
crepancy. The variables student and balance are correlated. Students tend
to hold higher levels of debt, which is in turn associated with higher prob-
ability of default. In other words, students are more likely to have large
credit card balances, which, as we know from the left-hand panel of Fig-
ure 4.3, tend to be associated with high default rates. Thus, even though
an individual student with a given credit card balance will tend to have a
lower probability of default than a non-student with the same credit card
balance, the fact that students on the whole tend to have higher credit card
balances means that overall, students tend to default at a higher rate than
non-students. This is an important distinction for a credit card company
that is trying to determine to whom they should offer credit. A student is](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-148-320.jpg)
![4.6 Generalized Linear Models 165
Coefficient Std. error z-statistic p-value
Intercept 73.60 5.13 14.34 0.00
workingday 1.27 1.78 0.71 0.48
temp 157.21 10.26 15.32 0.00
weathersit[cloudy/misty] -12.89 1.96 -6.56 0.00
weathersit[light rain/snow] -66.49 2.97 -22.43 0.00
weathersit[heavy rain/snow] -109.75 76.67 -1.43 0.15
TABLE 4.10. Results for a least squares linear model fit to predict bikers in
the Bikeshare data. The predictors mnth and hr are omitted from this table due
to space constraints, and can be seen in Figure 4.13. For the qualitative variable
weathersit, the baseline level corresponds to clear skies.
●
●
●
●
●
●
●
●
●
●
●
●
−40
−20
0
20
Month
Coefficient
J F M A M J J A S O N D
●
●
●
● ●
●
●
●
●
●
●
●
● ●
● ●
●
●
●
●
●
●
●
●
5 10 15 20
−100
0
50
100
200
Hour
Coefficient
FIGURE 4.13. A least squares linear regression model was fit to predict bikers
in the Bikeshare data set. Left: The coefficients associated with the month of the
year. Bike usage is highest in the spring and fall, and lowest in the winter. Right:
The coefficients associated with the hour of the day. Bike usage is highest during
peak commute times, and lowest overnight.
But upon more careful inspection, some issues become apparent. For
example, 9.6% of the fitted values in the Bikeshare data set are negative:
that is, the linear regression model predicts a negative number of users
during 9.6% of the hours in the data set. This calls into question our ability
to perform meaningful predictions on the data, and it also raises concerns
about the accuracy of the coefficient estimates, confidence intervals, and
other outputs of the regression model.
Furthermore, it is reasonable to suspect that when the expected value
of bikers is small, the variance of bikers should be small as well. For
instance, at 2 AM during a heavy December snow storm, we expect that
extremely few people will use a bike, and moreover that there should be
little variance associated with the number of users during those conditions.
This is borne out in the data: between 1 AM and 4 AM, in December,
January, and February, when it is raining, there are 5.05 users, on average,](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-175-320.jpg)
![168 4. Classification
Coefficient Std. error z-statistic p-value
Intercept 4.12 0.01 683.96 0.00
workingday 0.01 0.00 7.5 0.00
temp 0.79 0.01 68.43 0.00
weathersit[cloudy/misty] -0.08 0.00 -34.53 0.00
weathersit[light rain/snow] -0.58 0.00 -141.91 0.00
weathersit[heavy rain/snow] -0.93 0.17 -5.55 0.00
TABLE 4.11. Results for a Poisson regression model fit to predict bikers in
the Bikeshare data. The predictors mnth and hr are omitted from this table due
to space constraints, and can be seen in Figure 4.15. For the qualitative variable
weathersit, the baseline corresponds to clear skies.
or equivalently
λ(X1, . . . , Xp) = eβ0+β1X1+···+βpXp
. (4.37)
Here, β0, β1, . . . , βp are parameters to be estimated. Together, (4.35) and
(4.36) define the Poisson regression model. Notice that in (4.36), we take
the log of λ(X1, . . . , Xp) to be linear in X1, . . . , Xp, rather than hav-
ing λ(X1, . . . , Xp) itself be linear in X1, . . . , Xp, in order to ensure that
λ(X1, . . . , Xp) takes on nonnegative values for all values of the covariates.
To estimate the coefficients β0, β1, . . . , βp, we use the same maximum
likelihood approach that we adopted for logistic regression in Section 4.3.2.
Specifically, given n independent observations from the Poisson regression
model, the likelihood takes the form
ℓ(β0, β1, . . . , βp) =
n
E
i=1
e−λ(xi)
λ(xi)yi
yi!
, (4.38)
where λ(xi) = eβ0+β1xi1+···+βpxip
, due to (4.37). We estimate the coef-
ficients that maximize the likelihood ℓ(β0, β1, . . . , βp), i.e. that make the
observed data as likely as possible.
We now fit a Poisson regression model to the Bikeshare data set. The
results are shown in Table 4.11 and Figure 4.15. Qualitatively, the results
are similar to those from linear regression in Section 4.6.1. We again see
that bike usage is highest in the spring and fall and during rush hour,
and lowest during the winter and in the early morning hours. Moreover,
bike usage increases as the temperature increases, and decreases as the
weather worsens. Interestingly, the coefficient associated with workingday
is statistically significant under the Poisson regression model, but not under
the linear regression model.
Some important distinctions between the Poisson regression model and
the linear regression model are as follows:
• Interpretation: To interpret the coefficients in the Poisson regression
model, we must pay close attention to (4.37), which states that an
increase in Xj by one unit is associated with a change in E(Y ) = λ
by a factor of exp(βj). For example, a change in weather from clear](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-178-320.jpg)
![4.7 Lab: Classification Methods 171
4.7 Lab: Classification Methods
4.7.1 The Stock Market Data
We will begin by examining some numerical and graphical summaries of
the Smarket data, which is part of the ISLR2 library. This data set consists of
percentage returns for the S&P 500 stock index over 1, 250 days, from the
beginning of 2001 until the end of 2005. For each date, we have recorded
the percentage returns for each of the five previous trading days, Lag1
through Lag5. We have also recorded Volume (the number of shares traded
on the previous day, in billions), Today (the percentage return on the date
in question) and Direction (whether the market was Up or Down on this
date). Our goal is to predict Direction (a qualitative response) using the
other features.
> library(ISLR2)
> names(Smarket)
[1] "Year" "Lag1" "Lag2" "Lag3" "Lag4"
[6] "Lag5" "Volume" "Today" "Direction"
> dim(Smarket)
[1] 1250 9
> summary(Smarket)
Year Lag1 Lag2
Min. :2001 Min. : -4.92200 Min. : -4.92200
1st Qu .:2002 1st Qu .: -0.63950 1st Qu .: -0.63950
Median :2003 Median : 0.03900 Median : 0.03900
Mean :2003 Mean : 0.00383 Mean : 0.00392
3rd Qu .:2004 3rd Qu.: 0.59675 3rd Qu.: 0.59675
Max. :2005 Max. : 5.73300 Max. : 5.73300
Lag3 Lag4 Lag5
Min. : -4.92200 Min. : -4.92200 Min. : -4.92200
1st Qu .: -0.64000 1st Qu .: -0.64000 1st Qu .: -0.64000
Median : 0.03850 Median : 0.03850 Median : 0.03850
Mean : 0.00172 Mean : 0.00164 Mean : 0.00561
3rd Qu.: 0.59675 3rd Qu.: 0.59675 3rd Qu.: 0.59700
Max. : 5.73300 Max. : 5.73300 Max. : 5.73300
Volume Today Direction
Min. :0.356 Min. : -4.92200 Down :602
1st Qu .:1.257 1st Qu .: -0.63950 Up :648
Median :1.423 Median : 0.03850
Mean :1.478 Mean : 0.00314
3rd Qu .:1.642 3rd Qu.: 0.59675
Max. :3.152 Max. : 5.73300
> pairs(Smarket)
The cor() function produces a matrix that contains all of the pairwise
correlations among the predictors in a data set. The first command below
gives an error message because the Direction variable is qualitative.
> cor(Smarket)
Error in cor(Smarket) : ‘x’ must be numeric
> cor(Smarket[, -9])](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-181-320.jpg)
![4.7 Lab: Classification Methods 173
-1.45 -1.20 1.07 1.15 1.33
Coefficients :
Estimate Std. Error z value Pr(>|z|)
(Intercept) -0.12600 0.24074 -0.52 0.60
Lag1 -0.07307 0.05017 -1.46 0.15
Lag2 -0.04230 0.05009 -0.84 0.40
Lag3 0.01109 0.04994 0.22 0.82
Lag4 0.00936 0.04997 0.19 0.85
Lag5 0.01031 0.04951 0.21 0.83
Volume 0.13544 0.15836 0.86 0.39
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 1731.2 on 1249 degrees of freedom
Residual deviance: 1727.6 on 1243 degrees of freedom
AIC: 1742
Number of Fisher Scoring iterations: 3
The smallest p-value here is associated with Lag1. The negative coefficient
for this predictor suggests that if the market had a positive return yesterday,
then it is less likely to go up today. However, at a value of 0.15, the p-value
is still relatively large, and so there is no clear evidence of a real association
between Lag1 and Direction.
We use the coef() function in order to access just the coefficients for this
fitted model. We can also use the summary() function to access particular
aspects of the fitted model, such as the p-values for the coefficients.
> coef(glm.fits)
(Intercept) Lag1 Lag2 Lag3 Lag4
-0.12600 -0.07307 -0.04230 0.01109 0.00936
Lag5 Volume
0.01031 0.13544
> summary(glm.fits)$coef
Estimate Std. Error z value Pr(>|z|)
(Intercept) -0.12600 0.2407 -0.523 0.601
Lag1 -0.07307 0.0502 -1.457 0.145
Lag2 -0.04230 0.0501 -0.845 0.398
Lag3 0.01109 0.0499 0.222 0.824
Lag4 0.00936 0.0500 0.187 0.851
Lag5 0.01031 0.0495 0.208 0.835
Volume 0.13544 0.1584 0.855 0.392
> summary(glm.fits)$coef[, 4]
(Intercept) Lag1 Lag2 Lag3 Lag4
0.601 0.145 0.398 0.824 0.851
Lag5 Volume
0.835 0.392
The predict() function can be used to predict the probability that the
market will go up, given values of the predictors. The type = "response"
option tells R to output probabilities of the form P(Y = 1|X), as opposed
to other information such as the logit. If no data set is supplied to the](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-183-320.jpg)
![174 4. Classification
predict() function, then the probabilities are computed for the training
data that was used to fit the logistic regression model. Here we have printed
only the first ten probabilities. We know that these values correspond to
the probability of the market going up, rather than down, because the
contrasts() function indicates that R has created a dummy variable with
a 1 for Up.
> glm.probs <- predict(glm.fits , type = "response ")
> glm.probs [1:10]
1 2 3 4 5 6 7 8 9 10
0.507 0.481 0.481 0.515 0.511 0.507 0.493 0.509 0.518 0.489
> contrasts (Direction)
Up
Down 0
Up 1
In order to make a prediction as to whether the market will go up or
down on a particular day, we must convert these predicted probabilities
into class labels, Up or Down. The following two commands create a vector
of class predictions based on whether the predicted probability of a market
increase is greater than or less than 0.5.
> glm.pred <- rep("Down", 1250)
> glm.pred[glm.probs > .5] = "Up"
The first command creates a vector of 1,250 Down elements. The second line
transforms to Up all of the elements for which the predicted probability of a
market increase exceeds 0.5. Given these predictions, the table() function
table()
can be used to produce a confusion matrix in order to determine how many
observations were correctly or incorrectly classified.
> table(glm.pred , Direction)
Direction
glm.pred Down Up
Down 145 141
Up 457 507
> (507 + 145) / 1250
[1] 0.5216
> mean(glm.pred == Direction)
[1] 0.5216
The diagonal elements of the confusion matrix indicate correct predictions,
while the off-diagonals represent incorrect predictions. Hence our model
correctly predicted that the market would go up on 507 days and that
it would go down on 145 days, for a total of 507 + 145 = 652 correct
predictions. The mean() function can be used to compute the fraction of
days for which the prediction was correct. In this case, logistic regression
correctly predicted the movement of the market 52.2 % of the time.
At first glance, it appears that the logistic regression model is working
a little better than random guessing. However, this result is misleading
because we trained and tested the model on the same set of 1, 250 ob-
servations. In other words, 100% − 52.2% = 47.8%, is the training error](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-184-320.jpg)
![4.7 Lab: Classification Methods 175
rate. As we have seen previously, the training error rate is often overly
optimistic—it tends to underestimate the test error rate. In order to better
assess the accuracy of the logistic regression model in this setting, we can
fit the model using part of the data, and then examine how well it predicts
the held out data. This will yield a more realistic error rate, in the sense
that in practice we will be interested in our model’s performance not on
the data that we used to fit the model, but rather on days in the future for
which the market’s movements are unknown.
To implement this strategy, we will first create a vector corresponding
to the observations from 2001 through 2004. We will then use this vector
to create a held out data set of observations from 2005.
> train <- (Year < 2005)
> Smarket .2005 <- Smarket [!train , ]
> dim(Smarket .2005)
[1] 252 9
> Direction .2005 <- Direction [! train]
The object train is a vector of 1,250 elements, corresponding to the ob-
servations in our data set. The elements of the vector that correspond to
observations that occurred before 2005 are set to TRUE, whereas those that
correspond to observations in 2005 are set to FALSE. The object train is a
Boolean vector, since its elements are TRUE and FALSE. Boolean vectors can
boolean
be used to obtain a subset of the rows or columns of a matrix. For instance,
the command Smarket[train, ] would pick out a submatrix of the stock
market data set, corresponding only to the dates before 2005, since those
are the ones for which the elements of train are TRUE. The ! symbol can be
used to reverse all of the elements of a Boolean vector. That is, !train is
a vector similar to train, except that the elements that are TRUE in train
get swapped to FALSE in !train, and the elements that are FALSE in train
get swapped to TRUE in !train. Therefore, Smarket[!train, ] yields a sub-
matrix of the stock market data containing only the observations for which
train is FALSE—that is, the observations with dates in 2005. The output
above indicates that there are 252 such observations.
We now fit a logistic regression model using only the subset of the obser-
vations that correspond to dates before 2005, using the subset argument.
We then obtain predicted probabilities of the stock market going up for
each of the days in our test set—that is, for the days in 2005.
> glm.fits <- glm(
Direction ∼ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + Volume ,
data = Smarket , family = binomial , subset = train
)
> glm.probs <- predict(glm.fits , Smarket .2005 ,
type = "response")
Notice that we have trained and tested our model on two completely sep-
arate data sets: training was performed using only the dates before 2005,](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-185-320.jpg)
![176 4. Classification
and testing was performed using only the dates in 2005. Finally, we com-
pute the predictions for 2005 and compare them to the actual movements
of the market over that time period.
> glm.pred <- rep("Down", 252)
> glm.pred[glm.probs > .5] <- "Up"
> table(glm.pred , Direction .2005)
Direction .2005
glm.pred Down Up
Down 77 97
Up 34 44
> mean(glm.pred == Direction .2005)
[1] 0.48
> mean(glm.pred != Direction .2005)
[1] 0.52
The != notation means not equal to, and so the last command computes
the test set error rate. The results are rather disappointing: the test error
rate is 52 %, which is worse than random guessing! Of course this result
is not all that surprising, given that one would not generally expect to be
able to use previous days’ returns to predict future market performance.
(After all, if it were possible to do so, then the authors of this book would
be out striking it rich rather than writing a statistics textbook.)
We recall that the logistic regression model had very underwhelming p-
values associated with all of the predictors, and that the smallest p-value,
though not very small, corresponded to Lag1. Perhaps by removing the
variables that appear not to be helpful in predicting Direction, we can
obtain a more effective model. After all, using predictors that have no
relationship with the response tends to cause a deterioration in the test
error rate (since such predictors cause an increase in variance without a
corresponding decrease in bias), and so removing such predictors may in
turn yield an improvement. Below we have refit the logistic regression using
just Lag1 and Lag2, which seemed to have the highest predictive power in
the original logistic regression model.
> glm.fits <- glm(Direction ∼ Lag1 + Lag2 , data = Smarket ,
family = binomial , subset = train)
> glm.probs <- predict(glm.fits , Smarket .2005 ,
type = "response")
> glm.pred <- rep("Down", 252)
> glm.pred[glm.probs > .5] <- "Up"
> table(glm.pred , Direction .2005)
Direction .2005
glm.pred Down Up
Down 35 35
Up 76 106
> mean(glm.pred == Direction .2005)
[1] 0.56
> 106 / (106 + 76)
[1] 0.582](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-186-320.jpg)
![178 4. Classification
Coefficients of linear discriminants :
LD1
Lag1 -0.642
Lag2 -0.514
> plot(lda.fit)
The LDA output indicates that π̂1 = 0.492 and π̂2 = 0.508; in other words,
49.2 % of the training observations correspond to days during which the
market went down. It also provides the group means; these are the average
of each predictor within each class, and are used by LDA as estimates of
µk. These suggest that there is a tendency for the previous 2 days’ returns
to be negative on days when the market increases, and a tendency for the
previous days’ returns to be positive on days when the market declines. The
coefficients of linear discriminants output provides the linear combination
of Lag1 and Lag2 that are used to form the LDA decision rule. In other
words, these are the multipliers of the elements of X = x in (4.24). If
−0.642 × Lag1 − 0.514 × Lag2 is large, then the LDA classifier will predict
a market increase, and if it is small, then the LDA classifier will predict a
market decline.
The plot() function produces plots of the linear discriminants, obtained
by computing −0.642 × Lag1 − 0.514 × Lag2 for each of the training obser-
vations. The Up and Down observations are displayed separately.
The predict() function returns a list with three elements. The first ele-
ment, class, contains LDA’s predictions about the movement of the market.
The second element, posterior, is a matrix whose kth column contains the
posterior probability that the corresponding observation belongs to the kth
class, computed from (4.15). Finally, x contains the linear discriminants,
described earlier.
> lda.pred <- predict(lda.fit , Smarket .2005)
> names(lda.pred)
[1] "class" "posterior" "x"
As we observed in Section 4.5, the LDA and logistic regression predictions
are almost identical.
> lda.class <- lda.pred$class
> table(lda.class , Direction .2005)
Direction .2005
lda.pred Down Up
Down 35 35
Up 76 106
> mean(lda.class == Direction .2005)
[1] 0.56
Applying a 50 % threshold to the posterior probabilities allows us to recre-
ate the predictions contained in lda.pred$class.
> sum(lda.pred$posterior[, 1] >= .5)
[1] 70](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-188-320.jpg)
![4.7 Lab: Classification Methods 179
> sum(lda.pred$posterior[, 1] < .5)
[1] 182
Notice that the posterior probability output by the model corresponds to
the probability that the market will decrease:
> lda.pred$posterior [1:20 , 1]
> lda.class [1:20]
If we wanted to use a posterior probability threshold other than 50 % in
order to make predictions, then we could easily do so. For instance, suppose
that we wish to predict a market decrease only if we are very certain that the
market will indeed decrease on that day—say, if the posterior probability
is at least 90 %.
> sum(lda.pred$posterior[, 1] > .9)
[1] 0
No days in 2005 meet that threshold! In fact, the greatest posterior prob-
ability of decrease in all of 2005 was 52.02 %.
4.7.4 Quadratic Discriminant Analysis
We will now fit a QDA model to the Smarket data. QDA is implemented
in R using the qda() function, which is also part of the MASS library. The
qda()
syntax is identical to that of lda().
> qda.fit <- qda(Direction ∼ Lag1 + Lag2 , data = Smarket ,
subset = train)
> qda.fit
Call:
qda(Direction ∼ Lag1 + Lag2 , data = Smarket , subset = train)
Prior probabilities of groups:
Down Up
0.492 0.508
Group means:
Lag1 Lag2
Down 0.0428 0.0339
Up -0.0395 -0.0313
The output contains the group means. But it does not contain the coef-
ficients of the linear discriminants, because the QDA classifier involves a
quadratic, rather than a linear, function of the predictors. The predict()
function works in exactly the same fashion as for LDA.
> qda.class <- predict(qda.fit , Smarket .2005)$class
> table(qda.class , Direction .2005)
Direction .2005
qda.class Down Up
Down 30 20](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-189-320.jpg)
![180 4. Classification
Up 81 121
> mean(qda.class == Direction .2005)
[1] 0.599
Interestingly, the QDA predictions are accurate almost 60 % of the time,
even though the 2005 data was not used to fit the model. This level of accu-
racy is quite impressive for stock market data, which is known to be quite
hard to model accurately. This suggests that the quadratic form assumed
by QDA may capture the true relationship more accurately than the linear
forms assumed by LDA and logistic regression. However, we recommend
evaluating this method’s performance on a larger test set before betting
that this approach will consistently beat the market!
4.7.5 Naive Bayes
Next, we fit a naive Bayes model to the Smarket data. Naive Bayes is im-
plemented in R using the naiveBayes() function, which is part of the e1071
naiveBayes()
library. The syntax is identical to that of lda() and qda(). By default, this
implementation of the naive Bayes classifier models each quantitative fea-
ture using a Gaussian distribution. However, a kernel density method can
also be used to estimate the distributions.
> library(e1071)
> nb.fit <- naiveBayes (Direction ∼ Lag1 + Lag2 , data = Smarket ,
subset = train)
> nb.fit
Naive Bayes Classifier for Discrete Predictors
Call:
naiveBayes .default(x = X, y = Y, laplace = laplace)
A-priori probabilities :
Y
Down Up
0.492 0.508
Conditional probabilities :
Lag1
Y [,1] [,2]
Down 0.0428 1.23
Up -0.0395 1.23
Lag2
Y [,1] [,2]
Down 0.0339 1.24
Up -0.0313 1.22
The output contains the estimated mean and standard deviation for each
variable in each class. For example, the mean for Lag1 is 0.0428 for
Direction=Down, and the standard deviation is 1.23. We can easily verify
this:](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-190-320.jpg)
![4.7 Lab: Classification Methods 181
> mean(Lag1[train ][ Direction[train] == "Down"])
[1] 0.0428
> sd(Lag1[train ][ Direction[train] == "Down"])
[1] 1.23
The predict() function is straightforward.
> nb.class <- predict(nb.fit , Smarket .2005)
> table(nb.class , Direction .2005)
Direction .2005
nb.class Down Up
Down 28 20
Up 83 121
> mean(nb.class == Direction .2005)
[1] 0.591
Naive Bayes performs very well on this data, with accurate predictions over
59% of the time. This is slightly worse than QDA, but much better than
LDA.
The predict() function can also generate estimates of the probability
that each observation belongs to a particular class.
> nb.preds <- predict(nb.fit , Smarket .2005 , type = "raw")
> nb.preds [1:5 , ]
Down Up
[1,] 0.487 0.513
[2,] 0.476 0.524
[3,] 0.465 0.535
[4,] 0.475 0.525
[5,] 0.490 0.510
4.7.6 K-Nearest Neighbors
We will now perform KNN using the knn() function, which is part of the
knn()
class library. This function works rather differently from the other model-
fitting functions that we have encountered thus far. Rather than a two-step
approach in which we first fit the model and then we use the model to make
predictions, knn() forms predictions using a single command. The function
requires four inputs.
1. A matrix containing the predictors associated with the training data,
labeled train.X below.
2. A matrix containing the predictors associated with the data for which
we wish to make predictions, labeled test.X below.
3. A vector containing the class labels for the training observations,
labeled train.Direction below.
4. A value for K, the number of nearest neighbors to be used by the
classifier.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-191-320.jpg)
![182 4. Classification
We use the cbind() function, short for column bind, to bind the Lag1 and
cbind()
Lag2 variables together into two matrices, one for the training set and the
other for the test set.
> library(class)
> train.X <- cbind(Lag1 , Lag2)[train , ]
> test.X <- cbind(Lag1 , Lag2)[!train , ]
> train.Direction <- Direction[train]
Now the knn() function can be used to predict the market’s movement for
the dates in 2005. We set a random seed before we apply knn() because
if several observations are tied as nearest neighbors, then R will randomly
break the tie. Therefore, a seed must be set in order to ensure reproducibil-
ity of results.
> set.seed (1)
> knn.pred <- knn(train.X, test.X, train.Direction , k = 1)
> table(knn.pred , Direction .2005)
Direction .2005
knn.pred Down Up
Down 43 58
Up 68 83
> (83 + 43) / 252
[1] 0.5
The results using K = 1 are not very good, since only 50 % of the observa-
tions are correctly predicted. Of course, it may be that K = 1 results in an
overly flexible fit to the data. Below, we repeat the analysis using K = 3.
> knn.pred <- knn(train.X, test.X, train.Direction , k = 3)
> table(knn.pred , Direction .2005)
Direction .2005
knn.pred Down Up
Down 48 54
Up 63 87
> mean(knn.pred == Direction .2005)
[1] 0.536
The results have improved slightly. But increasing K further turns out
to provide no further improvements. It appears that for this data, QDA
provides the best results of the methods that we have examined so far.
KNN does not perform well on the Smarket data but it does often provide
impressive results. As an example we will apply the KNN approach to the
Caravan data set, which is part of the ISLR2 library. This data set includes 85
predictors that measure demographic characteristics for 5,822 individuals.
The response variable is Purchase, which indicates whether or not a given
individual purchases a caravan insurance policy. In this data set, only 6 %
of people purchased caravan insurance.
> dim(Caravan)
[1] 5822 86](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-192-320.jpg)
![4.7 Lab: Classification Methods 183
> attach(Caravan)
> summary(Purchase)
No Yes
5474 348
> 348 / 5822
[1] 0.0598
Because the KNN classifier predicts the class of a given test observation by
identifying the observations that are nearest to it, the scale of the variables
matters. Variables that are on a large scale will have a much larger effect
on the distance between the observations, and hence on the KNN classifier,
than variables that are on a small scale. For instance, imagine a data set
that contains two variables, salary and age (measured in dollars and years,
respectively). As far as KNN is concerned, a difference of $1,000 in salary
is enormous compared to a difference of 50 years in age. Consequently,
salary will drive the KNN classification results, and age will have almost
no effect. This is contrary to our intuition that a salary difference of $1,000
is quite small compared to an age difference of 50 years. Furthermore, the
importance of scale to the KNN classifier leads to another issue: if we
measured salary in Japanese yen, or if we measured age in minutes, then
we’d get quite different classification results from what we get if these two
variables are measured in dollars and years.
A good way to handle this problem is to standardize the data so that all
standardize
variables are given a mean of zero and a standard deviation of one. Then
all variables will be on a comparable scale. The scale() function does just
scale()
this. In standardizing the data, we exclude column 86, because that is the
qualitative Purchase variable.
> standardized .X <- scale(Caravan[, -86])
> var(Caravan[, 1])
[1] 165
> var(Caravan[, 2])
[1] 0.165
> var( standardized .X[, 1])
[1] 1
> var( standardized .X[, 2])
[1] 1
Now every column of standardized.X has a standard deviation of one and
a mean of zero.
We now split the observations into a test set, containing the first 1,000
observations, and a training set, containing the remaining observations.
We fit a KNN model on the training data using K = 1, and evaluate its
performance on the test data.
> test <- 1:1000
> train.X <- standardized .X[-test , ]
> test.X <- standardized .X[test , ]
> train.Y <- Purchase[-test]](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-193-320.jpg)
![184 4. Classification
> test.Y <- Purchase[test]
> set.seed (1)
> knn.pred <- knn(train.X, test.X, train.Y, k = 1)
> mean(test.Y != knn.pred)
[1] 0.118
> mean(test.Y != "No")
[1] 0.059
The vector test is numeric, with values from 1 through 1, 000. Typing
standardized.X[test, ] yields the submatrix of the data containing the
observations whose indices range from 1 to 1, 000, whereas typing
standardized.X[-test, ] yields the submatrix containing the observations
whose indices do not range from 1 to 1, 000. The KNN error rate on the
1,000 test observations is just under 12 %. At first glance, this may ap-
pear to be fairly good. However, since only 6 % of customers purchased
insurance, we could get the error rate down to 6 % by always predicting No
regardless of the values of the predictors!
Suppose that there is some non-trivial cost to trying to sell insurance
to a given individual. For instance, perhaps a salesperson must visit each
potential customer. If the company tries to sell insurance to a random
selection of customers, then the success rate will be only 6 %, which may
be far too low given the costs involved. Instead, the company would like
to try to sell insurance only to customers who are likely to buy it. So the
overall error rate is not of interest. Instead, the fraction of individuals that
are correctly predicted to buy insurance is of interest.
It turns out that KNN with K = 1 does far better than random guessing
among the customers that are predicted to buy insurance. Among 77 such
customers, 9, or 11.7 %, actually do purchase insurance. This is double the
rate that one would obtain from random guessing.
> table(knn.pred , test.Y)
test.Y
knn.pred No Yes
No 873 50
Yes 68 9
> 9 / (68 + 9)
[1] 0.117
Using K = 3, the success rate increases to 19 %, and with K = 5 the rate is
26.7 %. This is over four times the rate that results from random guessing.
It appears that KNN is finding some real patterns in a difficult data set!
> knn.pred <- knn(train.X, test.X, train.Y, k = 3)
> table(knn.pred , test.Y)
test.Y
knn.pred No Yes
No 920 54
Yes 21 5
> 5 / 26
[1] 0.192
> knn.pred <- knn(train.X, test.X, train.Y, k = 5)](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-194-320.jpg)
![4.7 Lab: Classification Methods 185
> table(knn.pred , test.Y)
test.Y
knn.pred No Yes
No 930 55
Yes 11 4
> 4 / 15
[1] 0.267
However, while this strategy is cost-effective, it is worth noting that only
15 customers are predicted to purchase insurance using KNN with K =
5. In practice, the insurance company may wish to expend resources on
convincing more than just 15 potential customers to buy insurance.
As a comparison, we can also fit a logistic regression model to the data.
If we use 0.5 as the predicted probability cut-off for the classifier, then
we have a problem: only seven of the test observations are predicted to
purchase insurance. Even worse, we are wrong about all of these! However,
we are not required to use a cut-off of 0.5. If we instead predict a purchase
any time the predicted probability of purchase exceeds 0.25, we get much
better results: we predict that 33 people will purchase insurance, and we
are correct for about 33 % of these people. This is over five times better
than random guessing!
> glm.fits <- glm(Purchase ∼ ., data = Caravan ,
family = binomial , subset = -test)
Warning message:
glm.fits: fitted probabilities numerically 0 or 1 occurred
> glm.probs <- predict(glm.fits , Caravan[test , ],
type = "response ")
> glm.pred <- rep("No", 1000)
> glm.pred[glm.probs > .5] <- "Yes"
> table(glm.pred , test.Y)
test.Y
glm.pred No Yes
No 934 59
Yes 7 0
> glm.pred <- rep("No", 1000)
> glm.pred[glm.probs > .25] <- "Yes"
> table(glm.pred , test.Y)
test.Y
glm.pred No Yes
No 919 48
Yes 22 11
> 11 / (22 + 11)
[1] 0.333
4.7.7 Poisson Regression
Finally, we fit a Poisson regression model to the Bikeshare data set, which
measures the number of bike rentals (bikers) per hour in Washington, DC.
The data can be found in the ISLR2 library.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-195-320.jpg)
![186 4. Classification
> attach(Bikeshare)
> dim(Bikeshare)
[1] 8645 15
> names(Bikeshare)
[1] "season" "mnth" "day" "hr"
[5] "holiday" "weekday" "workingday " "weathersit"
[9] "temp" "atemp" "hum" "windspeed"
[13] "casual" "registered " "bikers"
We begin by fitting a least squares linear regression model to the data.
> mod.lm <- lm(
bikers ∼ mnth + hr + workingday + temp + weathersit ,
data = Bikeshare
)
> summary(mod.lm)
Call:
lm(formula = bikers ∼ mnth + hr + workingday + temp +
weathersit , data = Bikeshare)
Residuals:
Min 1Q Median 3Q Max
-299.00 -45.70 -6.23 41.08 425.29
Coefficients :
Estimate Std. Error t value Pr(>|t|)
(Intercept) -68.632 5.307 -12.932 < 2e -16 ***
mnthFeb 6.845 4.287 1.597 0.110398
mnthMarch 16.551 4.301 3.848 0.000120 ***
mnthApril 41.425 4.972 8.331 < 2e -16 ***
mnthMay 72.557 5.641 12.862 < 2e -16 ***
Due to space constraints, we truncate the output of summary(mod.lm). In
mod.lm, the first level of hr (0) and mnth (Jan) are treated as the baseline
values, and so no coefficient estimates are provided for them: implicitly,
their coefficient estimates are zero, and all other levels are measured relative
to these baselines. For example, the Feb coefficient of 6.845 signifies that,
holding all other variables constant, there are on average about 7 more
riders in February than in January. Similarly there are about 16.5 more
riders in March than in January.
The results seen in Section 4.6.1 used a slightly different coding of the
variables hr and mnth, as follows:
> contrasts (Bikeshare$hr) = contr.sum (24)
> contrasts (Bikeshare$mnth) = contr.sum (12)
> mod.lm2 <- lm(
bikers ∼ mnth + hr + workingday + temp + weathersit ,
data = Bikeshare
)
> summary(mod.lm2)
Call:
lm(formula = bikers ∼ mnth + hr + workingday + temp +
weathersit , data = Bikeshare)](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-196-320.jpg)
![4.7 Lab: Classification Methods 187
Residuals:
Min 1Q Median 3Q Max
-299.00 -45.70 -6.23 41.08 425.29
Coefficients :
Estimate Std. Error t value Pr(>|t|)
(Intercept) 73.597 5.132 14.340 < 2e -16 ***
mnth1 -46.087 4.086 -11.281 < 2e-16 ***
mnth2 -39.242 3.539 -11.088 < 2e-16 ***
mnth3 -29.536 3.155 -9.361 < 2e -16 ***
mnth4 -4.662 2.741 -1.701 0.08895 .
What is the difference between the two codings? In mod.lm2, a coefficient
estimate is reported for all but the last level of hr and mnth. Importantly,
in mod.lm2, the coefficient estimate for the last level of mnth is not zero:
instead, it equals the negative of the sum of the coefficient estimates for
all of the other levels. Similarly, in mod.lm2, the coefficient estimate for the
last level of hr is the negative of the sum of the coefficient estimates for
all of the other levels. This means that the coefficients of hr and mnth in
mod.lm2 will always sum to zero, and can be interpreted as the difference
from the mean level. For example, the coefficient for January of −46.087
indicates that, holding all other variables constant, there are typically 46
fewer riders in January relative to the yearly average.
It is important to realize that the choice of coding really does not matter,
provided that we interpret the model output correctly in light of the coding
used. For example, we see that the predictions from the linear model are
the same regardless of coding:
> sum (( predict(mod.lm) - predict(mod.lm2))^2)
[1] 1.426e-18
The sum of squared differences is zero. We can also see this using the
all.equal() function:
all.equal()
> all.equal(predict(mod.lm), predict(mod.lm2))
To reproduce the left-hand side of Figure 4.13, we must first obtain
the coefficient estimates associated with mnth. The coefficients for January
through November can be obtained directly from the mod.lm2 object. The
coefficient for December must be explicitly computed as the negative sum
of all the other months.
> coef.months <- c(coef(mod.lm2)[2:12] ,
-sum(coef(mod.lm2)[2:12]))
To make the plot, we manually label the x-axis with the names of the
months.
> plot(coef.months , xlab = "Month", ylab = " Coefficient ",
xaxt = "n", col = "blue", pch = 19, type = "o")](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-197-320.jpg)
![188 4. Classification
> axis(side = 1, at = 1:12 , labels = c("J", "F", "M", "A",
"M", "J", "J", "A", "S", "O", "N", "D"))
Reproducing the right-hand side of Figure 4.13 follows a similar process.
> coef.hours <- c(coef(mod.lm2)[13:35] ,
-sum(coef(mod.lm2)[13:35]))
> plot(coef.hours , xlab = "Hour", ylab = " Coefficient ",
col = "blue", pch = 19, type = "o")
Now, we consider instead fitting a Poisson regression model to the Bikeshare
data. Very little changes, except that we now use the function glm() with
the argument family = poisson to specify that we wish to fit a Poisson
regression model:
> mod.pois <- glm(
bikers ∼ mnth + hr + workingday + temp + weathersit ,
data = Bikeshare , family = poisson
)
> summary(mod.pois)
Call:
glm(formula = bikers ∼ mnth + hr + workingday + temp +
weathersit , family = poisson , data = Bikeshare)
Deviance Residuals:
Min 1Q Median 3Q Max
-20.7574 -3.3441 -0.6549 2.6999 21.9628
Coefficients :
Estimate Std. Error z value Pr(>|z|)
(Intercept) 4.118245 0.006021 683.964 < 2e -16 ***
mnth1 -0.670170 0.005907 -113.445 < 2e-16 ***
mnth2 -0.444124 0.004860 -91.379 < 2e -16 ***
mnth3 -0.293733 0.004144 -70.886 < 2e -16 ***
mnth4 0.021523 0.003125 6.888 5.66e-12 ***
We can plot the coefficients associated with mnth and hr, in order to repro-
duce Figure 4.15:
> coef.mnth <- c(coef(mod.pois)[2:12] ,
-sum(coef(mod.pois)[2:12]))
> plot(coef.mnth , xlab = "Month", ylab = " Coefficient ",
xaxt = "n", col = "blue", pch = 19, type = "o")
> axis(side = 1, at = 1:12 , labels = c("J", "F", "M", "A", "M",
"J", "J", "A", "S", "O", "N", "D"))
> coef.hours <- c(coef(mod.pois)[13:35] ,
-sum(coef(mod.pois)[13:35]))
> plot(coef.hours , xlab = "Hour", ylab = " Coefficient ",
col = "blue", pch = 19, type = "o")
We can once again use the predict() function to obtain the fitted values
(predictions) from this Poisson regression model. However, we must use the
argument type = "response" to specify that we want R to output exp(β̂0 +
β̂1X1 +. . .+β̂pXp) rather than β̂0 +β̂1X1 +. . .+β̂pXp, which it will output
by default.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-198-320.jpg)
![190 4. Classification
perform prediction using only observations that are near the test ob-
servation for which a prediction must be made. This phenomenon is
known as the curse of dimensionality, and it ties into the fact that
curse of di-
mensionality
non-parametric approaches often perform poorly when p is large. We
will now investigate this curse.
(a) Suppose that we have a set of observations, each with measure-
ments on p = 1 feature, X. We assume that X is uniformly
(evenly) distributed on [0, 1]. Associated with each observation
is a response value. Suppose that we wish to predict a test obser-
vation’s response using only observations that are within 10 % of
the range of X closest to that test observation. For instance, in
order to predict the response for a test observation with X = 0.6,
we will use observations in the range [0.55, 0.65]. On average,
what fraction of the available observations will we use to make
the prediction?
(b) Now suppose that we have a set of observations, each with
measurements on p = 2 features, X1 and X2. We assume that
(X1, X2) are uniformly distributed on [0, 1] × [0, 1]. We wish to
predict a test observation’s response using only observations that
are within 10 % of the range of X1 and within 10 % of the range
of X2 closest to that test observation. For instance, in order to
predict the response for a test observation with X1 = 0.6 and
X2 = 0.35, we will use observations in the range [0.55, 0.65] for
X1 and in the range [0.3, 0.4] for X2. On average, what fraction
of the available observations will we use to make the prediction?
(c) Now suppose that we have a set of observations on p = 100 fea-
tures. Again the observations are uniformly distributed on each
feature, and again each feature ranges in value from 0 to 1. We
wish to predict a test observation’s response using observations
within the 10 % of each feature’s range that is closest to that test
observation. What fraction of the available observations will we
use to make the prediction?
(d) Using your answers to parts (a)–(c), argue that a drawback of
KNN when p is large is that there are very few training obser-
vations “near” any given test observation.
(e) Now suppose that we wish to make a prediction for a test obser-
vation by creating a p-dimensional hypercube centered around
the test observation that contains, on average, 10 % of the train-
ing observations. For p = 1, 2, and 100, what is the length of
each side of the hypercube? Comment on your answer.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-200-320.jpg)
![5.3 Lab: Cross-Validation and the Bootstrap 213
5.3.1 The Validation Set Approach
We explore the use of the validation set approach in order to estimate the
test error rates that result from fitting various linear models on the Auto
data set.
Before we begin, we use the set.seed() function in order to set a seed for
seed
R’s random number generator, so that the reader of this book will obtain
precisely the same results as those shown below. It is generally a good idea
to set a random seed when performing an analysis such as cross-validation
that contains an element of randomness, so that the results obtained can
be reproduced precisely at a later time.
We begin by using the sample() function to split the set of observations
sample()
into two halves, by selecting a random subset of 196 observations out of
the original 392 observations. We refer to these observations as the training
set.
> library(ISLR2)
> set.seed (1)
> train <- sample (392 , 196)
(Here we use a shortcut in the sample command; see ?sample for details.)
We then use the subset option in lm() to fit a linear regression using only
the observations corresponding to the training set.
> lm.fit <- lm(mpg ∼ horsepower , data = Auto , subset = train)
We now use the predict() function to estimate the response for all 392
observations, and we use the mean() function to calculate the MSE of the
196 observations in the validation set. Note that the -train index below
selects only the observations that are not in the training set.
> attach(Auto)
> mean (( mpg - predict(lm.fit , Auto))[-train ]^2)
[1] 23.27
Therefore, the estimated test MSE for the linear regression fit is 23.27. We
can use the poly() function to estimate the test error for the quadratic and
cubic regressions.
> lm.fit2 <- lm(mpg ∼ poly(horsepower , 2), data = Auto ,
subset = train)
> mean (( mpg - predict(lm.fit2 , Auto))[-train ]^2)
[1] 18.72
> lm.fit3 <- lm(mpg ∼ poly(horsepower , 3), data = Auto ,
subset = train)
> mean (( mpg - predict(lm.fit3 , Auto))[-train ]^2)
[1] 18.79
These error rates are 18.72 and 18.79, respectively. If we choose a different
training set instead, then we will obtain somewhat different errors on the
validation set.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-222-320.jpg)
![214 5. Resampling Methods
> set.seed (2)
> train <- sample (392 , 196)
> lm.fit <- lm(mpg ∼ horsepower , subset = train)
> mean (( mpg - predict(lm.fit , Auto))[-train ]^2)
[1] 25.73
> lm.fit2 <- lm(mpg ∼ poly(horsepower , 2), data = Auto ,
subset = train)
> mean (( mpg - predict(lm.fit2 , Auto))[-train ]^2)
[1] 20.43
> lm.fit3 <- lm(mpg ∼ poly(horsepower , 3), data = Auto ,
subset = train)
> mean (( mpg - predict(lm.fit3 , Auto))[-train ]^2)
[1] 20.39
Using this split of the observations into a training set and a validation
set, we find that the validation set error rates for the models with linear,
quadratic, and cubic terms are 25.73, 20.43, and 20.39, respectively.
These results are consistent with our previous findings: a model that
predicts mpg using a quadratic function of horsepower performs better than
a model that involves only a linear function of horsepower, and there is
little evidence in favor of a model that uses a cubic function of horsepower.
5.3.2 Leave-One-Out Cross-Validation
The LOOCV estimate can be automatically computed for any generalized
linear model using the glm() and cv.glm() functions. In the lab for Chap-
cv.glm()
ter 4, we used the glm() function to perform logistic regression by passing
in the family = "binomial" argument. But if we use glm() to fit a model
without passing in the family argument, then it performs linear regression,
just like the lm() function. So for instance,
> glm.fit <- glm(mpg ∼ horsepower , data = Auto)
> coef(glm.fit)
(Intercept) horsepower
39.936 -0.158
and
> lm.fit <- lm(mpg ∼ horsepower , data = Auto)
> coef(lm.fit)
(Intercept) horsepower
39.936 -0.158
yield identical linear regression models. In this lab, we will perform linear
regression using the glm() function rather than the lm() function because
the former can be used together with cv.glm(). The cv.glm() function is
part of the boot library.
> library(boot)
> glm.fit <- glm(mpg ∼ horsepower , data = Auto)
> cv.err <- cv.glm(Auto , glm.fit)
> cv.err$delta](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-223-320.jpg)
![5.3 Lab: Cross-Validation and the Bootstrap 215
1 1
24.23 24.23
The cv.glm() function produces a list with several components. The two
numbers in the delta vector contain the cross-validation results. In this
case the numbers are identical (up to two decimal places) and correspond
to the LOOCV statistic given in (5.1). Below, we discuss a situation in
which the two numbers differ. Our cross-validation estimate for the test
error is approximately 24.23.
We can repeat this procedure for increasingly complex polynomial fits.
To automate the process, we use the for() function to initiate a for loop
for()
for loop
which iteratively fits polynomial regressions for polynomials of order i = 1
to i = 10, computes the associated cross-validation error, and stores it in
the ith element of the vector cv.error. We begin by initializing the vector.
> cv.error <- rep(0, 10)
> for (i in 1:10) {
+ glm.fit <- glm(mpg ∼ poly(horsepower , i), data = Auto)
+ cv.error[i] <- cv.glm(Auto , glm.fit)$delta [1]
+ }
> cv.error
[1] 24.23 19.25 19.33 19.42 19.03 18.98 18.83 18.96 19.07 19.49
As in Figure 5.4, we see a sharp drop in the estimated test MSE between
the linear and quadratic fits, but then no clear improvement from using
higher-order polynomials.
5.3.3 k-Fold Cross-Validation
The cv.glm() function can also be used to implement k-fold CV. Below we
use k = 10, a common choice for k, on the Auto data set. We once again set
a random seed and initialize a vector in which we will store the CV errors
corresponding to the polynomial fits of orders one to ten.
> set.seed (17)
> cv.error .10 <- rep(0, 10)
> for (i in 1:10) {
+ glm.fit <- glm(mpg ∼ poly(horsepower , i), data = Auto)
+ cv.error .10[i] <- cv.glm(Auto , glm.fit , K = 10)$delta [1]
+ }
> cv.error .10
[1] 24.27 19.27 19.35 19.29 19.03 18.90 19.12 19.15 18.87 20.96
Notice that the computation time is shorter than that of LOOCV. (In
principle, the computation time for LOOCV for a least squares linear model
should be faster than for k-fold CV, due to the availability of the formula
(5.2) for LOOCV; however, unfortunately the cv.glm() function does not
make use of this formula.) We still see little evidence that using cubic or
higher-order polynomial terms leads to lower test error than simply using
a quadratic fit.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-224-320.jpg)
![216 5. Resampling Methods
We saw in Section 5.3.2 that the two numbers associated with delta are
essentially the same when LOOCV is performed. When we instead perform
k-fold CV, then the two numbers associated with delta differ slightly. The
first is the standard k-fold CV estimate, as in (5.3). The second is a bias-
corrected version. On this data set, the two estimates are very similar to
each other.
5.3.4 The Bootstrap
We illustrate the use of the bootstrap in the simple example of Section 5.2,
as well as on an example involving estimating the accuracy of the linear
regression model on the Auto data set.
Estimating the Accuracy of a Statistic of Interest
One of the great advantages of the bootstrap approach is that it can be
applied in almost all situations. No complicated mathematical calculations
are required. Performing a bootstrap analysis in R entails only two steps.
First, we must create a function that computes the statistic of interest.
Second, we use the boot() function, which is part of the boot library, to
boot()
perform the bootstrap by repeatedly sampling observations from the data
set with replacement.
The Portfolio data set in the ISLR2 package is simulated data of 100
pairs of returns, generated in the fashion described in Section 5.2. To illus-
trate the use of the bootstrap on this data, we must first create a function,
alpha.fn(), which takes as input the (X, Y ) data as well as a vector indi-
cating which observations should be used to estimate α. The function then
outputs the estimate for α based on the selected observations.
> alpha.fn <- function (data , index) {
+ X <- data$X[index]
+ Y <- data$Y[index]
+ (var(Y) - cov(X, Y)) / (var(X) + var(Y) - 2 * cov(X, Y))
+ }
This function returns, or outputs, an estimate for α based on applying
(5.7) to the observations indexed by the argument index. For instance, the
following command tells R to estimate α using all 100 observations.
> alpha.fn(Portfolio , 1:100)
[1] 0.576
The next command uses the sample() function to randomly select 100 ob-
servations from the range 1 to 100, with replacement. This is equivalent
to constructing a new bootstrap data set and recomputing α̂ based on the
new data set.
> set.seed (7)
> alpha.fn(Portfolio , sample (100 , 100, replace = T))
[1] 0.539](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-225-320.jpg)
![220 5. Resampling Methods
(d) When n = 5, what is the probability that the jth observation is
in the bootstrap sample?
(e) When n = 100, what is the probability that the jth observation
is in the bootstrap sample?
(f) When n = 10, 000, what is the probability that the jth observa-
tion is in the bootstrap sample?
(g) Create a plot that displays, for each integer value of n from 1
to 100, 000, the probability that the jth observation is in the
bootstrap sample. Comment on what you observe.
(h) We will now investigate numerically the probability that a boot-
strap sample of size n = 100 contains the jth observation. Here
j = 4. We repeatedly create bootstrap samples, and each time
we record whether or not the fourth observation is contained in
the bootstrap sample.
> store <- rep(NA , 10000)
> for(i in 1:10000){
store[i] <- sum(sample (1:100 , rep=TRUE) == 4) > 0
}
> mean(store)
Comment on the results obtained.
3. We now review k-fold cross-validation.
(a) Explain how k-fold cross-validation is implemented.
(b) What are the advantages and disadvantages of k-fold cross-
validation relative to:
i. The validation set approach?
ii. LOOCV?
4. Suppose that we use some statistical learning method to make a pre-
diction for the response Y for a particular value of the predictor X.
Carefully describe how we might estimate the standard deviation of
our prediction.
Applied
5. In Chapter 4, we used logistic regression to predict the probability of
default using income and balance on the Default data set. We will
now estimate the test error of this logistic regression model using the
validation set approach. Do not forget to set a random seed before
beginning your analysis.
(a) Fit a logistic regression model that uses income and balance to
predict default.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-229-320.jpg)
![5.4 Exercises 223
i. Y = β0 + β1X + ϵ
ii. Y = β0 + β1X + β2X2
+ ϵ
iii. Y = β0 + β1X + β2X2
+ β3X3
+ ϵ
iv. Y = β0 + β1X + β2X2
+ β3X3
+ β4X4
+ ϵ.
Note you may find it helpful to use the data.frame() function
to create a single data set containing both X and Y .
(d) Repeat (c) using another random seed, and report your results.
Are your results the same as what you got in (c)? Why?
(e) Which of the models in (c) had the smallest LOOCV error? Is
this what you expected? Explain your answer.
(f) Comment on the statistical significance of the coefficient esti-
mates that results from fitting each of the models in (c) using
least squares. Do these results agree with the conclusions drawn
based on the cross-validation results?
9. We will now consider the Boston housing data set, from the ISLR2
library.
(a) Based on this data set, provide an estimate for the population
mean of medv. Call this estimate µ̂.
(b) Provide an estimate of the standard error of µ̂. Interpret this
result.
Hint: We can compute the standard error of the sample mean by
dividing the sample standard deviation by the square root of the
number of observations.
(c) Now estimate the standard error of µ̂ using the bootstrap. How
does this compare to your answer from (b)?
(d) Based on your bootstrap estimate from (c), provide a 95 % con-
fidence interval for the mean of medv. Compare it to the results
obtained using t.test(Boston$medv).
Hint: You can approximate a 95 % confidence interval using the
formula [µ̂ − 2SE(µ̂), µ̂ + 2SE(µ̂)].
(e) Based on this data set, provide an estimate, µ̂med, for the median
value of medv in the population.
(f) We now would like to estimate the standard error of µ̂med. Unfor-
tunately, there is no simple formula for computing the standard
error of the median. Instead, estimate the standard error of the
median using the bootstrap. Comment on your findings.
(g) Based on this data set, provide an estimate for the tenth per-
centile of medv in Boston census tracts. Call this quantity µ̂0.1.
(You can use the quantile() function.)
(h) Use the bootstrap to estimate the standard error of µ̂0.1. Com-
ment on your findings.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-232-320.jpg)
![6.5 Lab: Linear Models and Regularization Methods 267
It is also important to be particularly careful in reporting errors and
measures of model fit in the high-dimensional setting. We have seen that
when p > n, it is easy to obtain a useless model that has zero residu-
als. Therefore, one should never use sum of squared errors, p-values, R2
statistics, or other traditional measures of model fit on the training data as
evidence of a good model fit in the high-dimensional setting. For instance,
as we saw in Figure 6.23, one can easily obtain a model with R2
= 1 when
p > n. Reporting this fact might mislead others into thinking that a sta-
tistically valid and useful model has been obtained, whereas in fact this
provides absolutely no evidence of a compelling model. It is important to
instead report results on an independent test set, or cross-validation errors.
For instance, the MSE or R2
on an independent test set is a valid measure
of model fit, but the MSE on the training set certainly is not.
6.5 Lab: Linear Models and Regularization
Methods
6.5.1 Subset Selection Methods
Best Subset Selection
Here we apply the best subset selection approach to the Hitters data. We
wish to predict a baseball player’s Salary on the basis of various statistics
associated with performance in the previous year.
First of all, we note that the Salary variable is missing for some of the
players. The is.na() function can be used to identify the missing observa-
is.na()
tions. It returns a vector of the same length as the input vector, with a TRUE
for any elements that are missing, and a FALSE for non-missing elements.
The sum() function can then be used to count all of the missing elements.
sum()
> library(ISLR2)
> View(Hitters)
> names(Hitters)
[1] "AtBat" "Hits" "HmRun" "Runs" "RBI"
[6] "Walks" "Years" "CAtBat" "CHits" "CHmRun"
[11] "CRuns" "CRBI" "CWalks" "League" "Division"
[16] "PutOuts" "Assists" "Errors" "Salary" "NewLeague"
> dim(Hitters)
[1] 322 20
> sum(is.na(Hitters$Salary))
[1] 59
Hence we see that Salary is missing for 59 players. The na.omit() function
removes all of the rows that have missing values in any variable.
> Hitters <- na.omit(Hitters)
> dim(Hitters)
[1] 263 20](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-275-320.jpg)
![268 6. Linear Model Selection and Regularization
> sum(is.na(Hitters))
[1] 0
The regsubsets() function (part of the leaps library) performs best sub-
regsubsets()
set selection by identifying the best model that contains a given number
of predictors, where best is quantified using RSS. The syntax is the same
as for lm(). The summary() command outputs the best set of variables for
each model size.
> library(leaps)
> regfit.full <- regsubsets (Salary ∼ ., Hitters)
> summary(regfit.full)
Subset selection object
Call: regsubsets.formula(Salary ∼ ., Hitters)
19 Variables (and intercept)
...
1 subsets of each size up to 8
Selection Algorithm: exhaustive
AtBat Hits HmRun Runs RBI Walks Years CAtBat CHits
1 ( 1 ) " " " " " " " " " " " " " " " " " "
2 ( 1 ) " " "*" " " " " " " " " " " " " " "
3 ( 1 ) " " "*" " " " " " " " " " " " " " "
4 ( 1 ) " " "*" " " " " " " " " " " " " " "
5 ( 1 ) "*" "*" " " " " " " " " " " " " " "
6 ( 1 ) "*" "*" " " " " " " "*" " " " " " "
7 ( 1 ) " " "*" " " " " " " "*" " " "*" "*"
8 ( 1 ) "*" "*" " " " " " " "*" " " " " " "
CHmRun CRuns CRBI CWalks LeagueN DivisionW PutOuts
1 ( 1 ) " " " " "*" " " " " " " " "
2 ( 1 ) " " " " "*" " " " " " " " "
3 ( 1 ) " " " " "*" " " " " " " "*"
4 ( 1 ) " " " " "*" " " " " "*" "*"
5 ( 1 ) " " " " "*" " " " " "*" "*"
6 ( 1 ) " " " " "*" " " " " "*" "*"
7 ( 1 ) "*" " " " " " " " " "*" "*"
8 ( 1 ) "*" "*" " " "*" " " "*" "*"
Assists Errors NewLeagueN
1 ( 1 ) " " " " " "
2 ( 1 ) " " " " " "
3 ( 1 ) " " " " " "
4 ( 1 ) " " " " " "
5 ( 1 ) " " " " " "
6 ( 1 ) " " " " " "
7 ( 1 ) " " " " " "
8 ( 1 ) " " " " " "
An asterisk indicates that a given variable is included in the corresponding
model. For instance, this output indicates that the best two-variable model
contains only Hits and CRBI. By default, regsubsets() only reports results
up to the best eight-variable model. But the nvmax option can be used
in order to return as many variables as are desired. Here we fit up to a
19-variable model.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-276-320.jpg)
![6.5 Lab: Linear Models and Regularization Methods 269
> regfit.full <- regsubsets (Salary ∼ ., data = Hitters ,
nvmax = 19)
> reg.summary <- summary(regfit.full)
The summary() function also returns R2
, RSS, adjusted R2
, Cp, and BIC.
We can examine these to try to select the best overall model.
> names(reg.summary)
[1] "which" "rsq" "rss" "adjr2" "cp" "bic"
[7] "outmat" "obj"
For instance, we see that the R2
statistic increases from 32 %, when only
one variable is included in the model, to almost 55 %, when all variables
are included. As expected, the R2
statistic increases monotonically as more
variables are included.
> reg.summary$rsq
[1] 0.321 0.425 0.451 0.475 0.491 0.509 0.514 0.529 0.535
[10] 0.540 0.543 0.544 0.544 0.545 0.545 0.546 0.546 0.546
[19] 0.546
Plotting RSS, adjusted R2
, Cp, and BIC for all of the models at once will
help us decide which model to select. Note the type = "l" option tells R to
connect the plotted points with lines.
> par(mfrow = c(2, 2))
> plot(reg.summary$rss , xlab = "Number of Variables ",
ylab = "RSS", type = "l")
> plot(reg.summary$adjr2 , xlab = "Number of Variables ",
ylab = "Adjusted RSq", type = "l")
The points() command works like the plot() command, except that it
points()
puts points on a plot that has already been created, instead of creating a
new plot. The which.max() function can be used to identify the location of
the maximum point of a vector. We will now plot a red dot to indicate the
model with the largest adjusted R2
statistic.
> which.max(reg.summary$adjr2)
[1] 11
> points (11, reg.summary$adjr2 [11] , col = "red", cex = 2,
pch = 20)
In a similar fashion we can plot the Cp and BIC statistics, and indicate the
models with the smallest statistic using which.min().
which.min()
> plot(reg.summary$cp , xlab = "Number of Variables ",
ylab = "Cp", type = "l")
> which.min(reg.summary$cp)
[1] 10
> points (10, reg.summary$cp[10] , col = "red", cex = 2,
pch = 20)
> which.min(reg.summary$bic)
[1] 6
> plot(reg.summary$bic , xlab = "Number of Variables ",
ylab = "BIC", type = "l")](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-277-320.jpg)
![270 6. Linear Model Selection and Regularization
> points (6, reg.summary$bic [6], col = "red", cex = 2,
pch = 20)
The regsubsets() function has a built-in plot() command which can
be used to display the selected variables for the best model with a given
number of predictors, ranked according to the BIC, Cp, adjusted R2
, or
AIC. To find out more about this function, type ?plot.regsubsets.
> plot(regfit.full , scale = "r2")
> plot(regfit.full , scale = "adjr2")
> plot(regfit.full , scale = "Cp")
> plot(regfit.full , scale = "bic")
The top row of each plot contains a black square for each variable selected
according to the optimal model associated with that statistic. For instance,
we see that several models share a BIC close to −150. However, the model
with the lowest BIC is the six-variable model that contains only AtBat,
Hits, Walks, CRBI, DivisionW, and PutOuts. We can use the coef() function
to see the coefficient estimates associated with this model.
> coef(regfit.full , 6)
(Intercept) AtBat Hits Walks CRBI
91.512 -1.869 7.604 3.698 0.643
DivisionW PutOuts
-122.952 0.264
Forward and Backward Stepwise Selection
We can also use the regsubsets() function to perform forward stepwise
or backward stepwise selection, using the argument method = "forward" or
method = "backward".
> regfit.fwd <- regsubsets (Salary ∼ ., data = Hitters ,
nvmax = 19, method = "forward")
> summary(regfit.fwd)
> regfit.bwd <- regsubsets (Salary ∼ ., data = Hitters ,
nvmax = 19, method = "backward ")
> summary(regfit.bwd)
For instance, we see that using forward stepwise selection, the best one-
variable model contains only CRBI, and the best two-variable model ad-
ditionally includes Hits. For this data, the best one-variable through six-
variable models are each identical for best subset and forward selection.
However, the best seven-variable models identified by forward stepwise se-
lection, backward stepwise selection, and best subset selection are different.
> coef(regfit.full , 7)
(Intercept) Hits Walks CAtBat CHits
79.451 1.283 3.227 -0.375 1.496
CHmRun DivisionW PutOuts
1.442 -129.987 0.237
> coef(regfit.fwd , 7)](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-278-320.jpg)
![6.5 Lab: Linear Models and Regularization Methods 271
(Intercept) AtBat Hits Walks CRBI
109.787 -1.959 7.450 4.913 0.854
CWalks DivisionW PutOuts
-0.305 -127.122 0.253
> coef(regfit.bwd , 7)
(Intercept) AtBat Hits Walks CRuns
105.649 -1.976 6.757 6.056 1.129
CWalks DivisionW PutOuts
-0.716 -116.169 0.303
Choosing Among Models Using the Validation-Set Approach and
Cross-Validation
We just saw that it is possible to choose among a set of models of different
sizes using Cp, BIC, and adjusted R2
. We will now consider how to do this
using the validation set and cross-validation approaches.
In order for these approaches to yield accurate estimates of the test
error, we must use only the training observations to perform all aspects of
model-fitting—including variable selection. Therefore, the determination of
which model of a given size is best must be made using only the training
observations. This point is subtle but important. If the full data set is used
to perform the best subset selection step, the validation set errors and
cross-validation errors that we obtain will not be accurate estimates of the
test error.
In order to use the validation set approach, we begin by splitting the
observations into a training set and a test set. We do this by creating
a random vector, train, of elements equal to TRUE if the corresponding
observation is in the training set, and FALSE otherwise. The vector test has
a TRUE if the observation is in the test set, and a FALSE otherwise. Note the
! in the command to create test causes TRUEs to be switched to FALSEs and
vice versa. We also set a random seed so that the user will obtain the same
training set/test set split.
> set.seed (1)
> train <- sample(c(TRUE , FALSE), nrow(Hitters),
replace = TRUE)
> test <- (! train)
Now, we apply regsubsets() to the training set in order to perform best
subset selection.
> regfit.best <- regsubsets (Salary ∼ .,
data = Hitters[train , ], nvmax = 19)
Notice that we subset the Hitters data frame directly in the call in or-
der to access only the training subset of the data, using the expression
Hitters[train, ]. We now compute the validation set error for the best
model of each model size. We first make a model matrix from the test
data.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-279-320.jpg)
![272 6. Linear Model Selection and Regularization
> test.mat <- model.matrix(Salary ∼ ., data = Hitters[test , ])
The model.matrix() function is used in many regression packages for build-
model.matrix()
ing an “X” matrix from data. Now we run a loop, and for each size i, we
extract the coefficients from regfit.best for the best model of that size,
multiply them into the appropriate columns of the test model matrix to
form the predictions, and compute the test MSE.
> val.errors <- rep(NA , 19)
> for (i in 1:19) {
+ coefi <- coef(regfit.best , id = i)
+ pred <- test.mat[, names(coefi)] %*% coefi
+ val.errors[i] <- mean (( Hitters$Salary[test] - pred)^2)
}
We find that the best model is the one that contains seven variables.
> val.errors
[1] 164377 144405 152176 145198 137902 139176 126849 136191
[9] 132890 135435 136963 140695 140691 141951 141508 142164
[17] 141767 142340 142238
> which.min(val.errors)
[1] 7
> coef(regfit.best , 7)
(Intercept) AtBat Hits Walks CRuns
67.109 -2.146 7.015 8.072 1.243
CWalks DivisionW PutOuts
-0.834 -118.436 0.253
This was a little tedious, partly because there is no predict() method
for regsubsets(). Since we will be using this function again, we can capture
our steps above and write our own predict method.
> predict.regsubsets <- function (object , newdata , id , ...) {
+ form <- as.formula(object$call [[2]])
+ mat <- model.matrix(form , newdata)
+ coefi <- coef(object , id = id)
+ xvars <- names(coefi)
+ mat[, xvars] %*% coefi
+ }
Our function pretty much mimics what we did above. The only complex
part is how we extracted the formula used in the call to regsubsets(). We
demonstrate how we use this function below, when we do cross-validation.
Finally, we perform best subset selection on the full data set, and select
the best seven-variable model. It is important that we make use of the
full data set in order to obtain more accurate coefficient estimates. Note
that we perform best subset selection on the full data set and select the
best seven-variable model, rather than simply using the variables that were
obtained from the training set, because the best seven-variable model on
the full data set may differ from the corresponding model on the training
set.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-280-320.jpg)
![6.5 Lab: Linear Models and Regularization Methods 273
> regfit.best <- regsubsets (Salary ∼ ., data = Hitters ,
nvmax = 19)
> coef(regfit.best , 7)
(Intercept) Hits Walks CAtBat CHits
79.451 1.283 3.227 -0.375 1.496
CHmRun DivisionW PutOuts
1.442 -129.987 0.237
In fact, we see that the best seven-variable model on the full data set has a
different set of variables than the best seven-variable model on the training
set.
We now try to choose among the models of different sizes using cross-
validation. This approach is somewhat involved, as we must perform best
subset selection within each of the k training sets. Despite this, we see that
with its clever subsetting syntax, R makes this job quite easy. First, we
create a vector that allocates each observation to one of k = 10 folds, and
we create a matrix in which we will store the results.
> k <- 10
> n <- nrow(Hitters)
> set.seed (1)
> folds <- sample(rep (1:k, length = n))
> cv.errors <- matrix(NA , k, 19,
dimnames = list(NULL , paste (1:19)))
Now we write a for loop that performs cross-validation. In the jth fold, the
elements of folds that equal j are in the test set, and the remainder are in
the training set. We make our predictions for each model size (using our new
predict() method), compute the test errors on the appropriate subset, and
store them in the appropriate slot in the matrix cv.errors. Note that in the
following code R will automatically use our predict.regsubsets() function
when we call predict() because the best.fit object has class regsubsets.
> for (j in 1:k) {
+ best.fit <- regsubsets (Salary ∼ .,
data = Hitters[folds != j, ],
nvmax = 19)
+ for (i in 1:19) {
+ pred <- predict(best.fit , Hitters[folds == j, ], id = i)
+ cv.errors[j, i] <-
mean (( Hitters$Salary[folds == j] - pred)^2)
+ }
+ }
This has given us a 10×19 matrix, of which the (j, i)th element corresponds
to the test MSE for the jth cross-validation fold for the best i-variable
model. We use the apply() function to average over the columns of this
apply()
matrix in order to obtain a vector for which the ith element is the cross-
validation error for the i-variable model.
> mean.cv.errors <- apply(cv.errors , 2, mean)
> mean.cv.errors](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-281-320.jpg)
![274 6. Linear Model Selection and Regularization
1 2 3 4 5 6 7 8
143440 126817 134214 131783 130766 120383 121443 114364
9 10 11 12 13 14 15 16
115163 109366 112738 113617 115558 115853 115631 116050
17 18 19
116117 116419 116299
> par(mfrow = c(1, 1))
> plot(mean.cv.errors , type = "b")
We see that cross-validation selects a 10-variable model. We now perform
best subset selection on the full data set in order to obtain the 10-variable
model.
> reg.best <- regsubsets (Salary ∼ ., data = Hitters ,
nvmax = 19)
> coef(reg.best , 10)
(Intercept) AtBat Hits Walks CAtBat
162.535 -2.169 6.918 5.773 -0.130
CRuns CRBI CWalks DivisionW PutOuts
1.408 0.774 -0.831 -112.380 0.297
Assists
0.283
6.5.2 Ridge Regression and the Lasso
We will use the glmnet package in order to perform ridge regression and
the lasso. The main function in this package is glmnet(), which can be used
glmnet()
to fit ridge regression models, lasso models, and more. This function has
slightly different syntax from other model-fitting functions that we have
encountered thus far in this book. In particular, we must pass in an x
matrix as well as a y vector, and we do not use the y ∼ x syntax. We will
now perform ridge regression and the lasso in order to predict Salary on
the Hitters data. Before proceeding ensure that the missing values have
been removed from the data, as described in Section 6.5.1.
> x <- model.matrix(Salary ∼ ., Hitters)[, -1]
> y <- Hitters$Salary
The model.matrix() function is particularly useful for creating x; not only
does it produce a matrix corresponding to the 19 predictors but it also
automatically transforms any qualitative variables into dummy variables.
The latter property is important because glmnet() can only take numerical,
quantitative inputs.
Ridge Regression
The glmnet() function has an alpha argument that determines what type
of model is fit. If alpha=0 then a ridge regression model is fit, and if alpha=1
then a lasso model is fit. We first fit a ridge regression model.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-282-320.jpg)
![6.5 Lab: Linear Models and Regularization Methods 275
> library(glmnet)
> grid <- 10^ seq (10, -2, length = 100)
> ridge.mod <- glmnet(x, y, alpha = 0, lambda = grid)
By default the glmnet() function performs ridge regression for an automati-
cally selected range of λ values. However, here we have chosen to implement
the function over a grid of values ranging from λ = 1010
to λ = 10−2
, es-
sentially covering the full range of scenarios from the null model containing
only the intercept, to the least squares fit. As we will see, we can also com-
pute model fits for a particular value of λ that is not one of the original
grid values. Note that by default, the glmnet() function standardizes the
variables so that they are on the same scale. To turn off this default setting,
use the argument standardize = FALSE.
Associated with each value of λ is a vector of ridge regression coefficients,
stored in a matrix that can be accessed by coef(). In this case, it is a 20×100
matrix, with 20 rows (one for each predictor, plus an intercept) and 100
columns (one for each value of λ).
> dim(coef(ridge.mod))
[1] 20 100
We expect the coefficient estimates to be much smaller, in terms of ℓ2 norm,
when a large value of λ is used, as compared to when a small value of λ is
used. These are the coefficients when λ = 11,498, along with their ℓ2 norm:
> ridge.mod$lambda [50]
[1] 11498
> coef(ridge.mod)[, 50]
(Intercept) AtBat Hits HmRun Runs
407.356 0.037 0.138 0.525 0.231
RBI Walks Years CAtBat CHits
0.240 0.290 1.108 0.003 0.012
CHmRun CRuns CRBI CWalks LeagueN
0.088 0.023 0.024 0.025 0.085
DivisionW PutOuts Assists Errors NewLeagueN
-6.215 0.016 0.003 -0.021 0.301
> sqrt(sum(coef(ridge.mod)[-1, 50]^2))
[1] 6.36
In contrast, here are the coefficients when λ = 705, along with their ℓ2
norm. Note the much larger ℓ2 norm of the coefficients associated with this
smaller value of λ.
> ridge.mod$lambda [60]
[1] 705
> coef(ridge.mod)[, 60]
(Intercept) AtBat Hits HmRun Runs
54.325 0.112 0.656 1.180 0.938
RBI Walks Years CAtBat CHits
0.847 1.320 2.596 0.011 0.047
CHmRun CRuns CRBI CWalks LeagueN
0.338 0.094 0.098 0.072 13.684](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-283-320.jpg)
![276 6. Linear Model Selection and Regularization
DivisionW PutOuts Assists Errors NewLeagueN
-54.659 0.119 0.016 -0.704 8.612
> sqrt(sum(coef(ridge.mod)[-1, 60]^2))
[1] 57.1
We can use the predict() function for a number of purposes. For instance,
we can obtain the ridge regression coefficients for a new value of λ, say 50:
> predict(ridge.mod , s = 50, type = " coefficients ")[1:20 , ]
(Intercept) AtBat Hits HmRun Runs
48.766 -0.358 1.969 -1.278 1.146
RBI Walks Years CAtBat CHits
0.804 2.716 -6.218 0.005 0.106
CHmRun CRuns CRBI CWalks LeagueN
0.624 0.221 0.219 -0.150 45.926
DivisionW PutOuts Assists Errors NewLeagueN
-118.201 0.250 0.122 -3.279 -9.497
We now split the samples into a training set and a test set in order
to estimate the test error of ridge regression and the lasso. There are two
common ways to randomly split a data set. The first is to produce a random
vector of TRUE, FALSE elements and select the observations corresponding to
TRUE for the training data. The second is to randomly choose a subset of
numbers between 1 and n; these can then be used as the indices for the
training observations. The two approaches work equally well. We used the
former method in Section 6.5.1. Here we demonstrate the latter approach.
We first set a random seed so that the results obtained will be repro-
ducible.
> set.seed (1)
> train <- sample (1: nrow(x), nrow(x) / 2)
> test <- (-train)
> y.test <- y[test]
Next we fit a ridge regression model on the training set, and evaluate
its MSE on the test set, using λ = 4. Note the use of the predict()
function again. This time we get predictions for a test set, by replacing
type="coefficients" with the newx argument.
> ridge.mod <- glmnet(x[train , ], y[train], alpha = 0,
lambda = grid , thresh = 1e -12)
> ridge.pred <- predict(ridge.mod , s = 4, newx = x[test , ])
> mean (( ridge.pred - y.test)^2)
[1] 142199
The test MSE is 142,199. Note that if we had instead simply fit a model
with just an intercept, we would have predicted each test observation using
the mean of the training observations. In that case, we could compute the
test set MSE like this:
> mean (( mean(y[train ]) - y.test)^2)
[1] 224670](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-284-320.jpg)
![6.5 Lab: Linear Models and Regularization Methods 277
We could also get the same result by fitting a ridge regression model with
a very large value of λ. Note that 1e10 means 1010
.
> ridge.pred <- predict(ridge.mod , s = 1e10 , newx = x[test , ])
> mean (( ridge.pred - y.test)^2)
[1] 224670
So fitting a ridge regression model with λ = 4 leads to a much lower test
MSE than fitting a model with just an intercept. We now check whether
there is any benefit to performing ridge regression with λ = 4 instead of
just performing least squares regression. Recall that least squares is simply
ridge regression with λ = 0.8
> ridge.pred <- predict(ridge.mod , s = 0, newx = x[test , ],
exact = T, x = x[train , ], y = y[train ])
> mean (( ridge.pred - y.test)^2)
[1] 168589
> lm(y ∼ x, subset = train)
> predict(ridge.mod , s = 0, exact = T, type = " coefficients ",
x = x[train , ], y = y[train ]) [1:20 , ]
In general, if we want to fit a (unpenalized) least squares model, then
we should use the lm() function, since that function provides more useful
outputs, such as standard errors and p-values for the coefficients.
In general, instead of arbitrarily choosing λ = 4, it would be better to
use cross-validation to choose the tuning parameter λ. We can do this using
the built-in cross-validation function, cv.glmnet(). By default, the function
cv.glmnet()
performs ten-fold cross-validation, though this can be changed using the
argument nfolds. Note that we set a random seed first so our results will
be reproducible, since the choice of the cross-validation folds is random.
> set.seed (1)
> cv.out <- cv.glmnet(x[train , ], y[train], alpha = 0)
> plot(cv.out)
> bestlam <- cv.out$lambda.min
> bestlam
[1] 326
Therefore, we see that the value of λ that results in the smallest cross-
validation error is 326. What is the test MSE associated with this value of
λ?
> ridge.pred <- predict(ridge.mod , s = bestlam ,
newx = x[test , ])
8In order for glmnet() to yield the exact least squares coefficients when λ = 0,
we use the argument exact = T when calling the predict() function. Otherwise,
the predict() function will interpolate over the grid of λ values used in fitting the
glmnet() model, yielding approximate results. When we use exact = T, there remains
a slight discrepancy in the third decimal place between the output of glmnet() when
λ = 0 and the output of lm(); this is due to numerical approximation on the part of
glmnet().](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-285-320.jpg)
![278 6. Linear Model Selection and Regularization
> mean (( ridge.pred - y.test)^2)
[1] 139857
This represents a further improvement over the test MSE that we got using
λ = 4. Finally, we refit our ridge regression model on the full data set,
using the value of λ chosen by cross-validation, and examine the coefficient
estimates.
> out <- glmnet(x, y, alpha = 0)
> predict(out , type = " coefficients ", s = bestlam)[1:20 , ]
(Intercept) AtBat Hits HmRun Runs
15.44 0.08 0.86 0.60 1.06
RBI Walks Years CAtBat CHits
0.88 1.62 1.35 0.01 0.06
CHmRun CRuns CRBI CWalks LeagueN
0.41 0.11 0.12 0.05 22.09
DivisionW PutOuts Assists Errors NewLeagueN
-79.04 0.17 0.03 -1.36 9.12
As expected, none of the coefficients are zero—ridge regression does not
perform variable selection!
The Lasso
We saw that ridge regression with a wise choice of λ can outperform least
squares as well as the null model on the Hitters data set. We now ask
whether the lasso can yield either a more accurate or a more interpretable
model than ridge regression. In order to fit a lasso model, we once again
use the glmnet() function; however, this time we use the argument alpha=1.
Other than that change, we proceed just as we did in fitting a ridge model.
> lasso.mod <- glmnet(x[train , ], y[train], alpha = 1,
lambda = grid)
> plot(lasso.mod)
We can see from the coefficient plot that depending on the choice of tuning
parameter, some of the coefficients will be exactly equal to zero. We now
perform cross-validation and compute the associated test error.
> set.seed (1)
> cv.out <- cv.glmnet(x[train , ], y[train], alpha = 1)
> plot(cv.out)
> bestlam <- cv.out$lambda.min
> lasso.pred <- predict(lasso.mod , s = bestlam ,
newx = x[test , ])
> mean (( lasso.pred - y.test)^2)
[1] 143674
This is substantially lower than the test set MSE of the null model and of
least squares, and very similar to the test MSE of ridge regression with λ
chosen by cross-validation.
However, the lasso has a substantial advantage over ridge regression in
that the resulting coefficient estimates are sparse. Here we see that 8 of the](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-286-320.jpg)
![6.5 Lab: Linear Models and Regularization Methods 279
19 coefficient estimates are exactly zero. So the lasso model with λ chosen
by cross-validation contains only eleven variables.
> out <- glmnet(x, y, alpha = 1, lambda = grid)
> lasso.coef <- predict(out , type = " coefficients ",
s = bestlam)[1:20 , ]
> lasso.coef
(Intercept) AtBat Hits HmRun Runs
1.27 -0.05 2.18 0.00 0.00
RBI Walks Years CAtBat CHits
0.00 2.29 -0.34 0.00 0.00
CHmRun CRuns CRBI CWalks LeagueN
0.03 0.22 0.42 0.00 20.29
DivisionW PutOuts Assists Errors NewLeagueN
-116.17 0.24 0.00 -0.86 0.00
> lasso.coef[lasso.coef != 0]
(Intercept) AtBat Hits Walks Years
1.27 -0.05 2.18 2.29 -0.34
CHmRun CRuns CRBI LeagueN DivisionW
0.03 0.22 0.42 20.29 -116.17
PutOuts Errors
0.24 -0.86
6.5.3 PCR and PLS Regression
Principal Components Regression
Principal components regression (PCR) can be performed using the pcr()
pcr()
function, which is part of the pls library. We now apply PCR to the Hitters
data, in order to predict Salary. Again, we ensure that the missing values
have been removed from the data, as described in Section 6.5.1.
> library(pls)
> set.seed (2)
> pcr.fit <- pcr(Salary ∼ ., data = Hitters , scale = TRUE ,
validation = "CV")
The syntax for the pcr() function is similar to that for lm(), with a few
additional options. Setting scale = TRUE has the effect of standardizing each
predictor, using (6.6), prior to generating the principal components, so that
the scale on which each variable is measured will not have an effect. Setting
validation = "CV" causes pcr() to compute the ten-fold cross-validation
error for each possible value of M, the number of principal components
used. The resulting fit can be examined using summary().
> summary(pcr.fit)
Data: X dimension: 263 19
Y dimension: 263 1
Fit method: svdpc
Number of components considered: 19
VALIDATION : RMSEP](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-287-320.jpg)
![280 6. Linear Model Selection and Regularization
Cross -validated using 10 random segments.
(Intercept) 1 comps 2 comps 3 comps 4 comps
CV 452 351.9 353.2 355.0 352.8
adjCV 452 351.6 352.7 354.4 352.1
...
TRAINING: % variance explained
1 comps 2 comps 3 comps 4 comps 5 comps
X 38.31 60.16 70.84 79.03 84.29
Salary 40.63 41.58 42.17 43.22 44.90
...
The CV score is provided for each possible number of components, ranging
from M = 0 onwards. (We have printed the CV output only up to M = 4.)
Note that pcr() reports the root mean squared error; in order to obtain
the usual MSE, we must square this quantity. For instance, a root mean
squared error of 352.8 corresponds to an MSE of 352.82
= 124,468.
One can also plot the cross-validation scores using the validationplot()
validationplot()
function. Using val.type = "MSEP" will cause the cross-validation MSE to
be plotted.
> validationplot (pcr.fit , val.type = "MSEP")
We see that the smallest cross-validation error occurs when M = 18 com-
ponents are used. This is barely fewer than M = 19, which amounts to
simply performing least squares, because when all of the components are
used in PCR no dimension reduction occurs. However, from the plot we
also see that the cross-validation error is roughly the same when only one
component is included in the model. This suggests that a model that uses
just a small number of components might suffice.
The summary() function also provides the percentage of variance explained
in the predictors and in the response using different numbers of compo-
nents. This concept is discussed in greater detail in Chapter 12. Briefly,
we can think of this as the amount of information about the predictors or
the response that is captured using M principal components. For example,
setting M = 1 only captures 38.31 % of all the variance, or information, in
the predictors. In contrast, using M = 5 increases the value to 84.29 %. If
we were to use all M = p = 19 components, this would increase to 100 %.
We now perform PCR on the training data and evaluate its test set
performance.
> set.seed (1)
> pcr.fit <- pcr(Salary ∼ ., data = Hitters , subset = train ,
scale = TRUE , validation = "CV")
> validationplot (pcr.fit , val.type = "MSEP")
Now we find that the lowest cross-validation error occurs when M = 5
components are used. We compute the test MSE as follows.
> pcr.pred <- predict(pcr.fit , x[test , ], ncomp = 5)
> mean (( pcr.pred - y.test)^2)](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-288-320.jpg)
![6.5 Lab: Linear Models and Regularization Methods 281
[1] 142812
This test set MSE is competitive with the results obtained using ridge re-
gression and the lasso. However, as a result of the way PCR is implemented,
the final model is more difficult to interpret because it does not perform
any kind of variable selection or even directly produce coefficient estimates.
Finally, we fit PCR on the full data set, using M = 5, the number of
components identified by cross-validation.
> pcr.fit <- pcr(y ∼ x, scale = TRUE , ncomp = 5)
> summary(pcr.fit)
Data: X dimension: 263 19
Y dimension: 263 1
Fit method: svdpc
Number of components considered: 5
TRAINING: % variance explained
1 comps 2 comps 3 comps 4 comps 5 comps
X 38.31 60.16 70.84 79.03 84.29
y 40.63 41.58 42.17 43.22 44.90
Partial Least Squares
We implement partial least squares (PLS) using the plsr() function, also
plsr()
in the pls library. The syntax is just like that of the pcr() function.
> set.seed (1)
> pls.fit <- plsr(Salary ∼ ., data = Hitters , subset = train ,
scale = TRUE , validation = "CV")
> summary(pls.fit)
Data: X dimension: 131 19
Y dimension: 131 1
Fit method: kernelpls
Number of components considered: 19
VALIDATION : RMSEP
Cross -validated using 10 random segments.
(Intercept) 1 comps 2 comps 3 comps 4 comps
CV 428.3 325.5 329.9 328.8 339.0
adjCV 428.3 325.0 328.2 327.2 336.6
...
TRAINING: % variance explained
1 comps 2 comps 3 comps 4 comps 5 comps
X 39.13 48.80 60.09 75.07 78.58
Salary 46.36 50.72 52.23 53.03 54.07
...
> validationplot (pls.fit , val.type = "MSEP")
The lowest cross-validation error occurs when only M = 1 partial least
squares directions are used. We now evaluate the corresponding test set
MSE.
> pls.pred <- predict(pls.fit , x[test , ], ncomp = 1)](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-289-320.jpg)
![282 6. Linear Model Selection and Regularization
> mean (( pls.pred - y.test)^2)
[1] 151995
The test MSE is comparable to, but slightly higher than, the test MSE
obtained using ridge regression, the lasso, and PCR.
Finally, we perform PLS using the full data set, using M = 1, the number
of components identified by cross-validation.
> pls.fit <- plsr(Salary ∼ ., data = Hitters , scale = TRUE ,
ncomp = 1)
> summary(pls.fit)
Data: X dimension: 263 19
Y dimension: 263 1
Fit method: kernelpls
Number of components considered: 1
TRAINING: % variance explained
1 comps
X 38.08
Salary 43.05
Notice that the percentage of variance in Salary that the one-component
PLS fit explains, 43.05 %, is almost as much as that explained using the final
five-component model PCR fit, 44.90 %. This is because PCR only attempts
to maximize the amount of variance explained in the predictors, while PLS
searches for directions that explain variance in both the predictors and the
response.
6.6 Exercises
Conceptual
1. We perform best subset, forward stepwise, and backward stepwise
selection on a single data set. For each approach, we obtain p + 1
models, containing 0, 1, 2, . . . , p predictors. Explain your answers:
(a) Which of the three models with k predictors has the smallest
training RSS?
(b) Which of the three models with k predictors has the smallest
test RSS?
(c) True or False:
i. The predictors in the k-variable model identified by forward
stepwise are a subset of the predictors in the (k+1)-variable
model identified by forward stepwise selection.
ii. The predictors in the k-variable model identified by back-
ward stepwise are a subset of the predictors in the (k + 1)-
variable model identified by backward stepwise selection.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-290-320.jpg)
![7.1 Polynomial Regression 291
20 30 40 50 60 70 80
50
100
150
200
250
300
Age
Wage
Degree−4 Polynomial
20 30 40 50 60 70 80
0.00
0.05
0.10
0.15
0.20
Age
| | |
| | |
|
| | |
|
| | | |
|
| | |
|| | |
|
|
|
|
| |
| |
|
| | | | |
| |
| |
| | |
| |
|
|
| | | |
|
| |
| |
| |
| | | |
| | |
|
| |
| |
|
| | | |
|
| |
| | |
|
| |
| | || |
|
|
| |
| |
| |
|
| |
| |
| |
|
| |
|
|
| |
| | | | |
|
|
|
| |
| |
|
| |
|
| |
| |
| |
|
| |
| |
|
| | |
| |
|
| |
|
|
| |
|
| |
| | | | |
|
| |
| |
|
|
| |
|
|
| |
| |
| | |
|
|
|
| | |
| |
| |
| | |
|
| | |
| |
|
| |
| | |
| |
|
|
|
|
| | |
| | |
| |
| | |
| | |
|
| |
| |
| | |
| |
|
| |
|
| | |
|
|
| |
|
| | |
| | |
|
|
| |
|
| |
| |
| |
|
| | | |
| |
| |
|
|
|
|
| |
| | |
| |
| | |
| |
| | | |
| |
| |
| | |
| |
|
| |
| | |
| |
| |
|
|
|
| |
|
|
|
| |
| |
| |
| |
| | |
|
|
| |
| |
| |
|
| | |
| |
| |
| |
|
| |
|
|
| |
|
| |
| |
| |
| |
|
| | | |
|
|
|
| |
|
| |
| |
| |
| |
|
| |
| |
| | |
| |
| |
| | |
| |
|
| |
| | |
|
| |
| | |
| |
| |
| | |
|
|
| |
| | |
|
|
| | |
|
|
| |
| | | | |
|
|
|
|
| |
| |
| |
| |
| |
| |
|
| |
|
|
| |
| |
| |
|
|
| | |
| |
| |
| |
|
| | |
| |
|
|
| |
| |
|
| | |
|
|| |
| |
| |
| |
|
|
| |
|
|
| |
|
| |
|
|
|
|
| |
| |
|
|
| | |
| |
| | | |
|
|
|
|
|
| |
| | |
| | |
| | |
| |
| |
|
|
|
| | | |
|| |
|
| | |
| |
| |
|
| |
|
| | |
| |
|
|
| | |
| |
| | |
|
|
| |
| | |
| |
| | |
|
| |
| |
| |
| |
| |
|
|
| |
| | |
|
||
| | | |
|
|
| | |
| |
|
| |
| |
|
| |
|
| |
|
| | |
| |
| | | |
| |
|
| | | |
|
| |
|
|
| |
| |
| | | |
| |
| | |
|
|
| | | | |
|
| | |
| | |
|
|
| |
|
| | |
|
|
| ||
|
|
| |
|
| | | |
| |
| |
| |
| |
| |
| |
| |
|
| |
| |
| |
| | |
|
| |
| |
| | |
| |
| |
|
|
| |
|
|
| |
|
| |
| |
| |
|
| |
| | | |
|
| |
| |
| | |
|
| |
|
| |
| |
| |
|
|
|
| |
| |
| |
| |
| |
| | | |
|
| | |
| |
|
| |
| |
| | |
|
|
| |
| | |
|
|
| |
| |
|
|
|
| |
| | |
| |
| |
|
| | |
|
| | |
| |
|
| |
|
|
|
| |
| |
|
|
|
| |
|
| | |
|
| |
|
| | |
| |
| |
| |
|
| |
| |
| | | |
| |
|
| |
| |
|
|
| |
|
| | |
|
|
|
| | | | |
| | |
|
| | | |
| | |
|
|
| |
| |
|
| |
| | |
|
| |
| | |
|
| |
|
|
|
| |
| |
|
|
| | |
|
| | |
| | | |
|
| |
|
|
| |
| |
|
| |
| |
| |
| | | |
| | |
| | |
| |
| |
| |
| | | |
| |
| | |
|
|
| |
|
| |
|
|
|
| |
|
| |
|
| |
| |
|
|
|
| |
| |
| | |
|
| |
| | | |
| | |
|
| | | |
|
| |
| |
| |
| |
| | |
|
| |
|
| |
| |
| |
|
|
|
| |
| |
| |
|
| | | ||
| | |
|
|| | |
| |
| |
|
|
|
|
| | |
| | |
| | |
| | |
|
|
|
| |
| |
| |
|
| | | |
|
| |
|
| | |
| |
|
| |
| |
| |
|
| | |
|
|
| | |
|
| | |
|
| | |
| | |
| | |
| | | |
| |
| |
| |
| |
| |
|
|
| |
| | | |
| | | | |
|
|
| |
| | |
|
| | |
| |
|
| |
| |
| |
|
| |
|
|
|
|
| |
|
| | |
| |
| |
| | | |
| | |
| |
| |
| |
| |
| |
| | |
| |
| | |
| |
| |
|
| |
| ||
| |
| |
| |
| |
|
| | |
| |
| |
|
| |
|
|
| |
|
| | |
|
|
| |
|
|
| |
|
| |
|
| |
|
| |
|
|
| |
|
| |
| |
| |
| |
| |
|
| | | | |
| |
| | |
|
| |
| | |
| | |
| | |
|
|
| | |
|
| |
|
| |
|
|
| | |
|
|
|
| |
|
| | | |
| |
| | | |
|
|
| | |
|
|
| |
|
| |
|
| | |
| | |
| |
| |
| | |
| | |
| |
| |
|
| | |
| |
|
| | |
| |
| |
|
| | | | |
|
|
| | |
| | | |
|
| |
|
| |
|
|
| | | |
|
| | |
|
|
| | |
| | |
| |
| | |
|
|
| |
| |
|
|
|
| |
| |
|
| |
|
| |
| | | |
|
|
|
| |
| |
|
|
| |
| | | |
|
| | |
| |
| |
| |
|
| |
|
|
| |
|
|
| | | |
| |
| |
| | |
|
| | |
| |
| |
| | | |
| |
| |
| |
|
|
| |
| |
|
| |
| |
|
| |
| |
|
| |
| |
| |
|
|
| |
| | |
|
| | |
|
| |
| | |
| |
|
|
| |
|
|
| |
|
| | | | |
| |
|
| | |
| | | |
|
|
|
| |
|
|
| |
|
| |
|
| |
| | | |
|
|
| |
|
| |
|
|
| |
| |
|
|
| |
| |
| |
|
| |
|
|
| | |
|
| | |
|
|
| | |
| |
| |
| | |
|
| | |
| |
|
| |
| |
|
| | |
| |
| |
|
| | |
| |
| |
| | |
| |
|
| | |
| |
| |
|
|
| |
| |
| | |
| |
|
| |
|
|
|
| |
| | |
| |
| |
| | |
| | |
|
| | |
|
| |
| | |
| |
| |
|
| |
| |
|
| |
|
| |
| | |
| |
| | | |
| |
| |
| |
|
| |
| |
| |
| | |
|
| |
| | |
|
|
|
|
| | |
|
|
|
| | | | |
| |
|
| |
| | |
|
|
|
| | | |
| |
| |
| | | |
| |
| |
| | | |
| |
|
| | |
|
|
| |
|
|
|
|
|| | | | |
| |
| | | |
|
| |
| | | |
| | |
| | |
|
| |
| |
|
|
| | |
|
| | |
| |
| |
|
|
|
| | |
| |
|
| |
|
| | |
| |
| |
| |
| |
| |
|
| | |
|
|
|
| |
| |
| |
|
| |
|
| |
| | | |
| |
| |
| |
|
|
| |
| | | |
| | |
| |
| | |
| |
| |
| | |
|
|
| | | | |
|
| | | |
|
|
| | |
| |
|
|
|
| | | |
|
| |
| |
|
| |
| |
| | |
| |
| |
| |
| | | |
| |
|
| |
| |
|
| |
| |
| |
| |
|
| | |
|
| |
| |
|
| |
| |
|
| | |
|
| | | | |
| |
| | |
|
| |
|
| |
|| |
|
|
|
|
|
| |
|
|
| | |
|
|
| |
|
| | |
| |
| |
|
| |
| |
|
| | |
|
|
| |
| |
|
|
| |
|
| |
|
| | |
|
| | |
| | | | |
|
| |
| | |
| |
| |
|
| | | | |
|
| |
| |
| |
|
| |
| | |
|
| | |
| |
|
|
| | |
|
|
| | |
|
| |
| |
| |
| |
| |
| | |
|
| |
| |
| | | |
|
| |
| |
| |
|
|
| |
|
| |
| | |
| |
|
| |
| | |
|
| |
|
| |
| | |
|
|
|
| | |
| |
| | |
|
| |
| |
|
| | |
|
| |
| |
| |
| | | |
| | |
|
| | | |
|
| |
| | | |
| | | |
| |
|
|
|
| |
| |
| |
||
| | |
|
| | | |
| |
| | | |
| | | |
|
|
| |
| |
|
|
| | |
| |
|
|
| |
|
|
| | | |
|
|
|
| |
|
|
|
|
| |
|
|
| | |
|
|
| |
| |
|
| | |
| |
|
|
|
| |
|
|
|
| | |
| |
| |
| | |
| |
|
| |
|
| | |
| |
|
| |
| |
|
| |
| | |
| |
| |
|
| |
| | | | |
| |
| |
|
| | |
| |
| | |
| |
| |
|
| |
| |
|
|
|
| |
| |
|
| |
| | | | |
| |
|
|
| |
| |
|
|
| |
|
| |
| |
|
|
| |
| |
| | |
|
| | |
|
|
|
|
| |
|
| | |
| |
|
| |
| | |
|
| |
| |
|
| | | |
|
|
| | |
|
| | |
|
|
| |
| |
|
| |
| |
| | |
|
| | |
|
| | |
|
|
| |
| | |
| |
|
|
| | |
|
|
|
| |
|
| | | |
|
| | |
|
| |
| |
|
| |
| | |
|
| |
| |
|
| | |
|
| |
| | |
|
| | | |
| |
| |
| |
| | |
|
| |
| |
| |
| |
|
|
| |
|
| | |
|
| |
| |
| |
|
|
| |
| |
|
|
|
| |
|
|
|
| |
|
|
| |
|
|
|
| | | |
| |
| |
|
|
|
|
|
|
| |
|
|
|
| |
| |
| |
| |
| |
| | | |
|
|
|
| | |
|
| |
| | |
|
| |
|
| |
| |
|
| |
| | | |
| |
| | |
| | |
| |
|
| |
|
|
| |
| |
| |
|
| |
|
|| |
|
| |
|
| |
|
| | | |
| |
|
| |
|
| |
| || |
| |
|
| |
|
| | |
|
| | |
| |
| |
|
| |
| |
| |
|
| |
|
|
| |
|
| | |
| | |
| |
| |
|
|
| |
| |
|
| |
| | | |
|
| |
| | |
| |
|
| |
| |
| | |
| |
|
|
| | |
|
| |
| | | | |
|
|
| | |
| | | |
|
| | |
| | |
| | |
|
| |
| |
| |
|
| | | |
| |
|
|
| |
|
| |
| |
| | |
| |
| |
|
|
| |
| |
| |
| | |
| |
| | |
| |
|
| | |
| |
|
| | |
| |
|
| |
| |
| | | |
| |
| |
|
| |
| |
| |
| |
|| |
|
|| |
Pr(Wage>250
|
Age)
FIGURE 7.1. The Wage data. Left: The solid blue curve is a degree-4 polynomial
of wage (in thousands of dollars) as a function of age, fit by least squares. The
dashed curves indicate an estimated 95 % confidence interval. Right: We model the
binary event wage>250 using logistic regression, again with a degree-4 polynomial.
The fitted posterior probability of wage exceeding $250,000 is shown in blue, along
with an estimated 95 % confidence interval.
The left-hand panel in Figure 7.1 is a plot of wage against age for the
Wage data set, which contains income and demographic information for
males who reside in the central Atlantic region of the United States. We
see the results of fitting a degree-4 polynomial using least squares (solid
blue curve). Even though this is a linear regression model like any other,
the individual coefficients are not of particular interest. Instead, we look at
the entire fitted function across a grid of 63 values for age from 18 to 80 in
order to understand the relationship between age and wage.
In Figure 7.1, a pair of dashed curves accompanies the fit; these are (2×)
standard error curves. Let’s see how these arise. Suppose we have computed
the fit at a particular value of age, x0:
ˆ
f(x0) = β̂0 + β̂1x0 + β̂2x2
0 + β̂3x3
0 + β̂4x4
0. (7.2)
What is the variance of the fit, i.e. Var ˆ
f(x0)? Least squares returns variance
estimates for each of the fitted coefficients β̂j, as well as the covariances
between pairs of coefficient estimates. We can use these to compute the
estimated variance of ˆ
f(x0).1
The estimated pointwise standard error of
1If Ĉ is the 5 × 5 covariance matrix of the β̂j, and if ℓT
0 = (1, x0, x2
0, x3
0, x4
0), then
Var[ ˆ
f(x0)] = ℓT
0 Ĉℓ0.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-299-320.jpg)
![7.8 Lab: Non-linear Modeling 313
There are several other equivalent ways of fitting this model, which show-
case the flexibility of the formula language in R. For example
> fit2a <- lm(wage ∼ age + I(age ^2) + I(age ^3) + I(age ^4) ,
data = Wage)
> coef(fit2a)
(Intercept) age I(age ^2) I(age ^3) I(age ^4)
-1.84e+02 2.12e+01 -5.64e-01 6.81e-03 -3.20e -05
This simply creates the polynomial basis functions on the fly, taking care
to protect terms like age^2 via the wrapper function I() (the ^ symbol has wrapper
a special meaning in formulas).
> fit2b <- lm(wage ∼ cbind(age , age^2, age^3, age ^4) ,
data = Wage)
This does the same more compactly, using the cbind() function for building
a matrix from a collection of vectors; any function call such as cbind() inside
a formula also serves as a wrapper.
We now create a grid of values for age at which we want predictions, and
then call the generic predict() function, specifying that we want standard
errors as well.
> agelims <- range(age)
> age.grid <- seq(from = agelims [1], to = agelims [2])
> preds <- predict(fit , newdata = list(age = age.grid),
se = TRUE)
> se.bands <- cbind(preds$fit + 2 * preds$se.fit ,
preds$fit - 2 * preds$se.fit)
Finally, we plot the data and add the fit from the degree-4 polynomial.
> par(mfrow = c(1, 2), mar = c(4.5 , 4.5, 1, 1),
oma = c(0, 0, 4, 0))
> plot(age , wage , xlim = agelims , cex = .5, col = "darkgrey")
> title("Degree -4 Polynomial ", outer = T)
> lines(age.grid , preds$fit , lwd = 2, col = "blue")
> matlines (age.grid , se.bands , lwd = 1, col = "blue", lty = 3)
Here the mar and oma arguments to par() allow us to control the margins
of the plot, and the title() function creates a figure title that spans both
title()
subplots.
We mentioned earlier that whether or not an orthogonal set of basis func-
tions is produced in the poly() function will not affect the model obtained
in a meaningful way. What do we mean by this? The fitted values obtained
in either case are identical:
> preds2 <- predict(fit2 , newdata = list(age = age.grid),
se = TRUE)
> max(abs(preds$fit - preds2$fit))
[1] 7.82e-11
In performing a polynomial regression we must decide on the degree of
the polynomial to use. One way to do this is by using hypothesis tests. We](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-321-320.jpg)
![7.8 Lab: Non-linear Modeling 315
poly(age , 5)2 -478.32 39.9161 -11.9830 2.368e -32
poly(age , 5)3 125.52 39.9161 3.1446 1.679e -03
poly(age , 5)4 -77.91 39.9161 -1.9519 5.105e-02
poly(age , 5)5 -35.81 39.9161 -0.8972 3.697e-01
Notice that the p-values are the same, and in fact the square of the
t-statistics are equal to the F-statistics from the anova() function; for
example:
> ( -11.983) ^2
[1] 143.6
However, the ANOVA method works whether or not we used orthogonal
polynomials; it also works when we have other terms in the model as well.
For example, we can use anova() to compare these three models:
> fit.1 <- lm(wage ∼ education + age , data = Wage)
> fit.2 <- lm(wage ∼ education + poly(age , 2), data = Wage)
> fit.3 <- lm(wage ∼ education + poly(age , 3), data = Wage)
> anova(fit.1, fit.2, fit .3)
As an alternative to using hypothesis tests and ANOVA, we could choose
the polynomial degree using cross-validation, as discussed in Chapter 5.
Next we consider the task of predicting whether an individual earns more
than $250,000 per year. We proceed much as before, except that first we
create the appropriate response vector, and then apply the glm() function
using family = "binomial" in order to fit a polynomial logistic regression
model.
> fit <- glm(I(wage > 250) ∼ poly(age , 4), data = Wage ,
family = binomial)
Note that we again use the wrapper I() to create this binary response
variable on the fly. The expression wage > 250 evaluates to a logical variable
containing TRUEs and FALSEs, which glm() coerces to binary by setting the
TRUEs to 1 and the FALSEs to 0.
Once again, we make predictions using the predict() function.
> preds <- predict(fit , newdata = list(age = age.grid), se = T)
However, calculating the confidence intervals is slightly more involved than
in the linear regression case. The default prediction type for a glm() model
is type = "link", which is what we use here. This means we get predictions
for the logit, or log-odds: that is, we have fit a model of the form
log
*
Pr(Y = 1|X)
1 − Pr(Y = 1|X)
+
= Xβ,
and the predictions given are of the form Xβ̂. The standard errors given
are also for Xβ̂. In order to obtain confidence intervals for Pr(Y = 1|X),
we use the transformation
Pr(Y = 1|X) =
exp(Xβ)
1 + exp(Xβ)
.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-323-320.jpg)
![316 7. Moving Beyond Linearity
> pfit <- exp(preds$fit) / (1 + exp(preds$fit))
> se.bands.logit <- cbind(preds$fit + 2 * preds$se.fit ,
preds$fit - 2 * preds$se.fit)
> se.bands <- exp(se.bands.logit) / (1 + exp(se.bands.logit))
Note that we could have directly computed the probabilities by selecting
the type = "response" option in the predict() function.
> preds <- predict(fit , newdata = list(age = age.grid),
type = "response", se = T)
However, the corresponding confidence intervals would not have been sen-
sible because we would end up with negative probabilities!
Finally, the right-hand plot from Figure 7.1 was made as follows:
> plot(age , I(wage > 250) , xlim = agelims , type = "n",
ylim = c(0, .2))
> points(jitter(age), I(( wage > 250) / 5), cex = .5, pch = "|",
col = "darkgrey")
> lines(age.grid , pfit , lwd = 2, col = "blue")
> matlines (age.grid , se.bands , lwd = 1, col = "blue", lty = 3)
We have drawn the age values corresponding to the observations with wage
values above 250 as gray marks on the top of the plot, and those with wage
values below 250 are shown as gray marks on the bottom of the plot. We
used the jitter() function to jitter the age values a bit so that observations
jitter()
with the same age value do not cover each other up. This is often called a
rug plot.
rug plot
In order to fit a step function, as discussed in Section 7.2, we use the
cut() function.
cut()
> table(cut(age , 4))
(17.9 ,33.5] (33.5 ,49] (49 ,64.5] (64.5 ,80.1]
750 1399 779 72
> fit <- lm(wage ∼ cut(age , 4), data = Wage)
> coef(summary(fit))
Estimate Std. Error t value Pr(>|t|)
(Intercept) 94.16 1.48 63.79 0.00e+00
cut(age , 4) (33.5 ,49] 24.05 1.83 13.15 1.98e-38
cut(age , 4) (49 ,64.5] 23.66 2.07 11.44 1.04e-29
cut(age , 4) (64.5 ,80.1] 7.64 4.99 1.53 1.26e -01
Here cut() automatically picked the cutpoints at 33.5, 49, and 64.5 years
of age. We could also have specified our own cutpoints directly using the
breaks option. The function cut() returns an ordered categorical variable;
the lm() function then creates a set of dummy variables for use in the re-
gression. The age < 33.5 category is left out, so the intercept coefficient of
$94,160 can be interpreted as the average salary for those under 33.5 years
of age, and the other coefficients can be interpreted as the average addi-
tional salary for those in the other age groups. We can produce predictions
and plots just as we did in the case of the polynomial fit.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-324-320.jpg)
![7.8 Lab: Non-linear Modeling 317
7.8.2 Splines
In order to fit regression splines in R, we use the splines library. In Section
7.4, we saw that regression splines can be fit by constructing an appropriate
matrix of basis functions. The bs() function generates the entire matrix of
bs()
basis functions for splines with the specified set of knots. By default, cubic
splines are produced. Fitting wage to age using a regression spline is simple:
> library(splines)
> fit <- lm(wage ∼ bs(age , knots = c(25, 40, 60)), data = Wage)
> pred <- predict(fit , newdata = list(age = age.grid), se = T)
> plot(age , wage , col = "gray")
> lines(age.grid , pred$fit , lwd = 2)
> lines(age.grid , pred$fit + 2 * pred$se, lty = "dashed")
> lines(age.grid , pred$fit - 2 * pred$se, lty = "dashed")
Here we have prespecified knots at ages 25, 40, and 60. This produces a
spline with six basis functions. (Recall that a cubic spline with three knots
has seven degrees of freedom; these degrees of freedom are used up by an
intercept, plus six basis functions.) We could also use the df option to
produce a spline with knots at uniform quantiles of the data.
> dim(bs(age , knots = c(25, 40, 60)))
[1] 3000 6
> dim(bs(age , df = 6))
[1] 3000 6
> attr(bs(age , df = 6), "knots")
25% 50% 75%
33.8 42.0 51.0
In this case R chooses knots at ages 33.8, 42.0, and 51.0, which correspond
to the 25th, 50th, and 75th percentiles of age. The function bs() also has
a degree argument, so we can fit splines of any degree, rather than the
default degree of 3 (which yields a cubic spline).
In order to instead fit a natural spline, we use the ns() function. Here
ns()
we fit a natural spline with four degrees of freedom.
> fit2 <- lm(wage ∼ ns(age , df = 4), data = Wage)
> pred2 <- predict(fit2 , newdata = list(age = age.grid),
se = T)
> lines(age.grid , pred2$fit , col = "red", lwd = 2)
As with the bs() function, we could instead specify the knots directly using
the knots option.
In order to fit a smoothing spline, we use the smooth.spline() function. smooth.
spline()
Figure 7.8 was produced with the following code:
> plot(age , wage , xlim = agelims , cex = .5, col = "darkgrey")
> title("Smoothing Spline")
> fit <- smooth.spline(age , wage , df = 16)
> fit2 <- smooth.spline(age , wage , cv = TRUE)
> fit2$df](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-325-320.jpg)
![318 7. Moving Beyond Linearity
[1] 6.8
> lines(fit , col = "red", lwd = 2)
> lines(fit2 , col = "blue", lwd = 2)
> legend("topright", legend = c("16 DF", "6.8 DF"),
col = c("red", "blue"), lty = 1, lwd = 2, cex = .8)
Notice that in the first call to smooth.spline(), we specified df = 16. The
function then determines which value of λ leads to 16 degrees of freedom. In
the second call to smooth.spline(), we select the smoothness level by cross-
validation; this results in a value of λ that yields 6.8 degrees of freedom.
In order to perform local regression, we use the loess() function.
loess()
> plot(age , wage , xlim = agelims , cex = .5, col = "darkgrey")
> title("Local Regression ")
> fit <- loess(wage ∼ age , span = .2, data = Wage)
> fit2 <- loess(wage ∼ age , span = .5, data = Wage)
> lines(age.grid , predict(fit , data.frame(age = age.grid)),
col = "red", lwd = 2)
> lines(age.grid , predict(fit2 , data.frame(age = age.grid)),
col = "blue", lwd = 2)
> legend("topright", legend = c("Span = 0.2", "Span = 0.5"),
col = c("red", "blue"), lty = 1, lwd = 2, cex = .8)
Here we have performed local linear regression using spans of 0.2 and 0.5:
that is, each neighborhood consists of 20 % or 50 % of the observations. The
larger the span, the smoother the fit. The locfit library can also be used
for fitting local regression models in R.
7.8.3 GAMs
We now fit a GAM to predict wage using natural spline functions of year
and age, treating education as a qualitative predictor, as in (7.16). Since
this is just a big linear regression model using an appropriate choice of
basis functions, we can simply do this using the lm() function.
> gam1 <- lm(wage ∼ ns(year , 4) + ns(age , 5) + education ,
data = Wage)
We now fit the model (7.16) using smoothing splines rather than natural
splines. In order to fit more general sorts of GAMs, using smoothing splines
or other components that cannot be expressed in terms of basis functions
and then fit using least squares regression, we will need to use the gam
library in R.
The s() function, which is part of the gam library, is used to indicate that
s()
we would like to use a smoothing spline. We specify that the function of
year should have 4 degrees of freedom, and that the function of age will
have 5 degrees of freedom. Since education is qualitative, we leave it as is,
and it is converted into four dummy variables. We use the gam() function in
gam()
order to fit a GAM using these components. All of the terms in (7.16) are
fit simultaneously, taking each other into account to explain the response.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-326-320.jpg)
![7.9 Exercises 325
(b) Fit a GAM on the training data, using out-of-state tuition as
the response and the features selected in the previous step as
the predictors. Plot the results, and explain your findings.
(c) Evaluate the model obtained on the test set, and explain the
results obtained.
(d) For which variables, if any, is there evidence of a non-linear
relationship with the response?
11. In Section 7.7, it was mentioned that GAMs are generally fit using
a backfitting approach. The idea behind backfitting is actually quite
simple. We will now explore backfitting in the context of multiple
linear regression.
Suppose that we would like to perform multiple linear regression, but
we do not have software to do so. Instead, we only have software
to perform simple linear regression. Therefore, we take the following
iterative approach: we repeatedly hold all but one coefficient esti-
mate fixed at its current value, and update only that coefficient
estimate using a simple linear regression. The process is continued un-
til convergence—that is, until the coefficient estimates stop changing.
We now try this out on a toy example.
(a) Generate a response Y and two predictors X1 and X2, with
n = 100.
(b) Initialize β̂1 to take on a value of your choice. It does not matter
what value you choose.
(c) Keeping β̂1 fixed, fit the model
Y − β̂1X1 = β0 + β2X2 + ϵ.
You can do this as follows:
> a <- y - beta1 * x1
> beta2 <- lm(a ∼ x2)$coef [2]
(d) Keeping β̂2 fixed, fit the model
Y − β̂2X2 = β0 + β1X1 + ϵ.
You can do this as follows:
> a <- y - beta2 * x2
> beta1 <- lm(a ∼ x1)$coef [2]
(e) Write a for loop to repeat (c) and (d) 1,000 times. Report the
estimates of β̂0, β̂1, and β̂2 at each iteration of the for loop.
Create a plot in which each of these values is displayed, with β̂0,
β̂1, and β̂2 each shown in a different color.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-333-320.jpg)
![8.3 Lab: Decision Trees 353
8.3 Lab: Decision Trees
8.3.1 Fitting Classification Trees
The tree library is used to construct classification and regression trees.
> library(tree)
We first use classification trees to analyze the Carseats data set. In these
data, Sales is a continuous variable, and so we begin by recoding it as a
binary variable. We use the ifelse() function to create a variable, called
ifelse()
High, which takes on a value of Yes if the Sales variable exceeds 8, and
takes on a value of No otherwise.
> library(ISLR2)
> attach(Carseats)
> High <- factor(ifelse(Sales <= 8, "No", "Yes"))
Finally, we use the data.frame() function to merge High with the rest of
the Carseats data.
> Carseats <- data.frame(Carseats , High)
We now use the tree() function to fit a classification tree in order to predict
tree()
High using all variables but Sales. The syntax of the tree() function is quite
similar to that of the lm() function.
> tree.carseats <- tree(High ∼ . - Sales , Carseats)
The summary() function lists the variables that are used as internal nodes
in the tree, the number of terminal nodes, and the (training) error rate.
> summary(tree.carseats)
Classification tree:
tree(formula = High ∼ . - Sales , data = Carseats)
Variables actually used in tree construction :
[1] "ShelveLoc" "Price" "Income" "CompPrice"
[5] "Population" " Advertising " "Age" "US"
Number of terminal nodes: 27
Residual mean deviance: 0.4575 = 170.7 / 373
Misclassification error rate: 0.09 = 36 / 400
We see that the training error rate is 9 %. For classification trees, the de-
viance reported in the output of summary() is given by
−2
0
m
0
k
nmk log p̂mk,
where nmk is the number of observations in the mth terminal node that
belong to the kth class. This is closely related to the entropy, defined in
(8.7). A small deviance indicates a tree that provides a good fit to the
(training) data. The residual mean deviance reported is simply the deviance
divided by n − |T0|, which in this case is 400 − 27 = 373.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-361-320.jpg)
![354 8. Tree-Based Methods
One of the most attractive properties of trees is that they can be
graphically displayed. We use the plot() function to display the tree struc-
ture, and the text() function to display the node labels. The argument
pretty = 0 instructs R to include the category names for any qualitative
predictors, rather than simply displaying a letter for each category.
> plot(tree.carseats)
> text(tree.carseats , pretty = 0)
The most important indicator of Sales appears to be shelving location,
since the first branch differentiates Good locations from Bad and Medium
locations.
If we just type the name of the tree object, R prints output corresponding
to each branch of the tree. R displays the split criterion (e.g. Price < 92.5),
the number of observations in that branch, the deviance, the overall pre-
diction for the branch (Yes or No), and the fraction of observations in that
branch that take on values of Yes and No. Branches that lead to terminal
nodes are indicated using asterisks.
> tree.carseats
node), split , n, deviance , yval , (yprob)
* denotes terminal node
1) root 400 541.5 No ( 0.590 0.410 )
2) ShelveLoc: Bad ,Medium 315 390.6 No ( 0.689 0.311 )
4) Price < 92.5 46 56.53 Yes ( 0.304 0.696 )
8) Income < 57 10 12.22 No ( 0.700 0.300 )
In order to properly evaluate the performance of a classification tree on
these data, we must estimate the test error rather than simply computing
the training error. We split the observations into a training set and a test
set, build the tree using the training set, and evaluate its performance on the
test data. The predict() function can be used for this purpose. In the case
of a classification tree, the argument type = "class" instructs R to return
the actual class prediction. This approach leads to correct predictions for
around 77 % of the locations in the test data set.
> set.seed (2)
> train <- sample (1: nrow(Carseats), 200)
> Carseats.test <- Carseats[-train , ]
> High.test <- High[-train]
> tree.carseats <- tree(High ∼ . - Sales , Carseats ,
subset = train)
> tree.pred <- predict(tree.carseats , Carseats.test ,
type = "class")
> table(tree.pred , High.test)
High.test
tree.pred No Yes
No 104 33
Yes 13 50
> (104 + 50) / 200
[1] 0.77](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-362-320.jpg)
![8.3 Lab: Decision Trees 355
(If you re-run the predict() function then you might get slightly different
results, due to “ties”: for instance, this can happen when the training ob-
servations corresponding to a terminal node are evenly split between Yes
and No response values.)
Next, we consider whether pruning the tree might lead to improved
results. The function cv.tree() performs cross-validation in order to
cv.tree()
determine the optimal level of tree complexity; cost complexity pruning
is used in order to select a sequence of trees for consideration. We use
the argument FUN = prune.misclass in order to indicate that we want the
classification error rate to guide the cross-validation and pruning process,
rather than the default for the cv.tree() function, which is deviance. The
cv.tree() function reports the number of terminal nodes of each tree con-
sidered (size) as well as the corresponding error rate and the value of the
cost-complexity parameter used (k, which corresponds to α in (8.4)).
> set.seed (7)
> cv.carseats <- cv.tree(tree.carseats , FUN = prune.misclass)
> names(cv.carseats)
[1] "size" "dev" "k" "method"
> cv.carseats
$size
[1] 21 19 14 9 8 5 3 2 1
$dev
[1] 75 75 75 74 82 83 83 85 82
$k
[1] -Inf 0.0 1.0 1.4 2.0 3.0 4.0 9.0 18.0
$method
[1] "misclass"
attr(," class ")
[1] "prune" "tree.sequence"
Despite its name, dev corresponds to the number of cross-validation errors.
The tree with 9 terminal nodes results in only 74 cross-validation errors.
We plot the error rate as a function of both size and k.
> par(mfrow = c(1, 2))
> plot(cv.carseats$size , cv.carseats$dev , type = "b")
> plot(cv.carseats$k, cv.carseats$dev , type = "b")
We now apply the prune.misclass() function in order to prune the tree to
obtain the nine-node tree.
> prune.carseats <- prune.misclass(tree.carseats , best = 9)
> plot(prune.carseats)
> text(prune.carseats , pretty = 0)
How well does this pruned tree perform on the test data set? Once again,
we apply the predict() function.
> tree.pred <- predict(prune.carseats , Carseats.test ,
type = "class")
> table(tree.pred , High.test)
High.test
prune.misclass()](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-363-320.jpg)
![356 8. Tree-Based Methods
tree.pred No Yes
No 97 25
Yes 20 58
> (97 + 58) / 200
[1] 0.775
Now 77.5 % of the test observations are correctly classified, so not only has
the pruning process produced a more interpretable tree, but it has also
slightly improved the classification accuracy.
If we increase the value of best, we obtain a larger pruned tree with lower
classification accuracy:
> prune.carseats <- prune.misclass(tree.carseats , best = 14)
> plot(prune.carseats)
> text(prune.carseats , pretty = 0)
> tree.pred <- predict(prune.carseats , Carseats.test ,
type = "class")
> table(tree.pred , High.test)
High.test
tree.pred No Yes
No 102 31
Yes 15 52
> (102 + 52) / 200
[1] 0.77
8.3.2 Fitting Regression Trees
Here we fit a regression tree to the Boston data set. First, we create a
training set, and fit the tree to the training data.
> set.seed (1)
> train <- sample (1: nrow(Boston), nrow(Boston) / 2)
> tree.boston <- tree(medv ∼ ., Boston , subset = train)
> summary(tree.boston)
Regression tree:
tree(formula = medv ∼ ., data = Boston , subset = train)
Variables actually used in tree construction :
[1] "rm" "lstat" "crim" "age"
Number of terminal nodes: 7
Residual mean deviance: 10.4 = 2550 / 246
Distribution of residuals:
Min. 1st Qu. Median Mean 3rd Qu. Max.
-10.200 -1.780 -0.177 0.000 1.920 16.600
Notice that the output of summary() indicates that only four of the variables
have been used in constructing the tree. In the context of a regression tree,
the deviance is simply the sum of squared errors for the tree. We now plot
the tree.
> plot(tree.boston)
> text(tree.boston , pretty = 0)](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-364-320.jpg)
![8.3 Lab: Decision Trees 357
The variable lstat measures the percentage of individuals with lower
socioeconomic status, while the variable rm corresponds to the average num-
ber of rooms. The tree indicates that larger values of rm, or lower values of
lstat, correspond to more expensive houses. For example, the tree predicts
a median house price of $45,400 for homes in census tracts in which rm >=
7.553.
It is worth noting that we could have fit a much bigger tree, by pass-
ing control = tree.control(nobs = length(train), mindev = 0) into the
tree() function.
Now we use the cv.tree() function to see whether pruning the tree will
improve performance.
> cv.boston <- cv.tree(tree.boston)
> plot(cv.boston$size , cv.boston$dev , type = "b")
In this case, the most complex tree under consideration is selected by cross-
validation. However, if we wish to prune the tree, we could do so as follows,
using the prune.tree() function:
prune.tree()
> prune.boston <- prune.tree(tree.boston , best = 5)
> plot(prune.boston)
> text(prune.boston , pretty = 0)
In keeping with the cross-validation results, we use the unpruned tree to
make predictions on the test set.
> yhat <- predict(tree.boston , newdata = Boston[-train , ])
> boston.test <- Boston[-train , "medv"]
> plot(yhat , boston.test)
> abline (0, 1)
> mean (( yhat - boston.test)^2)
[1] 35.29
In other words, the test set MSE associated with the regression tree is 35.29.
The square root of the MSE is therefore around 5.941, indicating that this
model leads to test predictions that are (on average) within approximately
$5,941 of the true median home value for the census tract.
8.3.3 Bagging and Random Forests
Here we apply bagging and random forests to the Boston data, using the
randomForest package in R. The exact results obtained in this section may
depend on the version of R and the version of the randomForest package
installed on your computer. Recall that bagging is simply a special case of
a random forest with m = p. Therefore, the randomForest() function can
randomForest()
be used to perform both random forests and bagging. We perform bagging
as follows:
> library( randomForest )
> set.seed (1)
> bag.boston <- randomForest (medv ∼ ., data = Boston ,](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-365-320.jpg)
![358 8. Tree-Based Methods
subset = train , mtry = 12, importance = TRUE)
> bag.boston
Call:
randomForest (formula = medv ∼ ., data = Boston , mtry = 12,
importance = TRUE , subset = train)
Type of random forest: regression
Number of trees: 500
No. of variables tried at each split: 12
Mean of squared residuals: 11.40
% Var explained: 85.17
The argument mtry = 12 indicates that all 12 predictors should be consid-
ered for each split of the tree—in other words, that bagging should be done.
How well does this bagged model perform on the test set?
> yhat.bag <- predict(bag.boston , newdata = Boston[-train , ])
> plot(yhat.bag , boston.test)
> abline (0, 1)
> mean (( yhat.bag - boston.test)^2)
[1] 23.42
The test set MSE associated with the bagged regression tree is 23.42, about
two-thirds of that obtained using an optimally-pruned single tree. We could
change the number of trees grown by randomForest() using the ntree argu-
ment:
> bag.boston <- randomForest (medv ∼ ., data = Boston ,
subset = train , mtry = 12, ntree = 25)
> yhat.bag <- predict(bag.boston , newdata = Boston[-train , ])
> mean (( yhat.bag - boston.test)^2)
[1] 25.75
Growing a random forest proceeds in exactly the same way, except that
we use a smaller value of the mtry argument. By default, randomForest()
uses p/3 variables when building a random forest of regression trees, and
√
p variables when building a random forest of classification trees. Here we
use mtry = 6.
> set.seed (1)
> rf.boston <- randomForest (medv ∼ ., data = Boston ,
subset = train , mtry = 6, importance = TRUE)
> yhat.rf <- predict(rf.boston , newdata = Boston[-train , ])
> mean (( yhat.rf - boston.test)^2)
[1] 20.07
The test set MSE is 20.07; this indicates that random forests yielded an
improvement over bagging in this case.
Using the importance() function, we can view the importance of each
importance()
variable.
> importance (rf.boston)
%IncMSE IncNodePurity
crim 19.436 1070.42](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-366-320.jpg)
![8.3 Lab: Decision Trees 359
zn 3.092 82.19
indus 6.141 590.10
chas 1.370 36.70
nox 13.263 859.97
rm 35.095 8270.34
age 15.145 634.31
dis 9.164 684.88
rad 4.794 83.19
tax 4.411 292.21
ptratio 8.613 902.20
lstat 28.725 5813.05
Two measures of variable importance are reported. The first is based upon
the mean decrease of accuracy in predictions on the out of bag samples when
a given variable is permuted. The second is a measure of the total decrease
in node impurity that results from splits over that variable, averaged over
all trees (this was plotted in Figure 8.9). In the case of regression trees,
the node impurity is measured by the training RSS, and for classification
trees by the deviance. Plots of these importance measures can be produced
using the varImpPlot() function.
varImpPlot()
> varImpPlot (rf.boston)
The results indicate that across all of the trees considered in the random
forest, the wealth of the community (lstat) and the house size (rm) are by
far the two most important variables.
8.3.4 Boosting
Here we use the gbm package, and within it the gbm() function, to fit boosted
gbm()
regression trees to the Boston data set. We run gbm() with the option
distribution = "gaussian" since this is a regression problem; if it were
a binary classification problem, we would use distribution = "bernoulli".
The argument n.trees = 5000 indicates that we want 5000 trees, and the
option interaction.depth = 4 limits the depth of each tree.
> library(gbm)
> set.seed (1)
> boost.boston <- gbm(medv ∼ ., data = Boston[train , ],
distribution = "gaussian ", n.trees = 5000 ,
interaction .depth = 4)
The summary() function produces a relative influence plot and also outputs
the relative influence statistics.
> summary(boost.boston)
var rel.inf
rm rm 44.482
lstat lstat 32.703
crim crim 4.851
dis dis 4.487
nox nox 3.752](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-367-320.jpg)
![360 8. Tree-Based Methods
age age 3.198
ptratio ptratio 2.814
tax tax 1.544
indus indus 1.034
rad rad 0.876
zn zn 0.162
chas chas 0.097
We see that lstat and rm are by far the most important variables. We can
also produce partial dependence plots for these two variables. These plots
partial
dependence
plot
illustrate the marginal effect of the selected variables on the response after
integrating out the other variables. In this case, as we might expect, median
house prices are increasing with rm and decreasing with lstat.
> plot(boost.boston , i = "rm")
> plot(boost.boston , i = "lstat")
We now use the boosted model to predict medv on the test set:
> yhat.boost <- predict(boost.boston ,
newdata = Boston[-train , ], n.trees = 5000)
> mean (( yhat.boost - boston.test)^2)
[1] 18.39
The test MSE obtained is 18.39: this is superior to the test MSE of random
forests and bagging. If we want to, we can perform boosting with a different
value of the shrinkage parameter λ in (8.10). The default value is 0.001,
but this is easily modified. Here we take λ = 0.2.
> boost.boston <- gbm(medv ∼ ., data = Boston[train , ],
distribution = "gaussian ", n.trees = 5000 ,
interaction .depth = 4, shrinkage = 0.2, verbose = F)
> yhat.boost <- predict(boost.boston ,
newdata = Boston[-train , ], n.trees = 5000)
> mean (( yhat.boost - boston.test)^2)
[1] 16.55
In this case, using λ = 0.2 leads to a lower test MSE than λ = 0.001.
8.3.5 Bayesian Additive Regression Trees
In this section we use the BART package, and within it the gbart() function,
gbart()
to fit a Bayesian additive regression tree model to the Boston housing data
set. The gbart() function is designed for quantitative outcome variables.
For binary outcomes, lbart() and pbart() are available.
lbart()
pbart()
To run the gbart() function, we must first create matrices of predictors
for the training and test data. We run BART with default settings.
> library(BART)
> x <- Boston[, 1:12]
> y <- Boston[, "medv"]
> xtrain <- x[train , ]
> ytrain <- y[train]](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-368-320.jpg)
![8.4 Exercises 361
> xtest <- x[-train , ]
> ytest <- y[-train]
> set.seed (1)
> bartfit <- gbart(xtrain , ytrain , x.test = xtest)
Next we compute the test error.
> yhat.bart <- bartfit$yhat.test.mean
> mean (( ytest - yhat.bart)^2)
[1] 15.95
On this data set, the test error of BART is lower than the test error of
random forests and boosting.
Now we can check how many times each variable appeared in the collec-
tion of trees.
> ord <- order(bartfit$varcount.mean , decreasing = T)
> bartfit$varcount.mean[ord]
nox lstat tax rad rm indus chas ptratio
22.95 21.33 21.25 20.78 19.89 19.82 19.05 18.98
age zn dis crim
18.27 15.95 14.46 11.01
8.4 Exercises
Conceptual
1. Draw an example (of your own invention) of a partition of two-
dimensional feature space that could result from recursive binary
splitting. Your example should contain at least six regions. Draw a
decision tree corresponding to this partition. Be sure to label all as-
pects of your figures, including the regions R1, R2, . . ., the cutpoints
t1, t2, . . ., and so forth.
Hint: Your result should look something like Figures 8.1 and 8.2.
2. It is mentioned in Section 8.2.3 that boosting using depth-one trees
(or stumps) leads to an additive model: that is, a model of the form
f(X) =
p
0
j=1
fj(Xj).
Explain why this is the case. You can begin with (8.12) in
Algorithm 8.2.
3. Consider the Gini index, classification error, and entropy in a simple
classification setting with two classes. Create a single plot that dis-
plays each of these quantities as a function of p̂m1. The x-axis should](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-369-320.jpg)
![386 9. Support Vector Machines
the K classes to the remaining K − 1 classes. Let β0k, β1k, . . . , βpk denote
the parameters that result from fitting an SVM comparing the kth class
(coded as +1) to the others (coded as −1). Let x∗
denote a test observation.
We assign the observation to the class for which β0k +β1kx∗
1 +β2kx∗
2 +· · ·+
βpkx∗
p is largest, as this amounts to a high level of confidence that the test
observation belongs to the kth class rather than to any of the other classes.
9.5 Relationship to Logistic Regression
When SVMs were first introduced in the mid-1990s, they made quite a
splash in the statistical and machine learning communities. This was due
in part to their good performance, good marketing, and also to the fact
that the underlying approach seemed both novel and mysterious. The idea
of finding a hyperplane that separates the data as well as possible, while al-
lowing some violations to this separation, seemed distinctly different from
classical approaches for classification, such as logistic regression and lin-
ear discriminant analysis. Moreover, the idea of using a kernel to expand
the feature space in order to accommodate non-linear class boundaries ap-
peared to be a unique and valuable characteristic.
However, since that time, deep connections between SVMs and other
more classical statistical methods have emerged. It turns out that one can
rewrite the criterion (9.12)–(9.15) for fitting the support vector classifier
f(X) = β0 + β1X1 + · · · + βpXp as
minimize
β0,β1,...,βp
⎧
⎨
⎩
n
0
i=1
max [0, 1 − yif(xi)] + λ
p
0
j=1
β2
j
⎫
⎬
⎭
, (9.25)
where λ is a nonnegative tuning parameter. When λ is large then β1, . . . , βp
are small, more violations to the margin are tolerated, and a low-variance
but high-bias classifier will result. When λ is small then few violations
to the margin will occur; this amounts to a high-variance but low-bias
classifier. Thus, a small value of λ in (9.25) amounts to a small value of C
in (9.15). Note that the λ
)p
j=1 β2
j term in (9.25) is the ridge penalty term
from Section 6.2.1, and plays a similar role in controlling the bias-variance
trade-off for the support vector classifier.
Now (9.25) takes the “Loss + Penalty” form that we have seen repeatedly
throughout this book:
minimize
β0,β1,...,βp
{L(X, y, β) + λP(β)} . (9.26)
In (9.26), L(X, y, β) is some loss function quantifying the extent to which
the model, parametrized by β, fits the data (X, y), and P(β) is a penalty](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-393-320.jpg)
![9.5 Relationship to Logistic Regression 387
function on the parameter vector β whose effect is controlled by a nonneg-
ative tuning parameter λ. For instance, ridge regression and the lasso both
take this form with
L(X, y, β) =
n
0
i=1
⎛
⎝yi − β0 −
p
0
j=1
xijβj
⎞
⎠
2
and with P(β) =
)p
j=1 β2
j for ridge regression and P(β) =
)p
j=1 |βj| for
the lasso. In the case of (9.25) the loss function instead takes the form
L(X, y, β) =
n
0
i=1
max [0, 1 − yi(β0 + β1xi1 + · · · + βpxip)] .
This is known as hinge loss, and is depicted in Figure 9.12. However, it
hinge loss
turns out that the hinge loss function is closely related to the loss function
used in logistic regression, also shown in Figure 9.12.
An interesting characteristic of the support vector classifier is that only
support vectors play a role in the classifier obtained; observations on the
correct side of the margin do not affect it. This is due to the fact that the
loss function shown in Figure 9.12 is exactly zero for observations for which
yi(β0 + β1xi1 + · · · + βpxip) ≥ 1; these correspond to observations that are
on the correct side of the margin.3
In contrast, the loss function for logistic
regression shown in Figure 9.12 is not exactly zero anywhere. But it is very
small for observations that are far from the decision boundary. Due to the
similarities between their loss functions, logistic regression and the support
vector classifier often give very similar results. When the classes are well
separated, SVMs tend to behave better than logistic regression; in more
overlapping regimes, logistic regression is often preferred.
When the support vector classifier and SVM were first introduced, it was
thought that the tuning parameter C in (9.15) was an unimportant “nui-
sance” parameter that could be set to some default value, like 1. However,
the “Loss + Penalty” formulation (9.25) for the support vector classifier
indicates that this is not the case. The choice of tuning parameter is very
important and determines the extent to which the model underfits or over-
fits the data, as illustrated, for example, in Figure 9.7.
We have established that the support vector classifier is closely related
to logistic regression and other preexisting statistical methods. Is the SVM
unique in its use of kernels to enlarge the feature space to accommodate
non-linear class boundaries? The answer to this question is “no”. We could
just as well perform logistic regression or many of the other classification
methods seen in this book using non-linear kernels; this is closely related to
3With this hinge-loss + penalty representation, the margin corresponds to the value
one, and the width of the margin is determined by
!
β2
j .](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-394-320.jpg)
![9.6 Lab: Support Vector Machines 389
9.6.1 Support Vector Classifier
The e1071 library contains implementations for a number of statistical
learning methods. In particular, the svm() function can be used to fit a
svm()
support vector classifier when the argument kernel = "linear" is used.
This function uses a slightly different formulation from (9.14) and (9.25)
for the support vector classifier. A cost argument allows us to specify the
cost of a violation to the margin. When the cost argument is small, then
the margins will be wide and many support vectors will be on the margin
or will violate the margin. When the cost argument is large, then the mar-
gins will be narrow and there will be few support vectors on the margin or
violating the margin.
We now use the svm() function to fit the support vector classifier for a
given value of the cost parameter. Here we demonstrate the use of this
function on a two-dimensional example so that we can plot the resulting
decision boundary. We begin by generating the observations, which belong
to two classes, and checking whether the classes are linearly separable.
> set.seed (1)
> x <- matrix(rnorm (20 * 2), ncol = 2)
> y <- c(rep(-1, 10) , rep(1, 10))
> x[y == 1, ] <- x[y == 1, ] + 1
> plot(x, col = (3 - y))
They are not. Next, we fit the support vector classifier. Note that in order
for the svm() function to perform classification (as opposed to SVM-based
regression), we must encode the response as a factor variable. We now
create a data frame with the response coded as a factor.
> dat <- data.frame(x = x, y = as.factor(y))
> library(e1071)
> svmfit <- svm(y ∼ ., data = dat , kernel = "linear",
cost = 10, scale = FALSE)
The argument scale = FALSE tells the svm() function not to scale each
feature to have mean zero or standard deviation one; depending on the
application, one might prefer to use scale = TRUE.
We can now plot the support vector classifier obtained:
> plot(svmfit , dat)
Note that the two arguments to the SVM plot() function are the output
of the call to svm(), as well as the data used in the call to svm(). The region
of feature space that will be assigned to the −1 class is shown in light
yellow, and the region that will be assigned to the +1 class is shown in
red. The decision boundary between the two classes is linear (because we
used the argument kernel = "linear"), though due to the way in which the
plotting function is implemented in this library the decision boundary looks
somewhat jagged in the plot. (Note that here the second feature is plotted
on the x-axis and the first feature is plotted on the y-axis, in contrast to](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-396-320.jpg)
![390 9. Support Vector Machines
the behavior of the usual plot() function in R.) The support vectors are
plotted as crosses and the remaining observations are plotted as circles;
we see here that there are seven support vectors. We can determine their
identities as follows:
> svmfit$index
[1] 1 2 5 7 14 16 17
We can obtain some basic information about the support vector classifier
fit using the summary() command:
> summary(svmfit)
Call:
svm(formula = y ∼ ., data = dat , kernel = "linear", cost = 10,
scale = FALSE)
Parameters :
SVM -Type: C- classification
SVM -Kernel: linear
cost: 10
Number of Support Vectors: 7
( 4 3 )
Number of Classes: 2
Levels:
-1 1
This tells us, for instance, that a linear kernel was used with cost = 10,
and that there were seven support vectors, four in one class and three in
the other.
What if we instead used a smaller value of the cost parameter?
> svmfit <- svm(y ∼ ., data = dat , kernel = "linear",
cost = 0.1, scale = FALSE)
> plot(svmfit , dat)
> svmfit$index
[1] 1 2 3 4 5 7 9 10 12 13 14 15 16 17 18 20
Now that a smaller value of the cost parameter is being used, we obtain a
larger number of support vectors, because the margin is now wider. Unfor-
tunately, the svm() function does not explicitly output the coefficients of
the linear decision boundary obtained when the support vector classifier is
fit, nor does it output the width of the margin.
The e1071 library includes a built-in function, tune(), to perform cross-
tune()
validation. By default, tune() performs ten-fold cross-validation on a set
of models of interest. In order to use this function, we pass in relevant
information about the set of models that are under consideration. The
following command indicates that we want to compare SVMs with a linear
kernel, using a range of values of the cost parameter.
> set.seed (1)
> tune.out <- tune(svm , y ∼ ., data = dat , kernel = "linear",
ranges = list(cost = c(0.001 , 0.01 , 0.1, 1, 5, 10, 100)))](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-397-320.jpg)
![9.6 Lab: Support Vector Machines 391
We can easily access the cross-validation errors for each of these models
using the summary() command:
> summary(tune.out)
Parameter tuning of ‘svm ’:
- sampling method: 10-fold cross validation
- best parameters:
cost
0.1
- best performance : 0.05
- Detailed performance results:
cost error dispersion
1 1e -03 0.55 0.438
2 1e -02 0.55 0.438
3 1e -01 0.05 0.158
4 1e+00 0.15 0.242
5 5e+00 0.15 0.242
6 1e+01 0.15 0.242
7 1e+02 0.15 0.242
We see that cost = 0.1 results in the lowest cross-validation error rate. The
tune() function stores the best model obtained, which can be accessed as
follows:
> bestmod <- tune.out$best.model
> summary(bestmod)
The predict() function can be used to predict the class label on a set of
test observations, at any given value of the cost parameter. We begin by
generating a test data set.
> xtest <- matrix(rnorm (20 * 2), ncol = 2)
> ytest <- sample(c(-1, 1), 20, rep = TRUE)
> xtest[ytest == 1, ] <- xtest[ytest == 1, ] + 1
> testdat <- data.frame(x = xtest , y = as.factor(ytest))
Now we predict the class labels of these test observations. Here we use the
best model obtained through cross-validation in order to make predictions.
> ypred <- predict(bestmod , testdat)
> table(predict = ypred , truth = testdat$y)
truth
predict -1 1
-1 9 1
1 2 8
Thus, with this value of cost, 17 of the test observations are correctly
classified. What if we had instead used cost = 0.01?
> svmfit <- svm(y ∼ ., data = dat , kernel = "linear",
cost = .01, scale = FALSE)
> ypred <- predict(svmfit , testdat)
> table(predict = ypred , truth = testdat$y)
truth
predict -1 1](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-398-320.jpg)
![392 9. Support Vector Machines
-1 11 6
1 0 3
In this case three additional observations are misclassified.
Now consider a situation in which the two classes are linearly separable.
Then we can find a separating hyperplane using the svm() function. We
first further separate the two classes in our simulated data so that they are
linearly separable:
> x[y == 1, ] <- x[y == 1, ] + 0.5
> plot(x, col = (y + 5) / 2, pch = 19)
Now the observations are just barely linearly separable. We fit the support
vector classifier and plot the resulting hyperplane, using a very large value
of cost so that no observations are misclassified.
> dat <- data.frame(x = x, y = as.factor(y))
> svmfit <- svm(y ∼ ., data = dat , kernel = "linear",
cost = 1e5)
> summary(svmfit)
Call:
svm(formula = y ∼ ., data = dat , kernel = "linear", cost =
1e+05)
Parameters :
SVM -Type: C- classification
SVM -Kernel: linear
cost: 1e+05
Number of Support Vectors: 3
( 1 2 )
Number of Classes: 2
Levels:
-1 1
> plot(svmfit , dat)
No training errors were made and only three support vectors were used.
However, we can see from the figure that the margin is very narrow (because
the observations that are not support vectors, indicated as circles, are very
close to the decision boundary). It seems likely that this model will perform
poorly on test data. We now try a smaller value of cost:
> svmfit <- svm(y ∼ ., data = dat , kernel = "linear", cost = 1)
> summary(svmfit)
> plot(svmfit , dat)
Using cost = 1, we misclassify a training observation, but we also obtain a
much wider margin and make use of seven support vectors. It seems likely
that this model will perform better on test data than the model with
cost = 1e5.
9.6.2 Support Vector Machine
In order to fit an SVM using a non-linear kernel, we once again use the svm()
function. However, now we use a different value of the parameter kernel.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-399-320.jpg)
![9.6 Lab: Support Vector Machines 393
To fit an SVM with a polynomial kernel we use kernel = "polynomial",
and to fit an SVM with a radial kernel we use kernel = "radial". In the
former case we also use the degree argument to specify a degree for the
polynomial kernel (this is d in (9.22)), and in the latter case we use gamma
to specify a value of γ for the radial basis kernel (9.24).
We first generate some data with a non-linear class boundary, as follows:
> set.seed (1)
> x <- matrix(rnorm (200 * 2), ncol = 2)
> x[1:100 , ] <- x[1:100 , ] + 2
> x[101:150 , ] <- x[101:150 , ] - 2
> y <- c(rep(1, 150) , rep(2, 50))
> dat <- data.frame(x = x, y = as.factor(y))
Plotting the data makes it clear that the class boundary is indeed non-
linear:
> plot(x, col = y)
The data is randomly split into training and testing groups. We then fit
the training data using the svm() function with a radial kernel and γ = 1:
> train <- sample (200 , 100)
> svmfit <- svm(y ∼ ., data = dat[train , ], kernel = "radial",
gamma = 1, cost = 1)
> plot(svmfit , dat[train , ])
The plot shows that the resulting SVM has a decidedly non-linear
boundary. The summary() function can be used to obtain some
information about the SVM fit:
> summary(svmfit)
Call:
svm(formula = y ∼ ., data = dat[train , ], kernel = "radial",
gamma = 1, cost = 1)
Parameters :
SVM -Type: C-classification
SVM -Kernel: radial
cost: 1
Number of Support Vectors: 31
( 16 15 )
Number of Classes: 2
Levels:
1 2
We can see from the figure that there are a fair number of training errors
in this SVM fit. If we increase the value of cost, we can reduce the number
of training errors. However, this comes at the price of a more irregular
decision boundary that seems to be at risk of overfitting the data.
> svmfit <- svm(y ∼ ., data = dat[train , ], kernel = "radial",
gamma = 1, cost = 1e5)
> plot(svmfit , dat[train , ])](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-400-320.jpg)
![394 9. Support Vector Machines
We can perform cross-validation using tune() to select the best choice of
γ and cost for an SVM with a radial kernel:
> set.seed (1)
> tune.out <- tune(svm , y ∼ ., data = dat[train , ],
kernel = "radial",
ranges = list(
cost = c(0.1 , 1, 10, 100, 1000) ,
gamma = c(0.5 , 1, 2, 3, 4)
)
)
> summary(tune.out)
Parameter tuning of ‘svm ’:
- sampling method: 10-fold cross validation
- best parameters:
cost gamma
1 0.5
- best performance : 0.07
- Detailed performance results:
cost gamma error dispersion
1 1e-01 0.5 0.26 0.158
2 1e+00 0.5 0.07 0.082
3 1e+01 0.5 0.07 0.082
4 1e+02 0.5 0.14 0.151
5 1e+03 0.5 0.11 0.074
6 1e-01 1.0 0.22 0.162
7 1e+00 1.0 0.07 0.082
. . .
Therefore, the best choice of parameters involves cost = 1 and gamma =
0.5. We can view the test set predictions for this model by applying the
predict() function to the data. Notice that to do this we subset the dataframe
dat using -train as an index set.
> table(
true = dat[-train , "y"],
pred = predict(
tune.out$best.model , newdata = dat[-train , ]
)
)
12 % of test observations are misclassified by this SVM.
9.6.3 ROC Curves
The ROCR package can be used to produce ROC curves such as those in
Figures 9.10 and 9.11. We first write a short function to plot an ROC curve
given a vector containing a numerical score for each observation, pred, and
a vector containing the class label for each observation, truth.
> library(ROCR)
> rocplot <- function (pred , truth , ...) {
+ predob <- prediction (pred , truth)](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-401-320.jpg)
![9.6 Lab: Support Vector Machines 395
+ perf <- performance (predob , "tpr", "fpr")
+ plot(perf , ...)
+ }
SVMs and support vector classifiers output class labels for each observa-
tion. However, it is also possible to obtain fitted values for each observation,
which are the numerical scores used to obtain the class labels. For instance,
in the case of a support vector classifier, the fitted value for an observation
X = (X1, X2, . . . , Xp)T
takes the form β̂0 + β̂1X1 + β̂2X2 + · · · + β̂pXp.
For an SVM with a non-linear kernel, the equation that yields the fitted
value is given in (9.23). In essence, the sign of the fitted value determines
on which side of the decision boundary the observation lies. Therefore, the
relationship between the fitted value and the class prediction for a given
observation is simple: if the fitted value exceeds zero then the observation
is assigned to one class, and if it is less than zero then it is assigned to the
other. In order to obtain the fitted values for a given SVM model fit, we use
decision.values = TRUE when fitting svm(). Then the predict() function
will output the fitted values.
> svmfit.opt <- svm(y ∼ ., data = dat[train , ],
kernel = "radial", gamma = 2, cost = 1,
decision.values = T)
> fitted <- attributes (
predict(svmfit.opt , dat[train , ], decision.values = TRUE)
)$decision.values
Now we can produce the ROC plot. Note we use the negative of the fitted
values so that negative values correspond to class 1 and positive values to
class 2.
> par(mfrow = c(1, 2))
> rocplot(-fitted , dat[train , "y"], main = "Training Data")
SVM appears to be producing accurate predictions. By increasing γ we can
produce a more flexible fit and generate further improvements in accuracy.
> svmfit.flex <- svm(y ∼ ., data = dat[train , ],
kernel = "radial", gamma = 50, cost = 1,
decision.values = T)
> fitted <- attributes (
predict(svmfit.flex , dat[train , ], decision.values = T)
)$decision.values
> rocplot(-fitted , dat[train , "y"], add = T, col = "red")
However, these ROC curves are all on the training data. We are really
more interested in the level of prediction accuracy on the test data. When
we compute the ROC curves on the test data, the model with γ = 2 appears
to provide the most accurate results.
> fitted <- attributes (
predict(svmfit.opt , dat[-train , ], decision.values = T)
)$decision.values](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-402-320.jpg)
![396 9. Support Vector Machines
> rocplot(-fitted , dat[-train , "y"], main = "Test Data")
> fitted <- attributes (
predict(svmfit.flex , dat[-train , ], decision.values = T)
)$decision.values
> rocplot(-fitted , dat[-train , "y"], add = T, col = "red")
9.6.4 SVM with Multiple Classes
If the response is a factor containing more than two levels, then the svm()
function will perform multi-class classification using the one-versus-one ap-
proach. We explore that setting here by generating a third class of obser-
vations.
> set.seed (1)
> x <- rbind(x, matrix(rnorm (50 * 2), ncol = 2))
> y <- c(y, rep(0, 50))
> x[y == 0, 2] <- x[y == 0, 2] + 2
> dat <- data.frame(x = x, y = as.factor(y))
> par(mfrow = c(1, 1))
> plot(x, col = (y + 1))
We now fit an SVM to the data:
> svmfit <- svm(y ∼ ., data = dat , kernel = "radial",
cost = 10, gamma = 1)
> plot(svmfit , dat)
The e1071 library can also be used to perform support vector regression,
if the response vector that is passed in to svm() is numerical rather than a
factor.
9.6.5 Application to Gene Expression Data
We now examine the Khan data set, which consists of a number of tissue
samples corresponding to four distinct types of small round blue cell tu-
mors. For each tissue sample, gene expression measurements are available.
The data set consists of training data, xtrain and ytrain, and testing data,
xtest and ytest.
We examine the dimension of the data:
> library(ISLR2)
> names(Khan)
[1] "xtrain" "xtest" "ytrain" "ytest"
> dim(Khan$xtrain)
[1] 63 2308
> dim(Khan$xtest)
[1] 20 2308
> length(Khan$ytrain)
[1] 63
> length(Khan$ytest)
[1] 20](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-403-320.jpg)
![10.8 Interpolation and Double Descent 439
• Regularization tuning parameters. These include the dropout rate φ
and the strength λ of lasso and ridge regularization, and are typically
set separately at each layer.
• Details of stochastic gradient descent. These includes the batch size,
the number of epochs, and if used, details of data augmentation (Sec-
tion 10.3.4.)
Choices such as these can make a difference. In preparing this MNIST exam-
ple, we achieved a respectable 1.8% misclassification error after some trial
and error. Finer tuning and training of a similar network can get under
1% error on these data, but the tinkering process can be tedious, and can
result in overfitting if done carelessly.
10.8 Interpolation and Double Descent
Throughout this book, we have repeatedly discussed the bias-variance trade-
off, first presented in Section 2.2.2. This trade-off indicates that statistical
learning methods tend to perform the best, in terms of test-set error, for an
intermediate level of model complexity. In particular, if we plot “flexibility”
on the x-axis and error on the y-axis, then we generally expect to see that
test error has a U-shape, whereas training error decreases monotonically.
Two “typical” examples of this behavior can be seen in the right-hand
panel of Figure 2.9 on page 31, and in Figure 2.17 on page 42. One implica-
tion of the bias-variance trade-off is that it is generally not a good idea to
interpolate the training data — that is, to get zero training error — since
interpolate
that will often result in very high test error.
However, it turns out that in certain specific settings it can be possible for
a statistical learning method that interpolates the training data to perform
well — or at least, better than a slightly less complex model that does not
quite interpolate the data. This phenomenon is known as double descent,
and is displayed in Figure 10.20. “Double descent” gets its name from the
fact that the test error has a U-shape before the interpolation threshold is
reached, and then it descends again (for a while, at least) as an increasingly
flexible model is fit.
We now describe the set-up that resulted in Figure 10.20. We simulated
n = 20 observations from the model
Y = sin(X) + ϵ,
where X ∼ U[−5, 5] (uniform distribution), and ϵ ∼ N(0, σ2
) with σ = 0.3.
We then fit a natural spline to the data, as described in Section 7.4, with d
degrees of freedom.23
Recall from Section 7.4 that fitting a natural spline
23This implies the choice of d knots, here chosen at d equi-probability quantiles of the
training data. When d > n, the quantiles are found by interpolation.](https://image.slidesharecdn.com/islrv2website-220603203225-800dff44/85/ISLRv2_website-pdf-446-320.jpg)
ISLRv2_website.pdf
- 1. Gareth James • Daniela Witten • Trevor Hastie • Robert Tibshirani An Introduction to Statistical Learning with Applications in R Second Edition 123 First Printing: August 4, 2021
- 2. To our parents: Alison and Michael James Chiara Nappi and Edward Witten Valerie and Patrick Hastie Vera and Sami Tibshirani and to our families: Michael, Daniel, and Catherine Tessa, Theo, Otto, and Ari Samantha, Timothy, and Lynda Charlie, Ryan, Julie, and Cheryl
- 3. Preface Statistical learning refers to a set of tools for making sense of complex datasets. In recent years, we have seen a staggering increase in the scale and scope of data collection across virtually all areas of science and industry. As a result, statistical learning has become a critical toolkit for anyone who wishes to understand data — and as more and more of today’s jobs involve data, this means that statistical learning is fast becoming a critical toolkit for everyone. One of the first books on statistical learning — The Elements of Statisti- cal Learning (ESL, by Hastie, Tibshirani, and Friedman) — was published in 2001, with a second edition in 2009. ESL has become a popular text not only in statistics but also in related fields. One of the reasons for ESL’s popularity is its relatively accessible style. But ESL is best-suited for indi- viduals with advanced training in the mathematical sciences. An Introduction to Statistical Learning (ISL) arose from the clear need for a broader and less technical treatment of the key topics in statistical learning. The intention behind ISL is to concentrate more on the applica- tions of the methods and less on the mathematical details. Beginning with Chapter 2, each chapter in ISL contains a lab illustrating how to implement the statistical learning methods seen in that chapter using the popular sta- tistical software package R. These labs provide the reader with valuable hands-on experience. ISL is appropriate for advanced undergraduates or master’s students in Statistics or related quantitative fields, or for individuals in other disciplines who wish to use statistical learning tools to analyze their data. It can be used as a textbook for a course spanning two semesters. vii
- 4. The first edition of ISL covered a number of important topics, including sparse methods for classification and regression, decision trees, boosting, support vector machines, and clustering. Since it was published in 2013, it has become a mainstay of undergraduate and graduate classrooms across the United States and worldwide, as well as a key reference book for data scientists. In this second edition of ISL, we have greatly expanded the set of topics covered. In particular, the second edition includes new chapters on deep learning (Chapter 10), survival analysis (Chapter 11), and multiple testing (Chapter 13). We have also substantially expanded some chapters that were part of the first edition: among other updates, we now include treatments of naive Bayes and generalized linear models in Chapter 4, Bayesian addi- tive regression trees in Chapter 8, and matrix completion in Chapter 12. Furthermore, we have updated the R code throughout the labs to ensure that the results that they produce agree with recent R releases. We are grateful to these readers for providing valuable comments on the first edition of this book: Pallavi Basu, Alexandra Chouldechova, Patrick Danaher, Will Fithian, Luella Fu, Sam Gross, Max Grazier G’Sell, Court- ney Paulson, Xinghao Qiao, Elisa Sheng, Noah Simon, Kean Ming Tan, Xin Lu Tan. We thank these readers for helpful input on the second edi- tion of this book: Alan Agresti, Iain Carmichael, Yiqun Chen, Erin Craig, Daisy Ding, Lucy Gao, Ismael Lemhadri, Bryan Martin, Anna Neufeld, Ge- off Tims, Carsten Voelkmann, Steve Yadlowsky, and James Zou. We also thank Anna Neufeld for her assistance in reformatting the R code through- out this book. We are immensely grateful to Balasubramanian “Naras” Narasimhan for his assistance on both editions of this textbook. It has been an honor and a privilege for us to see the considerable impact that the first edition of ISL has had on the way in which statistical learning is practiced, both in and out of the academic setting. We hope that this new edition will continue to give today’s and tomorrow’s applied statisticians and data scientists the tools they need for success in a data-driven world. It’s tough to make predictions, especially about the future. -Yogi Berra viii Preface
- 5. Contents Preface vii 1 Introduction 1 2 Statistical Learning 15 2.1 What Is Statistical Learning? . . . . . . . . . . . . . . . . . 15 2.1.1 Why Estimate f? . . . . . . . . . . . . . . . . . . . 17 2.1.2 How Do We Estimate f? . . . . . . . . . . . . . . . 21 2.1.3 The Trade-Off Between Prediction Accuracy and Model Interpretability . . . . . . . . . . . . . . 24 2.1.4 Supervised Versus Unsupervised Learning . . . . . 26 2.1.5 Regression Versus Classification Problems . . . . . 28 2.2 Assessing Model Accuracy . . . . . . . . . . . . . . . . . . 29 2.2.1 Measuring the Quality of Fit . . . . . . . . . . . . 29 2.2.2 The Bias-Variance Trade-Off . . . . . . . . . . . . . 33 2.2.3 The Classification Setting . . . . . . . . . . . . . . 37 2.3 Lab: Introduction to R . . . . . . . . . . . . . . . . . . . . 42 2.3.1 Basic Commands . . . . . . . . . . . . . . . . . . . 43 2.3.2 Graphics . . . . . . . . . . . . . . . . . . . . . . . . 45 2.3.3 Indexing Data . . . . . . . . . . . . . . . . . . . . . 47 2.3.4 Loading Data . . . . . . . . . . . . . . . . . . . . . 48 2.3.5 Additional Graphical and Numerical Summaries . . 50 2.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 3 Linear Regression 59 3.1 Simple Linear Regression . . . . . . . . . . . . . . . . . . . 60 3.1.1 Estimating the Coefficients . . . . . . . . . . . . . 61 3.1.2 Assessing the Accuracy of the Coefficient Estimates . . . . . . . . . . . . . . . . . . . . . . . 63 3.1.3 Assessing the Accuracy of the Model . . . . . . . . 68 3.2 Multiple Linear Regression . . . . . . . . . . . . . . . . . . 71 3.2.1 Estimating the Regression Coefficients . . . . . . . 72 3.2.2 Some Important Questions . . . . . . . . . . . . . . 75 3.3 Other Considerations in the Regression Model . . . . . . . 83 ix
- 6. 3.3.1 Qualitative Predictors . . . . . . . . . . . . . . . . 83 3.3.2 Extensions of the Linear Model . . . . . . . . . . . 87 3.3.3 Potential Problems . . . . . . . . . . . . . . . . . . 92 3.4 The Marketing Plan . . . . . . . . . . . . . . . . . . . . . . 103 3.5 Comparison of Linear Regression with K-Nearest Neighbors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 3.6 Lab: Linear Regression . . . . . . . . . . . . . . . . . . . . 110 3.6.1 Libraries . . . . . . . . . . . . . . . . . . . . . . . . 110 3.6.2 Simple Linear Regression . . . . . . . . . . . . . . . 111 3.6.3 Multiple Linear Regression . . . . . . . . . . . . . . 114 3.6.4 Interaction Terms . . . . . . . . . . . . . . . . . . . 116 3.6.5 Non-linear Transformations of the Predictors . . . 116 3.6.6 Qualitative Predictors . . . . . . . . . . . . . . . . 119 3.6.7 Writing Functions . . . . . . . . . . . . . . . . . . . 120 3.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 4 Classification 129 4.1 An Overview of Classification . . . . . . . . . . . . . . . . . 130 4.2 Why Not Linear Regression? . . . . . . . . . . . . . . . . . 131 4.3 Logistic Regression . . . . . . . . . . . . . . . . . . . . . . 133 4.3.1 The Logistic Model . . . . . . . . . . . . . . . . . . 133 4.3.2 Estimating the Regression Coefficients . . . . . . . 135 4.3.3 Making Predictions . . . . . . . . . . . . . . . . . . 136 4.3.4 Multiple Logistic Regression . . . . . . . . . . . . . 137 4.3.5 Multinomial Logistic Regression . . . . . . . . . . . 140 4.4 Generative Models for Classification . . . . . . . . . . . . . 141 4.4.1 Linear Discriminant Analysis for p = 1 . . . . . . . 142 4.4.2 Linear Discriminant Analysis for p >1 . . . . . . . 145 4.4.3 Quadratic Discriminant Analysis . . . . . . . . . . 152 4.4.4 Naive Bayes . . . . . . . . . . . . . . . . . . . . . . 153 4.5 A Comparison of Classification Methods . . . . . . . . . . 158 4.5.1 An Analytical Comparison . . . . . . . . . . . . . . 158 4.5.2 An Empirical Comparison . . . . . . . . . . . . . . 161 4.6 Generalized Linear Models . . . . . . . . . . . . . . . . . . 164 4.6.1 Linear Regression on the Bikeshare Data . . . . . . 164 4.6.2 Poisson Regression on the Bikeshare Data . . . . . 167 4.6.3 Generalized Linear Models in Greater Generality . 170 4.7 Lab: Classification Methods . . . . . . . . . . . . . . . . . . 171 4.7.1 The Stock Market Data . . . . . . . . . . . . . . . 171 4.7.2 Logistic Regression . . . . . . . . . . . . . . . . . . 172 4.7.3 Linear Discriminant Analysis . . . . . . . . . . . . 177 4.7.4 Quadratic Discriminant Analysis . . . . . . . . . . 179 4.7.5 Naive Bayes . . . . . . . . . . . . . . . . . . . . . . 180 4.7.6 K-Nearest Neighbors . . . . . . . . . . . . . . . . . 181 4.7.7 Poisson Regression . . . . . . . . . . . . . . . . . . 185 x CONTENTS
- 7. CONTENTS xi 4.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 5 Resampling Methods 197 5.1 Cross-Validation . . . . . . . . . . . . . . . . . . . . . . . . 198 5.1.1 The Validation Set Approach . . . . . . . . . . . . 198 5.1.2 Leave-One-Out Cross-Validation . . . . . . . . . . 200 5.1.3 k-Fold Cross-Validation . . . . . . . . . . . . . . . 203 5.1.4 Bias-Variance Trade-Off for k-Fold Cross-Validation . . . . . . . . . . . . . . . . . . . 205 5.1.5 Cross-Validation on Classification Problems . . . . 206 5.2 The Bootstrap . . . . . . . . . . . . . . . . . . . . . . . . . 209 5.3 Lab: Cross-Validation and the Bootstrap . . . . . . . . . . 212 5.3.1 The Validation Set Approach . . . . . . . . . . . . 213 5.3.2 Leave-One-Out Cross-Validation . . . . . . . . . . 214 5.3.3 k-Fold Cross-Validation . . . . . . . . . . . . . . . 215 5.3.4 The Bootstrap . . . . . . . . . . . . . . . . . . . . 216 5.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 6 Linear Model Selection and Regularization 225 6.1 Subset Selection . . . . . . . . . . . . . . . . . . . . . . . . 227 6.1.1 Best Subset Selection . . . . . . . . . . . . . . . . . 227 6.1.2 Stepwise Selection . . . . . . . . . . . . . . . . . . 229 6.1.3 Choosing the Optimal Model . . . . . . . . . . . . 232 6.2 Shrinkage Methods . . . . . . . . . . . . . . . . . . . . . . 237 6.2.1 Ridge Regression . . . . . . . . . . . . . . . . . . . 237 6.2.2 The Lasso . . . . . . . . . . . . . . . . . . . . . . . 241 6.2.3 Selecting the Tuning Parameter . . . . . . . . . . . 250 6.3 Dimension Reduction Methods . . . . . . . . . . . . . . . . 251 6.3.1 Principal Components Regression . . . . . . . . . . 252 6.3.2 Partial Least Squares . . . . . . . . . . . . . . . . . 259 6.4 Considerations in High Dimensions . . . . . . . . . . . . . 261 6.4.1 High-Dimensional Data . . . . . . . . . . . . . . . . 261 6.4.2 What Goes Wrong in High Dimensions? . . . . . . 262 6.4.3 Regression in High Dimensions . . . . . . . . . . . 264 6.4.4 Interpreting Results in High Dimensions . . . . . . 266 6.5 Lab: Linear Models and Regularization Methods . . . . . . 267 6.5.1 Subset Selection Methods . . . . . . . . . . . . . . 267 6.5.2 Ridge Regression and the Lasso . . . . . . . . . . . 274 6.5.3 PCR and PLS Regression . . . . . . . . . . . . . . 279 6.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 7 Moving Beyond Linearity 289 7.1 Polynomial Regression . . . . . . . . . . . . . . . . . . . . . 290 7.2 Step Functions . . . . . . . . . . . . . . . . . . . . . . . . . 292 7.3 Basis Functions . . . . . . . . . . . . . . . . . . . . . . . . 294
- 8. 7.4 Regression Splines . . . . . . . . . . . . . . . . . . . . . . . 295 7.4.1 Piecewise Polynomials . . . . . . . . . . . . . . . . 295 7.4.2 Constraints and Splines . . . . . . . . . . . . . . . 295 7.4.3 The Spline Basis Representation . . . . . . . . . . 297 7.4.4 Choosing the Number and Locations of the Knots . . . . . . . . . . . . . . . . . . . . . . 298 7.4.5 Comparison to Polynomial Regression . . . . . . . 300 7.5 Smoothing Splines . . . . . . . . . . . . . . . . . . . . . . . 301 7.5.1 An Overview of Smoothing Splines . . . . . . . . . 301 7.5.2 Choosing the Smoothing Parameter λ . . . . . . . 302 7.6 Local Regression . . . . . . . . . . . . . . . . . . . . . . . . 304 7.7 Generalized Additive Models . . . . . . . . . . . . . . . . . 306 7.7.1 GAMs for Regression Problems . . . . . . . . . . . 307 7.7.2 GAMs for Classification Problems . . . . . . . . . . 310 7.8 Lab: Non-linear Modeling . . . . . . . . . . . . . . . . . . . 311 7.8.1 Polynomial Regression and Step Functions . . . . . 312 7.8.2 Splines . . . . . . . . . . . . . . . . . . . . . . . . . 317 7.8.3 GAMs . . . . . . . . . . . . . . . . . . . . . . . . . 318 7.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 8 Tree-Based Methods 327 8.1 The Basics of Decision Trees . . . . . . . . . . . . . . . . . 327 8.1.1 Regression Trees . . . . . . . . . . . . . . . . . . . 328 8.1.2 Classification Trees . . . . . . . . . . . . . . . . . . 335 8.1.3 Trees Versus Linear Models . . . . . . . . . . . . . 338 8.1.4 Advantages and Disadvantages of Trees . . . . . . 339 8.2 Bagging, Random Forests, Boosting, and Bayesian Additive Regression Trees . . . . . . . . . . . . . . . . . . . . . . . . 340 8.2.1 Bagging . . . . . . . . . . . . . . . . . . . . . . . . 340 8.2.2 Random Forests . . . . . . . . . . . . . . . . . . . . 343 8.2.3 Boosting . . . . . . . . . . . . . . . . . . . . . . . . 345 8.2.4 Bayesian Additive Regression Trees . . . . . . . . . 348 8.2.5 Summary of Tree Ensemble Methods . . . . . . . . 351 8.3 Lab: Decision Trees . . . . . . . . . . . . . . . . . . . . . . 353 8.3.1 Fitting Classification Trees . . . . . . . . . . . . . . 353 8.3.2 Fitting Regression Trees . . . . . . . . . . . . . . . 356 8.3.3 Bagging and Random Forests . . . . . . . . . . . . 357 8.3.4 Boosting . . . . . . . . . . . . . . . . . . . . . . . . 359 8.3.5 Bayesian Additive Regression Trees . . . . . . . . . 360 8.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 9 Support Vector Machines 367 9.1 Maximal Margin Classifier . . . . . . . . . . . . . . . . . . 368 9.1.1 What Is a Hyperplane? . . . . . . . . . . . . . . . . 368 9.1.2 Classification Using a Separating Hyperplane . . . 369 xii CONTENTS
- 9. CONTENTS xiii 9.1.3 The Maximal Margin Classifier . . . . . . . . . . . 371 9.1.4 Construction of the Maximal Margin Classifier . . 372 9.1.5 The Non-separable Case . . . . . . . . . . . . . . . 373 9.2 Support Vector Classifiers . . . . . . . . . . . . . . . . . . . 373 9.2.1 Overview of the Support Vector Classifier . . . . . 373 9.2.2 Details of the Support Vector Classifier . . . . . . . 375 9.3 Support Vector Machines . . . . . . . . . . . . . . . . . . . 379 9.3.1 Classification with Non-Linear Decision Boundaries . . . . . . . . . . . . . . . . . . . . . . 379 9.3.2 The Support Vector Machine . . . . . . . . . . . . 380 9.3.3 An Application to the Heart Disease Data . . . . . 383 9.4 SVMs with More than Two Classes . . . . . . . . . . . . . 385 9.4.1 One-Versus-One Classification . . . . . . . . . . . . 385 9.4.2 One-Versus-All Classification . . . . . . . . . . . . 385 9.5 Relationship to Logistic Regression . . . . . . . . . . . . . 386 9.6 Lab: Support Vector Machines . . . . . . . . . . . . . . . . 388 9.6.1 Support Vector Classifier . . . . . . . . . . . . . . . 389 9.6.2 Support Vector Machine . . . . . . . . . . . . . . . 392 9.6.3 ROC Curves . . . . . . . . . . . . . . . . . . . . . . 394 9.6.4 SVM with Multiple Classes . . . . . . . . . . . . . 396 9.6.5 Application to Gene Expression Data . . . . . . . . 396 9.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 10 Deep Learning 403 10.1 Single Layer Neural Networks . . . . . . . . . . . . . . . . 404 10.2 Multilayer Neural Networks . . . . . . . . . . . . . . . . . . 407 10.3 Convolutional Neural Networks . . . . . . . . . . . . . . . . 411 10.3.1 Convolution Layers . . . . . . . . . . . . . . . . . . 412 10.3.2 Pooling Layers . . . . . . . . . . . . . . . . . . . . 415 10.3.3 Architecture of a Convolutional Neural Network . . 415 10.3.4 Data Augmentation . . . . . . . . . . . . . . . . . . 417 10.3.5 Results Using a Pretrained Classifier . . . . . . . . 417 10.4 Document Classification . . . . . . . . . . . . . . . . . . . . 419 10.5 Recurrent Neural Networks . . . . . . . . . . . . . . . . . . 421 10.5.1 Sequential Models for Document Classification . . 424 10.5.2 Time Series Forecasting . . . . . . . . . . . . . . . 427 10.5.3 Summary of RNNs . . . . . . . . . . . . . . . . . . 431 10.6 When to Use Deep Learning . . . . . . . . . . . . . . . . . 432 10.7 Fitting a Neural Network . . . . . . . . . . . . . . . . . . . 434 10.7.1 Backpropagation . . . . . . . . . . . . . . . . . . . 435 10.7.2 Regularization and Stochastic Gradient Descent . . 436 10.7.3 Dropout Learning . . . . . . . . . . . . . . . . . . . 438 10.7.4 Network Tuning . . . . . . . . . . . . . . . . . . . . 438 10.8 Interpolation and Double Descent . . . . . . . . . . . . . . 439 10.9 Lab: Deep Learning . . . . . . . . . . . . . . . . . . . . . . 443
- 10. 10.9.1 A Single Layer Network on the Hitters Data . . . . 443 10.9.2 A Multilayer Network on the MNIST Digit Data . 445 10.9.3 Convolutional Neural Networks . . . . . . . . . . . 448 10.9.4 Using Pretrained CNN Models . . . . . . . . . . . 451 10.9.5 IMDb Document Classification . . . . . . . . . . . 452 10.9.6 Recurrent Neural Networks . . . . . . . . . . . . . 454 10.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458 11 Survival Analysis and Censored Data 461 11.1 Survival and Censoring Times . . . . . . . . . . . . . . . . 462 11.2 A Closer Look at Censoring . . . . . . . . . . . . . . . . . . 463 11.3 The Kaplan-Meier Survival Curve . . . . . . . . . . . . . . 464 11.4 The Log-Rank Test . . . . . . . . . . . . . . . . . . . . . . 466 11.5 Regression Models With a Survival Response . . . . . . . . 469 11.5.1 The Hazard Function . . . . . . . . . . . . . . . . . 469 11.5.2 Proportional Hazards . . . . . . . . . . . . . . . . . 471 11.5.3 Example: Brain Cancer Data . . . . . . . . . . . . 475 11.5.4 Example: Publication Data . . . . . . . . . . . . . 475 11.6 Shrinkage for the Cox Model . . . . . . . . . . . . . . . . . 478 11.7 Additional Topics . . . . . . . . . . . . . . . . . . . . . . . 480 11.7.1 Area Under the Curve for Survival Analysis . . . . 480 11.7.2 Choice of Time Scale . . . . . . . . . . . . . . . . . 481 11.7.3 Time-Dependent Covariates . . . . . . . . . . . . . 481 11.7.4 Checking the Proportional Hazards Assumption . . 482 11.7.5 Survival Trees . . . . . . . . . . . . . . . . . . . . . 482 11.8 Lab: Survival Analysis . . . . . . . . . . . . . . . . . . . . . 483 11.8.1 Brain Cancer Data . . . . . . . . . . . . . . . . . . 483 11.8.2 Publication Data . . . . . . . . . . . . . . . . . . . 486 11.8.3 Call Center Data . . . . . . . . . . . . . . . . . . . 487 11.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490 12 Unsupervised Learning 497 12.1 The Challenge of Unsupervised Learning . . . . . . . . . . 497 12.2 Principal Components Analysis . . . . . . . . . . . . . . . . 498 12.2.1 What Are Principal Components? . . . . . . . . . . 499 12.2.2 Another Interpretation of Principal Components . 503 12.2.3 The Proportion of Variance Explained . . . . . . . 505 12.2.4 More on PCA . . . . . . . . . . . . . . . . . . . . . 507 12.2.5 Other Uses for Principal Components . . . . . . . . 510 12.3 Missing Values and Matrix Completion . . . . . . . . . . . 510 12.4 Clustering Methods . . . . . . . . . . . . . . . . . . . . . . 516 12.4.1 K-Means Clustering . . . . . . . . . . . . . . . . . 517 12.4.2 Hierarchical Clustering . . . . . . . . . . . . . . . . 521 12.4.3 Practical Issues in Clustering . . . . . . . . . . . . 530 12.5 Lab: Unsupervised Learning . . . . . . . . . . . . . . . . . 532 xiv CONTENTS
- 11. CONTENTS xv 12.5.1 Principal Components Analysis . . . . . . . . . . . 532 12.5.2 Matrix Completion . . . . . . . . . . . . . . . . . . 535 12.5.3 Clustering . . . . . . . . . . . . . . . . . . . . . . . 538 12.5.4 NCI60 Data Example . . . . . . . . . . . . . . . . . 542 12.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 548 13 Multiple Testing 553 13.1 A Quick Review of Hypothesis Testing . . . . . . . . . . . 554 13.1.1 Testing a Hypothesis . . . . . . . . . . . . . . . . . 555 13.1.2 Type I and Type II Errors . . . . . . . . . . . . . . 559 13.2 The Challenge of Multiple Testing . . . . . . . . . . . . . . 560 13.3 The Family-Wise Error Rate . . . . . . . . . . . . . . . . . 561 13.3.1 What is the Family-Wise Error Rate? . . . . . . . 562 13.3.2 Approaches to Control the Family-Wise Error Rate 564 13.3.3 Trade-Off Between the FWER and Power . . . . . 570 13.4 The False Discovery Rate . . . . . . . . . . . . . . . . . . . 571 13.4.1 Intuition for the False Discovery Rate . . . . . . . 571 13.4.2 The Benjamini-Hochberg Procedure . . . . . . . . 573 13.5 A Re-Sampling Approach to p-Values and False Discovery Rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575 13.5.1 A Re-Sampling Approach to the p-Value . . . . . . 576 13.5.2 A Re-Sampling Approach to the False Discovery Rate578 13.5.3 When Are Re-Sampling Approaches Useful? . . . . 581 13.6 Lab: Multiple Testing . . . . . . . . . . . . . . . . . . . . . 582 13.6.1 Review of Hypothesis Tests . . . . . . . . . . . . . 582 13.6.2 The Family-Wise Error Rate . . . . . . . . . . . . . 583 13.6.3 The False Discovery Rate . . . . . . . . . . . . . . 586 13.6.4 A Re-Sampling Approach . . . . . . . . . . . . . . 588 13.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591 Index 597
- 12. 1 Introduction An Overview of Statistical Learning Statistical learning refers to a vast set of tools for understanding data. These tools can be classified as supervised or unsupervised. Broadly speaking, supervised statistical learning involves building a statistical model for pre- dicting, or estimating, an output based on one or more inputs. Problems of this nature occur in fields as diverse as business, medicine, astrophysics, and public policy. With unsupervised statistical learning, there are inputs but no supervising output; nevertheless we can learn relationships and struc- ture from such data. To provide an illustration of some applications of statistical learning, we briefly discuss three real-world data sets that are considered in this book. Wage Data In this application (which we refer to as the Wage data set throughout this book), we examine a number of factors that relate to wages for a group of men from the Atlantic region of the United States. In particular, we wish to understand the association between an employee’s age and education, as well as the calendar year, on his wage. Consider, for example, the left-hand panel of Figure 1.1, which displays wage versus age for each of the individu- als in the data set. There is evidence that wage increases with age but then decreases again after approximately age 60. The blue line, which provides an estimate of the average wage for a given age, makes this trend clearer. © Springer Science+Business Media, LLC, part of Springer Nature 2021 G. James et al., An Introduction to Statistical Learning, Springer Texts in Statistics, https://doi.org/10.1007/978-1-0716-1418-1_1 1
- 13. 2 1. Introduction 20 40 60 80 50 100 200 300 Age Wage 2003 2006 2009 50 100 200 300 Year Wage 1 2 3 4 5 50 100 200 300 Education Level Wage FIGURE 1.1. Wage data, which contains income survey information for men from the central Atlantic region of the United States. Left: wage as a function of age. On average, wage increases with age until about 60 years of age, at which point it begins to decline. Center: wage as a function of year. There is a slow but steady increase of approximately $10,000 in the average wage between 2003 and 2009. Right: Boxplots displaying wage as a function of education, with 1 indicating the lowest level (no high school diploma) and 5 the highest level (an advanced graduate degree). On average, wage increases with the level of education. Given an employee’s age, we can use this curve to predict his wage. However, it is also clear from Figure 1.1 that there is a significant amount of vari- ability associated with this average value, and so age alone is unlikely to provide an accurate prediction of a particular man’s wage. We also have information regarding each employee’s education level and the year in which the wage was earned. The center and right-hand panels of Figure 1.1, which display wage as a function of both year and education, in- dicate that both of these factors are associated with wage. Wages increase by approximately $10,000, in a roughly linear (or straight-line) fashion, between 2003 and 2009, though this rise is very slight relative to the vari- ability in the data. Wages are also typically greater for individuals with higher education levels: men with the lowest education level (1) tend to have substantially lower wages than those with the highest education level (5). Clearly, the most accurate prediction of a given man’s wage will be obtained by combining his age, his education, and the year. In Chapter 3, we discuss linear regression, which can be used to predict wage from this data set. Ideally, we should predict wage in a way that accounts for the non-linear relationship between wage and age. In Chapter 7, we discuss a class of approaches for addressing this problem.
- 14. 1. Introduction 3 Down Up −4 −2 0 2 4 6 Yesterday Today’s Direction Percentage change in S&P Down Up −4 −2 0 2 4 6 Two Days Previous Today’s Direction Percentage change in S&P Down Up −4 −2 0 2 4 6 Three Days Previous Today’s Direction Percentage change in S&P FIGURE 1.2. Left: Boxplots of the previous day’s percentage change in the S&P index for the days for which the market increased or decreased, obtained from the Smarket data. Center and Right: Same as left panel, but the percentage changes for 2 and 3 days previous are shown. Stock Market Data The Wage data involves predicting a continuous or quantitative output value. This is often referred to as a regression problem. However, in certain cases we may instead wish to predict a non-numerical value—that is, a categorical or qualitative output. For example, in Chapter 4 we examine a stock market data set that contains the daily movements in the Standard & Poor’s 500 (S&P) stock index over a 5-year period between 2001 and 2005. We refer to this as the Smarket data. The goal is to predict whether the index will increase or decrease on a given day, using the past 5 days’ percentage changes in the index. Here the statistical learning problem does not involve predicting a numerical value. Instead it involves predicting whether a given day’s stock market performance will fall into the Up bucket or the Down bucket. This is known as a classification problem. A model that could accurately predict the direction in which the market will move would be very useful! The left-hand panel of Figure 1.2 displays two boxplots of the previous day’s percentage changes in the stock index: one for the 648 days for which the market increased on the subsequent day, and one for the 602 days for which the market decreased. The two plots look almost identical, suggest- ing that there is no simple strategy for using yesterday’s movement in the S&P to predict today’s returns. The remaining panels, which display box- plots for the percentage changes 2 and 3 days previous to today, similarly indicate little association between past and present returns. Of course, this lack of pattern is to be expected: in the presence of strong correlations be- tween successive days’ returns, one could adopt a simple trading strategy
- 15. 4 1. Introduction Down Up 0.46 0.48 0.50 0.52 Today’s Direction Predicted Probability FIGURE 1.3. We fit a quadratic discriminant analysis model to the subset of the Smarket data corresponding to the 2001–2004 time period, and predicted the probability of a stock market decrease using the 2005 data. On average, the predicted probability of decrease is higher for the days in which the market does decrease. Based on these results, we are able to correctly predict the direction of movement in the market 60% of the time. to generate profits from the market. Nevertheless, in Chapter 4, we explore these data using several different statistical learning methods. Interestingly, there are hints of some weak trends in the data that suggest that, at least for this 5-year period, it is possible to correctly predict the direction of movement in the market approximately 60% of the time (Figure 1.3). Gene Expression Data The previous two applications illustrate data sets with both input and output variables. However, another important class of problems involves situations in which we only observe input variables, with no corresponding output. For example, in a marketing setting, we might have demographic information for a number of current or potential customers. We may wish to understand which types of customers are similar to each other by grouping individuals according to their observed characteristics. This is known as a clustering problem. Unlike in the previous examples, here we are not trying to predict an output variable. We devote Chapter 12 to a discussion of statistical learning methods for problems in which no natural output variable is available. We consider the NCI60 data set, which consists of 6,830 gene expression measurements for each of 64 cancer cell lines. Instead of predicting a particular output variable, we are interested in determining whether there are groups, or clusters, among the cell lines based on their gene expression measurements. This is a difficult question to address, in part because there are thousands of gene expression measurements per cell line, making it hard to visualize the data.
- 16. 1. Introduction 5 −40 −20 0 20 40 60 −60 −40 −20 0 20 −40 −20 0 20 40 60 −60 −40 −20 0 20 Z1 Z1 Z 2 Z 2 FIGURE 1.4. Left: Representation of the NCI60 gene expression data set in a two-dimensional space, Z1 and Z2. Each point corresponds to one of the 64 cell lines. There appear to be four groups of cell lines, which we have represented using different colors. Right: Same as left panel except that we have represented each of the 14 different types of cancer using a different colored symbol. Cell lines corresponding to the same cancer type tend to be nearby in the two-dimensional space. The left-hand panel of Figure 1.4 addresses this problem by represent- ing each of the 64 cell lines using just two numbers, Z1 and Z2. These are the first two principal components of the data, which summarize the 6,830 expression measurements for each cell line down to two numbers or dimensions. While it is likely that this dimension reduction has resulted in some loss of information, it is now possible to visually examine the data for evidence of clustering. Deciding on the number of clusters is often a difficult problem. But the left-hand panel of Figure 1.4 suggests at least four groups of cell lines, which we have represented using separate colors. In this particular data set, it turns out that the cell lines correspond to 14 different types of cancer. (However, this information was not used to create the left-hand panel of Figure 1.4.) The right-hand panel of Fig- ure 1.4 is identical to the left-hand panel, except that the 14 cancer types are shown using distinct colored symbols. There is clear evidence that cell lines with the same cancer type tend to be located near each other in this two-dimensional representation. In addition, even though the cancer infor- mation was not used to produce the left-hand panel, the clustering obtained does bear some resemblance to some of the actual cancer types observed in the right-hand panel. This provides some independent verification of the accuracy of our clustering analysis.
- 17. 6 1. Introduction A Brief History of Statistical Learning Though the term statistical learning is fairly new, many of the concepts that underlie the field were developed long ago. At the beginning of the nine- teenth century, the method of least squares was developed, implementing the earliest form of what is now known as linear regression. The approach was first successfully applied to problems in astronomy. Linear regression is used for predicting quantitative values, such as an individual’s salary. In order to predict qualitative values, such as whether a patient survives or dies, or whether the stock market increases or decreases, linear discrim- inant analysis was proposed in 1936. In the 1940s, various authors put forth an alternative approach, logistic regression. In the early 1970s, the term generalized linear model was developed to describe an entire class of statistical learning methods that include both linear and logistic regression as special cases. By the end of the 1970s, many more techniques for learning from data were available. However, they were almost exclusively linear methods be- cause fitting non-linear relationships was computationally difficult at the time. By the 1980s, computing technology had finally improved sufficiently that non-linear methods were no longer computationally prohibitive. In the mid 1980s, classification and regression trees were developed, followed shortly by generalized additive models. Neural networks gained popularity in the 1980s, and support vector machines arose in the 1990s. Since that time, statistical learning has emerged as a new subfield in statistics, focused on supervised and unsupervised modeling and prediction. In recent years, progress in statistical learning has been marked by the increasing availability of powerful and relatively user-friendly software, such as the popular and freely available R system. This has the potential to continue the transformation of the field from a set of techniques used and developed by statisticians and computer scientists to an essential toolkit for a much broader community. This Book The Elements of Statistical Learning (ESL) by Hastie, Tibshirani, and Friedman was first published in 2001. Since that time, it has become an important reference on the fundamentals of statistical machine learning. Its success derives from its comprehensive and detailed treatment of many important topics in statistical learning, as well as the fact that (relative to many upper-level statistics textbooks) it is accessible to a wide audience. However, the greatest factor behind the success of ESL has been its topical nature. At the time of its publication, interest in the field of statistical
- 18. 1. Introduction 7 learning was starting to explode. ESL provided one of the first accessible and comprehensive introductions to the topic. Since ESL was first published, the field of statistical learning has con- tinued to flourish. The field’s expansion has taken two forms. The most obvious growth has involved the development of new and improved statis- tical learning approaches aimed at answering a range of scientific questions across a number of fields. However, the field of statistical learning has also expanded its audience. In the 1990s, increases in computational power generated a surge of interest in the field from non-statisticians who were eager to use cutting-edge statistical tools to analyze their data. Unfortu- nately, the highly technical nature of these approaches meant that the user community remained primarily restricted to experts in statistics, computer science, and related fields with the training (and time) to understand and implement them. In recent years, new and improved software packages have significantly eased the implementation burden for many statistical learning methods. At the same time, there has been growing recognition across a number of fields, from business to health care to genetics to the social sciences and beyond, that statistical learning is a powerful tool with important practical applications. As a result, the field has moved from one of primarily academic interest to a mainstream discipline, with an enormous potential audience. This trend will surely continue with the increasing availability of enormous quantities of data and the software to analyze it. The purpose of An Introduction to Statistical Learning (ISL) is to facili- tate the transition of statistical learning from an academic to a mainstream field. ISL is not intended to replace ESL, which is a far more comprehen- sive text both in terms of the number of approaches considered and the depth to which they are explored. We consider ESL to be an important companion for professionals (with graduate degrees in statistics, machine learning, or related fields) who need to understand the technical details behind statistical learning approaches. However, the community of users of statistical learning techniques has expanded to include individuals with a wider range of interests and backgrounds. Therefore, there is a place for a less technical and more accessible version of ESL. In teaching these topics over the years, we have discovered that they are of interest to master’s and PhD students in fields as disparate as business administration, biology, and computer science, as well as to quantitatively- oriented upper-division undergraduates. It is important for this diverse group to be able to understand the models, intuitions, and strengths and weaknesses of the various approaches. But for this audience, many of the technical details behind statistical learning methods, such as optimiza- tion algorithms and theoretical properties, are not of primary interest. We believe that these students do not need a deep understanding of these aspects in order to become informed users of the various methodologies, and
- 19. 8 1. Introduction in order to contribute to their chosen fields through the use of statistical learning tools. ISL is based on the following four premises. 1. Many statistical learning methods are relevant and useful in a wide range of academic and non-academic disciplines, beyond just the sta- tistical sciences. We believe that many contemporary statistical learn- ing procedures should, and will, become as widely available and used as is currently the case for classical methods such as linear regres- sion. As a result, rather than attempting to consider every possible approach (an impossible task), we have concentrated on presenting the methods that we believe are most widely applicable. 2. Statistical learning should not be viewed as a series of black boxes. No single approach will perform well in all possible applications. With- out understanding all of the cogs inside the box, or the interaction between those cogs, it is impossible to select the best box. Hence, we have attempted to carefully describe the model, intuition, assump- tions, and trade-offs behind each of the methods that we consider. 3. While it is important to know what job is performed by each cog, it is not necessary to have the skills to construct the machine inside the box! Thus, we have minimized discussion of technical details related to fitting procedures and theoretical properties. We assume that the reader is comfortable with basic mathematical concepts, but we do not assume a graduate degree in the mathematical sciences. For in- stance, we have almost completely avoided the use of matrix algebra, and it is possible to understand the entire book without a detailed knowledge of matrices and vectors. 4. We presume that the reader is interested in applying statistical learn- ing methods to real-world problems. In order to facilitate this, as well as to motivate the techniques discussed, we have devoted a section within each chapter to R computer labs. In each lab, we walk the reader through a realistic application of the methods considered in that chapter. When we have taught this material in our courses, we have allocated roughly one-third of classroom time to working through the labs, and we have found them to be extremely useful. Many of the less computationally-oriented students who were ini- tially intimidated by R’s command level interface got the hang of things over the course of the quarter or semester. We have used R because it is freely available and is powerful enough to implement all of the methods discussed in the book. It also has optional packages that can be downloaded to implement literally thousands of addi- tional methods. Most importantly, R is the language of choice for academic statisticians, and new approaches often become available in
- 20. 1. Introduction 9 R years before they are implemented in commercial packages. How- ever, the labs in ISL are self-contained, and can be skipped if the reader wishes to use a different software package or does not wish to apply the methods discussed to real-world problems. Who Should Read This Book? This book is intended for anyone who is interested in using modern statis- tical methods for modeling and prediction from data. This group includes scientists, engineers, data analysts, data scientists, and quants, but also less technical individuals with degrees in non-quantitative fields such as the social sciences or business. We expect that the reader will have had at least one elementary course in statistics. Background in linear regression is also useful, though not required, since we review the key concepts behind linear regression in Chapter 3. The mathematical level of this book is mod- est, and a detailed knowledge of matrix operations is not required. This book provides an introduction to the statistical programming language R. Previous exposure to a programming language, such as MATLAB or Python, is useful but not required. The first edition of this textbook has been used as to teach master’s and PhD students in business, economics, computer science, biology, earth sci- ences, psychology, and many other areas of the physical and social sciences. It has also been used to teach advanced undergraduates who have already taken a course on linear regression. In the context of a more mathemat- ically rigorous course in which ESL serves as the primary textbook, ISL could be used as a supplementary text for teaching computational aspects of the various approaches. Notation and Simple Matrix Algebra Choosing notation for a textbook is always a difficult task. For the most part we adopt the same notational conventions as ESL. We will use n to represent the number of distinct data points, or observa- tions, in our sample. We will let p denote the number of variables that are available for use in making predictions. For example, the Wage data set con- sists of 11 variables for 3,000 people, so we have n = 3,000 observations and p = 11 variables (such as year, age, race, and more). Note that throughout this book, we indicate variable names using colored font: Variable Name. In some examples, p might be quite large, such as on the order of thou- sands or even millions; this situation arises quite often, for example, in the analysis of modern biological data or web-based advertising data.
- 21. 10 1. Introduction In general, we will let xij represent the value of the jth variable for the ith observation, where i = 1, 2, . . . , n and j = 1, 2, . . . , p. Throughout this book, i will be used to index the samples or observations (from 1 to n) and j will be used to index the variables (from 1 to p). We let X denote an n × p matrix whose (i, j)th element is xij. That is, X = ⎛ ⎜ ⎜ ⎜ ⎝ x11 x12 . . . x1p x21 x22 . . . x2p . . . . . . ... . . . xn1 xn2 . . . xnp ⎞ ⎟ ⎟ ⎟ ⎠ . For readers who are unfamiliar with matrices, it is useful to visualize X as a spreadsheet of numbers with n rows and p columns. At times we will be interested in the rows of X, which we write as x1, x2, . . . , xn. Here xi is a vector of length p, containing the p variable measurements for the ith observation. That is, xi = ⎛ ⎜ ⎜ ⎜ ⎝ xi1 xi2 . . . xip ⎞ ⎟ ⎟ ⎟ ⎠ . (1.1) (Vectors are by default represented as columns.) For example, for the Wage data, xi is a vector of length 11, consisting of year, age, race, and other values for the ith individual. At other times we will instead be interested in the columns of X, which we write as x1, x2, . . . , xp. Each is a vector of length n. That is, xj = ⎛ ⎜ ⎜ ⎜ ⎝ x1j x2j . . . xnj ⎞ ⎟ ⎟ ⎟ ⎠ . For example, for the Wage data, x1 contains the n = 3,000 values for year. Using this notation, the matrix X can be written as X = ' x1 x2 · · · xp ( , or X = ⎛ ⎜ ⎜ ⎜ ⎝ xT 1 xT 2 . . . xT n ⎞ ⎟ ⎟ ⎟ ⎠ .
- 22. 1. Introduction 11 The T notation denotes the transpose of a matrix or vector. So, for example, XT = ⎛ ⎜ ⎜ ⎜ ⎝ x11 x21 . . . xn1 x12 x22 . . . xn2 . . . . . . . . . x1p x2p . . . xnp ⎞ ⎟ ⎟ ⎟ ⎠ , while xT i = ' xi1 xi2 · · · xip ( . We use yi to denote the ith observation of the variable on which we wish to make predictions, such as wage. Hence, we write the set of all n observations in vector form as y = ⎛ ⎜ ⎜ ⎜ ⎝ y1 y2 . . . yn ⎞ ⎟ ⎟ ⎟ ⎠ . Then our observed data consists of {(x1, y1), (x2, y2), . . . , (xn, yn)}, where each xi is a vector of length p. (If p = 1, then xi is simply a scalar.) In this text, a vector of length n will always be denoted in lower case bold; e.g. a = ⎛ ⎜ ⎜ ⎜ ⎝ a1 a2 . . . an ⎞ ⎟ ⎟ ⎟ ⎠ . However, vectors that are not of length n (such as feature vectors of length p, as in (1.1)) will be denoted in lower case normal font, e.g. a. Scalars will also be denoted in lower case normal font, e.g. a. In the rare cases in which these two uses for lower case normal font lead to ambiguity, we will clarify which use is intended. Matrices will be denoted using bold capitals, such as A. Random variables will be denoted using capital normal font, e.g. A, regardless of their dimensions. Occasionally we will want to indicate the dimension of a particular ob- ject. To indicate that an object is a scalar, we will use the notation a ∈ R. To indicate that it is a vector of length k, we will use a ∈ Rk (or a ∈ Rn if it is of length n). We will indicate that an object is an r × s matrix using A ∈ Rr×s . We have avoided using matrix algebra whenever possible. However, in a few instances it becomes too cumbersome to avoid it entirely. In these rare instances it is important to understand the concept of multiplying two matrices. Suppose that A ∈ Rr×d and B ∈ Rd×s . Then the product of A and B is denoted AB. The (i, j)th element of AB is computed by
- 23. 12 1. Introduction multiplying each element of the ith row of A by the corresponding element of the jth column of B. That is, (AB)ij = )d k=1 aikbkj. As an example, consider A = * 1 2 3 4 + and B = * 5 6 7 8 + . Then AB = * 1 2 3 4 + * 5 6 7 8 + = * 1 × 5 + 2 × 7 1 × 6 + 2 × 8 3 × 5 + 4 × 7 3 × 6 + 4 × 8 + = * 19 22 43 50 + . Note that this operation produces an r × s matrix. It is only possible to compute AB if the number of columns of A is the same as the number of rows of B. Organization of This Book Chapter 2 introduces the basic terminology and concepts behind statisti- cal learning. This chapter also presents the K-nearest neighbor classifier, a very simple method that works surprisingly well on many problems. Chap- ters 3 and 4 cover classical linear methods for regression and classification. In particular, Chapter 3 reviews linear regression, the fundamental start- ing point for all regression methods. In Chapter 4 we discuss two of the most important classical classification methods, logistic regression and lin- ear discriminant analysis. A central problem in all statistical learning situations involves choosing the best method for a given application. Hence, in Chapter 5 we intro- duce cross-validation and the bootstrap, which can be used to estimate the accuracy of a number of different methods in order to choose the best one. Much of the recent research in statistical learning has concentrated on non-linear methods. However, linear methods often have advantages over their non-linear competitors in terms of interpretability and sometimes also accuracy. Hence, in Chapter 6 we consider a host of linear methods, both classical and more modern, which offer potential improvements over stan- dard linear regression. These include stepwise selection, ridge regression, principal components regression, and the lasso. The remaining chapters move into the world of non-linear statistical learning. We first introduce in Chapter 7 a number of non-linear meth- ods that work well for problems with a single input variable. We then show how these methods can be used to fit non-linear additive models for which there is more than one input. In Chapter 8, we investigate tree-based methods, including bagging, boosting, and random forests. Support vector machines, a set of approaches for performing both linear and non-linear classification, are discussed in Chapter 9. We cover deep learning, an ap- proach for non-linear regression and classification that has received a lot
- 24. 1. Introduction 13 of attention in recent years, in Chapter 10. Chapter 11 explores survival analysis, a regression approach that is specialized to the setting in which the output variable is censored, i.e. not fully observed. In Chapter 12, we consider the unsupervised setting in which we have input variables but no output variable. In particular, we present princi- pal components analysis, K-means clustering, and hierarchical clustering. Finally, in Chapter 13 we cover the very important topic of multiple hy- pothesis testing. At the end of each chapter, we present one or more R lab sections in which we systematically work through applications of the various meth- ods discussed in that chapter. These labs demonstrate the strengths and weaknesses of the various approaches, and also provide a useful reference for the syntax required to implement the various methods. The reader may choose to work through the labs at his or her own pace, or the labs may be the focus of group sessions as part of a classroom environment. Within each R lab, we present the results that we obtained when we performed the lab at the time of writing this book. However, new versions of R are continuously released, and over time, the packages called in the labs will be updated. Therefore, in the future, it is possible that the results shown in the lab sections may no longer correspond precisely to the results obtained by the reader who performs the labs. As necessary, we will post updates to the labs on the book website. We use the symbol to denote sections or exercises that contain more challenging concepts. These can be easily skipped by readers who do not wish to delve as deeply into the material, or who lack the mathematical background. Data Sets Used in Labs and Exercises In this textbook, we illustrate statistical learning methods using applica- tions from marketing, finance, biology, and other areas. The ISLR2 package available on the book website and CRAN contains a number of data sets that are required in order to perform the labs and exercises associated with this book. One other data set is part of the base R distribution. Table 1.1 contains a summary of the data sets required to perform the labs and ex- ercises. A couple of these data sets are also available as text files on the book website, for use in Chapter 2.
- 25. 14 1. Introduction Name Description Auto Gas mileage, horsepower, and other information for cars. Bikeshare Hourly usage of a bike sharing program in Washington, DC. Boston Housing values and other information about Boston census tracts. BrainCancer Survival times for patients diagnosed with brain cancer. Caravan Information about individuals offered caravan insurance. Carseats Information about car seat sales in 400 stores. College Demographic characteristics, tuition, and more for USA colleges. Credit Information about credit card debt for 10,000 customers. Default Customer default records for a credit card company. Fund Returns of 2,000 hedge fund managers over 50 months. Hitters Records and salaries for baseball players. Khan Gene expression measurements for four cancer types. NCI60 Gene expression measurements for 64 cancer cell lines. NYSE Returns, volatility, and volume for the New York Stock Exchange. OJ Sales information for Citrus Hill and Minute Maid orange juice. Portfolio Past values of financial assets, for use in portfolio allocation. Publication Time to publication for 244 clinical trials. Smarket Daily percentage returns for S&P 500 over a 5-year period. USArrests Crime statistics per 100,000 residents in 50 states of USA. Wage Income survey data for men in central Atlantic region of USA. Weekly 1,089 weekly stock market returns for 21 years. TABLE 1.1. A list of data sets needed to perform the labs and exercises in this textbook. All data sets are available in the ISLR2 library, with the exception of USArrests, which is part of the base R distribution. Book Website The website for this book is located at It contains a number of resources, including the R package associated with this book, and some additional data sets. Acknowledgements A few of the plots in this book were taken from ESL: Figures 6.7, 8.3, and 12.14. All other plots are new to this book. www.statlearning.com
- 26. 2 Statistical Learning 2.1 What Is Statistical Learning? In order to motivate our study of statistical learning, we begin with a simple example. Suppose that we are statistical consultants hired by a client to investigate the association between advertising and sales of a particular product. The Advertising data set consists of the sales of that product in 200 different markets, along with advertising budgets for the product in each of those markets for three different media: TV, radio, and newspaper. The data are displayed in Figure 2.1. It is not possible for our client to directly increase sales of the product. On the other hand, they can control the advertising expenditure in each of the three media. Therefore, if we determine that there is an association between advertising and sales, then we can instruct our client to adjust advertising budgets, thereby indirectly increasing sales. In other words, our goal is to develop an accurate model that can be used to predict sales on the basis of the three media budgets. In this setting, the advertising budgets are input variables while sales input variable is an output variable. The input variables are typically denoted using the output variable symbol X, with a subscript to distinguish them. So X1 might be the TV budget, X2 the radio budget, and X3 the newspaper budget. The inputs go by different names, such as predictors, independent variables, features, predictor independent variable feature or sometimes just variables. The output variable—in this case, sales—is variable often called the response or dependent variable, and is typically denoted response dependent variable using the symbol Y . Throughout this book, we will use all of these terms interchangeably. © Springer Science+Business Media, LLC, part of Springer Nature 2021 G. James et al., An Introduction to Statistical Learning, Springer Texts in Statistics, https://doi.org/10.1007/978-1-0716-1418-1_2 15
- 27. 16 2. Statistical Learning 0 50 100 200 300 5 10 15 20 25 TV Sales 0 10 20 30 40 50 5 10 15 20 25 Radio Sales 0 20 40 60 80 100 5 10 15 20 25 Newspaper Sales FIGURE 2.1. The Advertising data set. The plot displays sales, in thousands of units, as a function of TV, radio, and newspaper budgets, in thousands of dollars, for 200 different markets. In each plot we show the simple least squares fit of sales to that variable, as described in Chapter 3. In other words, each blue line represents a simple model that can be used to predict sales using TV, radio, and newspaper, respectively. More generally, suppose that we observe a quantitative response Y and p different predictors, X1, X2, . . . , Xp. We assume that there is some relationship between Y and X = (X1, X2, . . . , Xp), which can be written in the very general form Y = f(X) + ϵ. (2.1) Here f is some fixed but unknown function of X1, . . . , Xp, and ϵ is a random error term, which is independent of X and has mean zero. In this formula- error term tion, f represents the systematic information that X provides about Y . systematic As another example, consider the left-hand panel of Figure 2.2, a plot of income versus years of education for 30 individuals in the Income data set. The plot suggests that one might be able to predict income using years of education. However, the function f that connects the input variable to the output variable is in general unknown. In this situation one must estimate f based on the observed points. Since Income is a simulated data set, f is known and is shown by the blue curve in the right-hand panel of Figure 2.2. The vertical lines represent the error terms ϵ. We note that some of the 30 observations lie above the blue curve and some lie below it; overall, the errors have approximately mean zero. In general, the function f may involve more than one input variable. In Figure 2.3 we plot income as a function of years of education and seniority. Here f is a two-dimensional surface that must be estimated based on the observed data.
- 28. 2.1 What Is Statistical Learning? 17 10 12 14 16 18 20 22 20 30 40 50 60 70 80 Years of Education Income 10 12 14 16 18 20 22 20 30 40 50 60 70 80 Years of Education Income FIGURE 2.2. The Income data set. Left: The red dots are the observed values of income (in tens of thousands of dollars) and years of education for 30 indi- viduals. Right: The blue curve represents the true underlying relationship between income and years of education, which is generally unknown (but is known in this case because the data were simulated). The black lines represent the error associated with each observation. Note that some errors are positive (if an ob- servation lies above the blue curve) and some are negative (if an observation lies below the curve). Overall, these errors have approximately mean zero. In essence, statistical learning refers to a set of approaches for estimating f. In this chapter we outline some of the key theoretical concepts that arise in estimating f, as well as tools for evaluating the estimates obtained. 2.1.1 Why Estimate f? There are two main reasons that we may wish to estimate f: prediction and inference. We discuss each in turn. Prediction In many situations, a set of inputs X are readily available, but the output Y cannot be easily obtained. In this setting, since the error term averages to zero, we can predict Y using Ŷ = ˆ f(X), (2.2) where ˆ f represents our estimate for f, and Ŷ represents the resulting pre- diction for Y . In this setting, ˆ f is often treated as a black box, in the sense that one is not typically concerned with the exact form of ˆ f, provided that it yields accurate predictions for Y . As an example, suppose that X1, . . . , Xp are characteristics of a patient’s blood sample that can be easily measured in a lab, and Y is a variable encoding the patient’s risk for a severe adverse reaction to a particular
- 29. 18 2. Statistical Learning Years of Education S e n i o r i t y In c o m e FIGURE 2.3. The plot displays income as a function of years of education and seniority in the Income data set. The blue surface represents the true un- derlying relationship between income and years of education and seniority, which is known since the data are simulated. The red dots indicate the observed values of these quantities for 30 individuals. drug. It is natural to seek to predict Y using X, since we can then avoid giving the drug in question to patients who are at high risk of an adverse reaction—that is, patients for whom the estimate of Y is high. The accuracy of Ŷ as a prediction for Y depends on two quantities, which we will call the reducible error and the irreducible error. In general, reducible error irreducible error ˆ f will not be a perfect estimate for f, and this inaccuracy will introduce some error. This error is reducible because we can potentially improve the accuracy of ˆ f by using the most appropriate statistical learning technique to estimate f. However, even if it were possible to form a perfect estimate for f, so that our estimated response took the form Ŷ = f(X), our prediction would still have some error in it! This is because Y is also a function of ϵ, which, by definition, cannot be predicted using X. Therefore, variability associated with ϵ also affects the accuracy of our predictions. This is known as the irreducible error, because no matter how well we estimate f, we cannot reduce the error introduced by ϵ. Why is the irreducible error larger than zero? The quantity ϵ may con- tain unmeasured variables that are useful in predicting Y : since we don’t measure them, f cannot use them for its prediction. The quantity ϵ may also contain unmeasurable variation. For example, the risk of an adverse reaction might vary for a given patient on a given day, depending on manufacturing variation in the drug itself or the patient’s general feeling of well-being on that day.
- 30. 2.1 What Is Statistical Learning? 19 Consider a given estimate ˆ f and a set of predictors X, which yields the prediction Ŷ = ˆ f(X). Assume for a moment that both ˆ f and X are fixed, so that the only variability comes from ϵ. Then, it is easy to show that E(Y − Ŷ )2 = E[f(X) + ϵ − ˆ f(X)]2 = [f(X) − ˆ f(X)]2 , -. / Reducible + Var(ϵ) , -. / Irreducible , (2.3) where E(Y − Ŷ )2 represents the average, or expected value, of the squared expected value difference between the predicted and actual value of Y , and Var(ϵ) repre- sents the variance associated with the error term ϵ. variance The focus of this book is on techniques for estimating f with the aim of minimizing the reducible error. It is important to keep in mind that the irreducible error will always provide an upper bound on the accuracy of our prediction for Y . This bound is almost always unknown in practice. Inference We are often interested in understanding the association between Y and X1, . . . , Xp. In this situation we wish to estimate f, but our goal is not necessarily to make predictions for Y . Now ˆ f cannot be treated as a black box, because we need to know its exact form. In this setting, one may be interested in answering the following questions: • Which predictors are associated with the response? It is often the case that only a small fraction of the available predictors are substantially associated with Y . Identifying the few important predictors among a large set of possible variables can be extremely useful, depending on the application. • What is the relationship between the response and each predictor? Some predictors may have a positive relationship with Y , in the sense that larger values of the predictor are associated with larger values of Y . Other predictors may have the opposite relationship. Depending on the complexity of f, the relationship between the response and a given predictor may also depend on the values of the other predictors. • Can the relationship between Y and each predictor be adequately sum- marized using a linear equation, or is the relationship more compli- cated? Historically, most methods for estimating f have taken a linear form. In some situations, such an assumption is reasonable or even de- sirable. But often the true relationship is more complicated, in which case a linear model may not provide an accurate representation of the relationship between the input and output variables. In this book, we will see a number of examples that fall into the prediction setting, the inference setting, or a combination of the two.
- 31. 20 2. Statistical Learning For instance, consider a company that is interested in conducting a direct-marketing campaign. The goal is to identify individuals who are likely to respond positively to a mailing, based on observations of demo- graphic variables measured on each individual. In this case, the demo- graphic variables serve as predictors, and response to the marketing cam- paign (either positive or negative) serves as the outcome. The company is not interested in obtaining a deep understanding of the relationships be- tween each individual predictor and the response; instead, the company simply wants to accurately predict the response using the predictors. This is an example of modeling for prediction. In contrast, consider the Advertising data illustrated in Figure 2.1. One may be interested in answering questions such as: – Which media are associated with sales? – Which media generate the biggest boost in sales? or – How large of an increase in sales is associated with a given increase in TV advertising? This situation falls into the inference paradigm. Another example involves modeling the brand of a product that a customer might purchase based on variables such as price, store location, discount levels, competition price, and so forth. In this situation one might really be most interested in the association between each variable and the probability of purchase. For in- stance, to what extent is the product’s price associated with sales? This is an example of modeling for inference. Finally, some modeling could be conducted both for prediction and in- ference. For example, in a real estate setting, one may seek to relate values of homes to inputs such as crime rate, zoning, distance from a river, air quality, schools, income level of community, size of houses, and so forth. In this case one might be interested in the association between each individ- ual input variable and housing price—for instance, how much extra will a house be worth if it has a view of the river? This is an inference problem. Alternatively, one may simply be interested in predicting the value of a home given its characteristics: is this house under- or over-valued? This is a prediction problem. Depending on whether our ultimate goal is prediction, inference, or a combination of the two, different methods for estimating f may be appro- priate. For example, linear models allow for relatively simple and inter- linear model pretable inference, but may not yield as accurate predictions as some other approaches. In contrast, some of the highly non-linear approaches that we discuss in the later chapters of this book can potentially provide quite accu- rate predictions for Y , but this comes at the expense of a less interpretable model for which inference is more challenging.
- 32. 2.1 What Is Statistical Learning? 21 2.1.2 How Do We Estimate f? Throughout this book, we explore many linear and non-linear approaches for estimating f. However, these methods generally share certain charac- teristics. We provide an overview of these shared characteristics in this section. We will always assume that we have observed a set of n different data points. For example in Figure 2.2 we observed n = 30 data points. These observations are called the training data because we will use these training data observations to train, or teach, our method how to estimate f. Let xij represent the value of the jth predictor, or input, for observation i, where i = 1, 2, . . . , n and j = 1, 2, . . . , p. Correspondingly, let yi represent the response variable for the ith observation. Then our training data consist of {(x1, y1), (x2, y2), . . . , (xn, yn)} where xi = (xi1, xi2, . . . , xip)T . Our goal is to apply a statistical learning method to the training data in order to estimate the unknown function f. In other words, we want to find a function ˆ f such that Y ≈ ˆ f(X) for any observation (X, Y ). Broadly speaking, most statistical learning methods for this task can be character- ized as either parametric or non-parametric. We now briefly discuss these parametric non- parametric two types of approaches. Parametric Methods Parametric methods involve a two-step model-based approach. 1. First, we make an assumption about the functional form, or shape, of f. For example, one very simple assumption is that f is linear in X: f(X) = β0 + β1X1 + β2X2 + · · · + βpXp. (2.4) This is a linear model, which will be discussed extensively in Chap- ter 3. Once we have assumed that f is linear, the problem of estimat- ing f is greatly simplified. Instead of having to estimate an entirely arbitrary p-dimensional function f(X), one only needs to estimate the p + 1 coefficients β0, β1, . . . , βp. 2. After a model has been selected, we need a procedure that uses the training data to fit or train the model. In the case of the linear model fit train (2.4), we need to estimate the parameters β0, β1, . . . , βp. That is, we want to find values of these parameters such that Y ≈ β0 + β1X1 + β2X2 + · · · + βpXp. The most common approach to fitting the model (2.4) is referred to as (ordinary) least squares, which we discuss in Chapter 3. However, least squares least squares is one of many possible ways to fit the linear model. In Chapter 6, we discuss other approaches for estimating the parameters in (2.4).
- 33. 22 2. Statistical Learning Years of Education S e n i o r i t y In c o m e FIGURE 2.4. A linear model fit by least squares to the Income data from Fig- ure 2.3. The observations are shown in red, and the yellow plane indicates the least squares fit to the data. The model-based approach just described is referred to as parametric; it reduces the problem of estimating f down to one of estimating a set of parameters. Assuming a parametric form for f simplifies the problem of estimating f because it is generally much easier to estimate a set of pa- rameters, such as β0, β1, . . . , βp in the linear model (2.4), than it is to fit an entirely arbitrary function f. The potential disadvantage of a paramet- ric approach is that the model we choose will usually not match the true unknown form of f. If the chosen model is too far from the true f, then our estimate will be poor. We can try to address this problem by choos- ing flexible models that can fit many different possible functional forms flexible for f. But in general, fitting a more flexible model requires estimating a greater number of parameters. These more complex models can lead to a phenomenon known as overfitting the data, which essentially means they overfitting follow the errors, or noise, too closely. These issues are discussed through- noise out this book. Figure 2.4 shows an example of the parametric approach applied to the Income data from Figure 2.3. We have fit a linear model of the form income ≈ β0 + β1 × education + β2 × seniority. Since we have assumed a linear relationship between the response and the two predictors, the entire fitting problem reduces to estimating β0, β1, and β2, which we do using least squares linear regression. Comparing Figure 2.3 to Figure 2.4, we can see that the linear fit given in Figure 2.4 is not quite right: the true f has some curvature that is not captured in the linear fit. However, the linear fit still appears to do a reasonable job of capturing the
- 34. 2.1 What Is Statistical Learning? 23 Years of Education S e n i o r i t y In c o m e FIGURE 2.5. A smooth thin-plate spline fit to the Income data from Figure 2.3 is shown in yellow; the observations are displayed in red. Splines are discussed in Chapter 7. positive relationship between years of education and income, as well as the slightly less positive relationship between seniority and income. It may be that with such a small number of observations, this is the best we can do. Non-Parametric Methods Non-parametric methods do not make explicit assumptions about the func- tional form of f. Instead they seek an estimate of f that gets as close to the data points as possible without being too rough or wiggly. Such approaches can have a major advantage over parametric approaches: by avoiding the assumption of a particular functional form for f, they have the potential to accurately fit a wider range of possible shapes for f. Any parametric approach brings with it the possibility that the functional form used to estimate f is very different from the true f, in which case the resulting model will not fit the data well. In contrast, non-parametric approaches completely avoid this danger, since essentially no assumption about the form of f is made. But non-parametric approaches do suffer from a major disadvantage: since they do not reduce the problem of estimating f to a small number of parameters, a very large number of observations (far more than is typically needed for a parametric approach) is required in order to obtain an accurate estimate for f. An example of a non-parametric approach to fitting the Income data is shown in Figure 2.5. A thin-plate spline is used to estimate f. This ap- thin-plate spline proach does not impose any pre-specified model on f. It instead attempts to produce an estimate for f that is as close as possible to the observed data, subject to the fit—that is, the yellow surface in Figure 2.5—being
- 35. 24 2. Statistical Learning Years of Education S e n i o r i t y In c o m e FIGURE 2.6. A rough thin-plate spline fit to the Income data from Figure 2.3. This fit makes zero errors on the training data. smooth. In this case, the non-parametric fit has produced a remarkably ac- curate estimate of the true f shown in Figure 2.3. In order to fit a thin-plate spline, the data analyst must select a level of smoothness. Figure 2.6 shows the same thin-plate spline fit using a lower level of smoothness, allowing for a rougher fit. The resulting estimate fits the observed data perfectly! However, the spline fit shown in Figure 2.6 is far more variable than the true function f, from Figure 2.3. This is an example of overfitting the data, which we discussed previously. It is an undesirable situation because the fit obtained will not yield accurate estimates of the response on new observations that were not part of the original training data set. We dis- cuss methods for choosing the correct amount of smoothness in Chapter 5. Splines are discussed in Chapter 7. As we have seen, there are advantages and disadvantages to parametric and non-parametric methods for statistical learning. We explore both types of methods throughout this book. 2.1.3 The Trade-Off Between Prediction Accuracy and Model Interpretability Of the many methods that we examine in this book, some are less flexible, or more restrictive, in the sense that they can produce just a relatively small range of shapes to estimate f. For example, linear regression is a relatively inflexible approach, because it can only generate linear functions such as the lines shown in Figure 2.1 or the plane shown in Figure 2.4. Other methods, such as the thin plate splines shown in Figures 2.5 and 2.6,
- 36. 2.1 What Is Statistical Learning? 25 Flexibility Interpretability Low High Low High Subset Selection Lasso Least Squares Generalized Additive Models Trees Bagging, Boosting Support Vector Machines Deep Learning FIGURE 2.7. A representation of the tradeoff between flexibility and inter- pretability, using different statistical learning methods. In general, as the flexibil- ity of a method increases, its interpretability decreases. are considerably more flexible because they can generate a much wider range of possible shapes to estimate f. One might reasonably ask the following question: why would we ever choose to use a more restrictive method instead of a very flexible approach? There are several reasons that we might prefer a more restrictive model. If we are mainly interested in inference, then restrictive models are much more interpretable. For instance, when inference is the goal, the linear model may be a good choice since it will be quite easy to understand the relationship between Y and X1, X2, . . . , Xp. In contrast, very flexible approaches, such as the splines discussed in Chapter 7 and displayed in Figures 2.5 and 2.6, and the boosting methods discussed in Chapter 8, can lead to such complicated estimates of f that it is difficult to understand how any individual predictor is associated with the response. Figure 2.7 provides an illustration of the trade-off between flexibility and interpretability for some of the methods that we cover in this book. Least squares linear regression, discussed in Chapter 3, is relatively inflexible but is quite interpretable. The lasso, discussed in Chapter 6, relies upon the lasso linear model (2.4) but uses an alternative fitting procedure for estimating the coefficients β0, β1, . . . , βp. The new procedure is more restrictive in es- timating the coefficients, and sets a number of them to exactly zero. Hence in this sense the lasso is a less flexible approach than linear regression. It is also more interpretable than linear regression, because in the final model the response variable will only be related to a small subset of the predictors—namely, those with nonzero coefficient estimates. Generalized additive models (GAMs), discussed in Chapter 7, instead extend the lin- generalized additive model ear model (2.4) to allow for certain non-linear relationships. Consequently,
- 37. 26 2. Statistical Learning GAMs are more flexible than linear regression. They are also somewhat less interpretable than linear regression, because the relationship between each predictor and the response is now modeled using a curve. Finally, fully non-linear methods such as bagging, boosting, support vector machines bagging boosting with non-linear kernels, and neural networks (deep learning), discussed in support vector machine Chapters 8, 9, and 10, are highly flexible approaches that are harder to interpret. We have established that when inference is the goal, there are clear ad- vantages to using simple and relatively inflexible statistical learning meth- ods. In some settings, however, we are only interested in prediction, and the interpretability of the predictive model is simply not of interest. For instance, if we seek to develop an algorithm to predict the price of a stock, our sole requirement for the algorithm is that it predict accurately— interpretability is not a concern. In this setting, we might expect that it will be best to use the most flexible model available. Surprisingly, this is not always the case! We will often obtain more accurate predictions using a less flexible method. This phenomenon, which may seem counterintuitive at first glance, has to do with the potential for overfitting in highly flexible methods. We saw an example of overfitting in Figure 2.6. We will discuss this very important concept further in Section 2.2 and throughout this book. 2.1.4 Supervised Versus Unsupervised Learning Most statistical learning problems fall into one of two categories: supervised supervised or unsupervised. The examples that we have discussed so far in this chap- unsupervised ter all fall into the supervised learning domain. For each observation of the predictor measurement(s) xi, i = 1, . . . , n there is an associated response measurement yi. We wish to fit a model that relates the response to the predictors, with the aim of accurately predicting the response for future observations (prediction) or better understanding the relationship between the response and the predictors (inference). Many classical statistical learn- ing methods such as linear regression and logistic regression (Chapter 4), as logistic regression well as more modern approaches such as GAM, boosting, and support vec- tor machines, operate in the supervised learning domain. The vast majority of this book is devoted to this setting. By contrast, unsupervised learning describes the somewhat more chal- lenging situation in which for every observation i = 1, . . . , n, we observe a vector of measurements xi but no associated response yi. It is not pos- sible to fit a linear regression model, since there is no response variable to predict. In this setting, we are in some sense working blind; the sit- uation is referred to as unsupervised because we lack a response vari- able that can supervise our analysis. What sort of statistical analysis is possible? We can seek to understand the relationships between the variables or between the observations. One statistical learning tool that we may use
- 38. 2.1 What Is Statistical Learning? 27 0 2 4 6 8 10 12 2 4 6 8 10 12 0 2 4 6 2 4 6 8 X1 X1 X 2 X 2 FIGURE 2.8. A clustering data set involving three groups. Each group is shown using a different colored symbol. Left: The three groups are well-separated. In this setting, a clustering approach should successfully identify the three groups. Right: There is some overlap among the groups. Now the clustering task is more challenging. in this setting is cluster analysis, or clustering. The goal of cluster analysis cluster analysis is to ascertain, on the basis of x1, . . . , xn, whether the observations fall into relatively distinct groups. For example, in a market segmentation study we might observe multiple characteristics (variables) for potential customers, such as zip code, family income, and shopping habits. We might believe that the customers fall into different groups, such as big spenders versus low spenders. If the information about each customer’s spending patterns were available, then a supervised analysis would be possible. However, this information is not available—that is, we do not know whether each poten- tial customer is a big spender or not. In this setting, we can try to cluster the customers on the basis of the variables measured, in order to identify distinct groups of potential customers. Identifying such groups can be of interest because it might be that the groups differ with respect to some property of interest, such as spending habits. Figure 2.8 provides a simple illustration of the clustering problem. We have plotted 150 observations with measurements on two variables, X1 and X2. Each observation corresponds to one of three distinct groups. For illustrative purposes, we have plotted the members of each group using dif- ferent colors and symbols. However, in practice the group memberships are unknown, and the goal is to determine the group to which each observa- tion belongs. In the left-hand panel of Figure 2.8, this is a relatively easy task because the groups are well-separated. By contrast, the right-hand panel illustrates a more challenging setting in which there is some overlap
- 39. 28 2. Statistical Learning between the groups. A clustering method could not be expected to assign all of the overlapping points to their correct group (blue, green, or orange). In the examples shown in Figure 2.8, there are only two variables, and so one can simply visually inspect the scatterplots of the observations in order to identify clusters. However, in practice, we often encounter data sets that contain many more than two variables. In this case, we cannot easily plot the observations. For instance, if there are p variables in our data set, then p(p − 1)/2 distinct scatterplots can be made, and visual inspection is simply not a viable way to identify clusters. For this reason, automated clustering methods are important. We discuss clustering and other unsupervised learning approaches in Chapter 12. Many problems fall naturally into the supervised or unsupervised learn- ing paradigms. However, sometimes the question of whether an analysis should be considered supervised or unsupervised is less clear-cut. For in- stance, suppose that we have a set of n observations. For m of the observa- tions, where m < n, we have both predictor measurements and a response measurement. For the remaining n − m observations, we have predictor measurements but no response measurement. Such a scenario can arise if the predictors can be measured relatively cheaply but the corresponding responses are much more expensive to collect. We refer to this setting as a semi-supervised learning problem. In this setting, we wish to use a sta- semi- supervised learning tistical learning method that can incorporate the m observations for which response measurements are available as well as the n − m observations for which they are not. Although this is an interesting topic, it is beyond the scope of this book. 2.1.5 Regression Versus Classification Problems Variables can be characterized as either quantitative or qualitative (also quantitative qualitative known as categorical). Quantitative variables take on numerical values. categorical Examples include a person’s age, height, or income, the value of a house, and the price of a stock. In contrast, qualitative variables take on values in one of K different classes, or categories. Examples of qualitative vari- class ables include a person’s marital status (married or not), the brand of prod- uct purchased (brand A, B, or C), whether a person defaults on a debt (yes or no), or a cancer diagnosis (Acute Myelogenous Leukemia, Acute Lymphoblastic Leukemia, or No Leukemia). We tend to refer to problems with a quantitative response as regression problems, while those involv- regression ing a qualitative response are often referred to as classification problems. classification However, the distinction is not always that crisp. Least squares linear re- gression (Chapter 3) is used with a quantitative response, whereas logistic regression (Chapter 4) is typically used with a qualitative (two-class, or binary) response. Thus, despite its name, logistic regression is a classifica- binary tion method. But since it estimates class probabilities, it can be thought of as a regression method as well. Some statistical methods, such as K-nearest
- 40. 2.2 Assessing Model Accuracy 29 neighbors (Chapters 2 and 4) and boosting (Chapter 8), can be used in the case of either quantitative or qualitative responses. We tend to select statistical learning methods on the basis of whether the response is quantitative or qualitative; i.e. we might use linear regres- sion when quantitative and logistic regression when qualitative. However, whether the predictors are qualitative or quantitative is generally consid- ered less important. Most of the statistical learning methods discussed in this book can be applied regardless of the predictor variable type, provided that any qualitative predictors are properly coded before the analysis is performed. This is discussed in Chapter 3. 2.2 Assessing Model Accuracy One of the key aims of this book is to introduce the reader to a wide range of statistical learning methods that extend far beyond the standard linear regression approach. Why is it necessary to introduce so many different statistical learning approaches, rather than just a single best method? There is no free lunch in statistics: no one method dominates all others over all possible data sets. On a particular data set, one specific method may work best, but some other method may work better on a similar but different data set. Hence it is an important task to decide for any given set of data which method produces the best results. Selecting the best approach can be one of the most challenging parts of performing statistical learning in practice. In this section, we discuss some of the most important concepts that arise in selecting a statistical learning procedure for a specific data set. As the book progresses, we will explain how the concepts presented here can be applied in practice. 2.2.1 Measuring the Quality of Fit In order to evaluate the performance of a statistical learning method on a given data set, we need some way to measure how well its predictions actually match the observed data. That is, we need to quantify the extent to which the predicted response value for a given observation is close to the true response value for that observation. In the regression setting, the most commonly-used measure is the mean squared error (MSE), given by mean squared error MSE = 1 n n 0 i=1 (yi − ˆ f(xi))2 , (2.5) where ˆ f(xi) is the prediction that ˆ f gives for the ith observation. The MSE will be small if the predicted responses are very close to the true responses,
- 41. 30 2. Statistical Learning and will be large if for some of the observations, the predicted and true responses differ substantially. The MSE in (2.5) is computed using the training data that was used to fit the model, and so should more accurately be referred to as the training MSE. But in general, we do not really care how well the method works training MSE on the training data. Rather, we are interested in the accuracy of the pre- dictions that we obtain when we apply our method to previously unseen test data. Why is this what we care about? Suppose that we are interested test data in developing an algorithm to predict a stock’s price based on previous stock returns. We can train the method using stock returns from the past 6 months. But we don’t really care how well our method predicts last week’s stock price. We instead care about how well it will predict tomorrow’s price or next month’s price. On a similar note, suppose that we have clinical measurements (e.g. weight, blood pressure, height, age, family history of disease) for a number of patients, as well as information about whether each patient has diabetes. We can use these patients to train a statistical learn- ing method to predict risk of diabetes based on clinical measurements. In practice, we want this method to accurately predict diabetes risk for future patients based on their clinical measurements. We are not very interested in whether or not the method accurately predicts diabetes risk for patients used to train the model, since we already know which of those patients have diabetes. To state it more mathematically, suppose that we fit our statistical learn- ing method on our training observations {(x1, y1), (x2, y2), . . . , (xn, yn)}, and we obtain the estimate ˆ f. We can then compute ˆ f(x1), ˆ f(x2), . . . , ˆ f(xn). If these are approximately equal to y1, y2, . . . , yn, then the training MSE given by (2.5) is small. However, we are really not interested in whether ˆ f(xi) ≈ yi; instead, we want to know whether ˆ f(x0) is approximately equal to y0, where (x0, y0) is a previously unseen test observation not used to train the statistical learning method. We want to choose the method that gives the lowest test MSE, as opposed to the lowest training MSE. In other words, test MSE if we had a large number of test observations, we could compute Ave(y0 − ˆ f(x0))2 , (2.6) the average squared prediction error for these test observations (x0, y0). We’d like to select the model for which this quantity is as small as possible. How can we go about trying to select a method that minimizes the test MSE? In some settings, we may have a test data set available—that is, we may have access to a set of observations that were not used to train the statistical learning method. We can then simply evaluate (2.6) on the test observations, and select the learning method for which the test MSE is smallest. But what if no test observations are available? In that case, one might imagine simply selecting a statistical learning method that minimizes the training MSE (2.5). This seems like it might be a sensible approach,
- 42. 2.2 Assessing Model Accuracy 31 0 20 40 60 80 100 2 4 6 8 10 12 X Y 2 5 10 20 0.0 0.5 1.0 1.5 2.0 2.5 Flexibility Mean Squared Error FIGURE 2.9. Left: Data simulated from f, shown in black. Three estimates of f are shown: the linear regression line (orange curve), and two smoothing spline fits (blue and green curves). Right: Training MSE (grey curve), test MSE (red curve), and minimum possible test MSE over all methods (dashed line). Squares represent the training and test MSEs for the three fits shown in the left-hand panel. since the training MSE and the test MSE appear to be closely related. Unfortunately, there is a fundamental problem with this strategy: there is no guarantee that the method with the lowest training MSE will also have the lowest test MSE. Roughly speaking, the problem is that many statistical methods specifically estimate coefficients so as to minimize the training set MSE. For these methods, the training set MSE can be quite small, but the test MSE is often much larger. Figure 2.9 illustrates this phenomenon on a simple example. In the left- hand panel of Figure 2.9, we have generated observations from (2.1) with the true f given by the black curve. The orange, blue and green curves illus- trate three possible estimates for f obtained using methods with increasing levels of flexibility. The orange line is the linear regression fit, which is rela- tively inflexible. The blue and green curves were produced using smoothing splines, discussed in Chapter 7, with different levels of smoothness. It is smoothing spline clear that as the level of flexibility increases, the curves fit the observed data more closely. The green curve is the most flexible and matches the data very well; however, we observe that it fits the true f (shown in black) poorly because it is too wiggly. By adjusting the level of flexibility of the smoothing spline fit, we can produce many different fits to this data. We now move on to the right-hand panel of Figure 2.9. The grey curve displays the average training MSE as a function of flexibility, or more for- mally the degrees of freedom, for a number of smoothing splines. The de- degrees of freedom grees of freedom is a quantity that summarizes the flexibility of a curve; it
- 43. 32 2. Statistical Learning is discussed more fully in Chapter 7. The orange, blue and green squares indicate the MSEs associated with the corresponding curves in the left- hand panel. A more restricted and hence smoother curve has fewer degrees of freedom than a wiggly curve—note that in Figure 2.9, linear regression is at the most restrictive end, with two degrees of freedom. The training MSE declines monotonically as flexibility increases. In this example the true f is non-linear, and so the orange linear fit is not flexible enough to estimate f well. The green curve has the lowest training MSE of all three methods, since it corresponds to the most flexible of the three curves fit in the left-hand panel. In this example, we know the true function f, and so we can also com- pute the test MSE over a very large test set, as a function of flexibility. (Of course, in general f is unknown, so this will not be possible.) The test MSE is displayed using the red curve in the right-hand panel of Figure 2.9. As with the training MSE, the test MSE initially declines as the level of flex- ibility increases. However, at some point the test MSE levels off and then starts to increase again. Consequently, the orange and green curves both have high test MSE. The blue curve minimizes the test MSE, which should not be surprising given that visually it appears to estimate f the best in the left-hand panel of Figure 2.9. The horizontal dashed line indicates Var(ϵ), the irreducible error in (2.3), which corresponds to the lowest achievable test MSE among all possible methods. Hence, the smoothing spline repre- sented by the blue curve is close to optimal. In the right-hand panel of Figure 2.9, as the flexibility of the statistical learning method increases, we observe a monotone decrease in the training MSE and a U-shape in the test MSE. This is a fundamental property of statistical learning that holds regardless of the particular data set at hand and regardless of the statistical method being used. As model flexibility increases, training MSE will decrease, but the test MSE may not. When a given method yields a small training MSE but a large test MSE, we are said to be overfitting the data. This happens because our statistical learning procedure is working too hard to find patterns in the training data, and may be picking up some patterns that are just caused by random chance rather than by true properties of the unknown function f. When we overfit the training data, the test MSE will be very large because the supposed patterns that the method found in the training data simply don’t exist in the test data. Note that regardless of whether or not overfitting has occurred, we almost always expect the training MSE to be smaller than the test MSE because most statistical learning methods either directly or indirectly seek to minimize the training MSE. Overfitting refers specifically to the case in which a less flexible model would have yielded a smaller test MSE. Figure 2.10 provides another example in which the true f is approxi- mately linear. Again we observe that the training MSE decreases mono- tonically as the model flexibility increases, and that there is a U-shape in
- 44. 2.2 Assessing Model Accuracy 33 0 20 40 60 80 100 2 4 6 8 10 12 X Y 2 5 10 20 0.0 0.5 1.0 1.5 2.0 2.5 Flexibility Mean Squared Error FIGURE 2.10. Details are as in Figure 2.9, using a different true f that is much closer to linear. In this setting, linear regression provides a very good fit to the data. the test MSE. However, because the truth is close to linear, the test MSE only decreases slightly before increasing again, so that the orange least squares fit is substantially better than the highly flexible green curve. Fi- nally, Figure 2.11 displays an example in which f is highly non-linear. The training and test MSE curves still exhibit the same general patterns, but now there is a rapid decrease in both curves before the test MSE starts to increase slowly. In practice, one can usually compute the training MSE with relative ease, but estimating test MSE is considerably more difficult because usually no test data are available. As the previous three examples illustrate, the flexibility level corresponding to the model with the minimal test MSE can vary considerably among data sets. Throughout this book, we discuss a variety of approaches that can be used in practice to estimate this minimum point. One important method is cross-validation (Chapter 5), which is a cross- validation method for estimating test MSE using the training data. 2.2.2 The Bias-Variance Trade-Off The U-shape observed in the test MSE curves (Figures 2.9–2.11) turns out to be the result of two competing properties of statistical learning methods. Though the mathematical proof is beyond the scope of this book, it is possible to show that the expected test MSE, for a given value x0, can always be decomposed into the sum of three fundamental quantities: the variance of ˆ f(x0), the squared bias of ˆ f(x0) and the variance of the error variance bias
- 45. 34 2. Statistical Learning 0 20 40 60 80 100 −10 0 10 20 X Y 2 5 10 20 0 5 10 15 20 Flexibility Mean Squared Error FIGURE 2.11. Details are as in Figure 2.9, using a different f that is far from linear. In this setting, linear regression provides a very poor fit to the data. terms ϵ. That is, E 1 y0 − ˆ f(x0) 22 = Var( ˆ f(x0)) + [Bias( ˆ f(x0))]2 + Var(ϵ). (2.7) Here the notation E 1 y0 − ˆ f(x0) 22 defines the expected test MSE at x0, expected test MSE and refers to the average test MSE that we would obtain if we repeatedly estimated f using a large number of training sets, and tested each at x0. The overall expected test MSE can be computed by averaging E 1 y0 − ˆ f(x0) 22 over all possible values of x0 in the test set. Equation 2.7 tells us that in order to minimize the expected test error, we need to select a statistical learning method that simultaneously achieves low variance and low bias. Note that variance is inherently a nonnegative quantity, and squared bias is also nonnegative. Hence, we see that the expected test MSE can never lie below Var(ϵ), the irreducible error from (2.3). What do we mean by the variance and bias of a statistical learning method? Variance refers to the amount by which ˆ f would change if we estimated it using a different training data set. Since the training data are used to fit the statistical learning method, different training data sets will result in a different ˆ f. But ideally the estimate for f should not vary too much between training sets. However, if a method has high variance then small changes in the training data can result in large changes in ˆ f. In general, more flexible statistical methods have higher variance. Consider the green and orange curves in Figure 2.9. The flexible green curve is following the observations very closely. It has high variance because changing any
- 46. 2.2 Assessing Model Accuracy 35 one of these data points may cause the estimate ˆ f to change considerably. In contrast, the orange least squares line is relatively inflexible and has low variance, because moving any single observation will likely cause only a small shift in the position of the line. On the other hand, bias refers to the error that is introduced by approxi- mating a real-life problem, which may be extremely complicated, by a much simpler model. For example, linear regression assumes that there is a linear relationship between Y and X1, X2, . . . , Xp. It is unlikely that any real-life problem truly has such a simple linear relationship, and so performing lin- ear regression will undoubtedly result in some bias in the estimate of f. In Figure 2.11, the true f is substantially non-linear, so no matter how many training observations we are given, it will not be possible to produce an accurate estimate using linear regression. In other words, linear regression results in high bias in this example. However, in Figure 2.10 the true f is very close to linear, and so given enough data, it should be possible for linear regression to produce an accurate estimate. Generally, more flexible methods result in less bias. As a general rule, as we use more flexible methods, the variance will increase and the bias will decrease. The relative rate of change of these two quantities determines whether the test MSE increases or decreases. As we increase the flexibility of a class of methods, the bias tends to initially decrease faster than the variance increases. Consequently, the expected test MSE declines. However, at some point increasing flexibility has little impact on the bias but starts to significantly increase the variance. When this happens the test MSE increases. Note that we observed this pattern of decreasing test MSE followed by increasing test MSE in the right-hand panels of Figures 2.9–2.11. The three plots in Figure 2.12 illustrate Equation 2.7 for the examples in Figures 2.9–2.11. In each case the blue solid curve represents the squared bias, for different levels of flexibility, while the orange curve corresponds to the variance. The horizontal dashed line represents Var(ϵ), the irreducible error. Finally, the red curve, corresponding to the test set MSE, is the sum of these three quantities. In all three cases, the variance increases and the bias decreases as the method’s flexibility increases. However, the flexibility level corresponding to the optimal test MSE differs considerably among the three data sets, because the squared bias and variance change at different rates in each of the data sets. In the left-hand panel of Figure 2.12, the bias initially decreases rapidly, resulting in an initial sharp decrease in the expected test MSE. On the other hand, in the center panel of Figure 2.12 the true f is close to linear, so there is only a small decrease in bias as flex- ibility increases, and the test MSE only declines slightly before increasing rapidly as the variance increases. Finally, in the right-hand panel of Fig- ure 2.12, as flexibility increases, there is a dramatic decline in bias because the true f is very non-linear. There is also very little increase in variance
- 47. 36 2. Statistical Learning 2 5 10 20 0.0 0.5 1.0 1.5 2.0 2.5 Flexibility 2 5 10 20 0.0 0.5 1.0 1.5 2.0 2.5 Flexibility 2 5 10 20 0 5 10 15 20 Flexibility MSE Bias Var FIGURE 2.12. Squared bias (blue curve), variance (orange curve), Var(ϵ) (dashed line), and test MSE (red curve) for the three data sets in Figures 2.9–2.11. The vertical dotted line indicates the flexibility level corresponding to the smallest test MSE. as flexibility increases. Consequently, the test MSE declines substantially before experiencing a small increase as model flexibility increases. The relationship between bias, variance, and test set MSE given in Equa- tion 2.7 and displayed in Figure 2.12 is referred to as the bias-variance trade-off. Good test set performance of a statistical learning method re- bias-variance trade-off quires low variance as well as low squared bias. This is referred to as a trade-off because it is easy to obtain a method with extremely low bias but high variance (for instance, by drawing a curve that passes through every single training observation) or a method with very low variance but high bias (by fitting a horizontal line to the data). The challenge lies in finding a method for which both the variance and the squared bias are low. This trade-off is one of the most important recurring themes in this book. In a real-life situation in which f is unobserved, it is generally not pos- sible to explicitly compute the test MSE, bias, or variance for a statistical learning method. Nevertheless, one should always keep the bias-variance trade-off in mind. In this book we explore methods that are extremely flexible and hence can essentially eliminate bias. However, this does not guarantee that they will outperform a much simpler method such as linear regression. To take an extreme example, suppose that the true f is linear. In this situation linear regression will have no bias, making it very hard for a more flexible method to compete. In contrast, if the true f is highly non-linear and we have an ample number of training observations, then we may do better using a highly flexible approach, as in Figure 2.11. In Chapter 5 we discuss cross-validation, which is a way to estimate the test MSE using the training data.
- 48. 2.2 Assessing Model Accuracy 37 2.2.3 The Classification Setting Thus far, our discussion of model accuracy has been focused on the regres- sion setting. But many of the concepts that we have encountered, such as the bias-variance trade-off, transfer over to the classification setting with only some modifications due to the fact that yi is no longer quan- titative. Suppose that we seek to estimate f on the basis of training obser- vations {(x1, y1), . . . , (xn, yn)}, where now y1, . . . , yn are qualitative. The most common approach for quantifying the accuracy of our estimate ˆ f is the training error rate, the proportion of mistakes that are made if we apply error rate our estimate ˆ f to the training observations: 1 n n 0 i=1 I(yi ̸= ŷi). (2.8) Here ŷi is the predicted class label for the ith observation using ˆ f. And I(yi ̸= ŷi) is an indicator variable that equals 1 if yi ̸= ŷi and zero if yi = ŷi. indicator variable If I(yi ̸= ŷi) = 0 then the ith observation was classified correctly by our classification method; otherwise it was misclassified. Hence Equation 2.8 computes the fraction of incorrect classifications. Equation 2.8 is referred to as the training error rate because it is com- training error puted based on the data that was used to train our classifier. As in the regression setting, we are most interested in the error rates that result from applying our classifier to test observations that were not used in training. The test error rate associated with a set of test observations of the form test error (x0, y0) is given by Ave (I(y0 ̸= ŷ0)) , (2.9) where ŷ0 is the predicted class label that results from applying the classifier to the test observation with predictor x0. A good classifier is one for which the test error (2.9) is smallest. The Bayes Classifier It is possible to show (though the proof is outside of the scope of this book) that the test error rate given in (2.9) is minimized, on average, by a very simple classifier that assigns each observation to the most likely class, given its predictor values. In other words, we should simply assign a test observation with predictor vector x0 to the class j for which Pr(Y = j|X = x0) (2.10) is largest. Note that (2.10) is a conditional probability: it is the probability conditional probability that Y = j, given the observed predictor vector x0. This very simple clas- sifier is called the Bayes classifier. In a two-class problem where there are Bayes classifier only two possible response values, say class 1 or class 2, the Bayes classifier
- 49. 38 2. Statistical Learning o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o X1 X 2 FIGURE 2.13. A simulated data set consisting of 100 observations in each of two groups, indicated in blue and in orange. The purple dashed line represents the Bayes decision boundary. The orange background grid indicates the region in which a test observation will be assigned to the orange class, and the blue background grid indicates the region in which a test observation will be assigned to the blue class. corresponds to predicting class one if Pr(Y = 1|X = x0) > 0.5, and class two otherwise. Figure 2.13 provides an example using a simulated data set in a two- dimensional space consisting of predictors X1 and X2. The orange and blue circles correspond to training observations that belong to two different classes. For each value of X1 and X2, there is a different probability of the response being orange or blue. Since this is simulated data, we know how the data were generated and we can calculate the conditional probabilities for each value of X1 and X2. The orange shaded region reflects the set of points for which Pr(Y = orange|X) is greater than 50 %, while the blue shaded region indicates the set of points for which the probability is below 50 %. The purple dashed line represents the points where the probability is exactly 50 %. This is called the Bayes decision boundary. The Bayes Bayes decision boundary classifier’s prediction is determined by the Bayes decision boundary; an observation that falls on the orange side of the boundary will be assigned to the orange class, and similarly an observation on the blue side of the boundary will be assigned to the blue class. The Bayes classifier produces the lowest possible test error rate, called the Bayes error rate. Since the Bayes classifier will always choose the class Bayes error rate for which (2.10) is largest, the error rate will be 1−maxj Pr(Y = j|X = x0)
- 50. 2.2 Assessing Model Accuracy 39 at X = x0. In general, the overall Bayes error rate is given by 1 − E * max j Pr(Y = j|X) + , (2.11) where the expectation averages the probability over all possible values of X. For our simulated data, the Bayes error rate is 0.133. It is greater than zero, because the classes overlap in the true population so maxj Pr(Y = j|X = x0) < 1 for some values of x0. The Bayes error rate is analogous to the irreducible error, discussed earlier. K-Nearest Neighbors In theory we would always like to predict qualitative responses using the Bayes classifier. But for real data, we do not know the conditional distri- bution of Y given X, and so computing the Bayes classifier is impossi- ble. Therefore, the Bayes classifier serves as an unattainable gold standard against which to compare other methods. Many approaches attempt to estimate the conditional distribution of Y given X, and then classify a given observation to the class with highest estimated probability. One such method is the K-nearest neighbors (KNN) classifier. Given a positive in- K-nearest neighbors teger K and a test observation x0, the KNN classifier first identifies the K points in the training data that are closest to x0, represented by N0. It then estimates the conditional probability for class j as the fraction of points in N0 whose response values equal j: Pr(Y = j|X = x0) = 1 K 0 i∈N0 I(yi = j). (2.12) Finally, KNN classifies the test observation x0 to the class with the largest probability from (2.12). Figure 2.14 provides an illustrative example of the KNN approach. In the left-hand panel, we have plotted a small training data set consisting of six blue and six orange observations. Our goal is to make a prediction for the point labeled by the black cross. Suppose that we choose K = 3. Then KNN will first identify the three observations that are closest to the cross. This neighborhood is shown as a circle. It consists of two blue points and one orange point, resulting in estimated probabilities of 2/3 for the blue class and 1/3 for the orange class. Hence KNN will predict that the black cross belongs to the blue class. In the right-hand panel of Figure 2.14 we have applied the KNN approach with K = 3 at all of the possible values for X1 and X2, and have drawn in the corresponding KNN decision boundary. Despite the fact that it is a very simple approach, KNN can often pro- duce classifiers that are surprisingly close to the optimal Bayes classifier. Figure 2.15 displays the KNN decision boundary, using K = 10, when ap- plied to the larger simulated data set from Figure 2.13. Notice that even
- 51. 40 2. Statistical Learning o o o o o o o o o o o o o o o o o o o o o o o o FIGURE 2.14. The KNN approach, using K = 3, is illustrated in a simple situation with six blue observations and six orange observations. Left: a test ob- servation at which a predicted class label is desired is shown as a black cross. The three closest points to the test observation are identified, and it is predicted that the test observation belongs to the most commonly-occurring class, in this case blue. Right: The KNN decision boundary for this example is shown in black. The blue grid indicates the region in which a test observation will be assigned to the blue class, and the orange grid indicates the region in which it will be assigned to the orange class. o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o X1 X 2 KNN: K=10 FIGURE 2.15. The black curve indicates the KNN decision boundary on the data from Figure 2.13, using K = 10. The Bayes decision boundary is shown as a purple dashed line. The KNN and Bayes decision boundaries are very similar.
- 52. 2.2 Assessing Model Accuracy 41 o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o KNN: K=1 KNN: K=100 FIGURE 2.16. A comparison of the KNN decision boundaries (solid black curves) obtained using K = 1 and K = 100 on the data from Figure 2.13. With K = 1, the decision boundary is overly flexible, while with K = 100 it is not sufficiently flexible. The Bayes decision boundary is shown as a purple dashed line. though the true distribution is not known by the KNN classifier, the KNN decision boundary is very close to that of the Bayes classifier. The test error rate using KNN is 0.1363, which is close to the Bayes error rate of 0.1304. The choice of K has a drastic effect on the KNN classifier obtained. Figure 2.16 displays two KNN fits to the simulated data from Figure 2.13, using K = 1 and K = 100. When K = 1, the decision boundary is overly flexible and finds patterns in the data that don’t correspond to the Bayes decision boundary. This corresponds to a classifier that has low bias but very high variance. As K grows, the method becomes less flexible and produces a decision boundary that is close to linear. This corresponds to a low-variance but high-bias classifier. On this simulated data set, neither K = 1 nor K = 100 give good predictions: they have test error rates of 0.1695 and 0.1925, respectively. Just as in the regression setting, there is not a strong relationship be- tween the training error rate and the test error rate. With K = 1, the KNN training error rate is 0, but the test error rate may be quite high. In general, as we use more flexible classification methods, the training error rate will decline but the test error rate may not. In Figure 2.17, we have plotted the KNN test and training errors as a function of 1/K. As 1/K in- creases, the method becomes more flexible. As in the regression setting, the training error rate consistently declines as the flexibility increases. However, the test error exhibits a characteristic U-shape, declining at first (with a minimum at approximately K = 10) before increasing again when the method becomes excessively flexible and overfits.
- 53. 42 2. Statistical Learning 0.01 0.02 0.05 0.10 0.20 0.50 1.00 0.00 0.05 0.10 0.15 0.20 1/K Error Rate Training Errors Test Errors FIGURE 2.17. The KNN training error rate (blue, 200 observations) and test error rate (orange, 5,000 observations) on the data from Figure 2.13, as the level of flexibility (assessed using 1/K on the log scale) increases, or equivalently as the number of neighbors K decreases. The black dashed line indicates the Bayes error rate. The jumpiness of the curves is due to the small size of the training data set. In both the regression and classification settings, choosing the correct level of flexibility is critical to the success of any statistical learning method. The bias-variance tradeoff, and the resulting U-shape in the test error, can make this a difficult task. In Chapter 5, we return to this topic and discuss various methods for estimating test error rates and thereby choosing the optimal level of flexibility for a given statistical learning method. 2.3 Lab: Introduction to R In this lab, we will introduce some simple R commands. The best way to learn a new language is to try out the commands. R can be downloaded from http://cran.r-project.org/ We recommend that you run R within an integrated development environ- ment (IDE) such as RStudio, which can be freely downloaded from http://rstudio.com
- 54. 2.3 Lab: Introduction to R 43 The RStudio website also provides a cloud-based version of R, which does not require installing any software. 2.3.1 Basic Commands R uses functions to perform operations. To run a function called funcname, function we type funcname(input1, input2), where the inputs (or arguments) input1 argument and input2 tell R how to run the function. A function can have any number of inputs. For example, to create a vector of numbers, we use the function c() (for concatenate). Any numbers inside the parentheses are joined to- c() gether. The following command instructs R to join together the numbers 1, 3, 2, and 5, and to save them as a vector named x. When we type x, it vector gives us back the vector. > x <- c(1, 3, 2, 5) > x [1] 1 3 2 5 Note that the > is not part of the command; rather, it is printed by R to indicate that it is ready for another command to be entered. We can also save things using = rather than <-: > x = c(1, 6, 2) > x [1] 1 6 2 > y = c(1, 4, 3) Hitting the up arrow multiple times will display the previous commands, which can then be edited. This is useful since one often wishes to repeat a similar command. In addition, typing ?funcname will always cause R to open a new help file window with additional information about the function funcname(). We can tell R to add two sets of numbers together. It will then add the first number from x to the first number from y, and so on. However, x and y should be the same length. We can check their length using the length() length() function. > length(x) [1] 3 > length(y) [1] 3 > x + y [1] 2 10 5 The ls() function allows us to look at a list of all of the objects, such ls() as data and functions, that we have saved so far. The rm() function can be rm() used to delete any that we don’t want. > ls() [1] "x" "y" > rm(x, y)
- 55. 44 2. Statistical Learning > ls() character (0) It’s also possible to remove all objects at once: > rm(list = ls()) The matrix() function can be used to create a matrix of numbers. Before matrix() we use the matrix() function, we can learn more about it: > ?matrix The help file reveals that the matrix() function takes a number of inputs, but for now we focus on the first three: the data (the entries in the matrix), the number of rows, and the number of columns. First, we create a simple matrix. > x <- matrix(data = c(1, 2, 3, 4), nrow = 2, ncol = 2) > x [,1] [,2] [1,] 1 3 [2,] 2 4 Note that we could just as well omit typing data=, nrow=, and ncol= in the matrix() command above: that is, we could just type > x <- matrix(c(1, 2, 3, 4), 2, 2) and this would have the same effect. However, it can sometimes be useful to specify the names of the arguments passed in, since otherwise R will assume that the function arguments are passed into the function in the same order that is given in the function’s help file. As this example illustrates, by default R creates matrices by successively filling in columns. Alternatively, the byrow = TRUE option can be used to populate the matrix in order of the rows. > matrix(c(1, 2, 3, 4), 2, 2, byrow = TRUE) [,1] [,2] [1,] 1 2 [2,] 3 4 Notice that in the above command we did not assign the matrix to a value such as x. In this case the matrix is printed to the screen but is not saved for future calculations. The sqrt() function returns the square root of each sqrt() element of a vector or matrix. The command x^2 raises each element of x to the power 2; any powers are possible, including fractional or negative powers. > sqrt(x) [,1] [,2] [1,] 1.00 1.73 [2,] 1.41 2.00 > x^2 [,1] [,2] [1,] 1 9 [2,] 4 16
- 56. 2.3 Lab: Introduction to R 45 The rnorm() function generates a vector of random normal variables, rnorm() with first argument n the sample size. Each time we call this function, we will get a different answer. Here we create two correlated sets of numbers, x and y, and use the cor() function to compute the correlation between cor() them. > x <- rnorm (50) > y <- x + rnorm (50, mean = 50, sd = .1) > cor(x, y) [1] 0.995 By default, rnorm() creates standard normal random variables with a mean of 0 and a standard deviation of 1. However, the mean and standard devi- ation can be altered using the mean and sd arguments, as illustrated above. Sometimes we want our code to reproduce the exact same set of random numbers; we can use the set.seed() function to do this. The set.seed() set.seed() function takes an (arbitrary) integer argument. > set.seed (1303) > rnorm (50) [1] -1.1440 1.3421 2.1854 0.5364 0.0632 0.5022 -0.0004 . . . We use set.seed() throughout the labs whenever we perform calculations involving random quantities. In general this should allow the user to re- produce our results. However, as new versions of R become available, small discrepancies may arise between this book and the output from R. The mean() and var() functions can be used to compute the mean and mean() var() variance of a vector of numbers. Applying sqrt() to the output of var() will give the standard deviation. Or we can simply use the sd() function. sd() > set.seed (3) > y <- rnorm (100) > mean(y) [1] 0.0110 > var(y) [1] 0.7329 > sqrt(var(y)) [1] 0.8561 > sd(y) [1] 0.8561 2.3.2 Graphics The plot() function is the primary way to plot data in R. For instance, plot() plot(x, y) produces a scatterplot of the numbers in x versus the numbers in y. There are many additional options that can be passed in to the plot() function. For example, passing in the argument xlab will result in a label on the x-axis. To find out more information about the plot() function, type ?plot.
- 57. 46 2. Statistical Learning > x <- rnorm (100) > y <- rnorm (100) > plot(x, y) > plot(x, y, xlab = "this is the x-axis", ylab = "this is the y-axis", main = "Plot of X vs Y") We will often want to save the output of an R plot. The command that we use to do this will depend on the file type that we would like to create. For instance, to create a pdf, we use the pdf() function, and to create a jpeg, pdf() we use the jpeg() function. jpeg() > pdf("Figure.pdf") > plot(x, y, col = "green") > dev.off () null device 1 The function dev.off() indicates to R that we are done creating the plot. dev.off() Alternatively, we can simply copy the plot window and paste it into an appropriate file type, such as a Word document. The function seq() can be used to create a sequence of numbers. For seq() instance, seq(a, b) makes a vector of integers between a and b. There are many other options: for instance, seq(0, 1, length = 10) makes a sequence of 10 numbers that are equally spaced between 0 and 1. Typing 3:11 is a shorthand for seq(3, 11) for integer arguments. > x <- seq(1, 10) > x [1] 1 2 3 4 5 6 7 8 9 10 > x <- 1:10 > x [1] 1 2 3 4 5 6 7 8 9 10 > x <- seq(-pi , pi , length = 50) We will now create some more sophisticated plots. The contour() func- contour() tion produces a contour plot in order to represent three-dimensional data; contour plot it is like a topographical map. It takes three arguments: 1. A vector of the x values (the first dimension), 2. A vector of the y values (the second dimension), and 3. A matrix whose elements correspond to the z value (the third dimen- sion) for each pair of (x, y) coordinates. As with the plot() function, there are many other inputs that can be used to fine-tune the output of the contour() function. To learn more about these, take a look at the help file by typing ?contour. > y <- x > f <- outer(x, y, function (x, y) cos(y) / (1 + x^2)) > contour(x, y, f) > contour(x, y, f, nlevels = 45, add = T)
- 58. 2.3 Lab: Introduction to R 47 > fa <- (f - t(f)) / 2 > contour(x, y, fa , nlevels = 15) The image() function works the same way as contour(), except that it image() produces a color-coded plot whose colors depend on the z value. This is known as a heatmap, and is sometimes used to plot temperature in weather heatmap forecasts. Alternatively, persp() can be used to produce a three-dimensional persp() plot. The arguments theta and phi control the angles at which the plot is viewed. > image(x, y, fa) > persp(x, y, fa) > persp(x, y, fa , theta = 30) > persp(x, y, fa , theta = 30, phi = 20) > persp(x, y, fa , theta = 30, phi = 70) > persp(x, y, fa , theta = 30, phi = 40) 2.3.3 Indexing Data We often wish to examine part of a set of data. Suppose that our data is stored in the matrix A. > A <- matrix (1:16 , 4, 4) > A [,1] [,2] [,3] [,4] [1,] 1 5 9 13 [2,] 2 6 10 14 [3,] 3 7 11 15 [4,] 4 8 12 16 Then, typing > A[2, 3] [1] 10 will select the element corresponding to the second row and the third col- umn. The first number after the open-bracket symbol [ always refers to the row, and the second number always refers to the column. We can also select multiple rows and columns at a time, by providing vectors as the indices. > A[c(1, 3), c(2, 4)] [,1] [,2] [1,] 5 13 [2,] 7 15 > A[1:3 , 2:4] [,1] [,2] [,3] [1,] 5 9 13 [2,] 6 10 14 [3,] 7 11 15 > A[1:2 , ] [,1] [,2] [,3] [,4] [1,] 1 5 9 13 [2,] 2 6 10 14
- 59. 48 2. Statistical Learning > A[, 1:2] [,1] [,2] [1,] 1 5 [2,] 2 6 [3,] 3 7 [4,] 4 8 The last two examples include either no index for the columns or no index for the rows. These indicate that R should include all columns or all rows, respectively. R treats a single row or column of a matrix as a vector. > A[1, ] [1] 1 5 9 13 The use of a negative sign - in the index tells R to keep all rows or columns except those indicated in the index. > A[-c(1, 3), ] [,1] [,2] [,3] [,4] [1,] 2 6 10 14 [2,] 4 8 12 16 > A[-c(1, 3), -c(1, 3, 4)] [1] 6 8 The dim() function outputs the number of rows followed by the number of dim() columns of a given matrix. > dim(A) [1] 4 4 2.3.4 Loading Data For most analyses, the first step involves importing a data set into R. The read.table() function is one of the primary ways to do this. The help file read.table() contains details about how to use this function. We can use the function write.table() to export data. write.table() Before attempting to load a data set, we must make sure that R knows to search for the data in the proper directory. For example, on a Windows system one could select the directory using the Change dir. . . option under the File menu. However, the details of how to do this depend on the oper- ating system (e.g. Windows, Mac, Unix) that is being used, and so we do not give further details here. We begin by loading in the Auto data set. This data is part of the ISLR2 library, discussed in Chapter 3. To illustrate the read.table() function, we load it now from a text file, Auto.data, which you can find on the textbook website. The following command will load the Auto.data file into R and store it as an object called Auto, in a format referred to as a data frame. data frame Once the data has been loaded, the View() function can be used to view
- 60. 2.3 Lab: Introduction to R 49 it in a spreadsheet-like window.1 The head() function can also be used to view the first few rows of the data. > Auto <- read.table("Auto.data") > View(Auto) > head(Auto) V1 V2 V3 V4 V5 1 mpg cylinders displacement horsepower weight 2 18.0 8 307.0 130.0 3504. 3 15.0 8 350.0 165.0 3693. 4 18.0 8 318.0 150.0 3436. 5 16.0 8 304.0 150.0 3433. 6 17.0 8 302.0 140.0 3449. V6 V7 V8 V9 1 acceleration year origin name 2 12.0 70 1 chevrolet chevelle malibu 3 11.5 70 1 buick skylark 320 4 11.0 70 1 plymouth satellite 5 12.0 70 1 amc rebel sst 6 10.5 70 1 ford torino Note that Auto.data is simply a text file, which you could alternatively open on your computer using a standard text editor. It is often a good idea to view a data set using a text editor or other software such as Excel before loading it into R. This particular data set has not been loaded correctly, because R has assumed that the variable names are part of the data and so has included them in the first row. The data set also includes a number of missing observations, indicated by a question mark ?. Missing values are a common occurrence in real data sets. Using the option header = T (or header = TRUE) in the read.table() function tells R that the first line of the file contains the variable names, and using the option na.strings tells R that any time it sees a particular character or set of characters (such as a question mark), it should be treated as a missing element of the data matrix. > Auto <- read.table("Auto.data", header = T, na.strings = "?", stringsAsFactors = T) > View(Auto) The stringsAsFactors = T argument tells R that any variable containing character strings should be interpreted as a qualitative variable, and that each distinct character string represents a distinct level for that qualitative variable. An easy way to load data from Excel into R is to save it as a csv (comma-separated values) file, and then use the read.csv() function. > Auto <- read.csv("Auto.csv", na.strings = "?", stringsAsFactors = T) > View(Auto) 1This function can sometimes be a bit finicky. If you have trouble using it, then try the head() function instead.
- 61. 50 2. Statistical Learning > dim(Auto) [1] 397 9 > Auto [1:4 , ] The dim() function tells us that the data has 397 observations, or rows, and dim() nine variables, or columns. There are various ways to deal with the missing data. In this case, only five of the rows contain missing observations, and so we choose to use the na.omit() function to simply remove these rows. na.omit() > Auto <- na.omit(Auto) > dim(Auto) [1] 392 9 Once the data are loaded correctly, we can use names() to check the names() variable names. > names(Auto) [1] "mpg" "cylinders" " displacement " "horsepower" [5] "weight" " acceleration " "year" "origin" [9] "name" 2.3.5 Additional Graphical and Numerical Summaries We can use the plot() function to produce scatterplots of the quantitative scatterplot variables. However, simply typing the variable names will produce an error message, because R does not know to look in the Auto data set for those variables. > plot(cylinders , mpg) Error in plot(cylinders , mpg) : object ‘cylinders ’ not found To refer to a variable, we must type the data set and the variable name joined with a $ symbol. Alternatively, we can use the attach() function in attach() order to tell R to make the variables in this data frame available by name. > plot(Auto$cylinders , Auto$mpg) > attach(Auto) > plot(cylinders , mpg) The cylinders variable is stored as a numeric vector, so R has treated it as quantitative. However, since there are only a small number of possible values for cylinders, one may prefer to treat it as a qualitative variable. The as.factor() function converts quantitative variables into qualitative as.factor() variables. > cylinders <- as.factor(cylinders) If the variable plotted on the x-axis is qualitative, then boxplots will boxplot automatically be produced by the plot() function. As usual, a number of options can be specified in order to customize the plots. > plot(cylinders , mpg) > plot(cylinders , mpg , col = "red") > plot(cylinders , mpg , col = "red", varwidth = T)
- 62. 2.3 Lab: Introduction to R 51 > plot(cylinders , mpg , col = "red", varwidth = T, horizontal = T) > plot(cylinders , mpg , col = "red", varwidth = T, xlab = "cylinders ", ylab = "MPG") The hist() function can be used to plot a histogram. Note that col = 2 hist() histogram has the same effect as col = "red". > hist(mpg) > hist(mpg , col = 2) > hist(mpg , col = 2, breaks = 15) The pairs() function creates a scatterplot matrix, i.e. a scatterplot for every pair of variables. We can also produce scatterplots for just a subset of the variables. > pairs(Auto) > pairs( ∼ mpg + displacement + horsepower + weight + acceleration , data = Auto ) In conjunction with the plot() function, identify() provides a useful identify() interactive method for identifying the value of a particular variable for points on a plot. We pass in three arguments to identify(): the x-axis variable, the y-axis variable, and the variable whose values we would like to see printed for each point. Then clicking one or more points in the plot and hitting Escape will cause R to print the values of the variable of interest. The numbers printed under the identify() function correspond to the rows for the selected points. > plot(horsepower , mpg) > identify (horsepower , mpg , name) The summary() function produces a numerical summary of each variable in summary() a particular data set. > summary(Auto) mpg cylinders displacement Min. : 9.00 Min. :3.000 Min. : 68.0 1st Qu .:17.00 1st Qu .:4.000 1st Qu .:105.0 Median :22.75 Median :4.000 Median :151.0 Mean :23.45 Mean :5.472 Mean :194.4 3rd Qu .:29.00 3rd Qu .:8.000 3rd Qu .:275.8 Max. :46.60 Max. :8.000 Max. :455.0 horsepower weight acceleration Min. : 46.0 Min. :1613 Min. : 8.00 1st Qu.: 75.0 1st Qu .:2225 1st Qu .:13.78 Median : 93.5 Median :2804 Median :15.50 Mean :104.5 Mean :2978 Mean :15.54 3rd Qu .:126.0 3rd Qu .:3615 3rd Qu .:17.02 Max. :230.0 Max. :5140 Max. :24.80 year origin name
- 63. 52 2. Statistical Learning Min. :70.00 Min. :1.000 amc matador : 5 1st Qu .:73.00 1st Qu .:1.000 ford pinto : 5 Median :76.00 Median :1.000 toyota corolla : 5 Mean :75.98 Mean :1.577 amc gremlin : 4 3rd Qu .:79.00 3rd Qu .:2.000 amc hornet : 4 Max. :82.00 Max. :3.000 chevrolet chevette: 4 (Other) :365 For qualitative variables such as name, R will list the number of observations that fall in each category. We can also produce a summary of just a single variable. > summary(mpg) Min. 1st Qu. Median Mean 3rd Qu. Max. 9.00 17.00 22.75 23.45 29.00 46.60 Once we have finished using R, we type q() in order to shut it down, or q() quit. When exiting R, we have the option to save the current workspace so workspace that all objects (such as data sets) that we have created in this R session will be available next time. Before exiting R, we may want to save a record of all of the commands that we typed in the most recent session; this can be accomplished using the savehistory() function. Next time we enter R, savehistory() we can load that history using the loadhistory() function, if we wish. loadhistory() 2.4 Exercises Conceptual 1. For each of parts (a) through (d), indicate whether we would generally expect the performance of a flexible statistical learning method to be better or worse than an inflexible method. Justify your answer. (a) The sample size n is extremely large, and the number of predic- tors p is small. (b) The number of predictors p is extremely large, and the number of observations n is small. (c) The relationship between the predictors and response is highly non-linear. (d) The variance of the error terms, i.e. σ2 = Var(ϵ), is extremely high. 2. Explain whether each scenario is a classification or regression prob- lem, and indicate whether we are most interested in inference or pre- diction. Finally, provide n and p. (a) We collect a set of data on the top 500 firms in the US. For each firm we record profit, number of employees, industry and the CEO salary. We are interested in understanding which factors affect CEO salary.
- 64. 2.4 Exercises 53 (b) We are considering launching a new product and wish to know whether it will be a success or a failure. We collect data on 20 similar products that were previously launched. For each prod- uct we have recorded whether it was a success or failure, price charged for the product, marketing budget, competition price, and ten other variables. (c) We are interested in predicting the % change in the USD/Euro exchange rate in relation to the weekly changes in the world stock markets. Hence we collect weekly data for all of 2012. For each week we record the % change in the USD/Euro, the % change in the US market, the % change in the British market, and the % change in the German market. 3. We now revisit the bias-variance decomposition. (a) Provide a sketch of typical (squared) bias, variance, training er- ror, test error, and Bayes (or irreducible) error curves, on a sin- gle plot, as we go from less flexible statistical learning methods towards more flexible approaches. The x-axis should represent the amount of flexibility in the method, and the y-axis should represent the values for each curve. There should be five curves. Make sure to label each one. (b) Explain why each of the five curves has the shape displayed in part (a). 4. You will now think of some real-life applications for statistical learn- ing. (a) Describe three real-life applications in which classification might be useful. Describe the response, as well as the predictors. Is the goal of each application inference or prediction? Explain your answer. (b) Describe three real-life applications in which regression might be useful. Describe the response, as well as the predictors. Is the goal of each application inference or prediction? Explain your answer. (c) Describe three real-life applications in which cluster analysis might be useful. 5. What are the advantages and disadvantages of a very flexible (versus a less flexible) approach for regression or classification? Under what circumstances might a more flexible approach be preferred to a less flexible approach? When might a less flexible approach be preferred?
- 65. 54 2. Statistical Learning 6. Describe the differences between a parametric and a non-parametric statistical learning approach. What are the advantages of a para- metric approach to regression or classification (as opposed to a non- parametric approach)? What are its disadvantages? 7. The table below provides a training data set containing six observa- tions, three predictors, and one qualitative response variable. Obs. X1 X2 X3 Y 1 0 3 0 Red 2 2 0 0 Red 3 0 1 3 Red 4 0 1 2 Green 5 −1 0 1 Green 6 1 1 1 Red Suppose we wish to use this data set to make a prediction for Y when X1 = X2 = X3 = 0 using K-nearest neighbors. (a) Compute the Euclidean distance between each observation and the test point, X1 = X2 = X3 = 0. (b) What is our prediction with K = 1? Why? (c) What is our prediction with K = 3? Why? (d) If the Bayes decision boundary in this problem is highly non- linear, then would we expect the best value for K to be large or small? Why? Applied 8. This exercise relates to the College data set, which can be found in the file College.csv on the book website. It contains a number of variables for 777 different universities and colleges in the US. The variables are • Private : Public/private indicator • Apps : Number of applications received • Accept : Number of applicants accepted • Enroll : Number of new students enrolled • Top10perc : New students from top 10 % of high school class • Top25perc : New students from top 25 % of high school class • F.Undergrad : Number of full-time undergraduates • P.Undergrad : Number of part-time undergraduates
- 66. 2.4 Exercises 55 • Outstate : Out-of-state tuition • Room.Board : Room and board costs • Books : Estimated book costs • Personal : Estimated personal spending • PhD : Percent of faculty with Ph.D.’s • Terminal : Percent of faculty with terminal degree • S.F.Ratio : Student/faculty ratio • perc.alumni : Percent of alumni who donate • Expend : Instructional expenditure per student • Grad.Rate : Graduation rate Before reading the data into R, it can be viewed in Excel or a text editor. (a) Use the read.csv() function to read the data into R. Call the loaded data college. Make sure that you have the directory set to the correct location for the data. (b) Look at the data using the View() function. You should notice that the first column is just the name of each university. We don’t really want R to treat this as data. However, it may be handy to have these names for later. Try the following commands: > rownames (college) <- college[, 1] > View(college) You should see that there is now a row.names column with the name of each university recorded. This means that R has given each row a name corresponding to the appropriate university. R will not try to perform calculations on the row names. However, we still need to eliminate the first column in the data where the names are stored. Try > college <- college[, -1] > View(college) Now you should see that the first data column is Private. Note that another column labeled row.names now appears before the Private column. However, this is not a data column but rather the name that R is giving to each row. (c) i. Use the summary() function to produce a numerical summary of the variables in the data set. ii. Use the pairs() function to produce a scatterplot matrix of the first ten columns or variables of the data. Recall that you can reference the first ten columns of a matrix A using A[,1:10].
- 67. 56 2. Statistical Learning iii. Use the plot() function to produce side-by-side boxplots of Outstate versus Private. iv. Create a new qualitative variable, called Elite, by binning the Top10perc variable. We are going to divide universities into two groups based on whether or not the proportion of students coming from the top 10 % of their high school classes exceeds 50 %. > Elite <- rep("No", nrow(college)) > Elite[college$Top10perc > 50] <- "Yes" > Elite <- as.factor(Elite) > college <- data.frame(college , Elite) Use the summary() function to see how many elite univer- sities there are. Now use the plot() function to produce side-by-side boxplots of Outstate versus Elite. v. Use the hist() function to produce some histograms with differing numbers of bins for a few of the quantitative vari- ables. You may find the command par(mfrow = c(2, 2)) useful: it will divide the print window into four regions so that four plots can be made simultaneously. Modifying the arguments to this function will divide the screen in other ways. vi. Continue exploring the data, and provide a brief summary of what you discover. 9. This exercise involves the Auto data set studied in the lab. Make sure that the missing values have been removed from the data. (a) Which of the predictors are quantitative, and which are quali- tative? (b) What is the range of each quantitative predictor? You can an- swer this using the range() function. range() (c) What is the mean and standard deviation of each quantitative predictor? (d) Now remove the 10th through 85th observations. What is the range, mean, and standard deviation of each predictor in the subset of the data that remains? (e) Using the full data set, investigate the predictors graphically, using scatterplots or other tools of your choice. Create some plots highlighting the relationships among the predictors. Comment on your findings. (f) Suppose that we wish to predict gas mileage (mpg) on the basis of the other variables. Do your plots suggest that any of the other variables might be useful in predicting mpg? Justify your answer.
- 68. 2.4 Exercises 57 10. This exercise involves the Boston housing data set. (a) To begin, load in the Boston data set. The Boston data set is part of the ISLR2 library. > library(ISLR2) Now the data set is contained in the object Boston. > Boston Read about the data set: > ?Boston How many rows are in this data set? How many columns? What do the rows and columns represent? (b) Make some pairwise scatterplots of the predictors (columns) in this data set. Describe your findings. (c) Are any of the predictors associated with per capita crime rate? If so, explain the relationship. (d) Do any of the census tracts of Boston appear to have particularly high crime rates? Tax rates? Pupil-teacher ratios? Comment on the range of each predictor. (e) How many of the census tracts in this data set bound the Charles river? (f) What is the median pupil-teacher ratio among the towns in this data set? (g) Which census tract of Boston has lowest median value of owner- occupied homes? What are the values of the other predictors for that census tract, and how do those values compare to the overall ranges for those predictors? Comment on your findings. (h) In this data set, how many of the census tracts average more than seven rooms per dwelling? More than eight rooms per dwelling? Comment on the census tracts that average more than eight rooms per dwelling.
- 69. 3 Linear Regression This chapter is about linear regression, a very simple approach for super- vised learning. In particular, linear regression is a useful tool for predicting a quantitative response. It has been around for a long time and is the topic of innumerable textbooks. Though it may seem somewhat dull compared to some of the more modern statistical learning approaches described in later chapters of this book, linear regression is still a useful and widely used sta- tistical learning method. Moreover, it serves as a good jumping-off point for newer approaches: as we will see in later chapters, many fancy statistical learning approaches can be seen as generalizations or extensions of linear regression. Consequently, the importance of having a good understanding of linear regression before studying more complex learning methods cannot be overstated. In this chapter, we review some of the key ideas underlying the linear regression model, as well as the least squares approach that is most commonly used to fit this model. Recall the Advertising data from Chapter 2. Figure 2.1 displays sales (in thousands of units) for a particular product as a function of advertis- ing budgets (in thousands of dollars) for TV, radio, and newspaper media. Suppose that in our role as statistical consultants we are asked to suggest, on the basis of this data, a marketing plan for next year that will result in high product sales. What information would be useful in order to provide such a recommendation? Here are a few important questions that we might seek to address: 1. Is there a relationship between advertising budget and sales? Our first goal should be to determine whether the data provide evi- © Springer Science+Business Media, LLC, part of Springer Nature 2021 G. James et al., An Introduction to Statistical Learning, Springer Texts in Statistics, 59 https://doi.org/10.1007/978-1-0716-1418-1_3
- 70. 60 3. Linear Regression dence of an association between advertising expenditure and sales. If the evidence is weak, then one might argue that no money should be spent on advertising! 2. How strong is the relationship between advertising budget and sales? Assuming that there is a relationship between advertising and sales, we would like to know the strength of this relationship. Does knowl- edge of the advertising budget provide a lot of information about product sales? 3. Which media are associated with sales? Are all three media—TV, radio, and newspaper—associated with sales, or are just one or two of the media associated? To answer this question, we must find a way to separate out the individual contribu- tion of each medium to sales when we have spent money on all three media. 4. How large is the association between each medium and sales? For every dollar spent on advertising in a particular medium, by what amount will sales increase? How accurately can we predict this amount of increase? 5. How accurately can we predict future sales? For any given level of television, radio, or newspaper advertising, what is our prediction for sales, and what is the accuracy of this prediction? 6. Is the relationship linear? If there is approximately a straight-line relationship between advertis- ing expenditure in the various media and sales, then linear regression is an appropriate tool. If not, then it may still be possible to trans- form the predictor or the response so that linear regression can be used. 7. Is there synergy among the advertising media? Perhaps spending $50,000 on television advertising and $50,000 on ra- dio advertising is associated with higher sales than allocating $100,000 to either television or radio individually. In marketing, this is known as a synergy effect, while in statistics it is called an interaction effect. synergy interaction It turns out that linear regression can be used to answer each of these questions. We will first discuss all of these questions in a general context, and then return to them in this specific context in Section 3.4. 3.1 Simple Linear Regression Simple linear regression lives up to its name: it is a very straightforward simple linear regression
- 71. 3.1 Simple Linear Regression 61 approach for predicting a quantitative response Y on the basis of a sin- gle predictor variable X. It assumes that there is approximately a linear relationship between X and Y . Mathematically, we can write this linear relationship as Y ≈ β0 + β1X. (3.1) You might read “≈” as “is approximately modeled as”. We will sometimes describe (3.1) by saying that we are regressing Y on X (or Y onto X). For example, X may represent TV advertising and Y may represent sales. Then we can regress sales onto TV by fitting the model sales ≈ β0 + β1 × TV. In Equation 3.1, β0 and β1 are two unknown constants that represent the intercept and slope terms in the linear model. Together, β0 and β1 are intercept slope known as the model coefficients or parameters. Once we have used our coefficient parameter training data to produce estimates β̂0 and β̂1 for the model coefficients, we can predict future sales on the basis of a particular value of TV advertising by computing ŷ = β̂0 + β̂1x, (3.2) where ŷ indicates a prediction of Y on the basis of X = x. Here we use a hat symbol, ˆ , to denote the estimated value for an unknown parameter or coefficient, or to denote the predicted value of the response. 3.1.1 Estimating the Coefficients In practice, β0 and β1 are unknown. So before we can use (3.1) to make predictions, we must use data to estimate the coefficients. Let (x1, y1), (x2, y2), . . . , (xn, yn) represent n observation pairs, each of which consists of a measurement of X and a measurement of Y . In the Advertising example, this data set consists of the TV advertising budget and product sales in n = 200 different markets. (Recall that the data are displayed in Figure 2.1.) Our goal is to obtain coefficient estimates β̂0 and β̂1 such that the linear model (3.1) fits the available data well—that is, so that yi ≈ β̂0 + β̂1xi for i = 1, . . . , n. In other words, we want to find an intercept β̂0 and a slope β̂1 such that the resulting line is as close as possible to the n = 200 data points. There are a number of ways of measuring closeness. However, by far the most common approach involves minimizing the least squares criterion, least squares and we take that approach in this chapter. Alternative approaches will be considered in Chapter 6. Let ŷi = β̂0 + β̂1xi be the prediction for Y based on the ith value of X. Then ei = yi −ŷi represents the ith residual—this is the difference between residual
- 72. 62 3. Linear Regression 0 50 100 150 200 250 300 5 10 15 20 25 TV Sales FIGURE 3.1. For the Advertising data, the least squares fit for the regression of sales onto TV is shown. The fit is found by minimizing the residual sum of squares. Each grey line segment represents a residual. In this case a linear fit captures the essence of the relationship, although it overestimates the trend in the left of the plot. the ith observed response value and the ith response value that is predicted by our linear model. We define the residual sum of squares (RSS) as residual sum of squares RSS = e2 1 + e2 2 + · · · + e2 n, or equivalently as RSS = (y1 −β̂0 −β̂1x1)2 +(y2 −β̂0 −β̂1x2)2 +· · ·+(yn −β̂0 −β̂1xn)2 . (3.3) The least squares approach chooses β̂0 and β̂1 to minimize the RSS. Using some calculus, one can show that the minimizers are β̂1 = )n i=1(xi − x̄)(yi − ȳ) )n i=1(xi − x̄)2 , β̂0 = ȳ − β̂1x̄, (3.4) where ȳ ≡ 1 n )n i=1 yi and x̄ ≡ 1 n )n i=1 xi are the sample means. In other words, (3.4) defines the least squares coefficient estimates for simple linear regression. Figure 3.1 displays the simple linear regression fit to the Advertising data, where β̂0 = 7.03 and β̂1 = 0.0475. In other words, according to this approximation, an additional $1,000 spent on TV advertising is asso- ciated with selling approximately 47.5 additional units of the product. In
- 73. 3.1 Simple Linear Regression 63 β0 β 1 2.11 2.15 2.2 2.3 2.5 2.5 3 3 5 6 7 8 9 0.03 0.04 0.05 0.06 ● R S S β1 β0 FIGURE 3.2. Contour and three-dimensional plots of the RSS on the Advertising data, using sales as the response and TV as the predictor. The red dots correspond to the least squares estimates β̂0 and β̂1, given by (3.4). Figure 3.2, we have computed RSS for a number of values of β0 and β1, using the advertising data with sales as the response and TV as the predic- tor. In each plot, the red dot represents the pair of least squares estimates (β̂0, β̂1) given by (3.4). These values clearly minimize the RSS. 3.1.2 Assessing the Accuracy of the Coefficient Estimates Recall from (2.1) that we assume that the true relationship between X and Y takes the form Y = f(X) + ϵ for some unknown function f, where ϵ is a mean-zero random error term. If f is to be approximated by a linear function, then we can write this relationship as Y = β0 + β1X + ϵ. (3.5) Here β0 is the intercept term—that is, the expected value of Y when X = 0, and β1 is the slope—the average increase in Y associated with a one-unit increase in X. The error term is a catch-all for what we miss with this simple model: the true relationship is probably not linear, there may be other variables that cause variation in Y , and there may be measurement error. We typically assume that the error term is independent of X. The model given by (3.5) defines the population regression line, which population regression line is the best linear approximation to the true relationship between X and Y .1 The least squares regression coefficient estimates (3.4) characterize the least squares line (3.2). The left-hand panel of Figure 3.3 displays these least squares line 1The assumption of linearity is often a useful working model. However, despite what many textbooks might tell us, we seldom believe that the true relationship is linear.
- 74. 64 3. Linear Regression −2 −1 0 1 2 −10 −5 0 5 10 X Y −2 −1 0 1 2 −10 −5 0 5 10 X Y FIGURE 3.3. A simulated data set. Left: The red line represents the true rela- tionship, f(X) = 2 + 3X, which is known as the population regression line. The blue line is the least squares line; it is the least squares estimate for f(X) based on the observed data, shown in black. Right: The population regression line is again shown in red, and the least squares line in dark blue. In light blue, ten least squares lines are shown, each computed on the basis of a separate random set of observations. Each least squares line is different, but on average, the least squares lines are quite close to the population regression line. two lines in a simple simulated example. We created 100 random Xs, and generated 100 corresponding Y s from the model Y = 2 + 3X + ϵ, (3.6) where ϵ was generated from a normal distribution with mean zero. The red line in the left-hand panel of Figure 3.3 displays the true relationship, f(X) = 2 + 3X, while the blue line is the least squares estimate based on the observed data. The true relationship is generally not known for real data, but the least squares line can always be computed using the coefficient estimates given in (3.4). In other words, in real applications, we have access to a set of observations from which we can compute the least squares line; however, the population regression line is unobserved. In the right-hand panel of Figure 3.3 we have generated ten different data sets from the model given by (3.6) and plotted the corresponding ten least squares lines. Notice that different data sets generated from the same true model result in slightly different least squares lines, but the unobserved population regression line does not change. At first glance, the difference between the population regression line and the least squares line may seem subtle and confusing. We only have one data set, and so what does it mean that two different lines describe the relationship between the predictor and the response? Fundamentally, the concept of these two lines is a natural extension of the standard statistical
- 75. 3.1 Simple Linear Regression 65 approach of using information from a sample to estimate characteristics of a large population. For example, suppose that we are interested in knowing the population mean µ of some random variable Y . Unfortunately, µ is unknown, but we do have access to n observations from Y , y1, . . . , yn, which we can use to estimate µ. A reasonable estimate is µ̂ = ȳ, where ȳ = 1 n )n i=1 yi is the sample mean. The sample mean and the population mean are different, but in general the sample mean will provide a good estimate of the population mean. In the same way, the unknown coefficients β0 and β1 in linear regression define the population regression line. We seek to estimate these unknown coefficients using β̂0 and β̂1 given in (3.4). These coefficient estimates define the least squares line. The analogy between linear regression and estimation of the mean of a random variable is an apt one based on the concept of bias. If we use the bias sample mean µ̂ to estimate µ, this estimate is unbiased, in the sense that unbiased on average, we expect µ̂ to equal µ. What exactly does this mean? It means that on the basis of one particular set of observations y1, . . . , yn, µ̂ might overestimate µ, and on the basis of another set of observations, µ̂ might underestimate µ. But if we could average a huge number of estimates of µ obtained from a huge number of sets of observations, then this average would exactly equal µ. Hence, an unbiased estimator does not systematically over- or under-estimate the true parameter. The property of unbiasedness holds for the least squares coefficient estimates given by (3.4) as well: if we estimate β0 and β1 on the basis of a particular data set, then our estimates won’t be exactly equal to β0 and β1. But if we could average the estimates obtained over a huge number of data sets, then the average of these estimates would be spot on! In fact, we can see from the right- hand panel of Figure 3.3 that the average of many least squares lines, each estimated from a separate data set, is pretty close to the true population regression line. We continue the analogy with the estimation of the population mean µ of a random variable Y . A natural question is as follows: how accurate is the sample mean µ̂ as an estimate of µ? We have established that the average of µ̂’s over many data sets will be very close to µ, but that a single estimate µ̂ may be a substantial underestimate or overestimate of µ. How far off will that single estimate of µ̂ be? In general, we answer this question by computing the standard error of µ̂, written as SE(µ̂). We have standard error the well-known formula Var(µ̂) = SE(µ̂) 2 = σ2 n , (3.7) where σ is the standard deviation of each of the realizations yi of Y .2 Roughly speaking, the standard error tells us the average amount that this estimate µ̂ differs from the actual value of µ. Equation 3.7 also tells us how 2This formula holds provided that the n observations are uncorrelated.
- 76. 66 3. Linear Regression this deviation shrinks with n—the more observations we have, the smaller the standard error of µ̂. In a similar vein, we can wonder how close β̂0 and β̂1 are to the true values β0 and β1. To compute the standard errors associated with β̂0 and β̂1, we use the following formulas: SE(β̂0) 2 = σ2 3 1 n + x̄2 )n i=1(xi − x̄)2 4 , SE(β̂1) 2 = σ2 )n i=1(xi − x̄)2 , (3.8) where σ2 = Var(ϵ). For these formulas to be strictly valid, we need to assume that the errors ϵi for each observation have common variance σ2 and are uncorrelated. This is clearly not true in Figure 3.1, but the formula still turns out to be a good approximation. Notice in the formula that SE(β̂1) is smaller when the xi are more spread out; intuitively we have more leverage to estimate a slope when this is the case. We also see that SE(β̂0) would be the same as SE(µ̂) if x̄ were zero (in which case β̂0 would be equal to ȳ). In general, σ2 is not known, but can be estimated from the data. This estimate of σ is known as the residual standard error, and is given by the formula residual standard error RSE = 5 RSS/(n − 2). Strictly speaking, when σ2 is estimated from the data we should write 6 SE(β̂1) to indicate that an estimate has been made, but for simplicity of notation we will drop this extra “hat”. Standard errors can be used to compute confidence intervals. A 95 % confidence interval confidence interval is defined as a range of values such that with 95 % probability, the range will contain the true unknown value of the param- eter. The range is defined in terms of lower and upper limits computed from the sample of data. A 95% confidence interval has the following prop- erty: if we take repeated samples and construct the confidence interval for each sample, 95% of the intervals will contain the true unknown value of the parameter. For linear regression, the 95 % confidence interval for β1 approximately takes the form β̂1 ± 2 · SE(β̂1). (3.9) That is, there is approximately a 95 % chance that the interval 7 β̂1 − 2 · SE(β̂1), β̂1 + 2 · SE(β̂1) 8 (3.10) will contain the true value of β1.3 Similarly, a confidence interval for β0 approximately takes the form β̂0 ± 2 · SE(β̂0). (3.11) 3Approximately for several reasons. Equation 3.10 relies on the assumption that the errors are Gaussian. Also, the factor of 2 in front of the SE(β̂1) term will vary slightly depending on the number of observations n in the linear regression. To be precise, rather than the number 2, (3.10) should contain the 97.5 % quantile of a t-distribution with n−2 degrees of freedom. Details of how to compute the 95 % confidence interval precisely in R will be provided later in this chapter.
- 77. 3.1 Simple Linear Regression 67 In the case of the advertising data, the 95 % confidence interval for β0 is [6.130, 7.935] and the 95 % confidence interval for β1 is [0.042, 0.053]. Therefore, we can conclude that in the absence of any advertising, sales will, on average, fall somewhere between 6,130 and 7,935 units. Furthermore, for each $1,000 increase in television advertising, there will be an average increase in sales of between 42 and 53 units. Standard errors can also be used to perform hypothesis tests on the hypothesis test coefficients. The most common hypothesis test involves testing the null hypothesis of null hypothesis H0 : There is no relationship between X and Y (3.12) versus the alternative hypothesis alternative hypothesis Ha : There is some relationship between X and Y . (3.13) Mathematically, this corresponds to testing H0 : β1 = 0 versus Ha : β1 ̸= 0, since if β1 = 0 then the model (3.5) reduces to Y = β0 + ϵ, and X is not associated with Y . To test the null hypothesis, we need to determine whether β̂1, our estimate for β1, is sufficiently far from zero that we can be confident that β1 is non-zero. How far is far enough? This of course depends on the accuracy of β̂1—that is, it depends on SE(β̂1). If SE(β̂1) is small, then even relatively small values of β̂1 may provide strong evidence that β1 ̸= 0, and hence that there is a relationship between X and Y . In contrast, if SE(β̂1) is large, then β̂1 must be large in absolute value in order for us to reject the null hypothesis. In practice, we compute a t-statistic, t-statistic given by t = β̂1 − 0 SE(β̂1) , (3.14) which measures the number of standard deviations that β̂1 is away from 0. If there really is no relationship between X and Y , then we expect that (3.14) will have a t-distribution with n − 2 degrees of freedom. The t-distribution has a bell shape and for values of n greater than approximately 30 it is quite similar to the standard normal distribution. Consequently, it is a simple matter to compute the probability of observing any number equal to |t| or larger in absolute value, assuming β1 = 0. We call this probability the p-value. Roughly speaking, we interpret the p-value as follows: a small p-value p-value indicates that it is unlikely to observe such a substantial association between the predictor and the response due to chance, in the absence of any real association between the predictor and the response. Hence, if we
- 78. 68 3. Linear Regression Coefficient Std. error t-statistic p-value Intercept 7.0325 0.4578 15.36 < 0.0001 TV 0.0475 0.0027 17.67 < 0.0001 TABLE 3.1. For the Advertising data, coefficients of the least squares model for the regression of number of units sold on TV advertising budget. An increase of $1,000 in the TV advertising budget is associated with an increase in sales by around 50 units. (Recall that the sales variable is in thousands of units, and the TV variable is in thousands of dollars.) see a small p-value, then we can infer that there is an association between the predictor and the response. We reject the null hypothesis—that is, we declare a relationship to exist between X and Y —if the p-value is small enough. Typical p-value cutoffs for rejecting the null hypothesis are 5% or 1%, although this topic will be explored in much greater detail in Chap- ter 13. When n = 30, these correspond to t-statistics (3.14) of around 2 and 2.75, respectively. Table 3.1 provides details of the least squares model for the regression of number of units sold on TV advertising budget for the Advertising data. Notice that the coefficients for β̂0 and β̂1 are very large relative to their standard errors, so the t-statistics are also large; the probabilities of seeing such values if H0 is true are virtually zero. Hence we can conclude that β0 ̸= 0 and β1 ̸= 0.4 3.1.3 Assessing the Accuracy of the Model Once we have rejected the null hypothesis (3.12) in favor of the alternative hypothesis (3.13), it is natural to want to quantify the extent to which the model fits the data. The quality of a linear regression fit is typically assessed using two related quantities: the residual standard error (RSE) and the R2 R2 statistic. Table 3.2 displays the RSE, the R2 statistic, and the F-statistic (to be described in Section 3.2.2) for the linear regression of number of units sold on TV advertising budget. Residual Standard Error Recall from the model (3.5) that associated with each observation is an error term ϵ. Due to the presence of these error terms, even if we knew the true regression line (i.e. even if β0 and β1 were known), we would not be 4In Table 3.1, a small p-value for the intercept indicates that we can reject the null hypothesis that β0 = 0, and a small p-value for TV indicates that we can reject the null hypothesis that β1 = 0. Rejecting the latter null hypothesis allows us to conclude that there is a relationship between TV and sales. Rejecting the former allows us to conclude that in the absence of TV expenditure, sales are non-zero.
- 79. 3.1 Simple Linear Regression 69 Quantity Value Residual standard error 3.26 R2 0.612 F-statistic 312.1 TABLE 3.2. For the Advertising data, more information about the least squares model for the regression of number of units sold on TV advertising budget. able to perfectly predict Y from X. The RSE is an estimate of the standard deviation of ϵ. Roughly speaking, it is the average amount that the response will deviate from the true regression line. It is computed using the formula RSE = 9 1 n − 2 RSS = : ; ; < 1 n − 2 n 0 i=1 (yi − ŷi)2. (3.15) Note that RSS was defined in Section 3.1.1, and is given by the formula RSS = n 0 i=1 (yi − ŷi)2 . (3.16) In the case of the advertising data, we see from the linear regression output in Table 3.2 that the RSE is 3.26. In other words, actual sales in each market deviate from the true regression line by approximately 3,260 units, on average. Another way to think about this is that even if the model were correct and the true values of the unknown coefficients β0 and β1 were known exactly, any prediction of sales on the basis of TV advertising would still be off by about 3,260 units on average. Of course, whether or not 3,260 units is an acceptable prediction error depends on the problem context. In the advertising data set, the mean value of sales over all markets is approximately 14,000 units, and so the percentage error is 3,260/14,000 = 23 %. The RSE is considered a measure of the lack of fit of the model (3.5) to the data. If the predictions obtained using the model are very close to the true outcome values—that is, if ŷi ≈ yi for i = 1, . . . , n—then (3.15) will be small, and we can conclude that the model fits the data very well. On the other hand, if ŷi is very far from yi for one or more observations, then the RSE may be quite large, indicating that the model doesn’t fit the data well. R2 Statistic The RSE provides an absolute measure of lack of fit of the model (3.5) to the data. But since it is measured in the units of Y , it is not always clear what constitutes a good RSE. The R2 statistic provides an alternative measure of fit. It takes the form of a proportion—the proportion of variance
- 80. 70 3. Linear Regression explained—and so it always takes on a value between 0 and 1, and is independent of the scale of Y . To calculate R2 , we use the formula R2 = TSS − RSS TSS = 1 − RSS TSS (3.17) where TSS = ) (yi − ȳ)2 is the total sum of squares, and RSS is defined total sum of squares in (3.16). TSS measures the total variance in the response Y , and can be thought of as the amount of variability inherent in the response before the regression is performed. In contrast, RSS measures the amount of variability that is left unexplained after performing the regression. Hence, TSS − RSS measures the amount of variability in the response that is explained (or removed) by performing the regression, and R2 measures the proportion of variability in Y that can be explained using X. An R2 statistic that is close to 1 indicates that a large proportion of the variability in the response is explained by the regression. A number near 0 indicates that the regression does not explain much of the variability in the response; this might occur because the linear model is wrong, or the error variance σ2 is high, or both. In Table 3.2, the R2 was 0.61, and so just under two-thirds of the variability in sales is explained by a linear regression on TV. The R2 statistic (3.17) has an interpretational advantage over the RSE (3.15), since unlike the RSE, it always lies between 0 and 1. However, it can still be challenging to determine what is a good R2 value, and in general, this will depend on the application. For instance, in certain problems in physics, we may know that the data truly comes from a linear model with a small residual error. In this case, we would expect to see an R2 value that is extremely close to 1, and a substantially smaller R2 value might indicate a serious problem with the experiment in which the data were generated. On the other hand, in typical applications in biology, psychology, marketing, and other domains, the linear model (3.5) is at best an extremely rough approximation to the data, and residual errors due to other unmeasured factors are often very large. In this setting, we would expect only a very small proportion of the variance in the response to be explained by the predictor, and an R2 value well below 0.1 might be more realistic! The R2 statistic is a measure of the linear relationship between X and Y . Recall that correlation, defined as correlation Cor(X, Y ) = )n i=1(xi − x)(yi − y) 5)n i=1(xi − x)2 5)n i=1(yi − y)2 , (3.18) is also a measure of the linear relationship between X and Y .5 This sug- gests that we might be able to use r = Cor(X, Y ) instead of R2 in order to 5We note that in fact, the right-hand side of (3.18) is the sample correlation; thus, it would be more correct to write ! Cor(X, Y ); however, we omit the “hat” for ease of notation.
- 81. 3.2 Multiple Linear Regression 71 assess the fit of the linear model. In fact, it can be shown that in the simple linear regression setting, R2 = r2 . In other words, the squared correlation and the R2 statistic are identical. However, in the next section we will discuss the multiple linear regression problem, in which we use several pre- dictors simultaneously to predict the response. The concept of correlation between the predictors and the response does not extend automatically to this setting, since correlation quantifies the association between a single pair of variables rather than between a larger number of variables. We will see that R2 fills this role. 3.2 Multiple Linear Regression Simple linear regression is a useful approach for predicting a response on the basis of a single predictor variable. However, in practice we often have more than one predictor. For example, in the Advertising data, we have examined the relationship between sales and TV advertising. We also have data for the amount of money spent advertising on the radio and in newspapers, and we may want to know whether either of these two media is associated with sales. How can we extend our analysis of the advertising data in order to accommodate these two additional predictors? One option is to run three separate simple linear regressions, each of which uses a different advertising medium as a predictor. For instance, we can fit a simple linear regression to predict sales on the basis of the amount spent on radio advertisements. Results are shown in Table 3.3 (top table). We find that a $1,000 increase in spending on radio advertising is associated with an increase in sales of around 203 units. Table 3.3 (bottom table) contains the least squares coefficients for a simple linear regression of sales onto newspaper advertising budget. A $1,000 increase in newspaper advertising budget is associated with an increase in sales of approximately 55 units. However, the approach of fitting a separate simple linear regression model for each predictor is not entirely satisfactory. First of all, it is unclear how to make a single prediction of sales given the three advertising media budgets, since each of the budgets is associated with a separate regression equation. Second, each of the three regression equations ignores the other two media in forming estimates for the regression coefficients. We will see shortly that if the media budgets are correlated with each other in the 200 markets in our data set, then this can lead to very misleading estimates of the association between each media budget and sales. Instead of fitting a separate simple linear regression model for each pre- dictor, a better approach is to extend the simple linear regression model (3.5) so that it can directly accommodate multiple predictors. We can do this by giving each predictor a separate slope coefficient in a single model. In general, suppose that we have p distinct predictors. Then the multiple
- 82. 72 3. Linear Regression Simple regression of sales on radio Coefficient Std. error t-statistic p-value Intercept 9.312 0.563 16.54 < 0.0001 radio 0.203 0.020 9.92 < 0.0001 Simple regression of sales on newspaper Coefficient Std. error t-statistic p-value Intercept 12.351 0.621 19.88 < 0.0001 newspaper 0.055 0.017 3.30 0.00115 TABLE 3.3. More simple linear regression models for the Advertising data. Co- efficients of the simple linear regression model for number of units sold on Top: radio advertising budget and Bottom: newspaper advertising budget. A $1,000 in- crease in spending on radio advertising is associated with an average increase in sales by around 203 units, while the same increase in spending on newspaper ad- vertising is associated with an average increase in sales by around 55 units. (Note that the sales variable is in thousands of units, and the radio and newspaper variables are in thousands of dollars.) linear regression model takes the form Y = β0 + β1X1 + β2X2 + · · · + βpXp + ϵ, (3.19) where Xj represents the jth predictor and βj quantifies the association between that variable and the response. We interpret βj as the average effect on Y of a one unit increase in Xj, holding all other predictors fixed. In the advertising example, (3.19) becomes sales = β0 + β1 × TV + β2 × radio + β3 × newspaper + ϵ. (3.20) 3.2.1 Estimating the Regression Coefficients As was the case in the simple linear regression setting, the regression coef- ficients β0, β1, . . . , βp in (3.19) are unknown, and must be estimated. Given estimates β̂0, β̂1, . . . , β̂p, we can make predictions using the formula ŷ = β̂0 + β̂1x1 + β̂2x2 + · · · + β̂pxp. (3.21) The parameters are estimated using the same least squares approach that we saw in the context of simple linear regression. We choose β0, β1, . . . , βp to minimize the sum of squared residuals RSS = n 0 i=1 (yi − ŷi)2 = n 0 i=1 (yi − β̂0 − β̂1xi1 − β̂2xi2 − · · · − β̂pxip)2 . (3.22)
- 83. 3.2 Multiple Linear Regression 73 X1 X2 Y FIGURE 3.4. In a three-dimensional setting, with two predictors and one re- sponse, the least squares regression line becomes a plane. The plane is chosen to minimize the sum of the squared vertical distances between each observation (shown in red) and the plane. The values β̂0, β̂1, . . . , β̂p that minimize (3.22) are the multiple least squares regression coefficient estimates. Unlike the simple linear regression esti- mates given in (3.4), the multiple regression coefficient estimates have somewhat complicated forms that are most easily represented using ma- trix algebra. For this reason, we do not provide them here. Any statistical software package can be used to compute these coefficient estimates, and later in this chapter we will show how this can be done in R. Figure 3.4 illustrates an example of the least squares fit to a toy data set with p = 2 predictors. Table 3.4 displays the multiple regression coefficient estimates when TV, radio, and newspaper advertising budgets are used to predict product sales using the Advertising data. We interpret these results as follows: for a given amount of TV and newspaper advertising, spending an additional $1,000 on radio advertising is associated with approximately 189 units of additional sales. Comparing these coefficient estimates to those displayed in Tables 3.1 and 3.3, we notice that the multiple regression coefficient estimates for TV and radio are pretty similar to the simple linear regression coefficient estimates. However, while the newspaper regression coefficient estimate in Table 3.3 was significantly non-zero, the coefficient estimate for newspaper
- 84. 74 3. Linear Regression in the multiple regression model is close to zero, and the corresponding p- value is no longer significant, with a value around 0.86. This illustrates that the simple and multiple regression coefficients can be quite different. This difference stems from the fact that in the simple regression case, the slope term represents the average increase in product sales associated with a $1,000 increase in newspaper advertising, ignoring other predictors such as TV and radio. By contrast, in the multiple regression setting, the coefficient for newspaper represents the average increase in product sales associated with increasing newspaper spending by $1,000 while holding TV and radio fixed. Coefficient Std. error t-statistic p-value Intercept 2.939 0.3119 9.42 < 0.0001 TV 0.046 0.0014 32.81 < 0.0001 radio 0.189 0.0086 21.89 < 0.0001 newspaper −0.001 0.0059 −0.18 0.8599 TABLE 3.4. For the Advertising data, least squares coefficient estimates of the multiple linear regression of number of units sold on TV, radio, and newspaper advertising budgets. Does it make sense for the multiple regression to suggest no relationship between sales and newspaper while the simple linear regression implies the opposite? In fact it does. Consider the correlation matrix for the three predictor variables and response variable, displayed in Table 3.5. Notice that the correlation between radio and newspaper is 0.35. This indicates that markets with high newspaper advertising tend to also have high ra- dio advertising. Now suppose that the multiple regression is correct and newspaper advertising is not associated with sales, but radio advertising is associated with sales. Then in markets where we spend more on radio our sales will tend to be higher, and as our correlation matrix shows, we also tend to spend more on newspaper advertising in those same mar- kets. Hence, in a simple linear regression which only examines sales versus newspaper, we will observe that higher values of newspaper tend to be as- sociated with higher values of sales, even though newspaper advertising is not directly associated with sales. So newspaper advertising is a surrogate for radio advertising; newspaper gets “credit” for the association between radio on sales. This slightly counterintuitive result is very common in many real life situations. Consider an absurd example to illustrate the point. Running a regression of shark attacks versus ice cream sales for data collected at a given beach community over a period of time would show a positive relationship, similar to that seen between sales and newspaper. Of course no one has (yet) suggested that ice creams should be banned at beaches to reduce shark attacks. In reality, higher temperatures cause more people
- 85. 3.2 Multiple Linear Regression 75 TV radio newspaper sales TV 1.0000 0.0548 0.0567 0.7822 radio 1.0000 0.3541 0.5762 newspaper 1.0000 0.2283 sales 1.0000 TABLE 3.5. Correlation matrix for TV, radio, newspaper, and sales for the Advertising data. to visit the beach, which in turn results in more ice cream sales and more shark attacks. A multiple regression of shark attacks onto ice cream sales and temperature reveals that, as intuition implies, ice cream sales is no longer a significant predictor after adjusting for temperature. 3.2.2 Some Important Questions When we perform multiple linear regression, we usually are interested in answering a few important questions. 1. Is at least one of the predictors X1, X2, . . . , Xp useful in predicting the response? 2. Do all the predictors help to explain Y , or is only a subset of the predictors useful? 3. How well does the model fit the data? 4. Given a set of predictor values, what response value should we predict, and how accurate is our prediction? We now address each of these questions in turn. One: Is There a Relationship Between the Response and Predictors? Recall that in the simple linear regression setting, in order to determine whether there is a relationship between the response and the predictor we can simply check whether β1 = 0. In the multiple regression setting with p predictors, we need to ask whether all of the regression coefficients are zero, i.e. whether β1 = β2 = · · · = βp = 0. As in the simple linear regression setting, we use a hypothesis test to answer this question. We test the null hypothesis, H0 : β1 = β2 = · · · = βp = 0 versus the alternative Ha : at least one βj is non-zero. This hypothesis test is performed by computing the F-statistic, F-statistic
- 86. 76 3. Linear Regression Quantity Value Residual standard error 1.69 R2 0.897 F-statistic 570 TABLE 3.6. More information about the least squares model for the regression of number of units sold on TV, newspaper, and radio advertising budgets in the Advertising data. Other information about this model was displayed in Table 3.4. F = (TSS − RSS)/p RSS/(n − p − 1) , (3.23) where, as with simple linear regression, TSS = ) (yi − ȳ)2 and RSS = ) (yi −ŷi)2 . If the linear model assumptions are correct, one can show that E{RSS/(n − p − 1)} = σ2 and that, provided H0 is true, E{(TSS − RSS)/p} = σ2 . Hence, when there is no relationship between the response and predictors, one would expect the F-statistic to take on a value close to 1. On the other hand, if Ha is true, then E{(TSS − RSS)/p} > σ2 , so we expect F to be greater than 1. The F-statistic for the multiple linear regression model obtained by re- gressing sales onto radio, TV, and newspaper is shown in Table 3.6. In this example the F-statistic is 570. Since this is far larger than 1, it provides compelling evidence against the null hypothesis H0. In other words, the large F-statistic suggests that at least one of the advertising media must be related to sales. However, what if the F-statistic had been closer to 1? How large does the F-statistic need to be before we can reject H0 and conclude that there is a relationship? It turns out that the answer depends on the values of n and p. When n is large, an F-statistic that is just a little larger than 1 might still provide evidence against H0. In contrast, a larger F-statistic is needed to reject H0 if n is small. When H0 is true and the errors ϵi have a normal distribution, the F-statistic follows an F-distribution.6 For any given value of n and p, any statistical software package can be used to compute the p-value associated with the F-statistic using this distribution. Based on this p-value, we can determine whether or not to reject H0. For the advertising data, the p-value associated with the F-statistic in Table 3.6 is essentially zero, so we have extremely strong evidence that at least one of the media is associated with increased sales. 6Even if the errors are not normally-distributed, the F-statistic approximately follows an F-distribution provided that the sample size n is large.
- 87. 3.2 Multiple Linear Regression 77 In (3.23) we are testing H0 that all the coefficients are zero. Sometimes we want to test that a particular subset of q of the coefficients are zero. This corresponds to a null hypothesis H0 : βp−q+1 = βp−q+2 = · · · = βp = 0, where for convenience we have put the variables chosen for omission at the end of the list. In this case we fit a second model that uses all the variables except those last q. Suppose that the residual sum of squares for that model is RSS0. Then the appropriate F-statistic is F = (RSS0 − RSS)/q RSS/(n − p − 1) . (3.24) Notice that in Table 3.4, for each individual predictor a t-statistic and a p-value were reported. These provide information about whether each individual predictor is related to the response, after adjusting for the other predictors. It turns out that each of these is exactly equivalent7 to the F- test that omits that single variable from the model, leaving all the others in—i.e. q=1 in (3.24). So it reports the partial effect of adding that variable to the model. For instance, as we discussed earlier, these p-values indicate that TV and radio are related to sales, but that there is no evidence that newspaper is associated with sales, when TV and radio are held fixed. Given these individual p-values for each variable, why do we need to look at the overall F-statistic? After all, it seems likely that if any one of the p-values for the individual variables is very small, then at least one of the predictors is related to the response. However, this logic is flawed, especially when the number of predictors p is large. For instance, consider an example in which p = 100 and H0 : β1 = β2 = · · · = βp = 0 is true, so no variable is truly associated with the response. In this situation, about 5 % of the p-values associated with each variable (of the type shown in Table 3.4) will be below 0.05 by chance. In other words, we expect to see approximately five small p-values even in the absence of any true association between the predictors and the response.8 In fact, it is likely that we will observe at least one p-value below 0.05 by chance! Hence, if we use the individual t-statistics and associated p-values in order to decide whether or not there is any association between the variables and the response, there is a very high chance that we will incorrectly conclude that there is a relationship. However, the F-statistic does not suffer from this problem because it adjusts for the number of predictors. Hence, if H0 is true, there is only a 5 % chance that the F-statistic will result in a p- value below 0.05, regardless of the number of predictors or the number of observations. 7The square of each t-statistic is the corresponding F-statistic. 8This is related to the important concept of multiple testing, which is the focus of Chapter 13.
- 88. 78 3. Linear Regression The approach of using an F-statistic to test for any association between the predictors and the response works when p is relatively small, and cer- tainly small compared to n. However, sometimes we have a very large num- ber of variables. If p > n then there are more coefficients βj to estimate than observations from which to estimate them. In this case we cannot even fit the multiple linear regression model using least squares, so the F- statistic cannot be used, and neither can most of the other concepts that we have seen so far in this chapter. When p is large, some of the approaches discussed in the next section, such as forward selection, can be used. This high-dimensional setting is discussed in greater detail in Chapter 6. high- dimensional Two: Deciding on Important Variables As discussed in the previous section, the first step in a multiple regression analysis is to compute the F-statistic and to examine the associated p- value. If we conclude on the basis of that p-value that at least one of the predictors is related to the response, then it is natural to wonder which are the guilty ones! We could look at the individual p-values as in Table 3.4, but as discussed (and as further explored in Chapter 13), if p is large we are likely to make some false discoveries. It is possible that all of the predictors are associated with the response, but it is more often the case that the response is only associated with a subset of the predictors. The task of determining which predictors are associated with the response, in order to fit a single model involving only those predictors, is referred to as variable selection. The variable selection variable selection problem is studied extensively in Chapter 6, and so here we will provide only a brief outline of some classical approaches. Ideally, we would like to perform variable selection by trying out a lot of different models, each containing a different subset of the predictors. For instance, if p = 2, then we can consider four models: (1) a model contain- ing no variables, (2) a model containing X1 only, (3) a model containing X2 only, and (4) a model containing both X1 and X2. We can then se- lect the best model out of all of the models that we have considered. How do we determine which model is best? Various statistics can be used to judge the quality of a model. These include Mallow’s Cp, Akaike informa- Mallow’s Cp tion criterion (AIC), Bayesian information criterion (BIC), and adjusted Akaike information criterion Bayesian information criterion R2 . These are discussed in more detail in Chapter 6. We can also deter- adjusted R2 mine which model is best by plotting various model outputs, such as the residuals, in order to search for patterns. Unfortunately, there are a total of 2p models that contain subsets of p variables. This means that even for moderate p, trying out every possible subset of the predictors is infeasible. For instance, we saw that if p = 2, then there are 22 = 4 models to consider. But if p = 30, then we must consider 230 = 1,073,741,824 models! This is not practical. Therefore, unless p is very small, we cannot consider all 2p models, and instead we need an automated
- 89. 3.2 Multiple Linear Regression 79 and efficient approach to choose a smaller set of models to consider. There are three classical approaches for this task: • Forward selection. We begin with the null model—a model that con- forward selection null model tains an intercept but no predictors. We then fit p simple linear re- gressions and add to the null model the variable that results in the lowest RSS. We then add to that model the variable that results in the lowest RSS for the new two-variable model. This approach is continued until some stopping rule is satisfied. • Backward selection. We start with all variables in the model, and backward selection remove the variable with the largest p-value—that is, the variable that is the least statistically significant. The new (p − 1)-variable model is fit, and the variable with the largest p-value is removed. This procedure continues until a stopping rule is reached. For instance, we may stop when all remaining variables have a p-value below some threshold. • Mixed selection. This is a combination of forward and backward se- mixed selection lection. We start with no variables in the model, and as with forward selection, we add the variable that provides the best fit. We con- tinue to add variables one-by-one. Of course, as we noted with the Advertising example, the p-values for variables can become larger as new predictors are added to the model. Hence, if at any point the p-value for one of the variables in the model rises above a certain threshold, then we remove that variable from the model. We con- tinue to perform these forward and backward steps until all variables in the model have a sufficiently low p-value, and all variables outside the model would have a large p-value if added to the model. Backward selection cannot be used if p > n, while forward selection can always be used. Forward selection is a greedy approach, and might include variables early that later become redundant. Mixed selection can remedy this. Three: Model Fit Two of the most common numerical measures of model fit are the RSE and R2 , the fraction of variance explained. These quantities are computed and interpreted in the same fashion as for simple linear regression. Recall that in simple regression, R2 is the square of the correlation of the response and the variable. In multiple linear regression, it turns out that it equals Cor(Y, Ŷ )2 , the square of the correlation between the response and the fitted linear model; in fact one property of the fitted linear model is that it maximizes this correlation among all possible linear models. An R2 value close to 1 indicates that the model explains a large portion of the variance in the response variable. As an example, we saw in Table 3.6
- 90. 80 3. Linear Regression that for the Advertising data, the model that uses all three advertising me- dia to predict sales has an R2 of 0.8972. On the other hand, the model that uses only TV and radio to predict sales has an R2 value of 0.89719. In other words, there is a small increase in R2 if we include newspaper advertising in the model that already contains TV and radio advertising, even though we saw earlier that the p-value for newspaper advertising in Table 3.4 is not significant. It turns out that R2 will always increase when more variables are added to the model, even if those variables are only weakly associated with the response. This is due to the fact that adding another variable always results in a decrease in the residual sum of squares on the training data (though not necessarily the testing data). Thus, the R2 statistic, which is also computed on the training data, must increase. The fact that adding newspaper advertising to the model containing only TV and radio advertising leads to just a tiny increase in R2 provides addi- tional evidence that newspaper can be dropped from the model. Essentially, newspaper provides no real improvement in the model fit to the training samples, and its inclusion will likely lead to poor results on independent test samples due to overfitting. By contrast, the model containing only TV as a predictor had an R2 of 0.61 (Table 3.2). Adding radio to the model leads to a substantial improve- ment in R2 . This implies that a model that uses TV and radio expenditures to predict sales is substantially better than one that uses only TV advertis- ing. We could further quantify this improvement by looking at the p-value for the radio coefficient in a model that contains only TV and radio as predictors. The model that contains only TV and radio as predictors has an RSE of 1.681, and the model that also contains newspaper as a predictor has an RSE of 1.686 (Table 3.6). In contrast, the model that contains only TV has an RSE of 3.26 (Table 3.2). This corroborates our previous conclusion that a model that uses TV and radio expenditures to predict sales is much more accurate (on the training data) than one that only uses TV spending. Furthermore, given that TV and radio expenditures are used as predictors, there is no point in also using newspaper spending as a predictor in the model. The observant reader may wonder how RSE can increase when newspaper is added to the model given that RSS must decrease. In general RSE is defined as RSE = 9 1 n − p − 1 RSS, (3.25) which simplifies to (3.15) for a simple linear regression. Thus, models with more variables can have higher RSE if the decrease in RSS is small relative to the increase in p. In addition to looking at the RSE and R2 statistics just discussed, it can be useful to plot the data. Graphical summaries can reveal problems with a model that are not visible from numerical statistics. For example,
- 91. 3.2 Multiple Linear Regression 81 Sales Radio TV FIGURE 3.5. For the Advertising data, a linear regression fit to sales using TV and radio as predictors. From the pattern of the residuals, we can see that there is a pronounced non-linear relationship in the data. The positive residuals (those visible above the surface), tend to lie along the 45-degree line, where TV and Radio budgets are split evenly. The negative residuals (most not visible), tend to lie away from this line, where budgets are more lopsided. Figure 3.5 displays a three-dimensional plot of TV and radio versus sales. We see that some observations lie above and some observations lie below the least squares regression plane. In particular, the linear model seems to overestimate sales for instances in which most of the advertising money was spent exclusively on either TV or radio. It underestimates sales for instances where the budget was split between the two media. This pro- nounced non-linear pattern suggests a synergy or interaction effect between interaction the advertising media, whereby combining the media together results in a bigger boost to sales than using any single medium. In Section 3.3.2, we will discuss extending the linear model to accommodate such synergistic effects through the use of interaction terms. Four: Predictions Once we have fit the multiple regression model, it is straightforward to apply (3.21) in order to predict the response Y on the basis of a set of values for the predictors X1, X2, . . . , Xp. However, there are three sorts of uncertainty associated with this prediction. 1. The coefficient estimates β̂0, β̂1, . . . , β̂p are estimates for β0, β1, . . . , βp. That is, the least squares plane Ŷ = β̂0 + β̂1X1 + · · · + β̂pXp
- 92. 82 3. Linear Regression is only an estimate for the true population regression plane f(X) = β0 + β1X1 + · · · + βpXp. The inaccuracy in the coefficient estimates is related to the reducible error from Chapter 2. We can compute a confidence interval in order to determine how close Ŷ will be to f(X). 2. Of course, in practice assuming a linear model for f(X) is almost always an approximation of reality, so there is an additional source of potentially reducible error which we call model bias. So when we use a linear model, we are in fact estimating the best linear approximation to the true surface. However, here we will ignore this discrepancy, and operate as if the linear model were correct. 3. Even if we knew f(X)—that is, even if we knew the true values for β0, β1, . . . , βp—the response value cannot be predicted perfectly because of the random error ϵ in the model (3.20). In Chapter 2, we referred to this as the irreducible error. How much will Y vary from Ŷ ? We use prediction intervals to answer this question. Prediction intervals are always wider than confidence intervals, because they incorporate both the error in the estimate for f(X) (the reducible error) and the uncertainty as to how much an individual point will differ from the population regression plane (the irreducible error). We use a confidence interval to quantify the uncertainty surrounding confidence interval the average sales over a large number of cities. For example, given that $100,000 is spent on TV advertising and $20,000 is spent on radio advertising in each city, the 95 % confidence interval is [10,985, 11,528]. We interpret this to mean that 95 % of intervals of this form will contain the true value of f(X).9 On the other hand, a prediction interval can be used to quantify the prediction interval uncertainty surrounding sales for a particular city. Given that $100,000 is spent on TV advertising and $20,000 is spent on radio advertising in that city the 95 % prediction interval is [7,930, 14,580]. We interpret this to mean that 95 % of intervals of this form will contain the true value of Y for this city. Note that both intervals are centered at 11,256, but that the prediction interval is substantially wider than the confidence interval, reflecting the increased uncertainty about sales for a given city in comparison to the average sales over many locations. 9In other words, if we collect a large number of data sets like the Advertising data set, and we construct a confidence interval for the average sales on the basis of each data set (given $100,000 in TV and $20,000 in radio advertising), then 95 % of these confidence intervals will contain the true value of average sales.
- 93. 3.3 Other Considerations in the Regression Model 83 3.3 Other Considerations in the Regression Model 3.3.1 Qualitative Predictors In our discussion so far, we have assumed that all variables in our linear regression model are quantitative. But in practice, this is not necessarily the case; often some predictors are qualitative. For example, the Credit data set displayed in Figure 3.6 records variables for a number of credit card holders. The response is balance (average credit card debt for each individual) and there are several quantitative predictors: age, cards (number of credit cards), education (years of education), income (in thousands of dollars), limit (credit limit), and rating (credit rating). Each panel of Figure 3.6 is a scatterplot for a pair of variables whose iden- tities are given by the corresponding row and column labels. For example, the scatterplot directly to the right of the word “Balance” depicts balance versus age, while the plot directly to the right of “Age” corresponds to age versus cards. In addition to these quantitative variables, we also have four qualitative variables: own (house ownership), student (student status), status (marital status), and region (East, West or South). Predictors with Only Two Levels Suppose that we wish to investigate differences in credit card balance be- tween those who own a house and those who don’t, ignoring the other vari- ables for the moment. If a qualitative predictor (also known as a factor) factor only has two levels, or possible values, then incorporating it into a regres- level sion model is very simple. We simply create an indicator or dummy variable dummy variable that takes on two possible numerical values.10 For example, based on the own variable, we can create a new variable that takes the form xi = = 1 if ith person owns a house 0 if ith person does not own a house, (3.26) and use this variable as a predictor in the regression equation. This results in the model yi = β0 + β1xi + ϵi = = β0 + β1 + ϵi if ith person owns a house β0 + ϵi if ith person does not. (3.27) Now β0 can be interpreted as the average credit card balance among those who do not own, β0 + β1 as the average credit card balance among those who do own their house, and β1 as the average difference in credit card balance between owners and non-owners. 10In the machine learning community, the creation of dummy variables to handle qualitative predictors is known as “one-hot encoding”.
- 94. 84 3. Linear Regression Balance 20 40 60 80 100 5 10 15 20 2000 8000 14000 0 500 1500 20 40 60 80 100 Age Cards 2 4 6 8 5 10 15 20 Education Income 50 100 150 2000 8000 14000 Limit 0 500 1500 2 4 6 8 50 100 150 200 600 1000 200 600 1000 Rating FIGURE 3.6. The Credit data set contains information about balance, age, cards, education, income, limit, and rating for a number of potential cus- tomers. Table 3.7 displays the coefficient estimates and other information asso- ciated with the model (3.27). The average credit card debt for non-owners is estimated to be $509.80, whereas owners are estimated to carry $19.73 in additional debt for a total of $509.80 + $19.73 = $529.53. However, we notice that the p-value for the dummy variable is very high. This indicates that there is no statistical evidence of a difference in average credit card balance based on house ownership. The decision to code owners as 1 and non-owners as 0 in (3.27) is ar- bitrary, and has no effect on the regression fit, but does alter the inter- pretation of the coefficients. If we had coded non-owners as 1 and own- ers as 0, then the estimates for β0 and β1 would have been 529.53 and −19.73, respectively, leading once again to a prediction of credit card debt of $529.53 − $19.73 = $509.80 for non-owners and a prediction of $529.53
- 95. 3.3 Other Considerations in the Regression Model 85 Coefficient Std. error t-statistic p-value Intercept 509.80 33.13 15.389 < 0.0001 own[Yes] 19.73 46.05 0.429 0.6690 TABLE 3.7. Least squares coefficient estimates associated with the regression of balance onto own in the Credit data set. The linear model is given in (3.27). That is, ownership is encoded as a dummy variable, as in (3.26). for owners. Alternatively, instead of a 0/1 coding scheme, we could create a dummy variable xi = = 1 if ith person owns a house −1 if ith person does not own a house and use this variable in the regression equation. This results in the model yi = β0 +β1xi +ϵi = = β0 + β1 + ϵi if ith person owns a house β0 − β1 + ϵi if ith person does not own a house. Now β0 can be interpreted as the overall average credit card balance (ig- noring the house ownership effect), and β1 is the amount by which house owners and non-owners have credit card balances that are above and below the average, respectively. In this example, the estimate for β0 is $519.665, halfway between the non-owner and owner averages of $509.80 and $529.53. The estimate for β1 is $9.865, which is half of $19.73, the average difference between owners and non-owners. It is important to note that the final pre- dictions for the credit balances of owners and non-owners will be identical regardless of the coding scheme used. The only difference is in the way that the coefficients are interpreted. Qualitative Predictors with More than Two Levels When a qualitative predictor has more than two levels, a single dummy variable cannot represent all possible values. In this situation, we can create additional dummy variables. For example, for the region variable we create two dummy variables. The first could be xi1 = = 1 if ith person is from the South 0 if ith person is not from the South, (3.28) and the second could be xi2 = = 1 if ith person is from the West 0 if ith person is not from the West. (3.29)
- 96. 86 3. Linear Regression Coefficient Std. error t-statistic p-value Intercept 531.00 46.32 11.464 < 0.0001 region[South] −18.69 65.02 −0.287 0.7740 region[West] −12.50 56.68 −0.221 0.8260 TABLE 3.8. Least squares coefficient estimates associated with the regression of balance onto region in the Credit data set. The linear model is given in (3.30). That is, region is encoded via two dummy variables (3.28) and (3.29). Then both of these variables can be used in the regression equation, in order to obtain the model yi = β0+β1xi1+β2xi2+ϵi = ⎧ ⎪ ⎨ ⎪ ⎩ β0+β1+ϵi if ith person is from the South β0+β2+ϵi if ith person is from the West β0+ϵi if ith person is from the East. (3.30) Now β0 can be interpreted as the average credit card balance for individuals from the East, β1 can be interpreted as the difference in the average balance between people from the South versus the East, and β2 can be interpreted as the difference in the average balance between those from the West versus the East. There will always be one fewer dummy variable than the number of levels. The level with no dummy variable—East in this example—is known as the baseline. baseline From Table 3.8, we see that the estimated balance for the baseline, East, is $531.00. It is estimated that those in the South will have $18.69 less debt than those in the East, and that those in the West will have $12.50 less debt than those in the East. However, the p-values associated with the coefficient estimates for the two dummy variables are very large, suggesting no statistical evidence of a real difference in average credit card balance between South and East or between West and East.11 Once again, the level selected as the baseline category is arbitrary, and the final predictions for each group will be the same regardless of this choice. However, the coefficients and their p-values do depend on the choice of dummy variable coding. Rather than rely on the individual coefficients, we can use an F-test to test H0 : β1 = β2 = 0; this does not depend on the coding. This F-test has a p-value of 0.96, indicating that we cannot reject the null hypothesis that there is no relationship between balance and region. Using this dummy variable approach presents no difficulties when in- corporating both quantitative and qualitative predictors. For example, to regress balance on both a quantitative variable such as income and a qual- itative variable such as student, we must simply create a dummy variable for student and then fit a multiple regression model using income and the dummy variable as predictors for credit card balance. 11There could still in theory be a difference between South and West, although the data here does not suggest any difference.
- 97. 3.3 Other Considerations in the Regression Model 87 There are many different ways of coding qualitative variables besides the dummy variable approach taken here. All of these approaches lead to equivalent model fits, but the coefficients are different and have different interpretations, and are designed to measure particular contrasts. This topic contrast is beyond the scope of the book. 3.3.2 Extensions of the Linear Model The standard linear regression model (3.19) provides interpretable results and works quite well on many real-world problems. However, it makes sev- eral highly restrictive assumptions that are often violated in practice. Two of the most important assumptions state that the relationship between the predictors and response are additive and linear. The additivity assumption additive linear means that the association between a predictor Xj and the response Y does not depend on the values of the other predictors. The linearity assumption states that the change in the response Y associated with a one-unit change in Xj is constant, regardless of the value of Xj. In later chapters of this book, we examine a number of sophisticated methods that relax these two assumptions. Here, we briefly examine some common classical approaches for extending the linear model. Removing the Additive Assumption In our previous analysis of the Advertising data, we concluded that both TV and radio seem to be associated with sales. The linear models that formed the basis for this conclusion assumed that the effect on sales of increasing one advertising medium is independent of the amount spent on the other media. For example, the linear model (3.20) states that the average increase in sales associated with a one-unit increase in TV is always β1, regardless of the amount spent on radio. However, this simple model may be incorrect. Suppose that spending money on radio advertising actually increases the effectiveness of TV ad- vertising, so that the slope term for TV should increase as radio increases. In this situation, given a fixed budget of $100,000, spending half on radio and half on TV may increase sales more than allocating the entire amount to either TV or to radio. In marketing, this is known as a synergy effect, and in statistics it is referred to as an interaction effect. Figure 3.5 sug- gests that such an effect may be present in the advertising data. Notice that when levels of either TV or radio are low, then the true sales are lower than predicted by the linear model. But when advertising is split between the two media, then the model tends to underestimate sales. Consider the standard linear regression model with two variables, Y = β0 + β1X1 + β2X2 + ϵ.
- 98. 88 3. Linear Regression According to this model, a one-unit increase in X1 is associated with an average increase in Y of β1 units. Notice that the presence of X2 does not alter this statement—that is, regardless of the value of X2, a one- unit increase in X1 is associated with a β1-unit increase in Y . One way of extending this model is to include a third predictor, called an interaction term, which is constructed by computing the product of X1 and X2. This results in the model Y = β0 + β1X1 + β2X2 + β3X1X2 + ϵ. (3.31) How does inclusion of this interaction term relax the additive assumption? Notice that (3.31) can be rewritten as Y = β0 + (β1 + β3X2)X1 + β2X2 + ϵ (3.32) = β0 + β̃1X1 + β2X2 + ϵ where β̃1 = β1 + β3X2. Since β̃1 is now a function of X2, the association between X1 and Y is no longer constant: a change in the value of X2 will change the association between X1 and Y . A similar argument shows that a change in the value of X1 changes the association between X2 and Y . For example, suppose that we are interested in studying the productiv- ity of a factory. We wish to predict the number of units produced on the basis of the number of production lines and the total number of workers. It seems likely that the effect of increasing the number of production lines will depend on the number of workers, since if no workers are available to operate the lines, then increasing the number of lines will not increase production. This suggests that it would be appropriate to include an inter- action term between lines and workers in a linear model to predict units. Suppose that when we fit the model, we obtain units ≈ 1.2 + 3.4 × lines + 0.22 × workers + 1.4 × (lines × workers) = 1.2 + (3.4 + 1.4 × workers) × lines + 0.22 × workers. In other words, adding an additional line will increase the number of units produced by 3.4 + 1.4 × workers. Hence the more workers we have, the stronger will be the effect of lines. We now return to the Advertising example. A linear model that uses radio, TV, and an interaction between the two to predict sales takes the form sales = β0 + β1 × TV + β2 × radio + β3 × (radio × TV) + ϵ = β0 + (β1 + β3 × radio) × TV + β2 × radio + ϵ. (3.33) We can interpret β3 as the increase in the effectiveness of TV advertising associated with a one-unit increase in radio advertising (or vice-versa). The coefficients that result from fitting the model (3.33) are given in Table 3.9.
- 99. 3.3 Other Considerations in the Regression Model 89 Coefficient Std. error t-statistic p-value Intercept 6.7502 0.248 27.23 < 0.0001 TV 0.0191 0.002 12.70 < 0.0001 radio 0.0289 0.009 3.24 0.0014 TV×radio 0.0011 0.000 20.73 < 0.0001 TABLE 3.9. For the Advertising data, least squares coefficient estimates asso- ciated with the regression of sales onto TV and radio, with an interaction term, as in (3.33). The results in Table 3.9 strongly suggest that the model that includes the interaction term is superior to the model that contains only main effects. main effect The p-value for the interaction term, TV×radio, is extremely low, indicating that there is strong evidence for Ha : β3 ̸= 0. In other words, it is clear that the true relationship is not additive. The R2 for the model (3.33) is 96.8 %, compared to only 89.7 % for the model that predicts sales using TV and radio without an interaction term. This means that (96.8 − 89.7)/(100 − 89.7) = 69 % of the variability in sales that remains after fitting the ad- ditive model has been explained by the interaction term. The coefficient estimates in Table 3.9 suggest that an increase in TV advertising of $1,000 is associated with increased sales of (β̂1 +β̂3 ×radio)×1,000 = 19+1.1×radio units. And an increase in radio advertising of $1,000 will be associated with an increase in sales of (β̂2 + β̂3 × TV) × 1,000 = 29 + 1.1 × TV units. In this example, the p-values associated with TV, radio, and the interac- tion term all are statistically significant (Table 3.9), and so it is obvious that all three variables should be included in the model. However, it is sometimes the case that an interaction term has a very small p-value, but the associated main effects (in this case, TV and radio) do not. The hier- archical principle states that if we include an interaction in a model, we hierarchical principle should also include the main effects, even if the p-values associated with their coefficients are not significant. In other words, if the interaction be- tween X1 and X2 seems important, then we should include both X1 and X2 in the model even if their coefficient estimates have large p-values. The rationale for this principle is that if X1 × X2 is related to the response, then whether or not the coefficients of X1 or X2 are exactly zero is of lit- tle interest. Also X1 × X2 is typically correlated with X1 and X2, and so leaving them out tends to alter the meaning of the interaction. In the previous example, we considered an interaction between TV and radio, both of which are quantitative variables. However, the concept of interactions applies just as well to qualitative variables, or to a combination of quantitative and qualitative variables. In fact, an interaction between a qualitative variable and a quantitative variable has a particularly nice interpretation. Consider the Credit data set from Section 3.3.1, and suppose that we wish to predict balance using the income (quantitative) and student (qualitative) variables. In the absence of an interaction term, the model
- 100. 90 3. Linear Regression 0 50 100 150 200 600 1000 1400 Income Balance 0 50 100 150 200 600 1000 1400 Income Balance student non−student FIGURE 3.7. For the Credit data, the least squares lines are shown for pre- diction of balance from income for students and non-students. Left: The model (3.34) was fit. There is no interaction between income and student. Right: The model (3.35) was fit. There is an interaction term between income and student. takes the form balancei ≈ β0 + β1 × incomei + = β2 if ith person is a student 0 if ith person is not a student = β1 × incomei + = β0 + β2 if ith person is a student β0 if ith person is not a student. (3.34) Notice that this amounts to fitting two parallel lines to the data, one for students and one for non-students. The lines for students and non-students have different intercepts, β0 + β2 versus β0, but the same slope, β1. This is illustrated in the left-hand panel of Figure 3.7. The fact that the lines are parallel means that the average effect on balance of a one-unit increase in income does not depend on whether or not the individual is a student. This represents a potentially serious limitation of the model, since in fact a change in income may have a very different effect on the credit card balance of a student versus a non-student. This limitation can be addressed by adding an interaction variable, cre- ated by multiplying income with the dummy variable for student. Our model now becomes balancei ≈ β0 + β1 × incomei + = β2 + β3 × incomei if student 0 if not student = = (β0 + β2) + (β1 + β3) × incomei if student β0 + β1 × incomei if not student. (3.35)
- 101. 3.3 Other Considerations in the Regression Model 91 50 100 150 200 10 20 30 40 50 Horsepower Miles per gallon Linear Degree 2 Degree 5 FIGURE 3.8. The Auto data set. For a number of cars, mpg and horsepower are shown. The linear regression fit is shown in orange. The linear regression fit for a model that includes horsepower2 is shown as a blue curve. The linear regression fit for a model that includes all polynomials of horsepower up to fifth-degree is shown in green. Once again, we have two different regression lines for the students and the non-students. But now those regression lines have different intercepts, β0+β2 versus β0, as well as different slopes, β1+β3 versus β1. This allows for the possibility that changes in income may affect the credit card balances of students and non-students differently. The right-hand panel of Figure 3.7 shows the estimated relationships between income and balance for students and non-students in the model (3.35). We note that the slope for students is lower than the slope for non-students. This suggests that increases in income are associated with smaller increases in credit card balance among students as compared to non-students. Non-linear Relationships As discussed previously, the linear regression model (3.19) assumes a linear relationship between the response and predictors. But in some cases, the true relationship between the response and the predictors may be non- linear. Here we present a very simple way to directly extend the linear model to accommodate non-linear relationships, using polynomial regression. In polynomial regression later chapters, we will present more complex approaches for performing non-linear fits in more general settings.
- 102. 92 3. Linear Regression Coefficient Std. error t-statistic p-value Intercept 56.9001 1.8004 31.6 < 0.0001 horsepower −0.4662 0.0311 −15.0 < 0.0001 horsepower2 0.0012 0.0001 10.1 < 0.0001 TABLE 3.10. For the Auto data set, least squares coefficient estimates associated with the regression of mpg onto horsepower and horsepower2 . Consider Figure 3.8, in which the mpg (gas mileage in miles per gallon) versus horsepower is shown for a number of cars in the Auto data set. The orange line represents the linear regression fit. There is a pronounced rela- tionship between mpg and horsepower, but it seems clear that this relation- ship is in fact non-linear: the data suggest a curved relationship. A simple approach for incorporating non-linear associations in a linear model is to include transformed versions of the predictors. For example, the points in Figure 3.8 seem to have a quadratic shape, suggesting that a model of the quadratic form mpg = β0 + β1 × horsepower + β2 × horsepower2 + ϵ (3.36) may provide a better fit. Equation 3.36 involves predicting mpg using a non-linear function of horsepower. But it is still a linear model! That is, (3.36) is simply a multiple linear regression model with X1 = horsepower and X2 = horsepower2 . So we can use standard linear regression software to estimate β0, β1, and β2 in order to produce a non-linear fit. The blue curve in Figure 3.8 shows the resulting quadratic fit to the data. The quadratic fit appears to be substantially better than the fit obtained when just the linear term is included. The R2 of the quadratic fit is 0.688, compared to 0.606 for the linear fit, and the p-value in Table 3.10 for the quadratic term is highly significant. If including horsepower2 led to such a big improvement in the model, why not include horsepower3 , horsepower4 , or even horsepower5 ? The green curve in Figure 3.8 displays the fit that results from including all polynomials up to fifth degree in the model (3.36). The resulting fit seems unnecessarily wiggly—that is, it is unclear that including the additional terms really has led to a better fit to the data. The approach that we have just described for extending the linear model to accommodate non-linear relationships is known as polynomial regres- sion, since we have included polynomial functions of the predictors in the regression model. We further explore this approach and other non-linear extensions of the linear model in Chapter 7. 3.3.3 Potential Problems When we fit a linear regression model to a particular data set, many prob- lems may occur. Most common among these are the following: 1. Non-linearity of the response-predictor relationships.
- 103. 3.3 Other Considerations in the Regression Model 93 2. Correlation of error terms. 3. Non-constant variance of error terms. 4. Outliers. 5. High-leverage points. 6. Collinearity. In practice, identifying and overcoming these problems is as much an art as a science. Many pages in countless books have been written on this topic. Since the linear regression model is not our primary focus here, we will provide only a brief summary of some key points. 1. Non-linearity of the Data 5 10 15 20 25 30 −15 −10 −5 0 5 10 15 20 Fitted values Residuals Residual Plot for Linear Fit 323 330 334 15 20 25 30 35 −15 −10 −5 0 5 10 15 Fitted values Residuals Residual Plot for Quadratic Fit 334 323 155 FIGURE 3.9. Plots of residuals versus predicted (or fitted) values for the Auto data set. In each plot, the red line is a smooth fit to the residuals, intended to make it easier to identify a trend. Left: A linear regression of mpg on horsepower. A strong pattern in the residuals indicates non-linearity in the data. Right: A linear regression of mpg on horsepower and horsepower2 . There is little pattern in the residuals. The linear regression model assumes that there is a straight-line rela- tionship between the predictors and the response. If the true relationship is far from linear, then virtually all of the conclusions that we draw from the fit are suspect. In addition, the prediction accuracy of the model can be significantly reduced. Residual plots are a useful graphical tool for identifying non-linearity. residual plot Given a simple linear regression model, we can plot the residuals, ei = yi − ŷi, versus the predictor xi. In the case of a multiple regression model,
- 104. 94 3. Linear Regression since there are multiple predictors, we instead plot the residuals versus the predicted (or fitted) values ŷi. Ideally, the residual plot will show no fitted discernible pattern. The presence of a pattern may indicate a problem with some aspect of the linear model. The left panel of Figure 3.9 displays a residual plot from the linear re- gression of mpg onto horsepower on the Auto data set that was illustrated in Figure 3.8. The red line is a smooth fit to the residuals, which is displayed in order to make it easier to identify any trends. The residuals exhibit a clear U-shape, which provides a strong indication of non-linearity in the data. In contrast, the right-hand panel of Figure 3.9 displays the residual plot that results from the model (3.36), which contains a quadratic term. There appears to be little pattern in the residuals, suggesting that the quadratic term improves the fit to the data. If the residual plot indicates that there are non-linear associations in the data, then a simple approach is to use non-linear transformations of the predictors, such as log X, √ X, and X2 , in the regression model. In the later chapters of this book, we will discuss other more advanced non-linear approaches for addressing this issue. 2. Correlation of Error Terms An important assumption of the linear regression model is that the error terms, ϵ1, ϵ2, . . . , ϵn, are uncorrelated. What does this mean? For instance, if the errors are uncorrelated, then the fact that ϵi is positive provides little or no information about the sign of ϵi+1. The standard errors that are computed for the estimated regression coefficients or the fitted values are based on the assumption of uncorrelated error terms. If in fact there is correlation among the error terms, then the estimated standard errors will tend to underestimate the true standard errors. As a result, confidence and prediction intervals will be narrower than they should be. For example, a 95 % confidence interval may in reality have a much lower probability than 0.95 of containing the true value of the parameter. In addition, p- values associated with the model will be lower than they should be; this could cause us to erroneously conclude that a parameter is statistically significant. In short, if the error terms are correlated, we may have an unwarranted sense of confidence in our model. As an extreme example, suppose we accidentally doubled our data, lead- ing to observations and error terms identical in pairs. If we ignored this, our standard error calculations would be as if we had a sample of size 2n, when in fact we have only n samples. Our estimated parameters would be the same for the 2n samples as for the n samples, but the confidence intervals would be narrower by a factor of √ 2! Why might correlations among the error terms occur? Such correlations frequently occur in the context of time series data, which consists of ob- time series servations for which measurements are obtained at discrete points in time.
- 105. 3.3 Other Considerations in the Regression Model 95 0 20 40 60 80 100 −3 −1 0 1 2 3 ρ=0.0 Residual 0 20 40 60 80 100 −4 −2 0 1 2 ρ=0.5 Residual 0 20 40 60 80 100 −1.5 −0.5 0.5 1.5 ρ=0.9 Residual Observation FIGURE 3.10. Plots of residuals from simulated time series data sets generated with differing levels of correlation ρ between error terms for adjacent time points. In many cases, observations that are obtained at adjacent time points will have positively correlated errors. In order to determine if this is the case for a given data set, we can plot the residuals from our model as a function of time. If the errors are uncorrelated, then there should be no discernible pat- tern. On the other hand, if the error terms are positively correlated, then we may see tracking in the residuals—that is, adjacent residuals may have tracking similar values. Figure 3.10 provides an illustration. In the top panel, we see the residuals from a linear regression fit to data generated with uncorre- lated errors. There is no evidence of a time-related trend in the residuals. In contrast, the residuals in the bottom panel are from a data set in which adjacent errors had a correlation of 0.9. Now there is a clear pattern in the residuals—adjacent residuals tend to take on similar values. Finally, the center panel illustrates a more moderate case in which the residuals had a correlation of 0.5. There is still evidence of tracking, but the pattern is less clear. Many methods have been developed to properly take account of corre- lations in the error terms in time series data. Correlation among the error
- 106. 96 3. Linear Regression 10 15 20 25 30 −10 −5 0 5 10 15 Fitted values Residuals Response Y 998 975 845 2.4 2.6 2.8 3.0 3.2 3.4 −0.8 −0.6 −0.4 −0.2 0.0 0.2 0.4 Fitted values Residuals Response log(Y) 437 671 605 FIGURE 3.11. Residual plots. In each plot, the red line is a smooth fit to the residuals, intended to make it easier to identify a trend. The blue lines track the outer quantiles of the residuals, and emphasize patterns. Left: The funnel shape indicates heteroscedasticity. Right: The response has been log transformed, and there is now no evidence of heteroscedasticity. terms can also occur outside of time series data. For instance, consider a study in which individuals’ heights are predicted from their weights. The assumption of uncorrelated errors could be violated if some of the indi- viduals in the study are members of the same family, eat the same diet, or have been exposed to the same environmental factors. In general, the assumption of uncorrelated errors is extremely important for linear regres- sion as well as for other statistical methods, and good experimental design is crucial in order to mitigate the risk of such correlations. 3. Non-constant Variance of Error Terms Another important assumption of the linear regression model is that the error terms have a constant variance, Var(ϵi) = σ2 . The standard errors, confidence intervals, and hypothesis tests associated with the linear model rely upon this assumption. Unfortunately, it is often the case that the variances of the error terms are non-constant. For instance, the variances of the error terms may increase with the value of the response. One can identify non-constant variances in the errors, or heteroscedasticity, from the presence of a funnel shape in hetero- scedasticity the residual plot. An example is shown in the left-hand panel of Figure 3.11, in which the magnitude of the residuals tends to increase with the fitted values. When faced with this problem, one possible solution is to trans- form the response Y using a concave function such as log Y or √ Y . Such a transformation results in a greater amount of shrinkage of the larger re- sponses, leading to a reduction in heteroscedasticity. The right-hand panel
- 107. 3.3 Other Considerations in the Regression Model 97 −2 −1 0 1 2 −4 −2 0 2 4 6 20 −2 0 2 4 6 −1 0 1 2 3 4 Fitted Values Residuals 20 −2 0 2 4 6 0 2 4 6 Fitted Values Studentized Residuals 20 X Y FIGURE 3.12. Left: The least squares regression line is shown in red, and the regression line after removing the outlier is shown in blue. Center: The residual plot clearly identifies the outlier. Right: The outlier has a studentized residual of 6; typically we expect values between −3 and 3. of Figure 3.11 displays the residual plot after transforming the response using log Y . The residuals now appear to have constant variance, though there is some evidence of a slight non-linear relationship in the data. Sometimes we have a good idea of the variance of each response. For example, the ith response could be an average of ni raw observations. If each of these raw observations is uncorrelated with variance σ2 , then their average has variance σ2 i = σ2 /ni. In this case a simple remedy is to fit our model by weighted least squares, with weights proportional to the inverse weighted least squares variances—i.e. wi = ni in this case. Most linear regression software allows for observation weights. 4. Outliers An outlier is a point for which yi is far from the value predicted by the outlier model. Outliers can arise for a variety of reasons, such as incorrect recording of an observation during data collection. The red point (observation 20) in the left-hand panel of Figure 3.12 illustrates a typical outlier. The red solid line is the least squares regression fit, while the blue dashed line is the least squares fit after removal of the outlier. In this case, removing the outlier has little effect on the least squares line: it leads to almost no change in the slope, and a miniscule reduction in the intercept. It is typical for an outlier that does not have an unusual predictor value to have little effect on the least squares fit. However, even if an outlier does not have much effect on the least squares fit, it can cause other problems. For instance, in this example, the RSE is 1.09 when the outlier is included in the regression, but it is only 0.77 when the outlier is removed. Since the RSE is used to compute all confidence intervals and p-values, such a dramatic increase caused by a single data point can have implications for the interpretation of the fit. Similarly, inclusion of the outlier causes the R2 to decline from 0.892 to 0.805.
- 108. 98 3. Linear Regression −2 −1 0 1 2 3 4 0 5 10 20 41 −2 −1 0 1 2 −2 −1 0 1 2 0.00 0.05 0.10 0.15 0.20 0.25 −1 0 1 2 3 4 5 Leverage Studentized Residuals 20 41 X Y X1 X 2 FIGURE 3.13. Left: Observation 41 is a high leverage point, while 20 is not. The red line is the fit to all the data, and the blue line is the fit with observation 41 removed. Center: The red observation is not unusual in terms of its X1 value or its X2 value, but still falls outside the bulk of the data, and hence has high leverage. Right: Observation 41 has a high leverage and a high residual. Residual plots can be used to identify outliers. In this example, the out- lier is clearly visible in the residual plot illustrated in the center panel of Figure 3.12. But in practice, it can be difficult to decide how large a resid- ual needs to be before we consider the point to be an outlier. To address this problem, instead of plotting the residuals, we can plot the studentized residuals, computed by dividing each residual ei by its estimated standard studentized residual error. Observations whose studentized residuals are greater than 3 in abso- lute value are possible outliers. In the right-hand panel of Figure 3.12, the outlier’s studentized residual exceeds 6, while all other observations have studentized residuals between −2 and 2. If we believe that an outlier has occurred due to an error in data collec- tion or recording, then one solution is to simply remove the observation. However, care should be taken, since an outlier may instead indicate a deficiency with the model, such as a missing predictor. 5. High Leverage Points We just saw that outliers are observations for which the response yi is unusual given the predictor xi. In contrast, observations with high leverage high leverage have an unusual value for xi. For example, observation 41 in the left-hand panel of Figure 3.13 has high leverage, in that the predictor value for this observation is large relative to the other observations. (Note that the data displayed in Figure 3.13 are the same as the data displayed in Figure 3.12, but with the addition of a single high leverage observation.) The red solid line is the least squares fit to the data, while the blue dashed line is the fit produced when observation 41 is removed. Comparing the left-hand panels of Figures 3.12 and 3.13, we observe that removing the high leverage observation has a much more substantial impact on the least squares line than removing the outlier. In fact, high leverage observations tend to have a sizable impact on the estimated regression line. It is cause for concern if
- 109. 3.3 Other Considerations in the Regression Model 99 the least squares line is heavily affected by just a couple of observations, because any problems with these points may invalidate the entire fit. For this reason, it is important to identify high leverage observations. In a simple linear regression, high leverage observations are fairly easy to identify, since we can simply look for observations for which the predictor value is outside of the normal range of the observations. But in a multiple linear regression with many predictors, it is possible to have an observation that is well within the range of each individual predictor’s values, but that is unusual in terms of the full set of predictors. An example is shown in the center panel of Figure 3.13, for a data set with two predictors, X1 and X2. Most of the observations’ predictor values fall within the blue dashed ellipse, but the red observation is well outside of this range. But neither its value for X1 nor its value for X2 is unusual. So if we examine just X1 or just X2, we will fail to notice this high leverage point. This problem is more pronounced in multiple regression settings with more than two predictors, because then there is no simple way to plot all dimensions of the data simultaneously. In order to quantify an observation’s leverage, we compute the leverage statistic. A large value of this statistic indicates an observation with high leverage statistic leverage. For a simple linear regression, hi = 1 n + (xi − x̄)2 )n i′=1(xi′ − x̄)2 . (3.37) It is clear from this equation that hi increases with the distance of xi from x̄. There is a simple extension of hi to the case of multiple predictors, though we do not provide the formula here. The leverage statistic hi is always between 1/n and 1, and the average leverage for all the observations is always equal to (p + 1)/n. So if a given observation has a leverage statistic that greatly exceeds (p+1)/n, then we may suspect that the corresponding point has high leverage. The right-hand panel of Figure 3.13 provides a plot of the studentized residuals versus hi for the data in the left-hand panel of Figure 3.13. Ob- servation 41 stands out as having a very high leverage statistic as well as a high studentized residual. In other words, it is an outlier as well as a high leverage observation. This is a particularly dangerous combination! This plot also reveals the reason that observation 20 had relatively little effect on the least squares fit in Figure 3.12: it has low leverage. 6. Collinearity Collinearity refers to the situation in which two or more predictor variables collinearity are closely related to one another. The concept of collinearity is illustrated in Figure 3.14 using the Credit data set. In the left-hand panel of Fig- ure 3.14, the two predictors limit and age appear to have no obvious rela- tionship. In contrast, in the right-hand panel of Figure 3.14, the predictors
- 110. 100 3. Linear Regression 2000 4000 6000 8000 12000 30 40 50 60 70 80 Limit Age 2000 4000 6000 8000 12000 200 400 600 800 Limit Rating FIGURE 3.14. Scatterplots of the observations from the Credit data set. Left: A plot of age versus limit. These two variables are not collinear. Right: A plot of rating versus limit. There is high collinearity. 21.25 21.5 21.8 0.16 0.17 0.18 0.19 −5 −4 −3 −2 −1 0 21.5 21.8 −0.1 0.0 0.1 0.2 0 1 2 3 4 5 βLimit βLimit β Age β Rating FIGURE 3.15. Contour plots for the RSS values as a function of the parameters β for various regressions involving the Credit data set. In each plot, the black dots represent the coefficient values corresponding to the minimum RSS. Left: A contour plot of RSS for the regression of balance onto age and limit. The minimum value is well defined. Right: A contour plot of RSS for the regression of balance onto rating and limit. Because of the collinearity, there are many pairs (βLimit, βRating) with a similar value for RSS. limit and rating are very highly correlated with each other, and we say that they are collinear. The presence of collinearity can pose problems in the regression context, since it can be difficult to separate out the indi- vidual effects of collinear variables on the response. In other words, since limit and rating tend to increase or decrease together, it can be difficult to determine how each one separately is associated with the response, balance.
- 111. 3.3 Other Considerations in the Regression Model 101 Figure 3.15 illustrates some of the difficulties that can result from collinear- ity. The left-hand panel of Figure 3.15 is a contour plot of the RSS (3.22) associated with different possible coefficient estimates for the regression of balance on limit and age. Each ellipse represents a set of coefficients that correspond to the same RSS, with ellipses nearest to the center tak- ing on the lowest values of RSS. The black dots and associated dashed lines represent the coefficient estimates that result in the smallest possible RSS—in other words, these are the least squares estimates. The axes for limit and age have been scaled so that the plot includes possible coeffi- cient estimates that are up to four standard errors on either side of the least squares estimates. Thus the plot includes all plausible values for the coefficients. For example, we see that the true limit coefficient is almost certainly somewhere between 0.15 and 0.20. In contrast, the right-hand panel of Figure 3.15 displays contour plots of the RSS associated with possible coefficient estimates for the regression of balance onto limit and rating, which we know to be highly collinear. Now the contours run along a narrow valley; there is a broad range of values for the coefficient estimates that result in equal values for RSS. Hence a small change in the data could cause the pair of coefficient values that yield the smallest RSS—that is, the least squares estimates—to move anywhere along this valley. This results in a great deal of uncertainty in the coefficient estimates. Notice that the scale for the limit coefficient now runs from roughly −0.2 to 0.2; this is an eight-fold increase over the plausible range of the limit coefficient in the regression with age. Interestingly, even though the limit and rating coefficients now have much more individual uncertainty, they will almost certainly lie somewhere in this contour valley. For example, we would not expect the true value of the limit and rating coefficients to be −0.1 and 1 respectively, even though such a value is plausible for each coefficient individually. Since collinearity reduces the accuracy of the estimates of the regression coefficients, it causes the standard error for β̂j to grow. Recall that the t-statistic for each predictor is calculated by dividing β̂j by its standard error. Consequently, collinearity results in a decline in the t-statistic. As a result, in the presence of collinearity, we may fail to reject H0 : βj = 0. This means that the power of the hypothesis test—the probability of correctly power detecting a non-zero coefficient—is reduced by collinearity. Table 3.11 compares the coefficient estimates obtained from two separate multiple regression models. The first is a regression of balance on age and limit, and the second is a regression of balance on rating and limit. In the first regression, both age and limit are highly significant with very small p- values. In the second, the collinearity between limit and rating has caused the standard error for the limit coefficient estimate to increase by a factor of 12 and the p-value to increase to 0.701. In other words, the importance of the limit variable has been masked due to the presence of collinearity.
- 112. 102 3. Linear Regression Coefficient Std. error t-statistic p-value Intercept −173.411 43.828 −3.957 < 0.0001 Model 1 age −2.292 0.672 −3.407 0.0007 limit 0.173 0.005 34.496 < 0.0001 Intercept −377.537 45.254 −8.343 < 0.0001 Model 2 rating 2.202 0.952 2.312 0.0213 limit 0.025 0.064 0.384 0.7012 TABLE 3.11. The results for two multiple regression models involving the Credit data set are shown. Model 1 is a regression of balance on age and limit, and Model 2 a regression of balance on rating and limit. The standard error of β̂limit increases 12-fold in the second regression, due to collinearity. To avoid such a situation, it is desirable to identify and address potential collinearity problems while fitting the model. A simple way to detect collinearity is to look at the correlation matrix of the predictors. An element of this matrix that is large in absolute value indicates a pair of highly correlated variables, and therefore a collinearity problem in the data. Unfortunately, not all collinearity problems can be detected by inspection of the correlation matrix: it is possible for collinear- ity to exist between three or more variables even if no pair of variables has a particularly high correlation. We call this situation multicollinearity. multi- collinearity Instead of inspecting the correlation matrix, a better way to assess multi- collinearity is to compute the variance inflation factor (VIF). The VIF is variance inflation factor the ratio of the variance of β̂j when fitting the full model divided by the variance of β̂j if fit on its own. The smallest possible value for VIF is 1, which indicates the complete absence of collinearity. Typically in practice there is a small amount of collinearity among the predictors. As a rule of thumb, a VIF value that exceeds 5 or 10 indicates a problematic amount of collinearity. The VIF for each variable can be computed using the formula VIF(β̂j) = 1 1 − R2 Xj |X−j , where R2 Xj |X−j is the R2 from a regression of Xj onto all of the other predictors. If R2 Xj |X−j is close to one, then collinearity is present, and so the VIF will be large. In the Credit data, a regression of balance on age, rating, and limit indicates that the predictors have VIF values of 1.01, 160.67, and 160.59. As we suspected, there is considerable collinearity in the data! When faced with the problem of collinearity, there are two simple solu- tions. The first is to drop one of the problematic variables from the regres- sion. This can usually be done without much compromise to the regression fit, since the presence of collinearity implies that the information that this variable provides about the response is redundant in the presence of the other variables. For instance, if we regress balance onto age and limit,
- 113. 3.4 The Marketing Plan 103 without the rating predictor, then the resulting VIF values are close to the minimum possible value of 1, and the R2 drops from 0.754 to 0.75. So dropping rating from the set of predictors has effectively solved the collinearity problem without compromising the fit. The second solution is to combine the collinear variables together into a single predictor. For in- stance, we might take the average of standardized versions of limit and rating in order to create a new variable that measures credit worthiness. 3.4 The Marketing Plan We now briefly return to the seven questions about the Advertising data that we set out to answer at the beginning of this chapter. 1. Is there a relationship between sales and advertising budget? This question can be answered by fitting a multiple regression model of sales onto TV, radio, and newspaper, as in (3.20), and testing the hypothesis H0 : βTV = βradio = βnewspaper = 0. In Section 3.2.2, we showed that the F-statistic can be used to determine whether or not we should reject this null hypothesis. In this case the p-value corresponding to the F-statistic in Table 3.6 is very low, indicating clear evidence of a relationship between advertising and sales. 2. How strong is the relationship? We discussed two measures of model accuracy in Section 3.1.3. First, the RSE estimates the standard deviation of the response from the population regression line. For the Advertising data, the RSE is 1.69 units while the mean value for the response is 14.022, indicating a percentage error of roughly 12 %. Second, the R2 statistic records the percentage of variability in the response that is explained by the predictors. The predictors explain almost 90 % of the variance in sales. The RSE and R2 statistics are displayed in Table 3.6. 3. Which media are associated with sales? To answer this question, we can examine the p-values associated with each predictor’s t-statistic (Section 3.1.2). In the multiple linear re- gression displayed in Table 3.4, the p-values for TV and radio are low, but the p-value for newspaper is not. This suggests that only TV and radio are related to sales. In Chapter 6 we explore this question in greater detail. 4. How large is the association between each medium and sales? We saw in Section 3.1.2 that the standard error of β̂j can be used to construct confidence intervals for βj. For the Advertising data, we
- 114. 104 3. Linear Regression can use the results in Table 3.4 to compute the 95 % confidence inter- vals for the coefficients in a multiple regression model using all three media budgets as predictors. The confidence intervals are as follows: (0.043, 0.049) for TV, (0.172, 0.206) for radio, and (−0.013, 0.011) for newspaper. The confidence intervals for TV and radio are narrow and far from zero, providing evidence that these media are related to sales. But the interval for newspaper includes zero, indicating that the variable is not statistically significant given the values of TV and radio. We saw in Section 3.3.3 that collinearity can result in very wide stan- dard errors. Could collinearity be the reason that the confidence in- terval associated with newspaper is so wide? The VIF scores are 1.005, 1.145, and 1.145 for TV, radio, and newspaper, suggesting no evidence of collinearity. In order to assess the association of each medium individually on sales, we can perform three separate simple linear regressions. Re- sults are shown in Tables 3.1 and 3.3. There is evidence of an ex- tremely strong association between TV and sales and between radio and sales. There is evidence of a mild association between newspaper and sales, when the values of TV and radio are ignored. 5. How accurately can we predict future sales? The response can be predicted using (3.21). The accuracy associ- ated with this estimate depends on whether we wish to predict an individual response, Y = f(X) + ϵ, or the average response, f(X) (Section 3.2.2). If the former, we use a prediction interval, and if the latter, we use a confidence interval. Prediction intervals will always be wider than confidence intervals because they account for the un- certainty associated with ϵ, the irreducible error. 6. Is the relationship linear? In Section 3.3.3, we saw that residual plots can be used in order to identify non-linearity. If the relationships are linear, then the residual plots should display no pattern. In the case of the Advertising data, we observe a non-linear effect in Figure 3.5, though this effect could also be observed in a residual plot. In Section 3.3.2, we discussed the inclusion of transformations of the predictors in the linear regression model in order to accommodate non-linear relationships. 7. Is there synergy among the advertising media? The standard linear regression model assumes an additive relation- ship between the predictors and the response. An additive model is easy to interpret because the association between each predictor and the response is unrelated to the values of the other predictors. However, the additive assumption may be unrealistic for certain data
- 115. 105 sets. In Section 3.3.2, we showed how to include an interaction term in the regression model in order to accommodate non-additive rela- tionships. A small p-value associated with the interaction term indi- cates the presence of such relationships. Figure 3.5 suggested that the Advertising data may not be additive. Including an interaction term in the model results in a substantial increase in R2 , from around 90 % to almost 97 %. 3.5 Comparison of Linear Regression with K-Nearest Neighbors As discussed in Chapter 2, linear regression is an example of a parametric approach because it assumes a linear functional form for f(X). Parametric methods have several advantages. They are often easy to fit, because one need estimate only a small number of coefficients. In the case of linear re- gression, the coefficients have simple interpretations, and tests of statistical significance can be easily performed. But parametric methods do have a disadvantage: by construction, they make strong assumptions about the form of f(X). If the specified functional form is far from the truth, and prediction accuracy is our goal, then the parametric method will perform poorly. For instance, if we assume a linear relationship between X and Y but the true relationship is far from linear, then the resulting model will provide a poor fit to the data, and any conclusions drawn from it will be suspect. In contrast, non-parametric methods do not explicitly assume a para- metric form for f(X), and thereby provide an alternative and more flexi- ble approach for performing regression. We discuss various non-parametric methods in this book. Here we consider one of the simplest and best-known non-parametric methods, K-nearest neighbors regression (KNN regression). K-nearest neighbors regression The KNN regression method is closely related to the KNN classifier dis- cussed in Chapter 2. Given a value for K and a prediction point x0, KNN regression first identifies the K training observations that are closest to x0, represented by N0. It then estimates f(x0) using the average of all the training responses in N0. In other words, ˆ f(x0) = 1 K 0 xi∈N0 yi. Figure 3.16 illustrates two KNN fits on a data set with p = 2 predictors. The fit with K = 1 is shown in the left-hand panel, while the right-hand panel corresponds to K = 9. We see that when K = 1, the KNN fit perfectly interpolates the training observations, and consequently takes the form of a step function. When K = 9, the KNN fit still is a step function, but averaging over nine observations results in much smaller regions of constant 3.5 Comparison of Linear Regression with K-Nearest Neighbors
- 116. 106 3. Linear Regression y y x1 x1 x 2 x 2 y y y FIGURE 3.16. Plots of ˆ f(X) using KNN regression on a two-dimensional data set with 64 observations (orange dots). Left: K = 1 results in a rough step func- tion fit. Right: K = 9 produces a much smoother fit. prediction, and consequently a smoother fit. In general, the optimal value for K will depend on the bias-variance tradeoff, which we introduced in Chapter 2. A small value for K provides the most flexible fit, which will have low bias but high variance. This variance is due to the fact that the prediction in a given region is entirely dependent on just one observation. In contrast, larger values of K provide a smoother and less variable fit; the prediction in a region is an average of several points, and so changing one observation has a smaller effect. However, the smoothing may cause bias by masking some of the structure in f(X). In Chapter 5, we introduce several approaches for estimating test error rates. These methods can be used to identify the optimal value of K in KNN regression. In what setting will a parametric approach such as least squares linear re- gression outperform a non-parametric approach such as KNN regression? The answer is simple: the parametric approach will outperform the non- parametric approach if the parametric form that has been selected is close to the true form of f. Figure 3.17 provides an example with data generated from a one-dimensional linear regression model. The black solid lines rep- resent f(X), while the blue curves correspond to the KNN fits using K = 1 and K = 9. In this case, the K = 1 predictions are far too variable, while the smoother K = 9 fit is much closer to f(X). However, since the true relationship is linear, it is hard for a non-parametric approach to compete with linear regression: a non-parametric approach incurs a cost in variance that is not offset by a reduction in bias. The blue dashed line in the left- hand panel of Figure 3.18 represents the linear regression fit to the same data. It is almost perfect. The right-hand panel of Figure 3.18 reveals that linear regression outperforms KNN for this data. The green solid line, plot-
- 117. 3.5 Comparison of Linear Regression with K-Nearest Neighbors 107 ted as a function of 1/K, represents the test set mean squared error (MSE) for KNN. The KNN errors are well above the black dashed line, which is the test MSE for linear regression. When the value of K is large, then KNN performs only a little worse than least squares regression in terms of MSE. It performs far worse when K is small. In practice, the true relationship between X and Y is rarely exactly lin- ear. Figure 3.19 examines the relative performances of least squares regres- sion and KNN under increasing levels of non-linearity in the relationship between X and Y . In the top row, the true relationship is nearly linear. In this case we see that the test MSE for linear regression is still superior to that of KNN for low values of K. However, for K ≥ 4, KNN out- performs linear regression. The second row illustrates a more substantial deviation from linearity. In this situation, KNN substantially outperforms linear regression for all values of K. Note that as the extent of non-linearity increases, there is little change in the test set MSE for the non-parametric KNN method, but there is a large increase in the test set MSE of linear regression. Figures 3.18 and 3.19 display situations in which KNN performs slightly worse than linear regression when the relationship is linear, but much bet- ter than linear regression for non-linear situations. In a real life situation in which the true relationship is unknown, one might suspect that KNN should be favored over linear regression because it will at worst be slightly inferior to linear regression if the true relationship is linear, and may give substantially better results if the true relationship is non-linear. But in re- ality, even when the true relationship is highly non-linear, KNN may still provide inferior results to linear regression. In particular, both Figures 3.18 and 3.19 illustrate settings with p = 1 predictor. But in higher dimensions, KNN often performs worse than linear regression. Figure 3.20 considers the same strongly non-linear situation as in the second row of Figure 3.19, except that we have added additional noise predictors that are not associated with the response. When p = 1 or p = 2, KNN outperforms linear regression. But for p = 3 the results are mixed, and for p ≥ 4 linear regression is superior to KNN. In fact, the increase in dimension has only caused a small deterioration in the linear regression test set MSE, but it has caused more than a ten-fold increase in the MSE for KNN. This decrease in performance as the dimension increases is a common problem for KNN, and results from the fact that in higher dimensions there is effectively a reduction in sample size. In this data set there are 50 training observations; when p = 1, this provides enough information to accurately estimate f(X). However, spreading 50 observations over p = 20 dimensions results in a phenomenon in which a given observation has no nearby neighbors—this is the so-called curse of dimensionality. That is, curse of di- mensionality the K observations that are nearest to a given test observation x0 may be very far away from x0 in p-dimensional space when p is large, leading to a very poor prediction of f(x0) and hence a poor KNN fit. As a general rule,
- 118. 108 3. Linear Regression −1.0 −0.5 0.0 0.5 1.0 1 2 3 4 −1.0 −0.5 0.0 0.5 1.0 1 2 3 4 y y x x FIGURE 3.17. Plots of ˆ f(X) using KNN regression on a one-dimensional data set with 50 observations. The true relationship is given by the black solid line. Left: The blue curve corresponds to K = 1 and interpolates (i.e. passes directly through) the training data. Right: The blue curve corresponds to K = 9, and represents a smoother fit. −1.0 −0.5 0.0 0.5 1.0 1 2 3 4 0.2 0.5 1.0 0.00 0.05 0.10 0.15 Mean Squared Error y x 1/K FIGURE 3.18. The same data set shown in Figure 3.17 is investigated further. Left: The blue dashed line is the least squares fit to the data. Since f(X) is in fact linear (displayed as the black line), the least squares regression line provides a very good estimate of f(X). Right: The dashed horizontal line represents the least squares test set MSE, while the green solid line corresponds to the MSE for KNN as a function of 1/K (on the log scale). Linear regression achieves a lower test MSE than does KNN regression, since f(X) is in fact linear. For KNN regression, the best results occur with a very large value of K, corresponding to a small value of 1/K.
- 119. 3.5 Comparison of Linear Regression with K-Nearest Neighbors 109 −1.0 −0.5 0.0 0.5 1.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 0.2 0.5 1.0 0.00 0.02 0.04 0.06 0.08 Mean Squared Error −1.0 −0.5 0.0 0.5 1.0 1.0 1.5 2.0 2.5 3.0 3.5 0.2 0.5 1.0 0.00 0.05 0.10 0.15 Mean Squared Error y y x x 1/K 1/K FIGURE 3.19. Top Left: In a setting with a slightly non-linear relationship between X and Y (solid black line), the KNN fits with K = 1 (blue) and K = 9 (red) are displayed. Top Right: For the slightly non-linear data, the test set MSE for least squares regression (horizontal black) and KNN with various values of 1/K (green) are displayed. Bottom Left and Bottom Right: As in the top panel, but with a strongly non-linear relationship between X and Y .
- 120. 110 3. Linear Regression 0.2 0.5 1.0 0.0 0.2 0.4 0.6 0.8 1.0 p=1 0.2 0.5 1.0 0.0 0.2 0.4 0.6 0.8 1.0 p=2 0.2 0.5 1.0 0.0 0.2 0.4 0.6 0.8 1.0 p=3 0.2 0.5 1.0 0.0 0.2 0.4 0.6 0.8 1.0 p=4 0.2 0.5 1.0 0.0 0.2 0.4 0.6 0.8 1.0 p=10 0.2 0.5 1.0 0.0 0.2 0.4 0.6 0.8 1.0 p=20 Mean Squared Error 1/K FIGURE 3.20. Test MSE for linear regression (black dashed lines) and KNN (green curves) as the number of variables p increases. The true function is non– linear in the first variable, as in the lower panel in Figure 3.19, and does not depend on the additional variables. The performance of linear regression deteri- orates slowly in the presence of these additional noise variables, whereas KNN’s performance degrades much more quickly as p increases. parametric methods will tend to outperform non-parametric approaches when there is a small number of observations per predictor. Even when the dimension is small, we might prefer linear regression to KNN from an interpretability standpoint. If the test MSE of KNN is only slightly lower than that of linear regression, we might be willing to forego a little bit of prediction accuracy for the sake of a simple model that can be described in terms of just a few coefficients, and for which p-values are available. 3.6 Lab: Linear Regression 3.6.1 Libraries The library() function is used to load libraries, or groups of functions library() and data sets that are not included in the base R distribution. Basic func- tions that perform least squares linear regression and other simple analyses come standard with the base distribution, but more exotic functions require additional libraries. Here we load the MASS package, which is a very large collection of data sets and functions. We also load the ISLR2 package, which includes the data sets associated with this book. > library(MASS) > library(ISLR2) If you receive an error message when loading any of these libraries, it likely indicates that the corresponding library has not yet been installed on your system. Some libraries, such as MASS, come with R and do not need to be separately installed on your computer. However, other packages, such as
- 121. 3.6 Lab: Linear Regression 111 ISLR2, must be downloaded the first time they are used. This can be done di- rectly from within R. For example, on a Windows system, select the Install package option under the Packages tab. After you select any mirror site, a list of available packages will appear. Simply select the package you wish to install and R will automatically download the package. Alternatively, this can be done at the R command line via install.packages("ISLR2"). This installation only needs to be done the first time you use a package. However, the library() function must be called within each R session. 3.6.2 Simple Linear Regression The ISLR2 library contains the Boston data set, which records medv (me- dian house value) for 506 census tracts in Boston. We will seek to predict medv using 12 predictors such as rm (average number of rooms per house), age (average age of houses), and lstat (percent of households with low socioeconomic status). > head(Boston) crim zn indus chas nox rm age dis rad tax 1 0.00632 18 2.31 0 0.538 6.575 65.2 4.0900 1 296 2 0.02731 0 7.07 0 0.469 6.421 78.9 4.9671 2 242 3 0.02729 0 7.07 0 0.469 7.185 61.1 4.9671 2 242 4 0.03237 0 2.18 0 0.458 6.998 45.8 6.0622 3 222 5 0.06905 0 2.18 0 0.458 7.147 54.2 6.0622 3 222 6 0.02985 0 2.18 0 0.458 6.430 58.7 6.0622 3 222 ptratio lstat medv 1 15.3 4.98 24.0 2 17.8 9.14 21.6 3 17.8 4.03 34.7 4 18.7 2.94 33.4 5 18.7 5.33 36.2 6 18.7 5.21 28.7 To find out more about the data set, we can type ?Boston. We will start by using the lm() function to fit a simple linear regression lm() model, with medv as the response and lstat as the predictor. The basic syntax is lm(y ∼ x, data), where y is the response, x is the predictor, and data is the data set in which these two variables are kept. > lm.fit <- lm(medv ∼ lstat) Error in eval(expr , envir , enclos) : Object "medv" not found The command causes an error because R does not know where to find the variables medv and lstat. The next line tells R that the variables are in Boston. If we attach Boston, the first line works fine because R now recognizes the variables. > lm.fit <- lm(medv ∼ lstat , data = Boston) > attach(Boston) > lm.fit <- lm(medv ∼ lstat)
- 122. 112 3. Linear Regression If we type lm.fit, some basic information about the model is output. For more detailed information, we use summary(lm.fit). This gives us p- values and standard errors for the coefficients, as well as the R2 statistic and F-statistic for the model. > lm.fit Call: lm(formula = medv ∼ lstat) Coefficients : (Intercept) lstat 34.55 -0.95 > summary(lm.fit) Call: lm(formula = medv ∼ lstat) Residuals: Min 1Q Median 3Q Max -15.17 -3.99 -1.32 2.03 24.50 Coefficients : Estimate Std. Error t value Pr(>|t|) (Intercept) 34.5538 0.5626 61.4 <2e -16 *** lstat -0.9500 0.0387 -24.5 <2e -16 *** --- Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1 Residual standard error: 6.22 on 504 degrees of freedom Multiple R-squared: 0.544 , Adjusted R-squared: 0.543 F-statistic: 602 on 1 and 504 DF , p-value: < 2e-16 We can use the names() function in order to find out what other pieces names() of information are stored in lm.fit. Although we can extract these quan- tities by name—e.g. lm.fit$coefficients—it is safer to use the extractor functions like coef() to access them. coef() > names(lm.fit) [1] " coefficients " "residuals" "effects" [4] "rank" "fitted.values" "assign" [7] "qr" "df.residual" "xlevels" [10] "call" "terms" "model" > coef(lm.fit) (Intercept) lstat 34.55 -0.95 In order to obtain a confidence interval for the coefficient estimates, we can use the confint() command. confint() > confint(lm.fit) 2.5 % 97.5 % (Intercept) 33.45 35.659
- 123. 3.6 Lab: Linear Regression 113 lstat -1.03 -0.874 The predict() function can be used to produce confidence intervals and predict() prediction intervals for the prediction of medv for a given value of lstat. > predict(lm.fit , data.frame(lstat = (c(5, 10, 15))), interval = "confidence ") fit lwr upr 1 29.80 29.01 30.60 2 25.05 24.47 25.63 3 20.30 19.73 20.87 > predict(lm.fit , data.frame(lstat = (c(5, 10, 15))), interval = "prediction ") fit lwr upr 1 29.80 17.566 42.04 2 25.05 12.828 37.28 3 20.30 8.078 32.53 For instance, the 95 % confidence interval associated with a lstat value of 10 is (24.47, 25.63), and the 95 % prediction interval is (12.828, 37.28). As expected, the confidence and prediction intervals are centered around the same point (a predicted value of 25.05 for medv when lstat equals 10), but the latter are substantially wider. We will now plot medv and lstat along with the least squares regression line using the plot() and abline() functions. abline() > plot(lstat , medv) > abline(lm.fit) There is some evidence for non-linearity in the relationship between lstat and medv. We will explore this issue later in this lab. The abline() function can be used to draw any line, not just the least squares regression line. To draw a line with intercept a and slope b, we type abline(a, b). Below we experiment with some additional settings for plotting lines and points. The lwd = 3 command causes the width of the regression line to be increased by a factor of 3; this works for the plot() and lines() functions also. We can also use the pch option to create different plotting symbols. > abline(lm.fit , lwd = 3) > abline(lm.fit , lwd = 3, col = "red") > plot(lstat , medv , col = "red") > plot(lstat , medv , pch = 20) > plot(lstat , medv , pch = "+") > plot (1:20 , 1:20 , pch = 1:20) Next we examine some diagnostic plots, several of which were discussed in Section 3.3.3. Four diagnostic plots are automatically produced by ap- plying the plot() function directly to the output from lm(). In general, this command will produce one plot at a time, and hitting Enter will generate the next plot. However, it is often convenient to view all four plots together. We can achieve this by using the par() and mfrow() functions, which tell R par() mfrow()
- 124. 114 3. Linear Regression to split the display screen into separate panels so that multiple plots can be viewed simultaneously. For example, par(mfrow = c(2, 2)) divides the plotting region into a 2 × 2 grid of panels. > par(mfrow = c(2, 2)) > plot(lm.fit) Alternatively, we can compute the residuals from a linear regression fit using the residuals() function. The function rstudent() will return the residuals() rstudent() studentized residuals, and we can use this function to plot the residuals against the fitted values. > plot(predict(lm.fit), residuals (lm.fit)) > plot(predict(lm.fit), rstudent (lm.fit)) On the basis of the residual plots, there is some evidence of non-linearity. Leverage statistics can be computed for any number of predictors using the hatvalues() function. hatvalues() > plot(hatvalues (lm.fit)) > which.max(hatvalues (lm.fit)) 375 The which.max() function identifies the index of the largest element of a which.max() vector. In this case, it tells us which observation has the largest leverage statistic. 3.6.3 Multiple Linear Regression In order to fit a multiple linear regression model using least squares, we again use the lm() function. The syntax lm(y ∼ x1 + x2 + x3) is used to fit a model with three predictors, x1, x2, and x3. The summary() function now outputs the regression coefficients for all the predictors. > lm.fit <- lm(medv ∼ lstat + age , data = Boston) > summary(lm.fit) Call: lm(formula = medv ∼ lstat + age , data = Boston) Residuals: Min 1Q Median 3Q Max -15.98 -3.98 -1.28 1.97 23.16 Coefficients : Estimate Std. Error t value Pr(>|t|) (Intercept) 33.2228 0.7308 45.46 <2e-16 *** lstat -1.0321 0.0482 -21.42 <2e-16 *** age 0.0345 0.0122 2.83 0.0049 ** --- Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1 Residual standard error: 6.17 on 503 degrees of freedom
- 125. 3.6 Lab: Linear Regression 115 Multiple R-squared: 0.551 , Adjusted R-squared: 0.549 F-statistic: 309 on 2 and 503 DF , p-value: < 2e-16 The Boston data set contains 12 variables, and so it would be cumbersome to have to type all of these in order to perform a regression using all of the predictors. Instead, we can use the following short-hand: > lm.fit <- lm(medv ∼ ., data = Boston) > summary(lm.fit) Call: lm(formula = medv ∼ ., data = Boston) Residuals: Min 1Q Median 3Q Max -15.130 -2.767 -0.581 1.941 26.253 Coefficients : Estimate Std. Error t value Pr(>|t|) (Intercept) 41.61727 4.93604 8.43 3.8e-16 *** crim -0.12139 0.03300 -3.68 0.00026 *** zn 0.04696 0.01388 3.38 0.00077 *** indus 0.01347 0.06214 0.22 0.82852 chas 2.83999 0.87001 3.26 0.00117 ** nox -18.75802 3.85135 -4.87 1.5e -06 *** rm 3.65812 0.42025 8.70 < 2e -16 *** age 0.00361 0.01333 0.27 0.78659 dis -1.49075 0.20162 -7.39 6.2e -13 *** rad 0.28940 0.06691 4.33 1.8e -05 *** tax -0.01268 0.00380 -3.34 0.00091 *** ptratio -0.93753 0.13221 -7.09 4.6e-12 *** lstat -0.55202 0.05066 -10.90 < 2e-16 *** --- Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1 Residual standard error: 4.8 on 493 degrees of freedom Multiple R-squared: 0.734 , Adjusted R-squared: 0.728 F-statistic: 114 on 12 and 493 DF , p-value: < 2e -16 We can access the individual components of a summary object by name (type ?summary.lm to see what is available). Hence summary(lm.fit)$r.sq gives us the R2 , and summary(lm.fit)$sigma gives us the RSE. The vif() vif() function, part of the car package, can be used to compute variance inflation factors. Most VIF’s are low to moderate for this data. The car package is not part of the base R installation so it must be downloaded the first time you use it via the install.packages() function in R. > library(car) > vif(lm.fit) crim zn indus chas nox rm age dis 1.77 2.30 3.99 1.07 4.37 1.91 3.09 3.95 rad tax ptratio lstat 7.45 9.00 1.80 2.87
- 126. 116 3. Linear Regression What if we would like to perform a regression using all of the variables but one? For example, in the above regression output, age has a high p-value. So we may wish to run a regression excluding this predictor. The following syntax results in a regression using all predictors except age. > lm.fit1 <- lm(medv ∼ . - age , data = Boston) > summary(lm.fit1) ... Alternatively, the update() function can be used. update() > lm.fit1 <- update(lm.fit , ∼ . - age) 3.6.4 Interaction Terms It is easy to include interaction terms in a linear model using the lm() function. The syntax lstat:black tells R to include an interaction term be- tween lstat and black. The syntax lstat * age simultaneously includes lstat, age, and the interaction term lstat×age as predictors; it is a short- hand for lstat + age + lstat:age. > summary(lm(medv ∼ lstat * age , data = Boston)) Call: lm(formula = medv ∼ lstat * age , data = Boston) Residuals: Min 1Q Median 3Q Max -15.81 -4.04 -1.33 2.08 27.55 Coefficients : Estimate Std. Error t value Pr(>|t|) (Intercept) 36.088536 1.469835 24.55 < 2e -16 *** lstat -1.392117 0.167456 -8.31 8.8e -16 *** age -0.000721 0.019879 -0.04 0.971 lstat:age 0.004156 0.001852 2.24 0.025 * --- Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1 Residual standard error: 6.15 on 502 degrees of freedom Multiple R-squared: 0.556 , Adjusted R-squared: 0.553 F-statistic: 209 on 3 and 502 DF , p-value: < 2e-16 3.6.5 Non-linear Transformations of the Predictors The lm() function can also accommodate non-linear transformations of the predictors. For instance, given a predictor X, we can create a predictor X2 using I(X^2). The function I() is needed since the ^ has a special meaning I() in a formula object; wrapping as we do allows the standard usage in R, which is to raise X to the power 2. We now perform a regression of medv onto lstat and lstat2 .
- 127. 3.6 Lab: Linear Regression 117 > lm.fit2 <- lm(medv ∼ lstat + I(lstat ^2)) > summary(lm.fit2) Call: lm(formula = medv ∼ lstat + I(lstat ^2)) Residuals: Min 1Q Median 3Q Max -15.28 -3.83 -0.53 2.31 25.41 Coefficients : Estimate Std. Error t value Pr(>|t|) (Intercept) 42.86201 0.87208 49.1 <2e -16 *** lstat -2.33282 0.12380 -18.8 <2e -16 *** I(lstat ^2) 0.04355 0.00375 11.6 <2e-16 *** --- Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1 Residual standard error: 5.52 on 503 degrees of freedom Multiple R-squared: 0.641 , Adjusted R-squared: 0.639 F-statistic: 449 on 2 and 503 DF , p-value: < 2e-16 The near-zero p-value associated with the quadratic term suggests that it leads to an improved model. We use the anova() function to further anova() quantify the extent to which the quadratic fit is superior to the linear fit. > lm.fit <- lm(medv ∼ lstat) > anova(lm.fit , lm.fit2) Analysis of Variance Table Model 1: medv ∼ lstat Model 2: medv ∼ lstat + I(lstat ^2) Res.Df RSS Df Sum of Sq F Pr(>F) 1 504 19472 2 503 15347 1 4125 135 <2e -16 *** --- Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1 Here Model 1 represents the linear submodel containing only one predictor, lstat, while Model 2 corresponds to the larger quadratic model that has two predictors, lstat and lstat2 . The anova() function performs a hypothesis test comparing the two models. The null hypothesis is that the two models fit the data equally well, and the alternative hypothesis is that the full model is superior. Here the F-statistic is 135 and the associated p-value is virtually zero. This provides very clear evidence that the model containing the predictors lstat and lstat2 is far superior to the model that only contains the predictor lstat. This is not surprising, since earlier we saw evidence for non-linearity in the relationship between medv and lstat. If we type > par(mfrow = c(2, 2)) > plot(lm.fit2)
- 128. 118 3. Linear Regression then we see that when the lstat2 term is included in the model, there is little discernible pattern in the residuals. In order to create a cubic fit, we can include a predictor of the form I(X^3). However, this approach can start to get cumbersome for higher- order polynomials. A better approach involves using the poly() function poly() to create the polynomial within lm(). For example, the following command produces a fifth-order polynomial fit: > lm.fit5 <- lm(medv ∼ poly(lstat , 5)) > summary(lm.fit5) Call: lm(formula = medv ∼ poly(lstat , 5)) Residuals: Min 1Q Median 3Q Max -13.543 -3.104 -0.705 2.084 27.115 Coefficients : Estimate Std. Error t value Pr(>|t|) (Intercept) 22.533 0.232 97.20 < 2e -16 *** poly(lstat , 5)1 -152.460 5.215 -29.24 < 2e-16 *** poly(lstat , 5)2 64.227 5.215 12.32 < 2e -16 *** poly(lstat , 5)3 -27.051 5.215 -5.19 3.1e-07 *** poly(lstat , 5)4 25.452 5.215 4.88 1.4e -06 *** poly(lstat , 5)5 -19.252 5.215 -3.69 0.00025 *** --- Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1 Residual standard error: 5.21 on 500 degrees of freedom Multiple R-squared: 0.682 , Adjusted R-squared: 0.679 F-statistic: 214 on 5 and 500 DF , p-value: < 2e-16 This suggests that including additional polynomial terms, up to fifth order, leads to an improvement in the model fit! However, further investigation of the data reveals that no polynomial terms beyond fifth order have signifi- cant p-values in a regression fit. By default, the poly() function orthogonalizes the predictors: this means that the features output by this function are not simply a sequence of powers of the argument. However, a linear model applied to the output of the poly() function will have the same fitted values as a linear model applied to the raw polynomials (although the coefficient estimates, standard errors, and p-values will differ). In order to obtain the raw polynomials from the poly() function, the argument raw = TRUE must be used. Of course, we are in no way restricted to using polynomial transforma- tions of the predictors. Here we try a log transformation. > summary(lm(medv ∼ log(rm), data = Boston)) ...
- 129. 3.6 Lab: Linear Regression 119 3.6.6 Qualitative Predictors We will now examine the Carseats data, which is part of the ISLR2 library. We will attempt to predict Sales (child car seat sales) in 400 locations based on a number of predictors. > head(Carseats) Sales CompPrice Income Advertising Population Price 1 9.50 138 73 11 276 120 2 11.22 111 48 16 260 83 3 10.06 113 35 10 269 80 4 7.40 117 100 4 466 97 5 4.15 141 64 3 340 128 6 10.81 124 113 13 501 72 ShelveLoc Age Education Urban US 1 Bad 42 17 Yes Yes 2 Good 65 10 Yes Yes 3 Medium 59 12 Yes Yes 4 Medium 55 14 Yes Yes 5 Bad 38 13 Yes No 6 Bad 78 16 No Yes The Carseats data includes qualitative predictors such as Shelveloc, an in- dicator of the quality of the shelving location—that is, the space within a store in which the car seat is displayed—at each location. The pre- dictor Shelveloc takes on three possible values: Bad, Medium, and Good. Given a qualitative variable such as Shelveloc, R generates dummy variables automatically. Below we fit a multiple regression model that includes some interaction terms. > lm.fit <- lm(Sales ∼ . + Income: Advertising + Price:Age , data = Carseats) > summary(lm.fit) Call: lm(formula = Sales ∼ . + Income: Advertising + Price:Age , data = Carseats) Residuals: Min 1Q Median 3Q Max -2.921 -0.750 0.018 0.675 3.341 Coefficients : Estimate Std. Error t value Pr(>|t|) (Intercept) 6.575565 1.008747 6.52 2.2e-10 *** CompPrice 0.092937 0.004118 22.57 < 2e-16 *** Income 0.010894 0.002604 4.18 3.6e -05 *** Advertising 0.070246 0.022609 3.11 0.00203 ** Population 0.000159 0.000368 0.43 0.66533 Price -0.100806 0.007440 -13.55 < 2e-16 *** ShelveLocGood 4.848676 0.152838 31.72 < 2e-16 *** ShelveLocMedium 1.953262 0.125768 15.53 < 2e -16 *** Age -0.057947 0.015951 -3.63 0.00032 ***
- 130. 120 3. Linear Regression Education -0.020852 0.019613 -1.06 0.28836 UrbanYes 0.140160 0.112402 1.25 0.21317 USYes -0.157557 0.148923 -1.06 0.29073 Income: Advertising 0.000751 0.000278 2.70 0.00729 ** Price:Age 0.000107 0.000133 0.80 0.42381 --- Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1 Residual standard error: 1.01 on 386 degrees of freedom Multiple R-squared: 0.876 , Adjusted R-squared: 0.872 F-statistic: 210 on 13 and 386 DF , p-value: < 2e -16 The contrasts() function returns the coding that R uses for the dummy contrasts() variables. > attach(Carseats) > contrasts (ShelveLoc) Good Medium Bad 0 0 Good 1 0 Medium 0 1 Use ?contrasts to learn about other contrasts, and how to set them. R has created a ShelveLocGood dummy variable that takes on a value of 1 if the shelving location is good, and 0 otherwise. It has also created a ShelveLocMedium dummy variable that equals 1 if the shelving location is medium, and 0 otherwise. A bad shelving location corresponds to a zero for each of the two dummy variables. The fact that the coefficient for ShelveLocGood in the regression output is positive indicates that a good shelving location is associated with high sales (relative to a bad location). And ShelveLocMedium has a smaller positive coefficient, indicating that a medium shelving location is associated with higher sales than a bad shelv- ing location but lower sales than a good shelving location. 3.6.7 Writing Functions As we have seen, R comes with many useful functions, and still more func- tions are available by way of R libraries. However, we will often be inter- ested in performing an operation for which no function is available. In this setting, we may want to write our own function. For instance, below we provide a simple function that reads in the ISLR2 and MASS libraries, called LoadLibraries(). Before we have created the function, R returns an error if we try to call it. > LoadLibraries Error: object ‘LoadLibraries ’ not found > LoadLibraries () Error: could not find function " LoadLibraries " We now create the function. Note that the + symbols are printed by R and should not be typed in. The { symbol informs R that multiple commands
- 131. 3.7 Exercises 121 are about to be input. Hitting Enter after typing { will cause R to print the + symbol. We can then input as many commands as we wish, hitting Enter after each one. Finally the } symbol informs R that no further commands will be entered. > LoadLibraries <- function () { + library(ISLR2) + library(MASS) + print("The libraries have been loaded.") + } Now if we type in LoadLibraries, R will tell us what is in the function. > LoadLibraries function () { library(ISLR2) library(MASS) print (" The libraries have been loaded .") } If we call the function, the libraries are loaded in and the print statement is output. > LoadLibraries () [1] "The libraries have been loaded ." 3.7 Exercises Conceptual 1. Describe the null hypotheses to which the p-values given in Table 3.4 correspond. Explain what conclusions you can draw based on these p-values. Your explanation should be phrased in terms of sales, TV, radio, and newspaper, rather than in terms of the coefficients of the linear model. 2. Carefully explain the differences between the KNN classifier and KNN regression methods. 3. Suppose we have a data set with five predictors, X1 = GPA, X2 = IQ, X3 = Level (1 for College and 0 for High School), X4 = Interac- tion between GPA and IQ, and X5 = Interaction between GPA and Level. The response is starting salary after graduation (in thousands of dollars). Suppose we use least squares to fit the model, and get β̂0 = 50, β̂1 = 20, β̂2 = 0.07, β̂3 = 35, β̂4 = 0.01, β̂5 = −10. (a) Which answer is correct, and why? i. For a fixed value of IQ and GPA, high school graduates earn more, on average, than college graduates.
- 132. 122 3. Linear Regression ii. For a fixed value of IQ and GPA, college graduates earn more, on average, than high school graduates. iii. For a fixed value of IQ and GPA, high school graduates earn more, on average, than college graduates provided that the GPA is high enough. iv. For a fixed value of IQ and GPA, college graduates earn more, on average, than high school graduates provided that the GPA is high enough. (b) Predict the salary of a college graduate with IQ of 110 and a GPA of 4.0. (c) True or false: Since the coefficient for the GPA/IQ interaction term is very small, there is very little evidence of an interaction effect. Justify your answer. 4. I collect a set of data (n = 100 observations) containing a single predictor and a quantitative response. I then fit a linear regression model to the data, as well as a separate cubic regression, i.e. Y = β0 + β1X + β2X2 + β3X3 + ϵ. (a) Suppose that the true relationship between X and Y is linear, i.e. Y = β0 + β1X + ϵ. Consider the training residual sum of squares (RSS) for the linear regression, and also the training RSS for the cubic regression. Would we expect one to be lower than the other, would we expect them to be the same, or is there not enough information to tell? Justify your answer. (b) Answer (a) using test rather than training RSS. (c) Suppose that the true relationship between X and Y is not linear, but we don’t know how far it is from linear. Consider the training RSS for the linear regression, and also the training RSS for the cubic regression. Would we expect one to be lower than the other, would we expect them to be the same, or is there not enough information to tell? Justify your answer. (d) Answer (c) using test rather than training RSS. 5. Consider the fitted values that result from performing linear regres- sion without an intercept. In this setting, the ith fitted value takes the form ŷi = xiβ̂, where β̂ = > n 0 i=1 xiyi ? / > n 0 i′=1 x2 i′ ? . (3.38)
- 133. 3.7 Exercises 123 Show that we can write ŷi = n 0 i′=1 ai′ yi′ . What is ai′ ? Note: We interpret this result by saying that the fitted values from linear regression are linear combinations of the response values. 6. Using (3.4), argue that in the case of simple linear regression, the least squares line always passes through the point (x̄, ȳ). 7. It is claimed in the text that in the case of simple linear regression of Y onto X, the R2 statistic (3.17) is equal to the square of the correlation between X and Y (3.18). Prove that this is the case. For simplicity, you may assume that x̄ = ȳ = 0. Applied 8. This question involves the use of simple linear regression on the Auto data set. (a) Use the lm() function to perform a simple linear regression with mpg as the response and horsepower as the predictor. Use the summary() function to print the results. Comment on the output. For example: i. Is there a relationship between the predictor and the re- sponse? ii. How strong is the relationship between the predictor and the response? iii. Is the relationship between the predictor and the response positive or negative? iv. What is the predicted mpg associated with a horsepower of 98? What are the associated 95 % confidence and prediction intervals? (b) Plot the response and the predictor. Use the abline() function to display the least squares regression line. (c) Use the plot() function to produce diagnostic plots of the least squares regression fit. Comment on any problems you see with the fit. 9. This question involves the use of multiple linear regression on the Auto data set.
- 134. 124 3. Linear Regression (a) Produce a scatterplot matrix which includes all of the variables in the data set. (b) Compute the matrix of correlations between the variables using the function cor(). You will need to exclude the name variable, cor() which is qualitative. (c) Use the lm() function to perform a multiple linear regression with mpg as the response and all other variables except name as the predictors. Use the summary() function to print the results. Comment on the output. For instance: i. Is there a relationship between the predictors and the re- sponse? ii. Which predictors appear to have a statistically significant relationship to the response? iii. What does the coefficient for the year variable suggest? (d) Use the plot() function to produce diagnostic plots of the linear regression fit. Comment on any problems you see with the fit. Do the residual plots suggest any unusually large outliers? Does the leverage plot identify any observations with unusually high leverage? (e) Use the * and : symbols to fit linear regression models with interaction effects. Do any interactions appear to be statistically significant? (f) Try a few different transformations of the variables, such as log(X), √ X, X2 . Comment on your findings. 10. This question should be answered using the Carseats data set. (a) Fit a multiple regression model to predict Sales using Price, Urban, and US. (b) Provide an interpretation of each coefficient in the model. Be careful—some of the variables in the model are qualitative! (c) Write out the model in equation form, being careful to handle the qualitative variables properly. (d) For which of the predictors can you reject the null hypothesis H0 : βj = 0? (e) On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome. (f) How well do the models in (a) and (e) fit the data? (g) Using the model from (e), obtain 95 % confidence intervals for the coefficient(s).
- 135. 3.7 Exercises 125 (h) Is there evidence of outliers or high leverage observations in the model from (e)? 11. In this problem we will investigate the t-statistic for the null hypoth- esis H0 : β = 0 in simple linear regression without an intercept. To begin, we generate a predictor x and a response y as follows. > set.seed (1) > x <- rnorm (100) > y <- 2 * x + rnorm (100) (a) Perform a simple linear regression of y onto x, without an in- tercept. Report the coefficient estimate β̂, the standard error of this coefficient estimate, and the t-statistic and p-value associ- ated with the null hypothesis H0 : β = 0. Comment on these results. (You can perform regression without an intercept using the command lm(y∼x+0).) (b) Now perform a simple linear regression of x onto y without an intercept, and report the coefficient estimate, its standard error, and the corresponding t-statistic and p-values associated with the null hypothesis H0 : β = 0. Comment on these results. (c) What is the relationship between the results obtained in (a) and (b)? (d) For the regression of Y onto X without an intercept, the t- statistic for H0 : β = 0 takes the form β̂/SE(β̂), where β̂ is given by (3.38), and where SE(β̂) = @ )n i=1(yi − xiβ̂)2 (n − 1) )n i′=1 x2 i′ . (These formulas are slightly different from those given in Sec- tions 3.1.1 and 3.1.2, since here we are performing regression without an intercept.) Show algebraically, and confirm numeri- cally in R, that the t-statistic can be written as ( √ n − 1) )n i=1 xiyi 5 ( )n i=1 x2 i )( )n i′=1 y2 i′ ) − ( )n i′=1 xi′ yi′ )2 . (e) Using the results from (d), argue that the t-statistic for the re- gression of y onto x is the same as the t-statistic for the regression of x onto y. (f) In R, show that when regression is performed with an intercept, the t-statistic for H0 : β1 = 0 is the same for the regression of y onto x as it is for the regression of x onto y.
- 136. 126 3. Linear Regression 12. This problem involves simple linear regression without an intercept. (a) Recall that the coefficient estimate β̂ for the linear regression of Y onto X without an intercept is given by (3.38). Under what circumstance is the coefficient estimate for the regression of X onto Y the same as the coefficient estimate for the regression of Y onto X? (b) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is different from the coefficient estimate for the regression of Y onto X. (c) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X. 13. In this exercise you will create some simulated data and will fit simple linear regression models to it. Make sure to use set.seed(1) prior to starting part (a) to ensure consistent results. (a) Using the rnorm() function, create a vector, x, containing 100 observations drawn from a N(0, 1) distribution. This represents a feature, X. (b) Using the rnorm() function, create a vector, eps, containing 100 observations drawn from a N(0, 0.25) distribution—a normal distribution with mean zero and variance 0.25. (c) Using x and eps, generate a vector y according to the model Y = −1 + 0.5X + ϵ. (3.39) What is the length of the vector y? What are the values of β0 and β1 in this linear model? (d) Create a scatterplot displaying the relationship between x and y. Comment on what you observe. (e) Fit a least squares linear model to predict y using x. Comment on the model obtained. How do β̂0 and β̂1 compare to β0 and β1? (f) Display the least squares line on the scatterplot obtained in (d). Draw the population regression line on the plot, in a different color. Use the legend() command to create an appropriate leg- end. (g) Now fit a polynomial regression model that predicts y using x and x2 . Is there evidence that the quadratic term improves the model fit? Explain your answer.
- 137. 3.7 Exercises 127 (h) Repeat (a)–(f) after modifying the data generation process in such a way that there is less noise in the data. The model (3.39) should remain the same. You can do this by decreasing the vari- ance of the normal distribution used to generate the error term ϵ in (b). Describe your results. (i) Repeat (a)–(f) after modifying the data generation process in such a way that there is more noise in the data. The model (3.39) should remain the same. You can do this by increasing the variance of the normal distribution used to generate the error term ϵ in (b). Describe your results. (j) What are the confidence intervals for β0 and β1 based on the original data set, the noisier data set, and the less noisy data set? Comment on your results. 14. This problem focuses on the collinearity problem. (a) Perform the following commands in R: > set.seed (1) > x1 <- runif (100) > x2 <- 0.5 * x1 + rnorm (100) / 10 > y <- 2 + 2 * x1 + 0.3 * x2 + rnorm (100) The last line corresponds to creating a linear model in which y is a function of x1 and x2. Write out the form of the linear model. What are the regression coefficients? (b) What is the correlation between x1 and x2? Create a scatterplot displaying the relationship between the variables. (c) Using this data, fit a least squares regression to predict y using x1 and x2. Describe the results obtained. What are β̂0, β̂1, and β̂2? How do these relate to the true β0, β1, and β2? Can you reject the null hypothesis H0 : β1 = 0? How about the null hypothesis H0 : β2 = 0? (d) Now fit a least squares regression to predict y using only x1. Comment on your results. Can you reject the null hypothesis H0 : β1 = 0? (e) Now fit a least squares regression to predict y using only x2. Comment on your results. Can you reject the null hypothesis H0 : β1 = 0? (f) Do the results obtained in (c)–(e) contradict each other? Explain your answer. (g) Now suppose we obtain one additional observation, which was unfortunately mismeasured.
- 138. 128 3. Linear Regression > x1 <- c(x1 , 0.1) > x2 <- c(x2 , 0.8) > y <- c(y, 6) Re-fit the linear models from (c) to (e) using this new data. What effect does this new observation have on the each of the models? In each model, is this observation an outlier? A high-leverage point? Both? Explain your answers. 15. This problem involves the Boston data set, which we saw in the lab for this chapter. We will now try to predict per capita crime rate using the other variables in this data set. In other words, per capita crime rate is the response, and the other variables are the predictors. (a) For each predictor, fit a simple linear regression model to predict the response. Describe your results. In which of the models is there a statistically significant association between the predictor and the response? Create some plots to back up your assertions. (b) Fit a multiple regression model to predict the response using all of the predictors. Describe your results. For which predictors can we reject the null hypothesis H0 : βj = 0? (c) How do your results from (a) compare to your results from (b)? Create a plot displaying the univariate regression coefficients from (a) on the x-axis, and the multiple regression coefficients from (b) on the y-axis. That is, each predictor is displayed as a single point in the plot. Its coefficient in a simple linear regres- sion model is shown on the x-axis, and its coefficient estimate in the multiple linear regression model is shown on the y-axis. (d) Is there evidence of non-linear association between any of the predictors and the response? To answer this question, for each predictor X, fit a model of the form Y = β0 + β1X + β2X2 + β3X3 + ϵ.
- 139. 4 Classification The linear regression model discussed in Chapter 3 assumes that the re- sponse variable Y is quantitative. But in many situations, the response variable is instead qualitative. For example, eye color is qualitative. Of- qualitative ten qualitative variables are referred to as categorical; we will use these terms interchangeably. In this chapter, we study approaches for predicting qualitative responses, a process that is known as classification. Predicting classification a qualitative response for an observation can be referred to as classifying that observation, since it involves assigning the observation to a category, or class. On the other hand, often the methods used for classification first predict the probability that the observation belongs to each of the cate- gories of a qualitative variable, as the basis for making the classification. In this sense they also behave like regression methods. There are many possible classification techniques, or classifiers, that one classifier might use to predict a qualitative response. We touched on some of these in Sections 2.1.5 and 2.2.3. In this chapter we discuss some widely-used classifiers: logistic regression, linear discriminant analysis, quadratic dis- logistic regression linear discriminant analysis criminant analysis, naive Bayes, and K-nearest neighbors. The discussion quadratic discriminant analysis naive Bayes K-nearest neighbors of logistic regression is used as a jumping-off point for a discussion of gen- eralized linear models, and in particular, Poisson regression. We discuss generalized linear models Poisson regression more computer-intensive classification methods in later chapters: these in- clude generalized additive models (Chapter 7); trees, random forests, and boosting (Chapter 8); and support vector machines (Chapter 9). © Springer Science+Business Media, LLC, part of Springer Nature 2021 G. James et al., An Introduction to Statistical Learning, Springer Texts in Statistics, https://doi.org/10.1007/978-1-0716-1418-1_4 129
- 140. 130 4. Classification 4.1 An Overview of Classification Classification problems occur often, perhaps even more so than regression problems. Some examples include: 1. A person arrives at the emergency room with a set of symptoms that could possibly be attributed to one of three medical conditions. Which of the three conditions does the individual have? 2. An online banking service must be able to determine whether or not a transaction being performed on the site is fraudulent, on the basis of the user’s IP address, past transaction history, and so forth. 3. On the basis of DNA sequence data for a number of patients with and without a given disease, a biologist would like to figure out which DNA mutations are deleterious (disease-causing) and which are not. Just as in the regression setting, in the classification setting we have a set of training observations (x1, y1), . . . , (xn, yn) that we can use to build a classifier. We want our classifier to perform well not only on the training data, but also on test observations that were not used to train the classifier. In this chapter, we will illustrate the concept of classification using the simulated Default data set. We are interested in predicting whether an individual will default on his or her credit card payment, on the basis of annual income and monthly credit card balance. The data set is displayed in Figure 4.1. In the left-hand panel of Figure 4.1, we have plotted annual income and monthly credit card balance for a subset of 10, 000 individuals. The individuals who defaulted in a given month are shown in orange, and those who did not in blue. (The overall default rate is about 3 %, so we have plotted only a fraction of the individuals who did not default.) It appears that individuals who defaulted tended to have higher credit card balances than those who did not. In the center and right-hand panels of Figure 4.1, two pairs of boxplots are shown. The first shows the distribution of balance split by the binary default variable; the second is a similar plot for income. In this chapter, we learn how to build a model to predict default (Y ) for any given value of balance (X1) and income (X2). Since Y is not quantitative, the simple linear regression model of Chapter 3 is not a good choice: we will elaborate on this further in Section 4.2. It is worth noting that Figure 4.1 displays a very pronounced relation- ship between the predictor balance and the response default. In most real applications, the relationship between the predictor and the response will not be nearly so strong. However, for the sake of illustrating the classifica- tion procedures discussed in this chapter, we use an example in which the relationship between the predictor and the response is somewhat exagger- ated.
- 141. 4.2 Why Not Linear Regression? 131 0 500 1000 1500 2000 2500 0 20000 40000 60000 Balance Income No Yes 0 500 1000 1500 2000 2500 Default Balance No Yes 0 20000 40000 60000 Default Income FIGURE 4.1. The Default data set. Left: The annual incomes and monthly credit card balances of a number of individuals. The individuals who defaulted on their credit card payments are shown in orange, and those who did not are shown in blue. Center: Boxplots of balance as a function of default status. Right: Boxplots of income as a function of default status. 4.2 Why Not Linear Regression? We have stated that linear regression is not appropriate in the case of a qualitative response. Why not? Suppose that we are trying to predict the medical condition of a patient in the emergency room on the basis of her symptoms. In this simplified example, there are three possible diagnoses: stroke, drug overdose, and epileptic seizure. We could consider encoding these values as a quantita- tive response variable, Y , as follows: Y = ⎧ ⎪ ⎨ ⎪ ⎩ 1 if stroke; 2 if drug overdose; 3 if epileptic seizure. Using this coding, least squares could be used to fit a linear regression model to predict Y on the basis of a set of predictors X1, . . . , Xp. Unfortunately, this coding implies an ordering on the outcomes, putting drug overdose in between stroke and epileptic seizure, and insisting that the difference between stroke and drug overdose is the same as the difference between drug overdose and epileptic seizure. In practice there is no particular reason that this needs to be the case. For instance, one could choose an
- 142. 132 4. Classification equally reasonable coding, Y = ⎧ ⎪ ⎨ ⎪ ⎩ 1 if epileptic seizure; 2 if stroke; 3 if drug overdose. which would imply a totally different relationship among the three condi- tions. Each of these codings would produce fundamentally different linear models that would ultimately lead to different sets of predictions on test observations. If the response variable’s values did take on a natural ordering, such as mild, moderate, and severe, and we felt the gap between mild and moderate was similar to the gap between moderate and severe, then a 1, 2, 3 coding would be reasonable. Unfortunately, in general there is no natural way to convert a qualitative response variable with more than two levels into a quantitative response that is ready for linear regression. For a binary (two level) qualitative response, the situation is better. For binary instance, perhaps there are only two possibilities for the patient’s med- ical condition: stroke and drug overdose. We could then potentially use the dummy variable approach from Section 3.3.1 to code the response as follows: Y = = 0 if stroke; 1 if drug overdose. We could then fit a linear regression to this binary response, and predict drug overdose if Ŷ > 0.5 and stroke otherwise. In the binary case it is not hard to show that even if we flip the above coding, linear regression will produce the same final predictions. For a binary response with a 0/1 coding as above, regression by least squares is not completely unreasonable: it can be shown that the Xβ̂ ob- tained using linear regression is in fact an estimate of Pr(drug overdose|X) in this special case. However, if we use linear regression, some of our es- timates might be outside the [0, 1] interval (see Figure 4.2), making them hard to interpret as probabilities! Nevertheless, the predictions provide an ordering and can be interpreted as crude probability estimates. Curiously, it turns out that the classifications that we get if we use linear regression to predict a binary response will be the same as for the linear discriminant analysis (LDA) procedure we discuss in Section 4.4. To summarize, there are at least two reasons not to perform classifica- tion using a regression method: (a) a regression method cannot accommo- date a qualitative response with more than two classes; (b) a regression method will not provide meaningful estimates of Pr(Y |X), even with just two classes. Thus, it is preferable to use a classification method that is truly suited for qualitative response values. In the next section, we present logistic regression, which is well-suited for the case of a binary qualita-
- 143. 4.3 Logistic Regression 133 0 500 1000 1500 2000 2500 0.0 0.2 0.4 0.6 0.8 1.0 Balance Probability of Default | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | || | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | ||| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | || | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | || | | | || | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | || | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | || | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || || | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0 500 1000 1500 2000 2500 0.0 0.2 0.4 0.6 0.8 1.0 Balance Probability of Default | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | || | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | ||| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | || | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | || | | | || | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | || | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | || | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || || | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | FIGURE 4.2. Classification using the Default data. Left: Estimated probabil- ity of default using linear regression. Some estimated probabilities are negative! The orange ticks indicate the 0/1 values coded for default(No or Yes). Right: Predicted probabilities of default using logistic regression. All probabilities lie between 0 and 1. tive response; in later sections we will cover classification methods that are appropriate when the qualitative response has two or more classes. 4.3 Logistic Regression Consider again the Default data set, where the response default falls into one of two categories, Yes or No. Rather than modeling this response Y directly, logistic regression models the probability that Y belongs to a par- ticular category. For the Default data, logistic regression models the probability of default. For example, the probability of default given balance can be written as Pr(default = Yes|balance). The values of Pr(default = Yes|balance), which we abbreviate p(balance), will range between 0 and 1. Then for any given value of balance, a prediction can be made for default. For example, one might predict default = Yes for any individual for whom p(balance) > 0.5. Alternatively, if a company wishes to be conservative in predicting individuals who are at risk for de- fault, then they may choose to use a lower threshold, such as p(balance) > 0.1. 4.3.1 The Logistic Model How should we model the relationship between p(X) = Pr(Y = 1|X) and X? (For convenience we are using the generic 0/1 coding for the response.)
- 144. 134 4. Classification In Section 4.2 we considered using a linear regression model to represent these probabilities: p(X) = β0 + β1X. (4.1) If we use this approach to predict default=Yes using balance, then we obtain the model shown in the left-hand panel of Figure 4.2. Here we see the problem with this approach: for balances close to zero we predict a negative probability of default; if we were to predict for very large balances, we would get values bigger than 1. These predictions are not sensible, since of course the true probability of default, regardless of credit card balance, must fall between 0 and 1. This problem is not unique to the credit default data. Any time a straight line is fit to a binary response that is coded as 0 or 1, in principle we can always predict p(X) < 0 for some values of X and p(X) > 1 for others (unless the range of X is limited). To avoid this problem, we must model p(X) using a function that gives outputs between 0 and 1 for all values of X. Many functions meet this description. In logistic regression, we use the logistic function, logistic function p(X) = eβ0+β1X 1 + eβ0+β1X . (4.2) To fit the model (4.2), we use a method called maximum likelihood, which maximum likelihood we discuss in the next section. The right-hand panel of Figure 4.2 illustrates the fit of the logistic regression model to the Default data. Notice that for low balances we now predict the probability of default as close to, but never below, zero. Likewise, for high balances we predict a default probability close to, but never above, one. The logistic function will always produce an S-shaped curve of this form, and so regardless of the value of X, we will obtain a sensible prediction. We also see that the logistic model is better able to capture the range of probabilities than is the linear regression model in the left-hand plot. The average fitted probability in both cases is 0.0333 (averaged over the training data), which is the same as the overall proportion of defaulters in the data set. After a bit of manipulation of (4.2), we find that p(X) 1 − p(X) = eβ0+β1X . (4.3) The quantity p(X)/[1−p(X)] is called the odds, and can take on any value odds between 0 and ∞. Values of the odds close to 0 and ∞ indicate very low and very high probabilities of default, respectively. For example, on average 1 in 5 people with an odds of 1/4 will default, since p(X) = 0.2 implies an odds of 0.2 1−0.2 = 1/4. Likewise, on average nine out of every ten people with an odds of 9 will default, since p(X) = 0.9 implies an odds of 0.9 1−0.9 = 9. Odds are traditionally used instead of probabilities in horse-racing, since they relate more naturally to the correct betting strategy.
- 145. 4.3 Logistic Regression 135 By taking the logarithm of both sides of (4.3), we arrive at log * p(X) 1 − p(X) + = β0 + β1X. (4.4) The left-hand side is called the log odds or logit. We see that the logistic log odds logit regression model (4.2) has a logit that is linear in X. Recall from Chapter 3 that in a linear regression model, β1 gives the average change in Y associated with a one-unit increase in X. By contrast, in a logistic regression model, increasing X by one unit changes the log odds by β1 (4.4). Equivalently, it multiplies the odds by eβ1 (4.3). However, because the relationship between p(X) and X in (4.2) is not a straight line, β1 does not correspond to the change in p(X) associated with a one-unit increase in X. The amount that p(X) changes due to a one-unit change in X depends on the current value of X. But regardless of the value of X, if β1 is positive then increasing X will be associated with increasing p(X), and if β1 is negative then increasing X will be associated with decreasing p(X). The fact that there is not a straight-line relationship between p(X) and X, and the fact that the rate of change in p(X) per unit change in X depends on the current value of X, can also be seen by inspection of the right-hand panel of Figure 4.2. 4.3.2 Estimating the Regression Coefficients The coefficients β0 and β1 in (4.2) are unknown, and must be estimated based on the available training data. In Chapter 3, we used the least squares approach to estimate the unknown linear regression coefficients. Although we could use (non-linear) least squares to fit the model (4.4), the more general method of maximum likelihood is preferred, since it has better sta- tistical properties. The basic intuition behind using maximum likelihood to fit a logistic regression model is as follows: we seek estimates for β0 and β1 such that the predicted probability p̂(xi) of default for each individual, using (4.2), corresponds as closely as possible to the individual’s observed default status. In other words, we try to find β̂0 and β̂1 such that plugging these estimates into the model for p(X), given in (4.2), yields a number close to one for all individuals who defaulted, and a number close to zero for all individuals who did not. This intuition can be formalized using a mathematical equation called a likelihood function: likelihood function ℓ(β0, β1) = E i:yi=1 p(xi) E i′:yi′ =0 (1 − p(xi′ )). (4.5) The estimates β̂0 and β̂1 are chosen to maximize this likelihood function. Maximum likelihood is a very general approach that is used to fit many of the non-linear models that we examine throughout this book. In the linear regression setting, the least squares approach is in fact a special case
- 146. 136 4. Classification Coefficient Std. error z-statistic p-value Intercept −10.6513 0.3612 −29.5 <0.0001 balance 0.0055 0.0002 24.9 <0.0001 TABLE 4.1. For the Default data, estimated coefficients of the logistic regres- sion model that predicts the probability of default using balance. A one-unit increase in balance is associated with an increase in the log odds of default by 0.0055 units. of maximum likelihood. The mathematical details of maximum likelihood are beyond the scope of this book. However, in general, logistic regression and other models can be easily fit using statistical software such as R, and so we do not need to concern ourselves with the details of the maximum likelihood fitting procedure. Table 4.1 shows the coefficient estimates and related information that result from fitting a logistic regression model on the Default data in order to predict the probability of default=Yes using balance. We see that β̂1 = 0.0055; this indicates that an increase in balance is associated with an increase in the probability of default. To be precise, a one-unit increase in balance is associated with an increase in the log odds of default by 0.0055 units. Many aspects of the logistic regression output shown in Table 4.1 are similar to the linear regression output of Chapter 3. For example, we can measure the accuracy of the coefficient estimates by computing their stan- dard errors. The z-statistic in Table 4.1 plays the same role as the t-statistic in the linear regression output, for example in Table 3.1 on page 68. For instance, the z-statistic associated with β1 is equal to β̂1/SE(β̂1), and so a large (absolute) value of the z-statistic indicates evidence against the null hypothesis H0 : β1 = 0. This null hypothesis implies that p(X) = eβ0 1+eβ0 : in other words, that the probability of default does not depend on balance. Since the p-value associated with balance in Table 4.1 is tiny, we can reject H0. In other words, we conclude that there is indeed an association between balance and probability of default. The estimated intercept in Table 4.1 is typically not of interest; its main purpose is to adjust the average fitted probabilities to the proportion of ones in the data (in this case, the overall default rate). 4.3.3 Making Predictions Once the coefficients have been estimated, we can compute the probability of default for any given credit card balance. For example, using the coeffi- cient estimates given in Table 4.1, we predict that the default probability for an individual with a balance of $1, 000 is p̂(X) = eβ̂0+β̂1X 1 + eβ̂0+β̂1X = e−10.6513+0.0055×1,000 1 + e−10.6513+0.0055×1,000 = 0.00576,
- 147. 4.3 Logistic Regression 137 Coefficient Std. error z-statistic p-value Intercept −3.5041 0.0707 −49.55 <0.0001 student[Yes] 0.4049 0.1150 3.52 0.0004 TABLE 4.2. For the Default data, estimated coefficients of the logistic regres- sion model that predicts the probability of default using student status. Student status is encoded as a dummy variable, with a value of 1 for a student and a value of 0 for a non-student, and represented by the variable student[Yes] in the table. which is below 1 %. In contrast, the predicted probability of default for an individual with a balance of $2, 000 is much higher, and equals 0.586 or 58.6 %. One can use qualitative predictors with the logistic regression model us- ing the dummy variable approach from Section 3.3.1. As an example, the Default data set contains the qualitative variable student. To fit a model that uses student status as a predictor variable, we simply create a dummy variable that takes on a value of 1 for students and 0 for non-students. The logistic regression model that results from predicting probability of default from student status can be seen in Table 4.2. The coefficient associated with the dummy variable is positive, and the associated p-value is statis- tically significant. This indicates that students tend to have higher default probabilities than non-students: 6 Pr(default=Yes|student=Yes) = e−3.5041+0.4049×1 1 + e−3.5041+0.4049×1 = 0.0431, 6 Pr(default=Yes|student=No) = e−3.5041+0.4049×0 1 + e−3.5041+0.4049×0 = 0.0292. 4.3.4 Multiple Logistic Regression We now consider the problem of predicting a binary response using multiple predictors. By analogy with the extension from simple to multiple linear regression in Chapter 3, we can generalize (4.4) as follows: log * p(X) 1 − p(X) + = β0 + β1X1 + · · · + βpXp, (4.6) where X = (X1, . . . , Xp) are p predictors. Equation 4.6 can be rewritten as p(X) = eβ0+β1X1+···+βpXp 1 + eβ0+β1X1+···+βpXp . (4.7) Just as in Section 4.3.2, we use the maximum likelihood method to estimate β0, β1, . . . , βp. Table 4.3 shows the coefficient estimates for a logistic regression model that uses balance, income (in thousands of dollars), and student status to predict probability of default. There is a surprising result here. The p-
- 148. 138 4. Classification Coefficient Std. error z-statistic p-value Intercept −10.8690 0.4923 −22.08 <0.0001 balance 0.0057 0.0002 24.74 <0.0001 income 0.0030 0.0082 0.37 0.7115 student[Yes] −0.6468 0.2362 −2.74 0.0062 TABLE 4.3. For the Default data, estimated coefficients of the logistic regres- sion model that predicts the probability of default using balance, income, and student status. Student status is encoded as a dummy variable student[Yes], with a value of 1 for a student and a value of 0 for a non-student. In fitting this model, income was measured in thousands of dollars. values associated with balance and the dummy variable for student status are very small, indicating that each of these variables is associated with the probability of default. However, the coefficient for the dummy variable is negative, indicating that students are less likely to default than non- students. In contrast, the coefficient for the dummy variable is positive in Table 4.2. How is it possible for student status to be associated with an increase in probability of default in Table 4.2 and a decrease in probability of default in Table 4.3? The left-hand panel of Figure 4.3 provides a graph- ical illustration of this apparent paradox. The orange and blue solid lines show the average default rates for students and non-students, respectively, as a function of credit card balance. The negative coefficient for student in the multiple logistic regression indicates that for a fixed value of balance and income, a student is less likely to default than a non-student. Indeed, we observe from the left-hand panel of Figure 4.3 that the student default rate is at or below that of the non-student default rate for every value of balance. But the horizontal broken lines near the base of the plot, which show the default rates for students and non-students averaged over all val- ues of balance and income, suggest the opposite effect: the overall student default rate is higher than the non-student default rate. Consequently, there is a positive coefficient for student in the single variable logistic regression output shown in Table 4.2. The right-hand panel of Figure 4.3 provides an explanation for this dis- crepancy. The variables student and balance are correlated. Students tend to hold higher levels of debt, which is in turn associated with higher prob- ability of default. In other words, students are more likely to have large credit card balances, which, as we know from the left-hand panel of Fig- ure 4.3, tend to be associated with high default rates. Thus, even though an individual student with a given credit card balance will tend to have a lower probability of default than a non-student with the same credit card balance, the fact that students on the whole tend to have higher credit card balances means that overall, students tend to default at a higher rate than non-students. This is an important distinction for a credit card company that is trying to determine to whom they should offer credit. A student is
- 149. 4.3 Logistic Regression 139 500 1000 1500 2000 0.0 0.2 0.4 0.6 0.8 Credit Card Balance Default Rate No Yes 0 500 1000 1500 2000 2500 Student Status Credit Card Balance FIGURE 4.3. Confounding in the Default data. Left: Default rates are shown for students (orange) and non-students (blue). The solid lines display default rate as a function of balance, while the horizontal broken lines display the overall default rates. Right: Boxplots of balance for students (orange) and non-students (blue) are shown. riskier than a non-student if no information about the student’s credit card balance is available. However, that student is less risky than a non-student with the same credit card balance! This simple example illustrates the dangers and subtleties associated with performing regressions involving only a single predictor when other predictors may also be relevant. As in the linear regression setting, the results obtained using one predictor may be quite different from those ob- tained using multiple predictors, especially when there is correlation among the predictors. In general, the phenomenon seen in Figure 4.3 is known as confounding. confounding By substituting estimates for the regression coefficients from Table 4.3 into (4.7), we can make predictions. For example, a student with a credit card balance of $1, 500 and an income of $40, 000 has an estimated proba- bility of default of p̂(X) = e−10.869+0.00574×1,500+0.003×40−0.6468×1 1 + e−10.869+0.00574×1,500+0.003×40−0.6468×1 = 0.058. (4.8) A non-student with the same balance and income has an estimated prob- ability of default of p̂(X) = e−10.869+0.00574×1,500+0.003×40−0.6468×0 1 + e−10.869+0.00574×1,500+0.003×40−0.6468×0 = 0.105. (4.9) (Here we multiply the income coefficient estimate from Table 4.3 by 40, rather than by 40,000, because in that table the model was fit with income measured in units of $1, 000.)
- 150. 140 4. Classification 4.3.5 Multinomial Logistic Regression We sometimes wish to classify a response variable that has more than two classes. For example, in Section 4.2 we had three categories of medical con- dition in the emergency room: stroke, drug overdose, epileptic seizure. However, the logistic regression approach that we have seen in this section only allows for K = 2 classes for the response variable. It turns out that it is possible to extend the two-class logistic regression approach to the setting of K > 2 classes. This extension is sometimes known as multinomial logistic regression. To do this, we first select a single multinomial logistic regression class to serve as the baseline; without loss of generality, we select the Kth class for this role. Then we replace the model (4.7) with the model Pr(Y = k|X = x) = eβk0+βk1x1+···+βkpxp 1 + )K−1 l=1 eβl0+βl1x1+···+βlpxp (4.10) for k = 1, . . . , K−1, and Pr(Y = K|X = x) = 1 1 + )K−1 l=1 eβl0+βl1x1+···+βlpxp . (4.11) It is not hard to show that for k = 1, . . . , K−1, log * Pr(Y = k|X = x) Pr(Y = K|X = x) + = βk0 + βk1x1 + · · · + βkpxp. (4.12) Notice that (4.12) is quite similar to (4.6). Equation 4.12 indicates that once again, the log odds between any pair of classes is linear in the features. It turns out that in (4.10)–(4.12), the decision to treat the Kth class as the baseline is unimportant. For example, when classifying emergency room visits into stroke, drug overdose, and epileptic seizure, suppose that we fit two multinomial logistic regression models: one treating stroke as the baseline, another treating drug overdose as the baseline. The coefficient estimates will differ between the two fitted models due to the differing choice of baseline, but the fitted values (predictions), the log odds between any pair of classes, and the other key model outputs will remain the same. Nonetheless, interpretation of the coefficients in a multinomial logistic regression model must be done with care, since it is tied to the choice of baseline. For example, if we set epileptic seizure to be the baseline, then we can interpret βstroke0 as the log odds of stroke versus epileptic seizure, given that x1 = . . . = xp = 0. Furthermore, a one-unit increase in Xj is associated with a βstrokej increase in the log odds of stroke over epileptic seizure. Stated another way, if Xj increases by one unit, then Pr(Y = stroke|X = x) Pr(Y = epileptic seizure|X = x) increases by eβstrokej .
- 151. 4.4 Generative Models for Classification 141 We now briefly present an alternative coding for multinomial logistic regression, known as the softmax coding. The softmax coding is equivalent softmax to the coding just described in the sense that the fitted values, log odds between any pair of classes, and other key model outputs will remain the same, regardless of coding. But the softmax coding is used extensively in some areas of the machine learning literature (and will appear again in Chapter 10), so it is worth being aware of it. In the softmax coding, rather than selecting a baseline class, we treat all K classes symmetrically, and assume that for k = 1, . . . , K, Pr(Y = k|X = x) = eβk0+βk1x1+···+βkpxp )K l=1 eβl0+βl1x1+···+βlpxp . (4.13) Thus, rather than estimating coefficients for K − 1 classes, we actually estimate coefficients for all K classes. It is not hard to see that as a result of (4.13), the log odds ratio between the kth and k′ th classes equals log % Pr(Y = k|X = x) Pr(Y = k′|X = x) & = (βk0 − βk′0) + (βk1 − βk′1)x1 + · · · + (βkp − βk′p)xp. (4.14) 4.4 Generative Models for Classification Logistic regression involves directly modeling Pr(Y = k|X = x) using the logistic function, given by (4.7) for the case of two response classes. In statistical jargon, we model the conditional distribution of the response Y , given the predictor(s) X. We now consider an alternative and less direct approach to estimating these probabilities. In this new approach, we model the distribution of the predictors X separately in each of the response classes (i.e. for each value of Y ). We then use Bayes’ theorem to flip these around into estimates for Pr(Y = k|X = x). When the distribution of X within each class is assumed to be normal, it turns out that the model is very similar in form to logistic regression. Why do we need another method, when we have logistic regression? There are several reasons: • When there is substantial separation between the two classes, the parameter estimates for the logistic regression model are surprisingly unstable. The methods that we consider in this section do not suffer from this problem. • If the distribution of the predictors X is approximately normal in each of the classes and the sample size is small, then the approaches in this section may be more accurate than logistic regression. • The methods in this section can be naturally extended to the case of more than two response classes. (In the case of more than two
- 152. 142 4. Classification response classes, we can also use multinomial logistic regression from Section 4.3.5.) Suppose that we wish to classify an observation into one of K classes, where K ≥ 2. In other words, the qualitative response variable Y can take on K possible distinct and unordered values. Let πk represent the overall or prior probability that a randomly chosen observation comes from the prior kth class. Let fk(X) ≡ Pr(X|Y = k)1 denote the density function of X density function for an observation that comes from the kth class. In other words, fk(x) is relatively large if there is a high probability that an observation in the kth class has X ≈ x, and fk(x) is small if it is very unlikely that an observation in the kth class has X ≈ x. Then Bayes’ theorem states that Bayes’ theorem Pr(Y = k|X = x) = πkfk(x) )K l=1 πlfl(x) . (4.15) In accordance with our earlier notation, we will use the abbreviation pk(x) = Pr(Y = k|X = x); this is the posterior probability that an observation posterior X = x belongs to the kth class. That is, it is the probability that the observation belongs to the kth class, given the predictor value for that observation. Equation 4.15 suggests that instead of directly computing the posterior probability pk(x) as in Section 4.3.1, we can simply plug in estimates of πk and fk(x) into (4.15). In general, estimating πk is easy if we have a random sample from the population: we simply compute the fraction of the training observations that belong to the kth class. However, estimating the density function fk(x) is much more challenging. As we will see, to estimate fk(x), we will typically have to make some simplifying assumptions. We know from Chapter 2 that the Bayes classifier, which classifies an observation x to the class for which pk(x) is largest, has the lowest possible error rate out of all classifiers. (Of course, this is only true if all of the terms in (4.15) are correctly specified.) Therefore, if we can find a way to estimate fk(x), then we can plug it into (4.15) in order to approximate the Bayes classifier. In the following sections, we discuss three classifiers that use different estimates of fk(x) in (4.15) to approximate the Bayes classifier: linear dis- criminant analysis, quadratic discriminant analysis, and naive Bayes. 4.4.1 Linear Discriminant Analysis for p = 1 For now, assume that p = 1—that is, we have only one predictor. We would like to obtain an estimate for fk(x) that we can plug into (4.15) in order to estimate pk(x). We will then classify an observation to the class for which 1Technically, this definition is only correct if X is a qualitative random variable. If X is quantitative, then fk(x)dx corresponds to the probability of X falling in a small region dx around x.
- 153. 4.4 Generative Models for Classification 143 pk(x) is greatest. To estimate fk(x), we will first make some assumptions about its form. In particular, we assume that fk(x) is normal or Gaussian. In the one- normal Gaussian dimensional setting, the normal density takes the form fk(x) = 1 √ 2πσk exp * − 1 2σ2 k (x − µk)2 + , (4.16) where µk and σ2 k are the mean and variance parameters for the kth class. For now, let us further assume that σ2 1 = · · · = σ2 K: that is, there is a shared variance term across all K classes, which for simplicity we can denote by σ2 . Plugging (4.16) into (4.15), we find that pk(x) = πk 1 √ 2πσ exp ' − 1 2σ2 (x − µk)2 ( )K l=1 πl 1 √ 2πσ exp ' − 1 2σ2 (x − µl)2 (. (4.17) (Note that in (4.17), πk denotes the prior probability that an observation belongs to the kth class, not to be confused with π ≈ 3.14159, the math- ematical constant.) The Bayes classifier2 involves assigning an observation X = x to the class for which (4.17) is largest. Taking the log of (4.17) and rearranging the terms, it is not hard to show3 that this is equivalent to assigning the observation to the class for which δk(x) = x · µk σ2 − µ2 k 2σ2 + log(πk) (4.18) is largest. For instance, if K = 2 and π1 = π2, then the Bayes classifier assigns an observation to class 1 if 2x (µ1 − µ2) > µ2 1 − µ2 2, and to class 2 otherwise. The Bayes decision boundary is the point for which δ1(x) = δ2(x); one can show that this amounts to x = µ2 1 − µ2 2 2(µ1 − µ2) = µ1 + µ2 2 . (4.19) An example is shown in the left-hand panel of Figure 4.4. The two normal density functions that are displayed, f1(x) and f2(x), represent two distinct classes. The mean and variance parameters for the two density functions are µ1 = −1.25, µ2 = 1.25, and σ2 1 = σ2 2 = 1. The two densities overlap, and so given that X = x, there is some uncertainty about the class to which the observation belongs. If we assume that an observation is equally likely to come from either class—that is, π1 = π2 = 0.5—then by inspection of (4.19), we see that the Bayes classifier assigns the observation to class 1 if x < 0 and class 2 otherwise. Note that in this case, we can compute the Bayes classifier because we know that X is drawn from a Gaussian distribution within each class, and we know all of the parameters involved. In a real-life situation, we are not able to calculate the Bayes classifier. 2Recall that the Bayes classifier assigns an observation to the class for which pk(x) is largest. This is different from Bayes’ theorem in (4.13), which allows us to manipulate conditional distributions. 3See Exercise 2 at the end of this chapter.
- 154. 144 4. Classification −4 −2 0 2 4 −3 −2 −1 0 1 2 3 4 0 1 2 3 4 5 FIGURE 4.4. Left: Two one-dimensional normal density functions are shown. The dashed vertical line represents the Bayes decision boundary. Right: 20 obser- vations were drawn from each of the two classes, and are shown as histograms. The Bayes decision boundary is again shown as a dashed vertical line. The solid vertical line represents the LDA decision boundary estimated from the training data. In practice, even if we are quite certain of our assumption that X is drawn from a Gaussian distribution within each class, to apply the Bayes classifier we still have to estimate the parameters µ1, . . . , µK, π1, . . . , πK, and σ2 . The linear discriminant analysis (LDA) method approximates the linear discriminant analysis Bayes classifier by plugging estimates for πk, µk, and σ2 into (4.18). In particular, the following estimates are used: µ̂k = 1 nk 0 i:yi=k xi σ̂2 = 1 n − K K 0 k=1 0 i:yi=k (xi − µ̂k)2 (4.20) where n is the total number of training observations, and nk is the number of training observations in the kth class. The estimate for µk is simply the average of all the training observations from the kth class, while σ̂2 can be seen as a weighted average of the sample variances for each of the K classes. Sometimes we have knowledge of the class membership probabili- ties π1, . . . , πK, which can be used directly. In the absence of any additional information, LDA estimates πk using the proportion of the training obser- vations that belong to the kth class. In other words, π̂k = nk/n. (4.21) The LDA classifier plugs the estimates given in (4.20) and (4.21) into (4.18), and assigns an observation X = x to the class for which δ̂k(x) = x · µ̂k σ̂2 − µ̂2 k 2σ̂2 + log(π̂k) (4.22)
- 155. 4.4 Generative Models for Classification 145 is largest. The word linear in the classifier’s name stems from the fact that the discriminant functions δ̂k(x) in (4.22) are linear functions of x (as discriminant function opposed to a more complex function of x). The right-hand panel of Figure 4.4 displays a histogram of a random sample of 20 observations from each class. To implement LDA, we began by estimating πk, µk, and σ2 using (4.20) and (4.21). We then computed the decision boundary, shown as a black solid line, that results from assigning an observation to the class for which (4.22) is largest. All points to the left of this line will be assigned to the green class, while points to the right of this line are assigned to the purple class. In this case, since n1 = n2 = 20, we have π̂1 = π̂2. As a result, the decision boundary corresponds to the midpoint between the sample means for the two classes, (µ̂1 + µ̂2)/2. The figure indicates that the LDA decision boundary is slightly to the left of the optimal Bayes decision boundary, which instead equals (µ1 + µ2)/2 = 0. How well does the LDA classifier perform on this data? Since this is simulated data, we can generate a large number of test observations in order to compute the Bayes error rate and the LDA test error rate. These are 10.6 % and 11.1 %, respectively. In other words, the LDA classifier’s error rate is only 0.5 % above the smallest possible error rate! This indicates that LDA is performing pretty well on this data set. To reiterate, the LDA classifier results from assuming that the obser- vations within each class come from a normal distribution with a class- specific mean and a common variance σ2 , and plugging estimates for these parameters into the Bayes classifier. In Section 4.4.3, we will consider a less stringent set of assumptions, by allowing the observations in the kth class to have a class-specific variance, σ2 k. 4.4.2 Linear Discriminant Analysis for p >1 We now extend the LDA classifier to the case of multiple predictors. To do this, we will assume that X = (X1, X2, . . . , Xp) is drawn from a multi- variate Gaussian (or multivariate normal) distribution, with a class-specific multivariate Gaussian mean vector and a common covariance matrix. We begin with a brief review of this distribution. The multivariate Gaussian distribution assumes that each individual pre- dictor follows a one-dimensional normal distribution, as in (4.16), with some correlation between each pair of predictors. Two examples of multivariate Gaussian distributions with p = 2 are shown in Figure 4.5. The height of the surface at any particular point represents the probability that both X1 and X2 fall in a small region around that point. In either panel, if the sur- face is cut along the X1 axis or along the X2 axis, the resulting cross-section will have the shape of a one-dimensional normal distribution. The left-hand panel of Figure 4.5 illustrates an example in which Var(X1) = Var(X2) and Cor(X1, X2) = 0; this surface has a characteristic bell shape. However, the bell shape will be distorted if the predictors are correlated or have unequal variances, as is illustrated in the right-hand panel of Figure 4.5. In this situation, the base of the bell will have an elliptical, rather than circular,
- 156. 146 4. Classification x1 x1 x 2 x 2 FIGURE 4.5. Two multivariate Gaussian density functions are shown, with p = 2. Left: The two predictors are uncorrelated. Right: The two variables have a correlation of 0.7. shape. To indicate that a p-dimensional random variable X has a multi- variate Gaussian distribution, we write X ∼ N(µ, Σ). Here E(X) = µ is the mean of X (a vector with p components), and Cov(X) = Σ is the p × p covariance matrix of X. Formally, the multivariate Gaussian density is defined as f(x) = 1 (2π)p/2|Σ|1/2 exp * − 1 2 (x − µ)T Σ−1 (x − µ) + . (4.23) In the case of p > 1 predictors, the LDA classifier assumes that the observations in the kth class are drawn from a multivariate Gaussian dis- tribution N(µk, Σ), where µk is a class-specific mean vector, and Σ is a covariance matrix that is common to all K classes. Plugging the density function for the kth class, fk(X = x), into (4.15) and performing a little bit of algebra reveals that the Bayes classifier assigns an observation X = x to the class for which δk(x) = xT Σ−1 µk − 1 2 µT k Σ−1 µk + log πk (4.24) is largest. This is the vector/matrix version of (4.18). An example is shown in the left-hand panel of Figure 4.6. Three equally- sized Gaussian classes are shown with class-specific mean vectors and a common covariance matrix. The three ellipses represent regions that con- tain 95 % of the probability for each of the three classes. The dashed lines are the Bayes decision boundaries. In other words, they represent the set of values x for which δk(x) = δℓ(x); i.e. xT Σ−1 µk − 1 2 µT k Σ−1 µk = xT Σ−1 µl − 1 2 µT l Σ−1 µl (4.25) for k ̸= l. (The log πk term from (4.24) has disappeared because each of the three classes has the same number of training observations; i.e. πk is
- 157. 4.4 Generative Models for Classification 147 −4 −2 0 2 4 −4 −2 0 2 4 −4 −2 0 2 4 −4 −2 0 2 4 X1 X1 X 2 X 2 FIGURE 4.6. An example with three classes. The observations from each class are drawn from a multivariate Gaussian distribution with p = 2, with a class-spe- cific mean vector and a common covariance matrix. Left: Ellipses that contain 95 % of the probability for each of the three classes are shown. The dashed lines are the Bayes decision boundaries. Right: 20 observations were generated from each class, and the corresponding LDA decision boundaries are indicated using solid black lines. The Bayes decision boundaries are once again shown as dashed lines. the same for each class.) Note that there are three lines representing the Bayes decision boundaries because there are three pairs of classes among the three classes. That is, one Bayes decision boundary separates class 1 from class 2, one separates class 1 from class 3, and one separates class 2 from class 3. These three Bayes decision boundaries divide the predictor space into three regions. The Bayes classifier will classify an observation according to the region in which it is located. Once again, we need to estimate the unknown parameters µ1, . . . , µK, π1, . . . , πK, and Σ; the formulas are similar to those used in the one- dimensional case, given in (4.20). To assign a new observation X = x, LDA plugs these estimates into (4.24) to obtain quantities δ̂k(x), and clas- sifies to the class for which δ̂k(x) is largest. Note that in (4.24) δk(x) is a linear function of x; that is, the LDA decision rule depends on x only through a linear combination of its elements. As previously discussed, this is the reason for the word linear in LDA. In the right-hand panel of Figure 4.6, 20 observations drawn from each of the three classes are displayed, and the resulting LDA decision boundaries are shown as solid black lines. Overall, the LDA decision boundaries are pretty close to the Bayes decision boundaries, shown again as dashed lines. The test error rates for the Bayes and LDA classifiers are 0.0746 and 0.0770, respectively. This indicates that LDA is performing well on this data. We can perform LDA on the Default data in order to predict whether or not an individual will default on the basis of credit card balance and
- 158. 148 4. Classification True default status No Yes Total Predicted No 9644 252 9896 default status Yes 23 81 104 Total 9667 333 10000 TABLE 4.4. A confusion matrix compares the LDA predictions to the true de- fault statuses for the 10,000 training observations in the Default data set. Ele- ments on the diagonal of the matrix represent individuals whose default statuses were correctly predicted, while off-diagonal elements represent individuals that were misclassified. LDA made incorrect predictions for 23 individuals who did not default and for 252 individuals who did default. student status.4 The LDA model fit to the 10,000 training samples results in a training error rate of 2.75 %. This sounds like a low error rate, but two caveats must be noted. • First of all, training error rates will usually be lower than test error rates, which are the real quantity of interest. In other words, we might expect this classifier to perform worse if we use it to predict whether or not a new set of individuals will default. The reason is that we specifically adjust the parameters of our model to do well on the training data. The higher the ratio of parameters p to number of samples n, the more we expect this overfitting to play a role. For overfitting these data we don’t expect this to be a problem, since p = 2 and n = 10, 000. • Second, since only 3.33 % of the individuals in the training sample defaulted, a simple but useless classifier that always predicts that an individual will not default, regardless of his or her credit card balance and student status, will result in an error rate of 3.33 %. In other words, the trivial null classifier will achieve an error rate that null is only a bit higher than the LDA training set error rate. In practice, a binary classifier such as this one can make two types of errors: it can incorrectly assign an individual who defaults to the no default category, or it can incorrectly assign an individual who does not default to the default category. It is often of interest to determine which of these two types of errors are being made. A confusion matrix, shown for the Default confusion matrix data in Table 4.4, is a convenient way to display this information. The table reveals that LDA predicted that a total of 104 people would default. Of these people, 81 actually defaulted and 23 did not. Hence only 23 out of 9,667 of the individuals who did not default were incorrectly labeled. 4The careful reader will notice that student status is qualitative — thus, the normality assumption made by LDA is clearly violated in this example! However, LDA is often remarkably robust to model violations, as this example shows. Naive Bayes, discussed in Section 4.4.4, provides an alternative to LDA that does not assume normally distributed predictors.
- 159. 4.4 Generative Models for Classification 149 This looks like a pretty low error rate! However, of the 333 individuals who defaulted, 252 (or 75.7 %) were missed by LDA. So while the overall error rate is low, the error rate among individuals who defaulted is very high. From the perspective of a credit card company that is trying to identify high-risk individuals, an error rate of 252/333 = 75.7 % among individuals who default may well be unacceptable. Class-specific performance is also important in medicine and biology, where the terms sensitivity and specificity characterize the performance of sensitivity specificity a classifier or screening test. In this case the sensitivity is the percentage of true defaulters that are identified; it equals 24.3 %. The specificity is the percentage of non-defaulters that are correctly identified; it equals (1 − 23/9667) = 99.8 %. Why does LDA do such a poor job of classifying the customers who de- fault? In other words, why does it have such low sensitivity? As we have seen, LDA is trying to approximate the Bayes classifier, which has the low- est total error rate out of all classifiers. That is, the Bayes classifier will yield the smallest possible total number of misclassified observations, re- gardless of the class from which the errors stem. Some misclassifications will result from incorrectly assigning a customer who does not default to the default class, and others will result from incorrectly assigning a customer who defaults to the non-default class. In contrast, a credit card company might particularly wish to avoid incorrectly classifying an individual who will default, whereas incorrectly classifying an individual who will not de- fault, though still to be avoided, is less problematic. We will now see that it is possible to modify LDA in order to develop a classifier that better meets the credit card company’s needs. The Bayes classifier works by assigning an observation to the class for which the posterior probability pk(X) is greatest. In the two-class case, this amounts to assigning an observation to the default class if Pr(default = Yes|X = x) > 0.5. (4.26) Thus, the Bayes classifier, and by extension LDA, uses a threshold of 50 % for the posterior probability of default in order to assign an observation to the default class. However, if we are concerned about incorrectly pre- dicting the default status for individuals who default, then we can consider lowering this threshold. For instance, we might label any customer with a posterior probability of default above 20 % to the default class. In other words, instead of assigning an observation to the default class if (4.26) holds, we could instead assign an observation to this class if Pr(default = Yes|X = x) > 0.2. (4.27) The error rates that result from taking this approach are shown in Table 4.5. Now LDA predicts that 430 individuals will default. Of the 333 individuals who default, LDA correctly predicts all but 138, or 41.4 %. This is a vast improvement over the error rate of 75.7 % that resulted from using the threshold of 50 %. However, this improvement comes at a cost: now 235
- 160. 150 4. Classification True default status No Yes Total Predicted No 9432 138 9570 default status Yes 235 195 430 Total 9667 333 10000 TABLE 4.5. A confusion matrix compares the LDA predictions to the true de- fault statuses for the 10,000 training observations in the Default data set, using a modified threshold value that predicts default for any individuals whose posterior default probability exceeds 20 %. 0.0 0.1 0.2 0.3 0.4 0.5 0.0 0.2 0.4 0.6 Threshold Error Rate FIGURE 4.7. For the Default data set, error rates are shown as a function of the threshold value for the posterior probability that is used to perform the assign- ment. The black solid line displays the overall error rate. The blue dashed line represents the fraction of defaulting customers that are incorrectly classified, and the orange dotted line indicates the fraction of errors among the non-defaulting customers. individuals who do not default are incorrectly classified. As a result, the overall error rate has increased slightly to 3.73 %. But a credit card company may consider this slight increase in the total error rate to be a small price to pay for more accurate identification of individuals who do indeed default. Figure 4.7 illustrates the trade-off that results from modifying the thresh- old value for the posterior probability of default. Various error rates are shown as a function of the threshold value. Using a threshold of 0.5, as in (4.26), minimizes the overall error rate, shown as a black solid line. This is to be expected, since the Bayes classifier uses a threshold of 0.5 and is known to have the lowest overall error rate. But when a threshold of 0.5 is used, the error rate among the individuals who default is quite high (blue dashed line). As the threshold is reduced, the error rate among individuals who default decreases steadily, but the error rate among the individuals who do not default increases. How can we decide which threshold value is best? Such a decision must be based on domain knowledge, such as detailed information about the costs associated with default. The ROC curve is a popular graphic for simultaneously displaying the ROC curve two types of errors for all possible thresholds. The name “ROC” is his- toric, and comes from communications theory. It is an acronym for receiver
- 161. 4.4 Generative Models for Classification 151 ROC Curve False positive rate True positive rate 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 FIGURE 4.8. A ROC curve for the LDA classifier on the Default data. It traces out two types of error as we vary the threshold value for the posterior probability of default. The actual thresholds are not shown. The true positive rate is the sensitivity: the fraction of defaulters that are correctly identified, using a given threshold value. The false positive rate is 1-specificity: the fraction of non-defaulters that we classify incorrectly as defaulters, using that same threshold value. The ideal ROC curve hugs the top left corner, indicating a high true positive rate and a low false positive rate. The dotted line represents the “no information” classifier; this is what we would expect if student status and credit card balance are not associated with probability of default. operating characteristics. Figure 4.8 displays the ROC curve for the LDA classifier on the training data. The overall performance of a classifier, sum- marized over all possible thresholds, is given by the area under the (ROC) curve (AUC). An ideal ROC curve will hug the top left corner, so the larger area under the (ROC) curve the AUC the better the classifier. For this data the AUC is 0.95, which is close to the maximum of one so would be considered very good. We expect a classifier that performs no better than chance to have an AUC of 0.5 (when evaluated on an independent test set not used in model training). ROC curves are useful for comparing different classifiers, since they take into account all possible thresholds. It turns out that the ROC curve for the logistic regression model of Section 4.3.4 fit to these data is virtually indis- tinguishable from this one for the LDA model, so we do not display it here. As we have seen above, varying the classifier threshold changes its true positive and false positive rate. These are also called the sensitivity and one sensitivity minus the specificity of our classifier. Since there is an almost bewildering specificity array of terms used in this context, we now give a summary. Table 4.6 shows the possible results when applying a classifier (or diagnostic test)
- 162. 152 4. Classification True class − or Null + or Non-null Total Predicted − or Null True Neg. (TN) False Neg. (FN) N∗ class + or Non-null False Pos. (FP) True Pos. (TP) P∗ Total N P TABLE 4.6. Possible results when applying a classifier or diagnostic test to a population. Name Definition Synonyms False Pos. rate FP/N Type I error, 1−Specificity True Pos. rate TP/P 1−Type II error, power, sensitivity, recall Pos. Pred. value TP/P∗ Precision, 1−false discovery proportion Neg. Pred. value TN/N∗ TABLE 4.7. Important measures for classification and diagnostic testing, derived from quantities in Table 4.6. to a population. To make the connection with the epidemiology literature, we think of “+” as the “disease” that we are trying to detect, and “−” as the “non-disease” state. To make the connection to the classical hypothesis testing literature, we think of “−” as the null hypothesis and “+” as the alternative (non-null) hypothesis. In the context of the Default data, “+” indicates an individual who defaults, and “−” indicates one who does not. Table 4.7 lists many of the popular performance measures that are used in this context. The denominators for the false positive and true positive rates are the actual population counts in each class. In contrast, the denominators for the positive predictive value and the negative predictive value are the total predicted counts for each class. 4.4.3 Quadratic Discriminant Analysis As we have discussed, LDA assumes that the observations within each class are drawn from a multivariate Gaussian distribution with a class- specific mean vector and a covariance matrix that is common to all K classes. Quadratic discriminant analysis (QDA) provides an alternative quadratic discriminant analysis approach. Like LDA, the QDA classifier results from assuming that the observations from each class are drawn from a Gaussian distribution, and plugging estimates for the parameters into Bayes’ theorem in order to per- form prediction. However, unlike LDA, QDA assumes that each class has its own covariance matrix. That is, it assumes that an observation from the kth class is of the form X ∼ N(µk, Σk), where Σk is a covariance matrix for the kth class. Under this assumption, the Bayes classifier assigns an
- 163. 4.4 Generative Models for Classification 153 observation X = x to the class for which δk(x) = − 1 2 (x − µk)T Σ−1 k (x − µk) − 1 2 log |Σk| + log πk = − 1 2 xT Σ−1 k x + xT Σ−1 k µk − 1 2 µT k Σ−1 k µk − 1 2 log |Σk| + log πk (4.28) is largest. So the QDA classifier involves plugging estimates for Σk, µk, and πk into (4.28), and then assigning an observation X = x to the class for which this quantity is largest. Unlike in (4.24), the quantity x appears as a quadratic function in (4.28). This is where QDA gets its name. Why does it matter whether or not we assume that the K classes share a common covariance matrix? In other words, why would one prefer LDA to QDA, or vice-versa? The answer lies in the bias-variance trade-off. When there are p predictors, then estimating a covariance matrix requires esti- mating p(p+1)/2 parameters. QDA estimates a separate covariance matrix for each class, for a total of Kp(p+1)/2 parameters. With 50 predictors this is some multiple of 1,275, which is a lot of parameters. By instead assum- ing that the K classes share a common covariance matrix, the LDA model becomes linear in x, which means there are Kp linear coefficients to esti- mate. Consequently, LDA is a much less flexible classifier than QDA, and so has substantially lower variance. This can potentially lead to improved prediction performance. But there is a trade-off: if LDA’s assumption that the K classes share a common covariance matrix is badly off, then LDA can suffer from high bias. Roughly speaking, LDA tends to be a better bet than QDA if there are relatively few training observations and so reducing variance is crucial. In contrast, QDA is recommended if the training set is very large, so that the variance of the classifier is not a major concern, or if the assumption of a common covariance matrix for the K classes is clearly untenable. Figure 4.9 illustrates the performances of LDA and QDA in two scenarios. In the left-hand panel, the two Gaussian classes have a common correla- tion of 0.7 between X1 and X2. As a result, the Bayes decision boundary is linear and is accurately approximated by the LDA decision boundary. The QDA decision boundary is inferior, because it suffers from higher vari- ance without a corresponding decrease in bias. In contrast, the right-hand panel displays a situation in which the orange class has a correlation of 0.7 between the variables and the blue class has a correlation of −0.7. Now the Bayes decision boundary is quadratic, and so QDA more accurately approximates this boundary than does LDA. 4.4.4 Naive Bayes In previous sections, we used Bayes’ theorem (4.15) to develop the LDA and QDA classifiers. Here, we use Bayes’ theorem to motivate the popular naive Bayes classifier. naive Bayes
- 164. 154 4. Classification −4 −2 0 2 4 −4 −3 −2 −1 0 1 2 −4 −2 0 2 4 −4 −3 −2 −1 0 1 2 X1 X1 X 2 X 2 FIGURE 4.9. Left: The Bayes (purple dashed), LDA (black dotted), and QDA (green solid) decision boundaries for a two-class problem with Σ1 = Σ2. The shading indicates the QDA decision rule. Since the Bayes decision boundary is linear, it is more accurately approximated by LDA than by QDA. Right: Details are as given in the left-hand panel, except that Σ1 ̸= Σ2. Since the Bayes decision boundary is non-linear, it is more accurately approximated by QDA than by LDA. Recall that Bayes’ theorem (4.15) provides an expression for the pos- terior probability pk(x) = Pr(Y = k|X = x) in terms of π1, . . . , πK and f1(x), . . . , fK(x). To use (4.15) in practice, we need estimates for π1, . . . , πK and f1(x), . . . , fK(x). As we saw in previous sections, estimating the prior probabilities π1, . . . , πK is typically straightforward: for instance, we can estimate π̂k as the proportion of training observations belonging to the kth class, for k = 1, . . . , K. However, estimating f1(x), . . . , fK(x) is more subtle. Recall that fk(x) is the p-dimensional density function for an observation in the kth class, for k = 1, . . . , K. In general, estimating a p-dimensional density function is challenging. In LDA, we make a very strong assumption that greatly sim- plifies the task: we assume that fk is the density function for a multivariate normal random variable with class-specific mean µk, and shared covariance matrix Σ. By contrast, in QDA, we assume that fk is the density function for a multivariate normal random variable with class-specific mean µk, and class-specific covariance matrix Σk. By making these very strong assump- tions, we are able to replace the very challenging problem of estimating K p-dimensional density functions with the much simpler problem of estimat- ing K p-dimensional mean vectors and one (in the case of LDA) or K (in the case of QDA) (p × p)-dimensional covariance matrices. The naive Bayes classifier takes a different tack for estimating f1(x), . . . , fK(x). Instead of assuming that these functions belong to a particular family of distributions (e.g. multivariate normal), we instead make a single assumption:
- 165. 4.4 Generative Models for Classification 155 Within the kth class, the p predictors are independent. Stated mathematically, this assumption means that for k = 1, . . . , K, fk(x) = fk1(x1) × fk2(x2) × · · · × fkp(xp), (4.29) where fkj is the density function of the jth predictor among observations in the kth class. Why is this assumption so powerful? Essentially, estimating a p-dimen- sional density function is challenging because we must consider not only the marginal distribution of each predictor — that is, the distribution of marginal distribution each predictor on its own — but also the joint distribution of the predictors joint distribution — that is, the association between the different predictors. In the case of a multivariate normal distribution, the association between the different predictors is summarized by the off-diagonal elements of the covariance matrix. However, in general, this association can be very hard to charac- terize, and exceedingly challenging to estimate. But by assuming that the p covariates are independent within each class, we completely eliminate the need to worry about the association between the p predictors, because we have simply assumed that there is no association between the predictors! Do we really believe the naive Bayes assumption that the p covariates are independent within each class? In most settings, we do not. But even though this modeling assumption is made for convenience, it often leads to pretty decent results, especially in settings where n is not large enough relative to p for us to effectively estimate the joint distribution of the predic- tors within each class. In fact, since estimating a joint distribution requires such a huge amount of data, naive Bayes is a good choice in a wide range of settings. Essentially, the naive Bayes assumption introduces some bias, but reduces variance, leading to a classifier that works quite well in practice as a result of the bias-variance trade-off. Once we have made the naive Bayes assumption, we can plug (4.29) into (4.15) to obtain an expression for the posterior probability, Pr(Y = k|X = x) = πk × fk1(x1) × fk2(x2) × · · · × fkp(xp) )K l=1 πl × fl1(x1) × fl2(x2) × · · · × flp(xp) (4.30) for k = 1, . . . , K. To estimate the one-dimensional density function fkj using training data x1j, . . . , xnj, we have a few options. • If Xj is quantitative, then we can assume that Xj|Y = k ∼ N(µjk, σ2 jk). In other words, we assume that within each class, the jth predictor is drawn from a (univariate) normal distribution. While this may sound a bit like QDA, there is one key difference, in that here we are assum- ing that the predictors are independent; this amounts to QDA with an additional assumption that the class-specific covariance matrix is diagonal.
- 166. 156 4. Classification True default status No Yes Total Predicted No 9615 241 9856 default status Yes 52 92 144 Total 9667 333 10000 TABLE 4.8. Comparison of the naive Bayes predictions to the true default status for the 10, 000 training observations in the Default data set, when we predict default for any observation for which P(Y = default|X = x) > 0.5. • If Xj is quantitative, then another option is to use a non-parametric estimate for fkj. A very simple way to do this is by making a his- togram for the observations of the jth predictor within each class. Then we can estimate fkj(xj) as the fraction of the training obser- vations in the kth class that belong to the same histogram bin as xj. Alternatively, we can use a kernel density estimator, which is essen- kernel density estimator tially a smoothed version of a histogram. • If Xj is qualitative, then we can simply count the proportion of train- ing observations for the jth predictor corresponding to each class. For instance, suppose that Xj ∈ {1, 2, 3}, and we have 100 observations in the kth class. Suppose that the jth predictor takes on values of 1, 2, and 3 in 32, 55, and 13 of those observations, respectively. Then we can estimate fkj as ˆ fkj(xj) = ⎧ ⎪ ⎨ ⎪ ⎩ 0.32 if xj = 1 0.55 if xj = 2 0.13 if xj = 3. We now consider the naive Bayes classifier in a toy example with p = 3 predictors and K = 2 classes. The first two predictors are quantitative, and the third predictor is qualitative with three levels. Suppose further that π̂1 = π̂2 = 0.5. The estimated density functions ˆ fkj for k = 1, 2 and j = 1, 2, 3 are displayed in Figure 4.10. Now suppose that we wish to classify a new observation, x∗ = (0.4, 1.5, 1)T . It turns out that in this example, ˆ f11(0.4) = 0.368, ˆ f12(1.5) = 0.484, ˆ f13(1) = 0.226, and ˆ f21(0.4) = 0.030, ˆ f22(1.5) = 0.130, ˆ f23(1) = 0.616. Plugging these estimates into (4.30) results in posterior probability estimates of Pr(Y = 1|X = x∗ ) = 0.944 and Pr(Y = 2|X = x∗ ) = 0.056. Table 4.8 provides the confusion matrix resulting from applying the naive Bayes classifier to the Default data set, where we predict a default if the posterior probability of a default — that is, P(Y = default|X = x) — ex- ceeds 0.5. Comparing this to the results for LDA in Table 4.4, our findings are mixed. While LDA has a slightly lower overall error rate, naive Bayes correctly predicts a higher fraction of the true defaulters. In this implemen- tation of naive Bayes, we have assumed that each quantitative predictor is
- 167. 4.4 Generative Models for Classification 157 Density estimates for class k=1 ˆ f11 ˆ f12 ˆ f13 −4 −2 0 2 4 Frequency −2 0 2 4 1 2 3 Density estimates for class k=2 ˆ f21 ˆ f22 ˆ f23 −4 −2 0 2 4 Frequency −2 0 2 4 1 2 3 FIGURE 4.10. In the toy example in Section 4.4.4, we generate data with p = 3 predictors and K = 2 classes. The first two predictors are quantitative, and the third predictor is qualitative with three levels. In each class, the estimated density for each of the three predictors is displayed. If the prior probabilities for the two classes are equal, then the observation x∗ = (0.4, 1.5, 1)T has a 94.4% posterior probability of belonging to the first class. True default status No Yes Total Predicted No 9320 128 9448 default status Yes 347 205 552 Total 9667 333 10000 TABLE 4.9. Comparison of the naive Bayes predictions to the true default status for the 10, 000 training observations in the Default data set, when we predict default for any observation for which P(Y = default|X = x) > 0.2. drawn from a Gaussian distribution (and, of course, that within each class, each predictor is independent). Just as with LDA, we can easily adjust the probability threshold for predicting a default. For example, Table 4.9 provides the confusion matrix resulting from predicting a default if P(Y = default|X = x) > 0.2. Again,
- 168. 158 4. Classification the results are mixed relative to LDA with the same threshold (Table 4.5). Naive Bayes has a higher error rate, but correctly predicts almost two-thirds of the true defaults. In this example, it should not be too surprising that naive Bayes does not convincingly outperform LDA: this data set has n = 10,000 and p = 4, and so the reduction in variance resulting from the naive Bayes assumption is not necessarily worthwhile. We expect to see a greater pay-off to using naive Bayes relative to LDA or QDA in instances where p is larger or n is smaller, so that reducing the variance is very important. 4.5 A Comparison of Classification Methods 4.5.1 An Analytical Comparison We now perform an analytical (or mathematical) comparison of LDA, QDA, naive Bayes, and logistic regression. We consider these approaches in a setting with K classes, so that we assign an observation to the class that maximizes Pr(Y = k|X = x). Equivalently, we can set K as the baseline class and assign an observation to the class that maximizes log * Pr(Y = k|X = x) Pr(Y = K|X = x) + (4.31) for k = 1, . . . , K. Examining the specific form of (4.31) for each method provides a clear understanding of their similarities and differences. First, for LDA, we can make use of Bayes’ Theorem (4.15) as well as the assumption that the predictors within each class are drawn from a multivariate normal density (4.23) with class-specific mean and shared co- variance matrix in order to show that log * Pr(Y = k|X = x) Pr(Y = K|X = x) + = log * πkfk(x) πKfK(x) + = log > πk exp ' −1 2 (x − µk)T Σ−1 (x − µk) ( πK exp ' −1 2 (x − µK)T Σ−1 (x − µK) ( ? = log * πk πK + − 1 2 (x − µk)T Σ−1 (x − µk) + 1 2 (x − µK)T Σ−1 (x − µK) = log * πk πK + − 1 2 (µk + µK)T Σ−1 (µk − µK) + xT Σ−1 (µk − µK) = ak + p 0 j=1 bkjxj, (4.32)
- 169. 4.5 A Comparison of Classification Methods 159 where ak = log 1 πk πK 2 − 1 2 (µk + µK)T Σ−1 (µk − µK) and bkj is the jth component of Σ−1 (µk − µK). Hence LDA, like logistic regression, assumes that the log odds of the posterior probabilities is linear in x. Using similar calculations, in the QDA setting (4.31) becomes log * Pr(Y = k|X = x) Pr(Y = K|X = x) + = ak + p 0 j=1 bkjxj + p 0 j=1 p 0 l=1 ckjlxjxl, (4.33) where ak, bkj, and ckjl are functions of πk, πK, µk, µK, Σk and ΣK. Again, as the name suggests, QDA assumes that the log odds of the posterior probabilities is quadratic in x. Finally, we examine (4.31) in the naive Bayes setting. Recall that in this setting, fk(x) is modeled as a product of p one-dimensional functions fkj(xj) for j = 1, . . . , p. Hence, log * Pr(Y = k|X = x) Pr(Y = K|X = x) + = log * πkfk(x) πKfK(x) + = log > πk Fp j=1 fkj(xj) πK Fp j=1 fKj(xj) ? = log * πk πK + + p 0 j=1 log * fkj(xj) fKj(xj) + = ak + p 0 j=1 gkj(xj), (4.34) where ak = log 1 πk πK 2 and gkj(xj) = log 1 fkj (xj ) fKj (xj ) 2 . Hence, the right-hand side of (4.34) takes the form of a generalized additive model, a topic that is discussed further in Chapter 7. Inspection of (4.32), (4.33), and (4.34) yields the following observations about LDA, QDA, and naive Bayes: • LDA is a special case of QDA with ckjl = 0 for all j = 1, . . . , p, l = 1, . . . , p, and k = 1, . . . , K. (Of course, this is not surprising, since LDA is simply a restricted version of QDA with Σ1 = · · · = ΣK = Σ.) • Any classifier with a linear decision boundary is a special case of naive Bayes with gkj(xj) = bkjxj. In particular, this means that LDA is a special case of naive Bayes! This is not at all obvious from the descriptions of LDA and naive Bayes earlier in the chapter, since each method makes very different assumptions: LDA assumes that the features are normally distributed with a common within-class covariance matrix, and naive Bayes instead assumes independence of the features.
- 170. 160 4. Classification • If we model fkj(xj) in the naive Bayes classifier using a one-dimensional Gaussian distribution N(µkj, σ2 j ), then we end up with gkj(xj) = bkjxj where bkj = (µkj − µKj)/σ2 j . In this case, naive Bayes is actu- ally a special case of LDA with Σ restricted to be a diagonal matrix with jth diagonal element equal to σ2 j . • Neither QDA nor naive Bayes is a special case of the other. Naive Bayes can produce a more flexible fit, since any choice can be made for gkj(xj). However, it is restricted to a purely additive fit, in the sense that in (4.34), a function of xj is added to a function of xl, for j ̸= l; however, these terms are never multiplied. By contrast, QDA includes multiplicative terms of the form ckjlxjxl. Therefore, QDA has the potential to be more accurate in settings where interactions among the predictors are important in discriminating between classes. None of these methods uniformly dominates the others: in any setting, the choice of method will depend on the true distribution of the predictors in each of the K classes, as well as other considerations, such as the values of n and p. The latter ties into the bias-variance trade-off. How does logistic regression tie into this story? Recall from (4.12) that multinomial logistic regression takes the form log * Pr(Y = k|X = x) Pr(Y = K|X = x) + = βk0 + p 0 j=1 βkjxj. This is identical to the linear form of LDA (4.32): in both cases, log 1 Pr(Y =k|X=x) Pr(Y =K|X=x) 2 is a linear function of the predictors. In LDA, the co- efficients in this linear function are functions of estimates for πk, πK, µk, µK, and Σ obtained by assuming that X1, . . . , Xp follow a normal distri- bution within each class. By contrast, in logistic regression, the coefficients are chosen to maximize the likelihood function (4.5). Thus, we expect LDA to outperform logistic regression when the normality assumption (approxi- mately) holds, and we expect logistic regression to perform better when it does not. We close with a brief discussion of K-nearest neighbors (KNN), intro- duced in Chapter 2. Recall that KNN takes a completely different approach from the classifiers seen in this chapter. In order to make a prediction for an observation X = x, the training observations that are closest to x are identified. Then X is assigned to the class to which the plurality of these observations belong. Hence KNN is a completely non-parametric approach: no assumptions are made about the shape of the decision boundary. We make the following observations about KNN: • Because KNN is completely non-parametric, we can expect this ap- proach to dominate LDA and logistic regression when the decision
- 171. 4.5 A Comparison of Classification Methods 161 boundary is highly non-linear, provided that n is very large and p is small. • In order to provide accurate classification, KNN requires a lot of ob- servations relative to the number of predictors—that is, n much larger than p. This has to do with the fact that KNN is non-parametric, and thus tends to reduce the bias while incurring a lot of variance. • In settings where the decision boundary is non-linear but n is only modest, or p is not very small, then QDA may be preferred to KNN. This is because QDA can provide a non-linear decision boundary while taking advantage of a parametric form, which means that it requires a smaller sample size for accurate classification, relative to KNN. • Unlike logistic regression, KNN does not tell us which predictors are important: we don’t get a table of coefficients as in Table 4.3. 4.5.2 An Empirical Comparison We now compare the empirical (practical) performance of logistic regres- sion, LDA, QDA, naive Bayes, and KNN. We generated data from six dif- ferent scenarios, each of which involves a binary (two-class) classification problem. In three of the scenarios, the Bayes decision boundary is linear, and in the remaining scenarios it is non-linear. For each scenario, we pro- duced 100 random training data sets. On each of these training sets, we fit each method to the data and computed the resulting test error rate on a large test set. Results for the linear scenarios are shown in Figure 4.11, and the results for the non-linear scenarios are in Figure 4.12. The KNN method requires selection of K, the number of neighbors (not to be con- fused with the number of classes in earlier sections of this chapter). We performed KNN with two values of K: K = 1, and a value of K that was chosen automatically using an approach called cross-validation, which we discuss further in Chapter 5. We applied naive Bayes assuming univariate Gaussian densities for the features within each class (and, of course — since this is the key characteristic of naive Bayes — assuming independence of the features). In each of the six scenarios, there were p = 2 quantitative predictors. The scenarios were as follows: Scenario 1: There were 20 training observations in each of two classes. The observations within each class were uncorrelated random normal variables with a different mean in each class. The left-hand panel of Figure 4.11 shows that LDA performed well in this setting, as one would expect since this is the model assumed by LDA. Logistic regression also performed quite well, since it assumes a linear decision boundary. KNN performed poorly because
- 172. 162 4. Classification KNN−1 KNN−CV LDA Logistic NBayes QDA SCENARIO 1 0.25 0.30 0.35 0.40 0.45 KNN−1 KNN−CV LDA Logistic NBayes QDA SCENARIO 2 0.15 0.20 0.25 0.30 KNN−1 KNN−CV LDA Logistic NBayes QDA SCENARIO 3 0.20 0.25 0.30 0.35 0.40 0.45 0.50 FIGURE 4.11. Boxplots of the test error rates for each of the linear scenarios described in the main text. it paid a price in terms of variance that was not offset by a reduction in bias. QDA also performed worse than LDA, since it fit a more flexible classifier than necessary. The performance of naive Bayes was slightly better than QDA, because the naive Bayes assumption of independent predictors is correct. Scenario 2: Details are as in Scenario 1, except that within each class, the two predictors had a correlation of −0.5. The center panel of Figure 4.11 indicates that the performance of most methods is similar to the previ- ous scenario. The notable exception is naive Bayes, which performs very poorly here, since the naive Bayes assumption of independent predictors is violated. Scenario 3: As in the previous scenario, there is substantial negative cor- relation between the predictors within each class. However, this time we generated X1 and X2 from the t-distribution, with 50 observations per class. t- distribution The t-distribution has a similar shape to the normal distribution, but it has a tendency to yield more extreme points—that is, more points that are far from the mean. In this setting, the decision boundary was still linear, and so fit into the logistic regression framework. The set-up violated the assumptions of LDA, since the observations were not drawn from a normal distribution. The right-hand panel of Figure 4.11 shows that logistic regres- sion outperformed LDA, though both methods were superior to the other approaches. In particular, the QDA results deteriorated considerably as a consequence of non-normality. Naive Bayes performed very poorly because the independence assumption is violated. Scenario 4: The data were generated from a normal distribution, with a correlation of 0.5 between the predictors in the first class, and correlation of −0.5 between the predictors in the second class. This setup corresponded to the QDA assumption, and resulted in quadratic decision boundaries. The left-hand panel of Figure 4.12 shows that QDA outperformed all of the other approaches. The naive Bayes assumption of independent predictors is violated, so naive Bayes performs poorly.
- 173. 4.5 A Comparison of Classification Methods 163 KNN−1 KNN−CV LDA Logistic NBayes QDA SCENARIO 4 0.30 0.35 0.40 KNN−1 KNN−CV LDA Logistic NBayes QDA SCENARIO 5 0.18 0.20 0.22 0.24 0.26 0.28 0.30 0.32 KNN−1 KNN−CV LDA Logistic NBayes QDA SCENARIO 6 0.15 0.20 0.25 0.30 0.35 0.40 0.45 FIGURE 4.12. Boxplots of the test error rates for each of the non-linear sce- narios described in the main text. Scenario 5: The data were generated from a normal distribution with un- correlated predictors. Then the responses were sampled from the logistic function applied to a complicated non-linear function of the predictors. The center panel of Figure 4.12 shows that both QDA and naive Bayes gave slightly better results than the linear methods, while the much more flexi- ble KNN-CV method gave the best results. But KNN with K = 1 gave the worst results out of all methods. This highlights the fact that even when the data exhibits a complex non-linear relationship, a non-parametric method such as KNN can still give poor results if the level of smoothness is not chosen correctly. Scenario 6: The observations were generated from a normal distribution with a different diagonal covariance matrix for each class. However, the sample size was very small: just n = 6 in each class. Naive Bayes performed very well, because its assumptions are met. LDA and logistic regression performed poorly because the true decision boundary is non-linear, due to the unequal covariance matrices. QDA performed a bit worse than naive Bayes, because given the very small sample size, the former incurred too much variance in estimating the correlation between the predictors within each class. KNN’s performance also suffered due to the very small sample size. These six examples illustrate that no one method will dominate the oth- ers in every situation. When the true decision boundaries are linear, then the LDA and logistic regression approaches will tend to perform well. When the boundaries are moderately non-linear, QDA or naive Bayes may give better results. Finally, for much more complicated decision boundaries, a non-parametric approach such as KNN can be superior. But the level of smoothness for a non-parametric approach must be chosen carefully. In the next chapter we examine a number of approaches for choosing the correct level of smoothness and, in general, for selecting the best overall method. Finally, recall from Chapter 3 that in the regression setting we can accom- modate a non-linear relationship between the predictors and the response
- 174. 164 4. Classification by performing regression using transformations of the predictors. A similar approach could be taken in the classification setting. For instance, we could create a more flexible version of logistic regression by including X2 , X3 , and even X4 as predictors. This may or may not improve logistic regres- sion’s performance, depending on whether the increase in variance due to the added flexibility is offset by a sufficiently large reduction in bias. We could do the same for LDA. If we added all possible quadratic terms and cross-products to LDA, the form of the model would be the same as the QDA model, although the parameter estimates would be different. This device allows us to move somewhere between an LDA and a QDA model. 4.6 Generalized Linear Models In Chapter 3, we assumed that the response Y is quantitative, and ex- plored the use of least squares linear regression to predict Y . Thus far in this chapter, we have instead assumed that Y is qualitative. However, we may sometimes be faced with situations in which Y is neither qualitative nor quantitative, and so neither linear regression from Chapter 3 nor the classification approaches covered in this chapter is applicable. As a concrete example, we consider the Bikeshare data set. The response is bikers, the number of hourly users of a bike sharing program in Wash- ington, DC. This response value is neither qualitative nor quantitative: instead, it takes on non-negative integer values, or counts. We will consider counts predicting bikers using the covariates mnth (month of the year), hr (hour of the day, from 0 to 23), workingday (an indicator variable that equals 1 if it is neither a weekend nor a holiday), temp (the normalized temperature, in Celsius), and weathersit (a qualitative variable that takes on one of four possible values: clear; misty or cloudy; light rain or light snow; or heavy rain or heavy snow.) In the analyses that follow, we will treat mnth, hr, and weathersit as qualitative variables. 4.6.1 Linear Regression on the Bikeshare Data To begin, we consider predicting bikers using linear regression. The results are shown in Table 4.10. We see, for example, that a progression of weather from clear to cloudy results in, on average, 12.89 fewer bikers per hour; however, if the weather progresses further to rain or snow, then this further results in 53.60 fewer bikers per hour. Figure 4.13 displays the coefficients associated with mnth and the coefficients associated with hr. We see that bike usage is highest in the spring and fall, and lowest during the winter months. Furthermore, bike usage is greatest around rush hour (9 AM and 6 PM), and lowest overnight. Thus, at first glance, fitting a linear regression model to the Bikeshare data set seems to provide reasonable and intuitive results.
- 175. 4.6 Generalized Linear Models 165 Coefficient Std. error z-statistic p-value Intercept 73.60 5.13 14.34 0.00 workingday 1.27 1.78 0.71 0.48 temp 157.21 10.26 15.32 0.00 weathersit[cloudy/misty] -12.89 1.96 -6.56 0.00 weathersit[light rain/snow] -66.49 2.97 -22.43 0.00 weathersit[heavy rain/snow] -109.75 76.67 -1.43 0.15 TABLE 4.10. Results for a least squares linear model fit to predict bikers in the Bikeshare data. The predictors mnth and hr are omitted from this table due to space constraints, and can be seen in Figure 4.13. For the qualitative variable weathersit, the baseline level corresponds to clear skies. ● ● ● ● ● ● ● ● ● ● ● ● −40 −20 0 20 Month Coefficient J F M A M J J A S O N D ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5 10 15 20 −100 0 50 100 200 Hour Coefficient FIGURE 4.13. A least squares linear regression model was fit to predict bikers in the Bikeshare data set. Left: The coefficients associated with the month of the year. Bike usage is highest in the spring and fall, and lowest in the winter. Right: The coefficients associated with the hour of the day. Bike usage is highest during peak commute times, and lowest overnight. But upon more careful inspection, some issues become apparent. For example, 9.6% of the fitted values in the Bikeshare data set are negative: that is, the linear regression model predicts a negative number of users during 9.6% of the hours in the data set. This calls into question our ability to perform meaningful predictions on the data, and it also raises concerns about the accuracy of the coefficient estimates, confidence intervals, and other outputs of the regression model. Furthermore, it is reasonable to suspect that when the expected value of bikers is small, the variance of bikers should be small as well. For instance, at 2 AM during a heavy December snow storm, we expect that extremely few people will use a bike, and moreover that there should be little variance associated with the number of users during those conditions. This is borne out in the data: between 1 AM and 4 AM, in December, January, and February, when it is raining, there are 5.05 users, on average,
- 176. 166 4. Classification 5 10 15 20 0 100 200 300 400 500 600 Hour Number of Bikers 5 10 15 20 0 1 2 3 4 5 6 Hour Log(Number of Bikers) FIGURE 4.14. Left: On the Bikeshare dataset, the number of bikers is dis- played on the y-axis, and the hour of the day is displayed on the x-axis. Jitter was applied for ease of visualization. For the most part, as the mean number of bikers increases, so does the variance in the number of bikers. A smoothing spline fit is shown in green. Right: The log of the number of bikers is now displayed on the y-axis. with a standard deviation of 3.73. By contrast, between 7 AM and 10 AM, in April, May, and June, when skies are clear, there are 243.59 users, on average, with a standard deviation of 131.7. The mean-variance relationship is displayed in the left-hand panel of Figure 4.14. This is a major violation of the assumptions of a linear model, which state that Y = )p j=1 Xjβj +ϵ, where ϵ is a mean-zero error term with variance σ2 that is constant, and not a function of the covariates. Therefore, the heteroscedasticity of the data calls into question the suitability of a linear regression model. Finally, the response bikers is integer-valued. But under a linear model, Y = β0 + )p j=1 Xjβj + ϵ, where ϵ is a continuous-valued error term. This means that in a linear model, the response Y is necessarily continuous- valued (quantitative). Thus, the integer nature of the response bikers sug- gests that a linear regression model is not entirely satisfactory for this data set. Some of the problems that arise when fitting a linear regression model to the Bikeshare data can be overcome by transforming the response; for instance, we can fit the model log(Y ) = p 0 j=1 Xjβj + ϵ. Transforming the response avoids the possibility of negative predictions, and it overcomes much of the heteroscedasticity in the untransformed data, as is shown in the right-hand panel of Figure 4.14. However, it is not quite a satisfactory solution, since predictions and inference are made in terms of the log of the response, rather than the response. This leads to challenges in interpretation, e.g. “a one-unit increase in Xj is associated with an increase in the mean of the log of Y by an amount βj”. Furthermore, a
- 177. 4.6 Generalized Linear Models 167 log transformation of the response cannot be applied in settings where the response can take on a value of 0. Thus, while fitting a linear model to a transformation of the response may be an adequate approach for some count-valued data sets, it often leaves something to be desired. We will see in the next section that a Poisson regression model provides a much more natural and elegant approach for this task. 4.6.2 Poisson Regression on the Bikeshare Data To overcome the inadequacies of linear regression for analyzing the Bikeshare data set, we will make use of an alternative approach, called Poisson regression. Before we can talk about Poisson regression, we must first in- Poisson regression troduce the Poisson distribution. Poisson distribution Suppose that a random variable Y takes on nonnegative integer values, i.e. Y ∈ {0, 1, 2, . . .}. If Y follows the Poisson distribution, then Pr(Y = k) = e−λ λk k! for k = 0, 1, 2, . . . . (4.35) Here, λ > 0 is the expected value of Y , i.e. E(Y ). It turns out that λ also equals the variance of Y , i.e. λ = E(Y ) = Var(Y ). This means that if Y follows the Poisson distribution, then the larger the mean of Y , the larger its variance. (In (4.35), the notation k!, pronounced “k factorial”, is defined as k! = k × (k − 1) × (k − 2) × . . . × 3 × 2 × 1.) The Poisson distribution is typically used to model counts; this is a natural choice for a number of reasons, including the fact that counts, like the Poisson distribution, take on nonnegative integer values. To see how we might use the Poisson distribution in practice, let Y denote the number of users of the bike sharing program during a particular hour of the day, under a particular set of weather conditions, and during a particular month of the year. We might model Y as a Poisson distribution with mean E(Y ) = λ = 5. This means that the probability of no users during this particular hour is Pr(Y = 0) = e−5 50 0! = e−5 = 0.0067 (where 0! = 1 by convention). The probability that there is exactly one user is Pr(Y = 1) = e−5 51 1! = 5e−5 = 0.034, the probability of two users is Pr(Y = 2) = e−5 52 2! = 0.084, and so on. Of course, in reality, we expect the mean number of users of the bike sharing program, λ = E(Y ), to vary as a function of the hour of the day, the month of the year, the weather conditions, and so forth. So rather than modeling the number of bikers, Y , as a Poisson distribution with a fixed mean value like λ = 5, we would like to allow the mean to vary as a function of the covariates. In particular, we consider the following model for the mean λ = E(Y ), which we now write as λ(X1, . . . , Xp) to emphasize that it is a function of the covariates X1, . . . , Xp: log(λ(X1, . . . , Xp)) = β0 + β1X1 + · · · + βpXp (4.36)
- 178. 168 4. Classification Coefficient Std. error z-statistic p-value Intercept 4.12 0.01 683.96 0.00 workingday 0.01 0.00 7.5 0.00 temp 0.79 0.01 68.43 0.00 weathersit[cloudy/misty] -0.08 0.00 -34.53 0.00 weathersit[light rain/snow] -0.58 0.00 -141.91 0.00 weathersit[heavy rain/snow] -0.93 0.17 -5.55 0.00 TABLE 4.11. Results for a Poisson regression model fit to predict bikers in the Bikeshare data. The predictors mnth and hr are omitted from this table due to space constraints, and can be seen in Figure 4.15. For the qualitative variable weathersit, the baseline corresponds to clear skies. or equivalently λ(X1, . . . , Xp) = eβ0+β1X1+···+βpXp . (4.37) Here, β0, β1, . . . , βp are parameters to be estimated. Together, (4.35) and (4.36) define the Poisson regression model. Notice that in (4.36), we take the log of λ(X1, . . . , Xp) to be linear in X1, . . . , Xp, rather than hav- ing λ(X1, . . . , Xp) itself be linear in X1, . . . , Xp, in order to ensure that λ(X1, . . . , Xp) takes on nonnegative values for all values of the covariates. To estimate the coefficients β0, β1, . . . , βp, we use the same maximum likelihood approach that we adopted for logistic regression in Section 4.3.2. Specifically, given n independent observations from the Poisson regression model, the likelihood takes the form ℓ(β0, β1, . . . , βp) = n E i=1 e−λ(xi) λ(xi)yi yi! , (4.38) where λ(xi) = eβ0+β1xi1+···+βpxip , due to (4.37). We estimate the coef- ficients that maximize the likelihood ℓ(β0, β1, . . . , βp), i.e. that make the observed data as likely as possible. We now fit a Poisson regression model to the Bikeshare data set. The results are shown in Table 4.11 and Figure 4.15. Qualitatively, the results are similar to those from linear regression in Section 4.6.1. We again see that bike usage is highest in the spring and fall and during rush hour, and lowest during the winter and in the early morning hours. Moreover, bike usage increases as the temperature increases, and decreases as the weather worsens. Interestingly, the coefficient associated with workingday is statistically significant under the Poisson regression model, but not under the linear regression model. Some important distinctions between the Poisson regression model and the linear regression model are as follows: • Interpretation: To interpret the coefficients in the Poisson regression model, we must pay close attention to (4.37), which states that an increase in Xj by one unit is associated with a change in E(Y ) = λ by a factor of exp(βj). For example, a change in weather from clear
- 179. 4.6 Generalized Linear Models 169 ● ● ● ● ● ● ● ● ● ● ● ● −0.6 −0.4 −0.2 0.0 0.2 Month Coefficient J F M A M J J A S O N D ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5 10 15 20 −2 −1 0 1 Hour Coefficient FIGURE 4.15. A Poisson regression model was fit to predict bikers in the Bikeshare data set. Left: The coefficients associated with the month of the year. Bike usage is highest in the spring and fall, and lowest in the winter. Right: The coefficients associated with the hour of the day. Bike usage is highest during peak commute times, and lowest overnight. to cloudy skies is associated with a change in mean bike usage by a factor of exp(−0.08) = 0.923, i.e. on average, only 92.3% as many people will use bikes when it is cloudy relative to when it is clear. If the weather worsens further and it begins to rain, then the mean bike usage will further change by a factor of exp(−0.5) = 0.607, i.e. on average only 60.7% as many people will use bikes when it is rainy relative to when it is cloudy. • Mean-variance relationship: As mentioned earlier, under the Poisson model, λ = E(Y ) = Var(Y ). Thus, by modeling bike usage with a Poisson regression, we implicitly assume that mean bike usage in a given hour equals the variance of bike usage during that hour. By contrast, under a linear regression model, the variance of bike usage always takes on a constant value. Recall from Figure 4.14 that in the Bikeshare data, when biking conditions are favorable, both the mean and the variance in bike usage are much higher than when conditions are unfavorable. Thus, the Poisson regression model is able to handle the mean-variance relationship seen in the Bikeshare data in a way that the linear regression model is not.5 overdispersion • nonnegative fitted values: There are no negative predictions using the Poisson regression model. This is because the Poisson model itself only allows for nonnegative values; see (4.35). By contrast, when we 5In fact, the variance in the Bikeshare data appears to be much higher than the mean, a situation referred to as overdispersion. This causes the Z-values to be inflated in Table 4.11. A more careful analysis should account for this overdispersion to obtain more accurate Z-values, and there are a variety of methods for doing this. But they are beyond the scope of this book.
- 180. 170 4. Classification fit a linear regression model to the Bikeshare data set, almost 10% of the predictions were negative. 4.6.3 Generalized Linear Models in Greater Generality We have now discussed three types of regression models: linear, logistic and Poisson. These approaches share some common characteristics: 1. Each approach uses predictors X1, . . . , Xp to predict a response Y . We assume that, conditional on X1, . . . , Xp, Y belongs to a certain family of distributions. For linear regression, we typically assume that Y follows a Gaussian or normal distribution. For logistic regression, we assume that Y follows a Bernoulli distribution. Finally, for Poisson regression, we assume that Y follows a Poisson distribution. 2. Each approach models the mean of Y as a function of the predictors. In linear regression, the mean of Y takes the form E(Y |X1, . . . , Xp) = β0 + β1X1 + · · · + βpXp, (4.39) i.e. it is a linear function of the predictors. For logistic regression, the mean instead takes the form E(Y |X1, . . . , Xp) = Pr(Y = 1|X1, . . . , Xp) = eβ0+β1X1+···+βpXp 1 + eβ0+β1X1+···+βpXp , (4.40) while for Poisson regression it takes the form E(Y |X1, . . . , Xp) = λ(X1, . . . , Xp) = eβ0+β1X1+···+βpXp . (4.41) Equations (4.39)–(4.41) can be expressed using a link function, η, which link function applies a transformation to E(Y |X1, . . . , Xp) so that the transformed mean is a linear function of the predictors. That is, η(E(Y |X1, . . . , Xp)) = β0 + β1X1 + · · · + βpXp. (4.42) The link functions for linear, logistic and Poisson regression are η(µ) = µ, η(µ) = log(µ/(1 − µ)), and η(µ) = log(µ), respectively. The Gaussian, Bernoulli and Poisson distributions are all members of a wider class of distributions, known as the exponential family. Other well- exponential family known members of this family are the exponential distribution, the Gamma exponential distribution distribution, and the negative binomial distribution. In general, we can per- Gamma distribution negative binomial distribution form a regression by modeling the response Y as coming from a particular member of the exponential family, and then transforming the mean of the response so that the transformed mean is a linear function of the predictors via (4.42). Any regression approach that follows this very general recipe is known as a generalized linear model (GLM). Thus, linear regression, logistic generalized linear model regression, and Poisson regression are three examples of GLMs. Other ex- amples not covered here include Gamma regression and negative binomial Gamma regression regression. negative binomial regression
- 181. 4.7 Lab: Classification Methods 171 4.7 Lab: Classification Methods 4.7.1 The Stock Market Data We will begin by examining some numerical and graphical summaries of the Smarket data, which is part of the ISLR2 library. This data set consists of percentage returns for the S&P 500 stock index over 1, 250 days, from the beginning of 2001 until the end of 2005. For each date, we have recorded the percentage returns for each of the five previous trading days, Lag1 through Lag5. We have also recorded Volume (the number of shares traded on the previous day, in billions), Today (the percentage return on the date in question) and Direction (whether the market was Up or Down on this date). Our goal is to predict Direction (a qualitative response) using the other features. > library(ISLR2) > names(Smarket) [1] "Year" "Lag1" "Lag2" "Lag3" "Lag4" [6] "Lag5" "Volume" "Today" "Direction" > dim(Smarket) [1] 1250 9 > summary(Smarket) Year Lag1 Lag2 Min. :2001 Min. : -4.92200 Min. : -4.92200 1st Qu .:2002 1st Qu .: -0.63950 1st Qu .: -0.63950 Median :2003 Median : 0.03900 Median : 0.03900 Mean :2003 Mean : 0.00383 Mean : 0.00392 3rd Qu .:2004 3rd Qu.: 0.59675 3rd Qu.: 0.59675 Max. :2005 Max. : 5.73300 Max. : 5.73300 Lag3 Lag4 Lag5 Min. : -4.92200 Min. : -4.92200 Min. : -4.92200 1st Qu .: -0.64000 1st Qu .: -0.64000 1st Qu .: -0.64000 Median : 0.03850 Median : 0.03850 Median : 0.03850 Mean : 0.00172 Mean : 0.00164 Mean : 0.00561 3rd Qu.: 0.59675 3rd Qu.: 0.59675 3rd Qu.: 0.59700 Max. : 5.73300 Max. : 5.73300 Max. : 5.73300 Volume Today Direction Min. :0.356 Min. : -4.92200 Down :602 1st Qu .:1.257 1st Qu .: -0.63950 Up :648 Median :1.423 Median : 0.03850 Mean :1.478 Mean : 0.00314 3rd Qu .:1.642 3rd Qu.: 0.59675 Max. :3.152 Max. : 5.73300 > pairs(Smarket) The cor() function produces a matrix that contains all of the pairwise correlations among the predictors in a data set. The first command below gives an error message because the Direction variable is qualitative. > cor(Smarket) Error in cor(Smarket) : ‘x’ must be numeric > cor(Smarket[, -9])
- 182. 172 4. Classification Year Lag1 Lag2 Lag3 Lag4 Lag5 Year 1.0000 0.02970 0.03060 0.03319 0.03569 0.02979 Lag1 0.0297 1.00000 -0.02629 -0.01080 -0.00299 -0.00567 Lag2 0.0306 -0.02629 1.00000 -0.02590 -0.01085 -0.00356 Lag3 0.0332 -0.01080 -0.02590 1.00000 -0.02405 -0.01881 Lag4 0.0357 -0.00299 -0.01085 -0.02405 1.00000 -0.02708 Lag5 0.0298 -0.00567 -0.00356 -0.01881 -0.02708 1.00000 Volume 0.5390 0.04091 -0.04338 -0.04182 -0.04841 -0.02200 Today 0.0301 -0.02616 -0.01025 -0.00245 -0.00690 -0.03486 Volume Today Year 0.5390 0.03010 Lag1 0.0409 -0.02616 Lag2 -0.0434 -0.01025 Lag3 -0.0418 -0.00245 Lag4 -0.0484 -0.00690 Lag5 -0.0220 -0.03486 Volume 1.0000 0.01459 Today 0.0146 1.00000 As one would expect, the correlations between the lag variables and to- day’s returns are close to zero. In other words, there appears to be little correlation between today’s returns and previous days’ returns. The only substantial correlation is between Year and Volume. By plotting the data, which is ordered chronologically, we see that Volume is increasing over time. In other words, the average number of shares traded daily increased from 2001 to 2005. > attach(Smarket) > plot(Volume) 4.7.2 Logistic Regression Next, we will fit a logistic regression model in order to predict Direction using Lag1 through Lag5 and Volume. The glm() function can be used to fit glm() many types of generalized linear models, including logistic regression. The generalized linear model syntax of the glm() function is similar to that of lm(), except that we must pass in the argument family = binomial in order to tell R to run a logistic regression rather than some other type of generalized linear model. > glm.fits <- glm( Direction ∼ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + Volume , data = Smarket , family = binomial ) > summary(glm.fits) Call: glm(formula = Direction ∼ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + Volume , family = binomial , data = Smarket) Deviance Residuals: Min 1Q Median 3Q Max
- 183. 4.7 Lab: Classification Methods 173 -1.45 -1.20 1.07 1.15 1.33 Coefficients : Estimate Std. Error z value Pr(>|z|) (Intercept) -0.12600 0.24074 -0.52 0.60 Lag1 -0.07307 0.05017 -1.46 0.15 Lag2 -0.04230 0.05009 -0.84 0.40 Lag3 0.01109 0.04994 0.22 0.82 Lag4 0.00936 0.04997 0.19 0.85 Lag5 0.01031 0.04951 0.21 0.83 Volume 0.13544 0.15836 0.86 0.39 (Dispersion parameter for binomial family taken to be 1) Null deviance: 1731.2 on 1249 degrees of freedom Residual deviance: 1727.6 on 1243 degrees of freedom AIC: 1742 Number of Fisher Scoring iterations: 3 The smallest p-value here is associated with Lag1. The negative coefficient for this predictor suggests that if the market had a positive return yesterday, then it is less likely to go up today. However, at a value of 0.15, the p-value is still relatively large, and so there is no clear evidence of a real association between Lag1 and Direction. We use the coef() function in order to access just the coefficients for this fitted model. We can also use the summary() function to access particular aspects of the fitted model, such as the p-values for the coefficients. > coef(glm.fits) (Intercept) Lag1 Lag2 Lag3 Lag4 -0.12600 -0.07307 -0.04230 0.01109 0.00936 Lag5 Volume 0.01031 0.13544 > summary(glm.fits)$coef Estimate Std. Error z value Pr(>|z|) (Intercept) -0.12600 0.2407 -0.523 0.601 Lag1 -0.07307 0.0502 -1.457 0.145 Lag2 -0.04230 0.0501 -0.845 0.398 Lag3 0.01109 0.0499 0.222 0.824 Lag4 0.00936 0.0500 0.187 0.851 Lag5 0.01031 0.0495 0.208 0.835 Volume 0.13544 0.1584 0.855 0.392 > summary(glm.fits)$coef[, 4] (Intercept) Lag1 Lag2 Lag3 Lag4 0.601 0.145 0.398 0.824 0.851 Lag5 Volume 0.835 0.392 The predict() function can be used to predict the probability that the market will go up, given values of the predictors. The type = "response" option tells R to output probabilities of the form P(Y = 1|X), as opposed to other information such as the logit. If no data set is supplied to the
- 184. 174 4. Classification predict() function, then the probabilities are computed for the training data that was used to fit the logistic regression model. Here we have printed only the first ten probabilities. We know that these values correspond to the probability of the market going up, rather than down, because the contrasts() function indicates that R has created a dummy variable with a 1 for Up. > glm.probs <- predict(glm.fits , type = "response ") > glm.probs [1:10] 1 2 3 4 5 6 7 8 9 10 0.507 0.481 0.481 0.515 0.511 0.507 0.493 0.509 0.518 0.489 > contrasts (Direction) Up Down 0 Up 1 In order to make a prediction as to whether the market will go up or down on a particular day, we must convert these predicted probabilities into class labels, Up or Down. The following two commands create a vector of class predictions based on whether the predicted probability of a market increase is greater than or less than 0.5. > glm.pred <- rep("Down", 1250) > glm.pred[glm.probs > .5] = "Up" The first command creates a vector of 1,250 Down elements. The second line transforms to Up all of the elements for which the predicted probability of a market increase exceeds 0.5. Given these predictions, the table() function table() can be used to produce a confusion matrix in order to determine how many observations were correctly or incorrectly classified. > table(glm.pred , Direction) Direction glm.pred Down Up Down 145 141 Up 457 507 > (507 + 145) / 1250 [1] 0.5216 > mean(glm.pred == Direction) [1] 0.5216 The diagonal elements of the confusion matrix indicate correct predictions, while the off-diagonals represent incorrect predictions. Hence our model correctly predicted that the market would go up on 507 days and that it would go down on 145 days, for a total of 507 + 145 = 652 correct predictions. The mean() function can be used to compute the fraction of days for which the prediction was correct. In this case, logistic regression correctly predicted the movement of the market 52.2 % of the time. At first glance, it appears that the logistic regression model is working a little better than random guessing. However, this result is misleading because we trained and tested the model on the same set of 1, 250 ob- servations. In other words, 100% − 52.2% = 47.8%, is the training error
- 185. 4.7 Lab: Classification Methods 175 rate. As we have seen previously, the training error rate is often overly optimistic—it tends to underestimate the test error rate. In order to better assess the accuracy of the logistic regression model in this setting, we can fit the model using part of the data, and then examine how well it predicts the held out data. This will yield a more realistic error rate, in the sense that in practice we will be interested in our model’s performance not on the data that we used to fit the model, but rather on days in the future for which the market’s movements are unknown. To implement this strategy, we will first create a vector corresponding to the observations from 2001 through 2004. We will then use this vector to create a held out data set of observations from 2005. > train <- (Year < 2005) > Smarket .2005 <- Smarket [!train , ] > dim(Smarket .2005) [1] 252 9 > Direction .2005 <- Direction [! train] The object train is a vector of 1,250 elements, corresponding to the ob- servations in our data set. The elements of the vector that correspond to observations that occurred before 2005 are set to TRUE, whereas those that correspond to observations in 2005 are set to FALSE. The object train is a Boolean vector, since its elements are TRUE and FALSE. Boolean vectors can boolean be used to obtain a subset of the rows or columns of a matrix. For instance, the command Smarket[train, ] would pick out a submatrix of the stock market data set, corresponding only to the dates before 2005, since those are the ones for which the elements of train are TRUE. The ! symbol can be used to reverse all of the elements of a Boolean vector. That is, !train is a vector similar to train, except that the elements that are TRUE in train get swapped to FALSE in !train, and the elements that are FALSE in train get swapped to TRUE in !train. Therefore, Smarket[!train, ] yields a sub- matrix of the stock market data containing only the observations for which train is FALSE—that is, the observations with dates in 2005. The output above indicates that there are 252 such observations. We now fit a logistic regression model using only the subset of the obser- vations that correspond to dates before 2005, using the subset argument. We then obtain predicted probabilities of the stock market going up for each of the days in our test set—that is, for the days in 2005. > glm.fits <- glm( Direction ∼ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + Volume , data = Smarket , family = binomial , subset = train ) > glm.probs <- predict(glm.fits , Smarket .2005 , type = "response") Notice that we have trained and tested our model on two completely sep- arate data sets: training was performed using only the dates before 2005,
- 186. 176 4. Classification and testing was performed using only the dates in 2005. Finally, we com- pute the predictions for 2005 and compare them to the actual movements of the market over that time period. > glm.pred <- rep("Down", 252) > glm.pred[glm.probs > .5] <- "Up" > table(glm.pred , Direction .2005) Direction .2005 glm.pred Down Up Down 77 97 Up 34 44 > mean(glm.pred == Direction .2005) [1] 0.48 > mean(glm.pred != Direction .2005) [1] 0.52 The != notation means not equal to, and so the last command computes the test set error rate. The results are rather disappointing: the test error rate is 52 %, which is worse than random guessing! Of course this result is not all that surprising, given that one would not generally expect to be able to use previous days’ returns to predict future market performance. (After all, if it were possible to do so, then the authors of this book would be out striking it rich rather than writing a statistics textbook.) We recall that the logistic regression model had very underwhelming p- values associated with all of the predictors, and that the smallest p-value, though not very small, corresponded to Lag1. Perhaps by removing the variables that appear not to be helpful in predicting Direction, we can obtain a more effective model. After all, using predictors that have no relationship with the response tends to cause a deterioration in the test error rate (since such predictors cause an increase in variance without a corresponding decrease in bias), and so removing such predictors may in turn yield an improvement. Below we have refit the logistic regression using just Lag1 and Lag2, which seemed to have the highest predictive power in the original logistic regression model. > glm.fits <- glm(Direction ∼ Lag1 + Lag2 , data = Smarket , family = binomial , subset = train) > glm.probs <- predict(glm.fits , Smarket .2005 , type = "response") > glm.pred <- rep("Down", 252) > glm.pred[glm.probs > .5] <- "Up" > table(glm.pred , Direction .2005) Direction .2005 glm.pred Down Up Down 35 35 Up 76 106 > mean(glm.pred == Direction .2005) [1] 0.56 > 106 / (106 + 76) [1] 0.582
- 187. 4.7 Lab: Classification Methods 177 Now the results appear to be a little better: 56% of the daily movements have been correctly predicted. It is worth noting that in this case, a much simpler strategy of predicting that the market will increase every day will also be correct 56% of the time! Hence, in terms of overall error rate, the logistic regression method is no better than the naive approach. However, the confusion matrix shows that on days when logistic regression predicts an increase in the market, it has a 58% accuracy rate. This suggests a possible trading strategy of buying on days when the model predicts an in- creasing market, and avoiding trades on days when a decrease is predicted. Of course one would need to investigate more carefully whether this small improvement was real or just due to random chance. Suppose that we want to predict the returns associated with particular values of Lag1 and Lag2. In particular, we want to predict Direction on a day when Lag1 and Lag2 equal 1.2 and 1.1, respectively, and on a day when they equal 1.5 and −0.8. We do this using the predict() function. > predict(glm.fits , newdata = data.frame(Lag1 = c(1.2 , 1.5) , Lag2 = c(1.1 , -0.8)), type = "response" ) 1 2 0.4791 0.4961 4.7.3 Linear Discriminant Analysis Now we will perform LDA on the Smarket data. In R, we fit an LDA model using the lda() function, which is part of the MASS library. Notice that the lda() syntax for the lda() function is identical to that of lm(), and to that of glm() except for the absence of the family option. We fit the model using only the observations before 2005. > library(MASS) > lda.fit <- lda(Direction ∼ Lag1 + Lag2 , data = Smarket , subset = train) > lda.fit Call: lda(Direction ∼ Lag1 + Lag2 , data = Smarket , subset = train) Prior probabilities of groups: Down Up 0.492 0.508 Group means: Lag1 Lag2 Down 0.0428 0.0339 Up -0.0395 -0.0313
- 188. 178 4. Classification Coefficients of linear discriminants : LD1 Lag1 -0.642 Lag2 -0.514 > plot(lda.fit) The LDA output indicates that π̂1 = 0.492 and π̂2 = 0.508; in other words, 49.2 % of the training observations correspond to days during which the market went down. It also provides the group means; these are the average of each predictor within each class, and are used by LDA as estimates of µk. These suggest that there is a tendency for the previous 2 days’ returns to be negative on days when the market increases, and a tendency for the previous days’ returns to be positive on days when the market declines. The coefficients of linear discriminants output provides the linear combination of Lag1 and Lag2 that are used to form the LDA decision rule. In other words, these are the multipliers of the elements of X = x in (4.24). If −0.642 × Lag1 − 0.514 × Lag2 is large, then the LDA classifier will predict a market increase, and if it is small, then the LDA classifier will predict a market decline. The plot() function produces plots of the linear discriminants, obtained by computing −0.642 × Lag1 − 0.514 × Lag2 for each of the training obser- vations. The Up and Down observations are displayed separately. The predict() function returns a list with three elements. The first ele- ment, class, contains LDA’s predictions about the movement of the market. The second element, posterior, is a matrix whose kth column contains the posterior probability that the corresponding observation belongs to the kth class, computed from (4.15). Finally, x contains the linear discriminants, described earlier. > lda.pred <- predict(lda.fit , Smarket .2005) > names(lda.pred) [1] "class" "posterior" "x" As we observed in Section 4.5, the LDA and logistic regression predictions are almost identical. > lda.class <- lda.pred$class > table(lda.class , Direction .2005) Direction .2005 lda.pred Down Up Down 35 35 Up 76 106 > mean(lda.class == Direction .2005) [1] 0.56 Applying a 50 % threshold to the posterior probabilities allows us to recre- ate the predictions contained in lda.pred$class. > sum(lda.pred$posterior[, 1] >= .5) [1] 70
- 189. 4.7 Lab: Classification Methods 179 > sum(lda.pred$posterior[, 1] < .5) [1] 182 Notice that the posterior probability output by the model corresponds to the probability that the market will decrease: > lda.pred$posterior [1:20 , 1] > lda.class [1:20] If we wanted to use a posterior probability threshold other than 50 % in order to make predictions, then we could easily do so. For instance, suppose that we wish to predict a market decrease only if we are very certain that the market will indeed decrease on that day—say, if the posterior probability is at least 90 %. > sum(lda.pred$posterior[, 1] > .9) [1] 0 No days in 2005 meet that threshold! In fact, the greatest posterior prob- ability of decrease in all of 2005 was 52.02 %. 4.7.4 Quadratic Discriminant Analysis We will now fit a QDA model to the Smarket data. QDA is implemented in R using the qda() function, which is also part of the MASS library. The qda() syntax is identical to that of lda(). > qda.fit <- qda(Direction ∼ Lag1 + Lag2 , data = Smarket , subset = train) > qda.fit Call: qda(Direction ∼ Lag1 + Lag2 , data = Smarket , subset = train) Prior probabilities of groups: Down Up 0.492 0.508 Group means: Lag1 Lag2 Down 0.0428 0.0339 Up -0.0395 -0.0313 The output contains the group means. But it does not contain the coef- ficients of the linear discriminants, because the QDA classifier involves a quadratic, rather than a linear, function of the predictors. The predict() function works in exactly the same fashion as for LDA. > qda.class <- predict(qda.fit , Smarket .2005)$class > table(qda.class , Direction .2005) Direction .2005 qda.class Down Up Down 30 20
- 190. 180 4. Classification Up 81 121 > mean(qda.class == Direction .2005) [1] 0.599 Interestingly, the QDA predictions are accurate almost 60 % of the time, even though the 2005 data was not used to fit the model. This level of accu- racy is quite impressive for stock market data, which is known to be quite hard to model accurately. This suggests that the quadratic form assumed by QDA may capture the true relationship more accurately than the linear forms assumed by LDA and logistic regression. However, we recommend evaluating this method’s performance on a larger test set before betting that this approach will consistently beat the market! 4.7.5 Naive Bayes Next, we fit a naive Bayes model to the Smarket data. Naive Bayes is im- plemented in R using the naiveBayes() function, which is part of the e1071 naiveBayes() library. The syntax is identical to that of lda() and qda(). By default, this implementation of the naive Bayes classifier models each quantitative fea- ture using a Gaussian distribution. However, a kernel density method can also be used to estimate the distributions. > library(e1071) > nb.fit <- naiveBayes (Direction ∼ Lag1 + Lag2 , data = Smarket , subset = train) > nb.fit Naive Bayes Classifier for Discrete Predictors Call: naiveBayes .default(x = X, y = Y, laplace = laplace) A-priori probabilities : Y Down Up 0.492 0.508 Conditional probabilities : Lag1 Y [,1] [,2] Down 0.0428 1.23 Up -0.0395 1.23 Lag2 Y [,1] [,2] Down 0.0339 1.24 Up -0.0313 1.22 The output contains the estimated mean and standard deviation for each variable in each class. For example, the mean for Lag1 is 0.0428 for Direction=Down, and the standard deviation is 1.23. We can easily verify this:
- 191. 4.7 Lab: Classification Methods 181 > mean(Lag1[train ][ Direction[train] == "Down"]) [1] 0.0428 > sd(Lag1[train ][ Direction[train] == "Down"]) [1] 1.23 The predict() function is straightforward. > nb.class <- predict(nb.fit , Smarket .2005) > table(nb.class , Direction .2005) Direction .2005 nb.class Down Up Down 28 20 Up 83 121 > mean(nb.class == Direction .2005) [1] 0.591 Naive Bayes performs very well on this data, with accurate predictions over 59% of the time. This is slightly worse than QDA, but much better than LDA. The predict() function can also generate estimates of the probability that each observation belongs to a particular class. > nb.preds <- predict(nb.fit , Smarket .2005 , type = "raw") > nb.preds [1:5 , ] Down Up [1,] 0.487 0.513 [2,] 0.476 0.524 [3,] 0.465 0.535 [4,] 0.475 0.525 [5,] 0.490 0.510 4.7.6 K-Nearest Neighbors We will now perform KNN using the knn() function, which is part of the knn() class library. This function works rather differently from the other model- fitting functions that we have encountered thus far. Rather than a two-step approach in which we first fit the model and then we use the model to make predictions, knn() forms predictions using a single command. The function requires four inputs. 1. A matrix containing the predictors associated with the training data, labeled train.X below. 2. A matrix containing the predictors associated with the data for which we wish to make predictions, labeled test.X below. 3. A vector containing the class labels for the training observations, labeled train.Direction below. 4. A value for K, the number of nearest neighbors to be used by the classifier.
- 192. 182 4. Classification We use the cbind() function, short for column bind, to bind the Lag1 and cbind() Lag2 variables together into two matrices, one for the training set and the other for the test set. > library(class) > train.X <- cbind(Lag1 , Lag2)[train , ] > test.X <- cbind(Lag1 , Lag2)[!train , ] > train.Direction <- Direction[train] Now the knn() function can be used to predict the market’s movement for the dates in 2005. We set a random seed before we apply knn() because if several observations are tied as nearest neighbors, then R will randomly break the tie. Therefore, a seed must be set in order to ensure reproducibil- ity of results. > set.seed (1) > knn.pred <- knn(train.X, test.X, train.Direction , k = 1) > table(knn.pred , Direction .2005) Direction .2005 knn.pred Down Up Down 43 58 Up 68 83 > (83 + 43) / 252 [1] 0.5 The results using K = 1 are not very good, since only 50 % of the observa- tions are correctly predicted. Of course, it may be that K = 1 results in an overly flexible fit to the data. Below, we repeat the analysis using K = 3. > knn.pred <- knn(train.X, test.X, train.Direction , k = 3) > table(knn.pred , Direction .2005) Direction .2005 knn.pred Down Up Down 48 54 Up 63 87 > mean(knn.pred == Direction .2005) [1] 0.536 The results have improved slightly. But increasing K further turns out to provide no further improvements. It appears that for this data, QDA provides the best results of the methods that we have examined so far. KNN does not perform well on the Smarket data but it does often provide impressive results. As an example we will apply the KNN approach to the Caravan data set, which is part of the ISLR2 library. This data set includes 85 predictors that measure demographic characteristics for 5,822 individuals. The response variable is Purchase, which indicates whether or not a given individual purchases a caravan insurance policy. In this data set, only 6 % of people purchased caravan insurance. > dim(Caravan) [1] 5822 86
- 193. 4.7 Lab: Classification Methods 183 > attach(Caravan) > summary(Purchase) No Yes 5474 348 > 348 / 5822 [1] 0.0598 Because the KNN classifier predicts the class of a given test observation by identifying the observations that are nearest to it, the scale of the variables matters. Variables that are on a large scale will have a much larger effect on the distance between the observations, and hence on the KNN classifier, than variables that are on a small scale. For instance, imagine a data set that contains two variables, salary and age (measured in dollars and years, respectively). As far as KNN is concerned, a difference of $1,000 in salary is enormous compared to a difference of 50 years in age. Consequently, salary will drive the KNN classification results, and age will have almost no effect. This is contrary to our intuition that a salary difference of $1,000 is quite small compared to an age difference of 50 years. Furthermore, the importance of scale to the KNN classifier leads to another issue: if we measured salary in Japanese yen, or if we measured age in minutes, then we’d get quite different classification results from what we get if these two variables are measured in dollars and years. A good way to handle this problem is to standardize the data so that all standardize variables are given a mean of zero and a standard deviation of one. Then all variables will be on a comparable scale. The scale() function does just scale() this. In standardizing the data, we exclude column 86, because that is the qualitative Purchase variable. > standardized .X <- scale(Caravan[, -86]) > var(Caravan[, 1]) [1] 165 > var(Caravan[, 2]) [1] 0.165 > var( standardized .X[, 1]) [1] 1 > var( standardized .X[, 2]) [1] 1 Now every column of standardized.X has a standard deviation of one and a mean of zero. We now split the observations into a test set, containing the first 1,000 observations, and a training set, containing the remaining observations. We fit a KNN model on the training data using K = 1, and evaluate its performance on the test data. > test <- 1:1000 > train.X <- standardized .X[-test , ] > test.X <- standardized .X[test , ] > train.Y <- Purchase[-test]
- 194. 184 4. Classification > test.Y <- Purchase[test] > set.seed (1) > knn.pred <- knn(train.X, test.X, train.Y, k = 1) > mean(test.Y != knn.pred) [1] 0.118 > mean(test.Y != "No") [1] 0.059 The vector test is numeric, with values from 1 through 1, 000. Typing standardized.X[test, ] yields the submatrix of the data containing the observations whose indices range from 1 to 1, 000, whereas typing standardized.X[-test, ] yields the submatrix containing the observations whose indices do not range from 1 to 1, 000. The KNN error rate on the 1,000 test observations is just under 12 %. At first glance, this may ap- pear to be fairly good. However, since only 6 % of customers purchased insurance, we could get the error rate down to 6 % by always predicting No regardless of the values of the predictors! Suppose that there is some non-trivial cost to trying to sell insurance to a given individual. For instance, perhaps a salesperson must visit each potential customer. If the company tries to sell insurance to a random selection of customers, then the success rate will be only 6 %, which may be far too low given the costs involved. Instead, the company would like to try to sell insurance only to customers who are likely to buy it. So the overall error rate is not of interest. Instead, the fraction of individuals that are correctly predicted to buy insurance is of interest. It turns out that KNN with K = 1 does far better than random guessing among the customers that are predicted to buy insurance. Among 77 such customers, 9, or 11.7 %, actually do purchase insurance. This is double the rate that one would obtain from random guessing. > table(knn.pred , test.Y) test.Y knn.pred No Yes No 873 50 Yes 68 9 > 9 / (68 + 9) [1] 0.117 Using K = 3, the success rate increases to 19 %, and with K = 5 the rate is 26.7 %. This is over four times the rate that results from random guessing. It appears that KNN is finding some real patterns in a difficult data set! > knn.pred <- knn(train.X, test.X, train.Y, k = 3) > table(knn.pred , test.Y) test.Y knn.pred No Yes No 920 54 Yes 21 5 > 5 / 26 [1] 0.192 > knn.pred <- knn(train.X, test.X, train.Y, k = 5)
- 195. 4.7 Lab: Classification Methods 185 > table(knn.pred , test.Y) test.Y knn.pred No Yes No 930 55 Yes 11 4 > 4 / 15 [1] 0.267 However, while this strategy is cost-effective, it is worth noting that only 15 customers are predicted to purchase insurance using KNN with K = 5. In practice, the insurance company may wish to expend resources on convincing more than just 15 potential customers to buy insurance. As a comparison, we can also fit a logistic regression model to the data. If we use 0.5 as the predicted probability cut-off for the classifier, then we have a problem: only seven of the test observations are predicted to purchase insurance. Even worse, we are wrong about all of these! However, we are not required to use a cut-off of 0.5. If we instead predict a purchase any time the predicted probability of purchase exceeds 0.25, we get much better results: we predict that 33 people will purchase insurance, and we are correct for about 33 % of these people. This is over five times better than random guessing! > glm.fits <- glm(Purchase ∼ ., data = Caravan , family = binomial , subset = -test) Warning message: glm.fits: fitted probabilities numerically 0 or 1 occurred > glm.probs <- predict(glm.fits , Caravan[test , ], type = "response ") > glm.pred <- rep("No", 1000) > glm.pred[glm.probs > .5] <- "Yes" > table(glm.pred , test.Y) test.Y glm.pred No Yes No 934 59 Yes 7 0 > glm.pred <- rep("No", 1000) > glm.pred[glm.probs > .25] <- "Yes" > table(glm.pred , test.Y) test.Y glm.pred No Yes No 919 48 Yes 22 11 > 11 / (22 + 11) [1] 0.333 4.7.7 Poisson Regression Finally, we fit a Poisson regression model to the Bikeshare data set, which measures the number of bike rentals (bikers) per hour in Washington, DC. The data can be found in the ISLR2 library.
- 196. 186 4. Classification > attach(Bikeshare) > dim(Bikeshare) [1] 8645 15 > names(Bikeshare) [1] "season" "mnth" "day" "hr" [5] "holiday" "weekday" "workingday " "weathersit" [9] "temp" "atemp" "hum" "windspeed" [13] "casual" "registered " "bikers" We begin by fitting a least squares linear regression model to the data. > mod.lm <- lm( bikers ∼ mnth + hr + workingday + temp + weathersit , data = Bikeshare ) > summary(mod.lm) Call: lm(formula = bikers ∼ mnth + hr + workingday + temp + weathersit , data = Bikeshare) Residuals: Min 1Q Median 3Q Max -299.00 -45.70 -6.23 41.08 425.29 Coefficients : Estimate Std. Error t value Pr(>|t|) (Intercept) -68.632 5.307 -12.932 < 2e -16 *** mnthFeb 6.845 4.287 1.597 0.110398 mnthMarch 16.551 4.301 3.848 0.000120 *** mnthApril 41.425 4.972 8.331 < 2e -16 *** mnthMay 72.557 5.641 12.862 < 2e -16 *** Due to space constraints, we truncate the output of summary(mod.lm). In mod.lm, the first level of hr (0) and mnth (Jan) are treated as the baseline values, and so no coefficient estimates are provided for them: implicitly, their coefficient estimates are zero, and all other levels are measured relative to these baselines. For example, the Feb coefficient of 6.845 signifies that, holding all other variables constant, there are on average about 7 more riders in February than in January. Similarly there are about 16.5 more riders in March than in January. The results seen in Section 4.6.1 used a slightly different coding of the variables hr and mnth, as follows: > contrasts (Bikeshare$hr) = contr.sum (24) > contrasts (Bikeshare$mnth) = contr.sum (12) > mod.lm2 <- lm( bikers ∼ mnth + hr + workingday + temp + weathersit , data = Bikeshare ) > summary(mod.lm2) Call: lm(formula = bikers ∼ mnth + hr + workingday + temp + weathersit , data = Bikeshare)
- 197. 4.7 Lab: Classification Methods 187 Residuals: Min 1Q Median 3Q Max -299.00 -45.70 -6.23 41.08 425.29 Coefficients : Estimate Std. Error t value Pr(>|t|) (Intercept) 73.597 5.132 14.340 < 2e -16 *** mnth1 -46.087 4.086 -11.281 < 2e-16 *** mnth2 -39.242 3.539 -11.088 < 2e-16 *** mnth3 -29.536 3.155 -9.361 < 2e -16 *** mnth4 -4.662 2.741 -1.701 0.08895 . What is the difference between the two codings? In mod.lm2, a coefficient estimate is reported for all but the last level of hr and mnth. Importantly, in mod.lm2, the coefficient estimate for the last level of mnth is not zero: instead, it equals the negative of the sum of the coefficient estimates for all of the other levels. Similarly, in mod.lm2, the coefficient estimate for the last level of hr is the negative of the sum of the coefficient estimates for all of the other levels. This means that the coefficients of hr and mnth in mod.lm2 will always sum to zero, and can be interpreted as the difference from the mean level. For example, the coefficient for January of −46.087 indicates that, holding all other variables constant, there are typically 46 fewer riders in January relative to the yearly average. It is important to realize that the choice of coding really does not matter, provided that we interpret the model output correctly in light of the coding used. For example, we see that the predictions from the linear model are the same regardless of coding: > sum (( predict(mod.lm) - predict(mod.lm2))^2) [1] 1.426e-18 The sum of squared differences is zero. We can also see this using the all.equal() function: all.equal() > all.equal(predict(mod.lm), predict(mod.lm2)) To reproduce the left-hand side of Figure 4.13, we must first obtain the coefficient estimates associated with mnth. The coefficients for January through November can be obtained directly from the mod.lm2 object. The coefficient for December must be explicitly computed as the negative sum of all the other months. > coef.months <- c(coef(mod.lm2)[2:12] , -sum(coef(mod.lm2)[2:12])) To make the plot, we manually label the x-axis with the names of the months. > plot(coef.months , xlab = "Month", ylab = " Coefficient ", xaxt = "n", col = "blue", pch = 19, type = "o")
- 198. 188 4. Classification > axis(side = 1, at = 1:12 , labels = c("J", "F", "M", "A", "M", "J", "J", "A", "S", "O", "N", "D")) Reproducing the right-hand side of Figure 4.13 follows a similar process. > coef.hours <- c(coef(mod.lm2)[13:35] , -sum(coef(mod.lm2)[13:35])) > plot(coef.hours , xlab = "Hour", ylab = " Coefficient ", col = "blue", pch = 19, type = "o") Now, we consider instead fitting a Poisson regression model to the Bikeshare data. Very little changes, except that we now use the function glm() with the argument family = poisson to specify that we wish to fit a Poisson regression model: > mod.pois <- glm( bikers ∼ mnth + hr + workingday + temp + weathersit , data = Bikeshare , family = poisson ) > summary(mod.pois) Call: glm(formula = bikers ∼ mnth + hr + workingday + temp + weathersit , family = poisson , data = Bikeshare) Deviance Residuals: Min 1Q Median 3Q Max -20.7574 -3.3441 -0.6549 2.6999 21.9628 Coefficients : Estimate Std. Error z value Pr(>|z|) (Intercept) 4.118245 0.006021 683.964 < 2e -16 *** mnth1 -0.670170 0.005907 -113.445 < 2e-16 *** mnth2 -0.444124 0.004860 -91.379 < 2e -16 *** mnth3 -0.293733 0.004144 -70.886 < 2e -16 *** mnth4 0.021523 0.003125 6.888 5.66e-12 *** We can plot the coefficients associated with mnth and hr, in order to repro- duce Figure 4.15: > coef.mnth <- c(coef(mod.pois)[2:12] , -sum(coef(mod.pois)[2:12])) > plot(coef.mnth , xlab = "Month", ylab = " Coefficient ", xaxt = "n", col = "blue", pch = 19, type = "o") > axis(side = 1, at = 1:12 , labels = c("J", "F", "M", "A", "M", "J", "J", "A", "S", "O", "N", "D")) > coef.hours <- c(coef(mod.pois)[13:35] , -sum(coef(mod.pois)[13:35])) > plot(coef.hours , xlab = "Hour", ylab = " Coefficient ", col = "blue", pch = 19, type = "o") We can once again use the predict() function to obtain the fitted values (predictions) from this Poisson regression model. However, we must use the argument type = "response" to specify that we want R to output exp(β̂0 + β̂1X1 +. . .+β̂pXp) rather than β̂0 +β̂1X1 +. . .+β̂pXp, which it will output by default.
- 199. 4.8 Exercises 189 > plot(predict(mod.lm2), predict(mod.pois , type = "response")) > abline (0, 1, col = 2, lwd = 3) The predictions from the Poisson regression model are correlated with those from the linear model; however, the former are non-negative. As a result the Poisson regression predictions tend to be larger than those from the linear model for either very low or very high levels of ridership. In this section, we used the glm() function with the argument family = poisson in order to perform Poisson regression. Earlier in this lab we used the glm() function with family = binomial to perform logistic regression. Other choices for the family argument can be used to fit other types of GLMs. For instance, family = Gamma fits a gamma regression model. 4.8 Exercises Conceptual 1. Using a little bit of algebra, prove that (4.2) is equivalent to (4.3). In other words, the logistic function representation and logit represen- tation for the logistic regression model are equivalent. 2. It was stated in the text that classifying an observation to the class for which (4.17) is largest is equivalent to classifying an observation to the class for which (4.18) is largest. Prove that this is the case. In other words, under the assumption that the observations in the kth class are drawn from a N(µk, σ2 ) distribution, the Bayes classifier assigns an observation to the class for which the discriminant function is maximized. 3. This problem relates to the QDA model, in which the observations within each class are drawn from a normal distribution with a class- specific mean vector and a class specific covariance matrix. We con- sider the simple case where p = 1; i.e. there is only one feature. Suppose that we have K classes, and that if an observation belongs to the kth class then X comes from a one-dimensional normal dis- tribution, X ∼ N(µk, σ2 k). Recall that the density function for the one-dimensional normal distribution is given in (4.16). Prove that in this case, the Bayes classifier is not linear. Argue that it is in fact quadratic. Hint: For this problem, you should follow the arguments laid out in Section 4.4.1, but without making the assumption that σ2 1 = . . . = σ2 K. 4. When the number of features p is large, there tends to be a deteri- oration in the performance of KNN and other local approaches that
- 200. 190 4. Classification perform prediction using only observations that are near the test ob- servation for which a prediction must be made. This phenomenon is known as the curse of dimensionality, and it ties into the fact that curse of di- mensionality non-parametric approaches often perform poorly when p is large. We will now investigate this curse. (a) Suppose that we have a set of observations, each with measure- ments on p = 1 feature, X. We assume that X is uniformly (evenly) distributed on [0, 1]. Associated with each observation is a response value. Suppose that we wish to predict a test obser- vation’s response using only observations that are within 10 % of the range of X closest to that test observation. For instance, in order to predict the response for a test observation with X = 0.6, we will use observations in the range [0.55, 0.65]. On average, what fraction of the available observations will we use to make the prediction? (b) Now suppose that we have a set of observations, each with measurements on p = 2 features, X1 and X2. We assume that (X1, X2) are uniformly distributed on [0, 1] × [0, 1]. We wish to predict a test observation’s response using only observations that are within 10 % of the range of X1 and within 10 % of the range of X2 closest to that test observation. For instance, in order to predict the response for a test observation with X1 = 0.6 and X2 = 0.35, we will use observations in the range [0.55, 0.65] for X1 and in the range [0.3, 0.4] for X2. On average, what fraction of the available observations will we use to make the prediction? (c) Now suppose that we have a set of observations on p = 100 fea- tures. Again the observations are uniformly distributed on each feature, and again each feature ranges in value from 0 to 1. We wish to predict a test observation’s response using observations within the 10 % of each feature’s range that is closest to that test observation. What fraction of the available observations will we use to make the prediction? (d) Using your answers to parts (a)–(c), argue that a drawback of KNN when p is large is that there are very few training obser- vations “near” any given test observation. (e) Now suppose that we wish to make a prediction for a test obser- vation by creating a p-dimensional hypercube centered around the test observation that contains, on average, 10 % of the train- ing observations. For p = 1, 2, and 100, what is the length of each side of the hypercube? Comment on your answer.
- 201. 4.8 Exercises 191 Note: A hypercube is a generalization of a cube to an arbitrary number of dimensions. When p = 1, a hypercube is simply a line segment, when p = 2 it is a square, and when p = 100 it is a 100-dimensional cube. 5. We now examine the differences between LDA and QDA. (a) If the Bayes decision boundary is linear, do we expect LDA or QDA to perform better on the training set? On the test set? (b) If the Bayes decision boundary is non-linear, do we expect LDA or QDA to perform better on the training set? On the test set? (c) In general, as the sample size n increases, do we expect the test prediction accuracy of QDA relative to LDA to improve, decline, or be unchanged? Why? (d) True or False: Even if the Bayes decision boundary for a given problem is linear, we will probably achieve a superior test er- ror rate using QDA rather than LDA because QDA is flexible enough to model a linear decision boundary. Justify your answer. 6. Suppose we collect data for a group of students in a statistics class with variables X1 = hours studied, X2 = undergrad GPA, and Y = receive an A. We fit a logistic regression and produce estimated coefficient, β̂0 = −6, β̂1 = 0.05, β̂2 = 1. (a) Estimate the probability that a student who studies for 40 h and has an undergrad GPA of 3.5 gets an A in the class. (b) How many hours would the student in part (a) need to study to have a 50 % chance of getting an A in the class? 7. Suppose that we wish to predict whether a given stock will issue a dividend this year (“Yes” or “No”) based on X, last year’s percent profit. We examine a large number of companies and discover that the mean value of X for companies that issued a dividend was X̄ = 10, while the mean for those that didn’t was X̄ = 0. In addition, the variance of X for these two sets of companies was σ̂2 = 36. Finally, 80 % of companies issued dividends. Assuming that X follows a nor- mal distribution, predict the probability that a company will issue a dividend this year given that its percentage profit was X = 4 last year. Hint: Recall that the density function for a normal random variable is f(x) = 1 √ 2πσ2 e−(x−µ)2 /2σ2 . You will need to use Bayes’ theorem. 8. Suppose that we take a data set, divide it into equally-sized training and test sets, and then try out two different classification procedures.
- 202. 192 4. Classification First we use logistic regression and get an error rate of 20 % on the training data and 30 % on the test data. Next we use 1-nearest neigh- bors (i.e. K = 1) and get an average error rate (averaged over both test and training data sets) of 18 %. Based on these results, which method should we prefer to use for classification of new observations? Why? 9. This problem has to do with odds. (a) On average, what fraction of people with an odds of 0.37 of defaulting on their credit card payment will in fact default? (b) Suppose that an individual has a 16 % chance of defaulting on her credit card payment. What are the odds that she will de- fault? 10. Equation 4.32 derived an expression for log 1 Pr(Y =k|X=x) Pr(Y =K|X=x) 2 in the setting where p > 1, so that the mean for the kth class, µk, is a p- dimensional vector, and the shared covariance Σ is a p × p matrix. However, in the setting with p = 1, (4.32) takes a simpler form, since the means µ1, . . . , µK and the variance σ2 are scalars. In this simpler setting, repeat the calculation in (4.32), and provide expressions for ak and bkj in terms of πk, πK, µk, µK, and σ2 . 11. Work out the detailed forms of ak, bkj, and bkjl in (4.33). Your answer should involve πk, πK, µk, µK, Σk, and ΣK. 12. Suppose that you wish to classify an observation X ∈ R into apples and oranges. You fit a logistic regression model and find that 6 Pr(Y = orange|X = x) = exp(β̂0 + β̂1x) 1 + exp(β̂0 + β̂1x) . Your friend fits a logistic regression model to the same data using the softmax formulation in (4.13), and finds that 6 Pr(Y = orange|X = x) = exp(α̂orange0 + α̂orange1x) exp(α̂orange0 + α̂orange1x) + exp(α̂apple0 + α̂apple1x) . (a) What is the log odds of orange versus apple in your model? (b) What is the log odds of orange versus apple in your friend’s model? (c) Suppose that in your model, β̂0 = 2 and β̂1 = −1. What are the coefficient estimates in your friend’s model? Be as specific as possible.
- 203. 4.8 Exercises 193 (d) Now suppose that you and your friend fit the same two models on a different data set. This time, your friend gets the coefficient estimates α̂orange0 = 1.2, α̂orange1 = −2, α̂apple0 = 3, α̂apple1 = 0.6. What are the coefficient estimates in your model? (e) Finally, suppose you apply both models from (d) to a data set with 2,000 test observations. What fraction of the time do you expect the predicted class labels from your model to agree with those from your friend’s model? Explain your answer. Applied 13. This question should be answered using the Weekly data set, which is part of the ISLR2 package. This data is similar in nature to the Smarket data from this chapter’s lab, except that it contains 1, 089 weekly returns for 21 years, from the beginning of 1990 to the end of 2010. (a) Produce some numerical and graphical summaries of the Weekly data. Do there appear to be any patterns? (b) Use the full data set to perform a logistic regression with Direction as the response and the five lag variables plus Volume as predictors. Use the summary function to print the results. Do any of the predictors appear to be statistically significant? If so, which ones? (c) Compute the confusion matrix and overall fraction of correct predictions. Explain what the confusion matrix is telling you about the types of mistakes made by logistic regression. (d) Now fit the logistic regression model using a training data period from 1990 to 2008, with Lag2 as the only predictor. Compute the confusion matrix and the overall fraction of correct predictions for the held out data (that is, the data from 2009 and 2010). (e) Repeat (d) using LDA. (f) Repeat (d) using QDA. (g) Repeat (d) using KNN with K = 1. (h) Repeat (d) using naive Bayes. (i) Which of these methods appears to provide the best results on this data? (j) Experiment with different combinations of predictors, includ- ing possible transformations and interactions, for each of the methods. Report the variables, method, and associated confu- sion matrix that appears to provide the best results on the held out data. Note that you should also experiment with values for K in the KNN classifier.
- 204. 194 4. Classification 14. In this problem, you will develop a model to predict whether a given car gets high or low gas mileage based on the Auto data set. (a) Create a binary variable, mpg01, that contains a 1 if mpg contains a value above its median, and a 0 if mpg contains a value below its median. You can compute the median using the median() function. Note you may find it helpful to use the data.frame() function to create a single data set containing both mpg01 and the other Auto variables. (b) Explore the data graphically in order to investigate the associ- ation between mpg01 and the other features. Which of the other features seem most likely to be useful in predicting mpg01? Scat- terplots and boxplots may be useful tools to answer this ques- tion. Describe your findings. (c) Split the data into a training set and a test set. (d) Perform LDA on the training data in order to predict mpg01 using the variables that seemed most associated with mpg01 in (b). What is the test error of the model obtained? (e) Perform QDA on the training data in order to predict mpg01 using the variables that seemed most associated with mpg01 in (b). What is the test error of the model obtained? (f) Perform logistic regression on the training data in order to pre- dict mpg01 using the variables that seemed most associated with mpg01 in (b). What is the test error of the model obtained? (g) Perform naive Bayes on the training data in order to predict mpg01 using the variables that seemed most associated with mpg01 in (b). What is the test error of the model obtained? (h) Perform KNN on the training data, with several values of K, in order to predict mpg01. Use only the variables that seemed most associated with mpg01 in (b). What test errors do you obtain? Which value of K seems to perform the best on this data set? 15. This problem involves writing functions. (a) Write a function, Power(), that prints out the result of raising 2 to the 3rd power. In other words, your function should compute 23 and print out the results. Hint: Recall that x^a raises x to the power a. Use the print() function to output the result. (b) Create a new function, Power2(), that allows you to pass any two numbers, x and a, and prints out the value of x^a. You can do this by beginning your function with the line > Power2 <- function (x, a) {
- 205. 4.8 Exercises 195 You should be able to call your function by entering, for instance, > Power2 (3, 8) on the command line. This should output the value of 38 , namely, 6, 561. (c) Using the Power2() function that you just wrote, compute 103 , 817 , and 1313 . (d) Now create a new function, Power3(), that actually returns the result x^a as an R object, rather than simply printing it to the screen. That is, if you store the value x^a in an object called result within your function, then you can simply return() this return() result, using the following line: return(result) The line above should be the last line in your function, before the } symbol. (e) Now using the Power3() function, create a plot of f(x) = x2 . The x-axis should display a range of integers from 1 to 10, and the y-axis should display x2 . Label the axes appropriately, and use an appropriate title for the figure. Consider displaying either the x-axis, the y-axis, or both on the log-scale. You can do this by using log = "x", log = "y", or log = "xy" as arguments to the plot() function. (f) Create a function, PlotPower(), that allows you to create a plot of x against x^a for a fixed a and for a range of values of x. For instance, if you call > PlotPower (1:10 , 3) then a plot should be created with an x-axis taking on values 1, 2, . . . , 10, and a y-axis taking on values 13 , 23 , . . . , 103 . 16. Using the Boston data set, fit classification models in order to predict whether a given census tract has a crime rate above or below the me- dian. Explore logistic regression, LDA, naive Bayes, and KNN models using various subsets of the predictors. Describe your findings. Hint: You will have to create the response variable yourself, using the variables that are contained in the Boston data set.
- 206. 5 Resampling Methods Resampling methods are an indispensable tool in modern statistics. They involve repeatedly drawing samples from a training set and refitting a model of interest on each sample in order to obtain additional information about the fitted model. For example, in order to estimate the variability of a linear regression fit, we can repeatedly draw different samples from the training data, fit a linear regression to each new sample, and then examine the extent to which the resulting fits differ. Such an approach may allow us to obtain information that would not be available from fitting the model only once using the original training sample. Resampling approaches can be computationally expensive, because they involve fitting the same statistical method multiple times using different subsets of the training data. However, due to recent advances in computing power, the computational requirements of resampling methods generally are not prohibitive. In this chapter, we discuss two of the most commonly used resampling methods, cross-validation and the bootstrap. Both methods are important tools in the practical application of many statistical learning procedures. For example, cross-validation can be used to estimate the test error associated with a given statistical learning method in order to evaluate its performance, or to select the appropriate level of flexibility. The process of evaluating a model’s performance is known as model assessment, whereas model assessment the process of selecting the proper level of flexibility for a model is known as model selection. The bootstrap is used in several contexts, most commonly model selection to provide a measure of accuracy of a parameter estimate or of a given statistical learning method. © Springer Science+Business Media, LLC, part of Springer Nature 2021 G. James et al., An Introduction to Statistical Learning, Springer Texts in Statistics, 197 https://doi.org/10.1007/978-1-0716-1418-1_5
- 207. 198 5. Resampling Methods 5.1 Cross-Validation In Chapter 2 we discuss the distinction between the test error rate and the training error rate. The test error is the average error that results from using a statistical learning method to predict the response on a new observation— that is, a measurement that was not used in training the method. Given a data set, the use of a particular statistical learning method is warranted if it results in a low test error. The test error can be easily calculated if a designated test set is available. Unfortunately, this is usually not the case. In contrast, the training error can be easily calculated by applying the statistical learning method to the observations used in its training. But as we saw in Chapter 2, the training error rate often is quite different from the test error rate, and in particular the former can dramatically underestimate the latter. In the absence of a very large designated test set that can be used to directly estimate the test error rate, a number of techniques can be used to estimate this quantity using the available training data. Some methods make a mathematical adjustment to the training error rate in order to estimate the test error rate. Such approaches are discussed in Chapter 6. In this section, we instead consider a class of methods that estimate the test error rate by holding out a subset of the training observations from the fitting process, and then applying the statistical learning method to those held out observations. In Sections 5.1.1–5.1.4, for simplicity we assume that we are interested in performing regression with a quantitative response. In Section 5.1.5 we consider the case of classification with a qualitative response. As we will see, the key concepts remain the same regardless of whether the response is quantitative or qualitative. 5.1.1 The Validation Set Approach Suppose that we would like to estimate the test error associated with fit- ting a particular statistical learning method on a set of observations. The validation set approach, displayed in Figure 5.1, is a very simple strategy validation set approach for this task. It involves randomly dividing the available set of observa- tions into two parts, a training set and a validation set or hold-out set. The validation set hold-out set model is fit on the training set, and the fitted model is used to predict the responses for the observations in the validation set. The resulting validation set error rate—typically assessed using MSE in the case of a quantitative response—provides an estimate of the test error rate. We illustrate the validation set approach on the Auto data set. Recall from Chapter 3 that there appears to be a non-linear relationship between mpg and horsepower, and that a model that predicts mpg using horsepower and horsepower2 gives better results than a model that uses only a linear term. It is natural to wonder whether a cubic or higher-order fit might provide
- 208. 5.1 Cross-Validation 199 !"#"$""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""%" &""##""!$"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""'!" FIGURE 5.1. A schematic display of the validation set approach. A set of n observations are randomly split into a training set (shown in blue, containing observations 7, 22, and 13, among others) and a validation set (shown in beige, and containing observation 91, among others). The statistical learning method is fit on the training set, and its performance is evaluated on the validation set. even better results. We answer this question in Chapter 3 by looking at the p-values associated with a cubic term and higher-order polynomial terms in a linear regression. But we could also answer this question using the validation method. We randomly split the 392 observations into two sets, a training set containing 196 of the data points, and a validation set containing the remaining 196 observations. The validation set error rates that result from fitting various regression models on the training sample and evaluating their performance on the validation sample, using MSE as a measure of validation set error, are shown in the left-hand panel of Figure 5.2. The validation set MSE for the quadratic fit is considerably smaller than for the linear fit. However, the validation set MSE for the cubic fit is actually slightly larger than for the quadratic fit. This implies that including a cubic term in the regression does not lead to better prediction than simply using a quadratic term. Recall that in order to create the left-hand panel of Figure 5.2, we ran- domly divided the data set into two parts, a training set and a validation set. If we repeat the process of randomly splitting the sample set into two parts, we will get a somewhat different estimate for the test MSE. As an illustration, the right-hand panel of Figure 5.2 displays ten different vali- dation set MSE curves from the Auto data set, produced using ten different random splits of the observations into training and validation sets. All ten curves indicate that the model with a quadratic term has a dramatically smaller validation set MSE than the model with only a linear term. Fur- thermore, all ten curves indicate that there is not much benefit in including cubic or higher-order polynomial terms in the model. But it is worth noting that each of the ten curves results in a different test MSE estimate for each of the ten regression models considered. And there is no consensus among the curves as to which model results in the smallest validation set MSE. Based on the variability among these curves, all that we can conclude with any confidence is that the linear fit is not adequate for this data. The validation set approach is conceptually simple and is easy to imple- ment. But it has two potential drawbacks:
- 209. 200 5. Resampling Methods 2 4 6 8 10 16 18 20 22 24 26 28 Degree of Polynomial Mean Squared Error 2 4 6 8 10 16 18 20 22 24 26 28 Degree of Polynomial Mean Squared Error FIGURE 5.2. The validation set approach was used on the Auto data set in order to estimate the test error that results from predicting mpg using polynomial functions of horsepower. Left: Validation error estimates for a single split into training and validation data sets. Right: The validation method was repeated ten times, each time using a different random split of the observations into a training set and a validation set. This illustrates the variability in the estimated test MSE that results from this approach. 1. As is shown in the right-hand panel of Figure 5.2, the validation esti- mate of the test error rate can be highly variable, depending on pre- cisely which observations are included in the training set and which observations are included in the validation set. 2. In the validation approach, only a subset of the observations—those that are included in the training set rather than in the validation set—are used to fit the model. Since statistical methods tend to per- form worse when trained on fewer observations, this suggests that the validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set. In the coming subsections, we will present cross-validation, a refinement of the validation set approach that addresses these two issues. 5.1.2 Leave-One-Out Cross-Validation Leave-one-out cross-validation (LOOCV) is closely related to the validation leave-one- out cross- validation set approach of Section 5.1.1, but it attempts to address that method’s drawbacks. Like the validation set approach, LOOCV involves splitting the set of observations into two parts. However, instead of creating two subsets of comparable size, a single observation (x1, y1) is used for the validation set, and the remaining observations {(x2, y2), . . . , (xn, yn)} make up the training set. The statistical learning method is fit on the n − 1 training observations, and a prediction ŷ1 is made for the excluded observation,
- 210. 5.1 Cross-Validation 201 !"#"$"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""%" !"#"$"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""%" !"#"$"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""%" !"#"$"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""%" !"#"$"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""%" &" &" &" FIGURE 5.3. A schematic display of LOOCV. A set of n data points is repeat- edly split into a training set (shown in blue) containing all but one observation, and a validation set that contains only that observation (shown in beige). The test error is then estimated by averaging the n resulting MSE’s. The first training set contains all but observation 1, the second training set contains all but observation 2, and so forth. using its value x1. Since (x1, y1) was not used in the fitting process, MSE1 = (y1 − ŷ1)2 provides an approximately unbiased estimate for the test error. But even though MSE1 is unbiased for the test error, it is a poor estimate because it is highly variable, since it is based upon a single observation (x1, y1). We can repeat the procedure by selecting (x2, y2) for the validation data, training the statistical learning procedure on the n − 1 observations {(x1, y1), (x3, y3), . . . , (xn, yn)}, and computing MSE2 = (y2−ŷ2)2 . Repeat- ing this approach n times produces n squared errors, MSE1, . . . , MSEn. The LOOCV estimate for the test MSE is the average of these n test error estimates: CV(n) = 1 n n 0 i=1 MSEi. (5.1) A schematic of the LOOCV approach is illustrated in Figure 5.3. LOOCV has a couple of major advantages over the validation set ap- proach. First, it has far less bias. In LOOCV, we repeatedly fit the sta- tistical learning method using training sets that contain n − 1 observa- tions, almost as many as are in the entire data set. This is in contrast to the validation set approach, in which the training set is typically around half the size of the original data set. Consequently, the LOOCV approach tends not to overestimate the test error rate as much as the validation set approach does. Second, in contrast to the validation approach which will yield different results when applied repeatedly due to randomness in the training/validation set splits, performing LOOCV multiple times will
- 211. 202 5. Resampling Methods 2 4 6 8 10 16 18 20 22 24 26 28 LOOCV Degree of Polynomial Mean Squared Error 2 4 6 8 10 16 18 20 22 24 26 28 10−fold CV Degree of Polynomial Mean Squared Error FIGURE 5.4. Cross-validation was used on the Auto data set in order to es- timate the test error that results from predicting mpg using polynomial functions of horsepower. Left: The LOOCV error curve. Right: 10-fold CV was run nine separate times, each with a different random split of the data into ten parts. The figure shows the nine slightly different CV error curves. always yield the same results: there is no randomness in the training/vali- dation set splits. We used LOOCV on the Auto data set in order to obtain an estimate of the test set MSE that results from fitting a linear regression model to predict mpg using polynomial functions of horsepower. The results are shown in the left-hand panel of Figure 5.4. LOOCV has the potential to be expensive to implement, since the model has to be fit n times. This can be very time consuming if n is large, and if each individual model is slow to fit. With least squares linear or polynomial regression, an amazing shortcut makes the cost of LOOCV the same as that of a single model fit! The following formula holds: CV(n) = 1 n n 0 i=1 * yi − ŷi 1 − hi +2 , (5.2) where ŷi is the ith fitted value from the original least squares fit, and hi is the leverage defined in (3.37) on page 99.1 This is like the ordinary MSE, except the ith residual is divided by 1 − hi. The leverage lies between 1/n and 1, and reflects the amount that an observation influences its own fit. Hence the residuals for high-leverage points are inflated in this formula by exactly the right amount for this equality to hold. LOOCV is a very general method, and can be used with any kind of predictive modeling. For example we could use it with logistic regression 1In the case of multiple linear regression, the leverage takes a slightly more compli- cated form than (3.37), but (5.2) still holds.
- 212. 5.1 Cross-Validation 203 !"#"$""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""%" !!"&'"(""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""")&" !!"&'"(""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""")&" !!"&'"(""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""")&" !!"&'"(""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""")&" !!"&'"(""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""")&" !!"&'"(""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""")&" FIGURE 5.5. A schematic display of 5-fold CV. A set of n observations is randomly split into five non-overlapping groups. Each of these fifths acts as a validation set (shown in beige), and the remainder as a training set (shown in blue). The test error is estimated by averaging the five resulting MSE estimates. or linear discriminant analysis, or any of the methods discussed in later chapters. The magic formula (5.2) does not hold in general, in which case the model has to be refit n times. 5.1.3 k-Fold Cross-Validation An alternative to LOOCV is k-fold CV. This approach involves randomly k-fold CV dividing the set of observations into k groups, or folds, of approximately equal size. The first fold is treated as a validation set, and the method is fit on the remaining k − 1 folds. The mean squared error, MSE1, is then computed on the observations in the held-out fold. This procedure is repeated k times; each time, a different group of observations is treated as a validation set. This process results in k estimates of the test error, MSE1, MSE2, . . . , MSEk. The k-fold CV estimate is computed by averaging these values, CV(k) = 1 k k 0 i=1 MSEi. (5.3) Figure 5.5 illustrates the k-fold CV approach. It is not hard to see that LOOCV is a special case of k-fold CV in which k is set to equal n. In practice, one typically performs k-fold CV using k = 5 or k = 10. What is the advantage of using k = 5 or k = 10 rather than k = n? The most obvious advantage is computational. LOOCV requires fitting the statistical learning method n times. This has the potential to be computationally expensive (except for linear models fit by least squares, in which case formula (5.2) can be used). But cross-validation is a very general approach that can be applied to almost any statistical learning method. Some statistical learning methods have computationally intensive
- 213. 204 5. Resampling Methods 2 5 10 20 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Flexibility Mean Squared Error 2 5 10 20 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Flexibility Mean Squared Error 2 5 10 20 0 5 10 15 20 Flexibility Mean Squared Error FIGURE 5.6. True and estimated test MSE for the simulated data sets in Fig- ures 2.9 ( left), 2.10 ( center), and 2.11 ( right). The true test MSE is shown in blue, the LOOCV estimate is shown as a black dashed line, and the 10-fold CV estimate is shown in orange. The crosses indicate the minimum of each of the MSE curves. fitting procedures, and so performing LOOCV may pose computational problems, especially if n is extremely large. In contrast, performing 10-fold CV requires fitting the learning procedure only ten times, which may be much more feasible. As we see in Section 5.1.4, there also can be other non-computational advantages to performing 5-fold or 10-fold CV, which involve the bias-variance trade-off. The right-hand panel of Figure 5.4 displays nine different 10-fold CV estimates for the Auto data set, each resulting from a different random split of the observations into ten folds. As we can see from the figure, there is some variability in the CV estimates as a result of the variability in how the observations are divided into ten folds. But this variability is typically much lower than the variability in the test error estimates that results from the validation set approach (right-hand panel of Figure 5.2). When we examine real data, we do not know the true test MSE, and so it is difficult to determine the accuracy of the cross-validation estimate. However, if we examine simulated data, then we can compute the true test MSE, and can thereby evaluate the accuracy of our cross-validation results. In Figure 5.6, we plot the cross-validation estimates and true test error rates that result from applying smoothing splines to the simulated data sets illustrated in Figures 2.9–2.11 of Chapter 2. The true test MSE is displayed in blue. The black dashed and orange solid lines respectively show the estimated LOOCV and 10-fold CV estimates. In all three plots, the two cross-validation estimates are very similar. In the right-hand panel of Figure 5.6, the true test MSE and the cross-validation curves are almost identical. In the center panel of Figure 5.6, the two sets of curves are similar at the lower degrees of flexibility, while the CV curves overestimate the test set MSE for higher degrees of flexibility. In the left-hand panel of Figure 5.6,
- 214. 5.1 Cross-Validation 205 the CV curves have the correct general shape, but they underestimate the true test MSE. When we perform cross-validation, our goal might be to determine how well a given statistical learning procedure can be expected to perform on independent data; in this case, the actual estimate of the test MSE is of interest. But at other times we are interested only in the location of the minimum point in the estimated test MSE curve. This is because we might be performing cross-validation on a number of statistical learning methods, or on a single method using different levels of flexibility, in order to identify the method that results in the lowest test error. For this purpose, the location of the minimum point in the estimated test MSE curve is important, but the actual value of the estimated test MSE is not. We find in Figure 5.6 that despite the fact that they sometimes underestimate the true test MSE, all of the CV curves come close to identifying the correct level of flexibility—that is, the flexibility level corresponding to the smallest test MSE. 5.1.4 Bias-Variance Trade-Off for k-Fold Cross-Validation We mentioned in Section 5.1.3 that k-fold CV with k < n has a compu- tational advantage to LOOCV. But putting computational issues aside, a less obvious but potentially more important advantage of k-fold CV is that it often gives more accurate estimates of the test error rate than does LOOCV. This has to do with a bias-variance trade-off. It was mentioned in Section 5.1.1 that the validation set approach can lead to overestimates of the test error rate, since in this approach the training set used to fit the statistical learning method contains only half the observations of the entire data set. Using this logic, it is not hard to see that LOOCV will give approximately unbiased estimates of the test error, since each training set contains n−1 observations, which is almost as many as the number of observations in the full data set. And performing k-fold CV for, say, k = 5 or k = 10 will lead to an intermediate level of bias, since each training set contains approximately (k − 1)n/k observations— fewer than in the LOOCV approach, but substantially more than in the validation set approach. Therefore, from the perspective of bias reduction, it is clear that LOOCV is to be preferred to k-fold CV. However, we know that bias is not the only source for concern in an esti- mating procedure; we must also consider the procedure’s variance. It turns out that LOOCV has higher variance than does k-fold CV with k < n. Why is this the case? When we perform LOOCV, we are in effect averaging the outputs of n fitted models, each of which is trained on an almost identical set of observations; therefore, these outputs are highly (positively) corre- lated with each other. In contrast, when we perform k-fold CV with k < n, we are averaging the outputs of k fitted models that are somewhat less correlated with each other, since the overlap between the training sets in
- 215. 206 5. Resampling Methods each model is smaller. Since the mean of many highly correlated quantities has higher variance than does the mean of many quantities that are not as highly correlated, the test error estimate resulting from LOOCV tends to have higher variance than does the test error estimate resulting from k-fold CV. To summarize, there is a bias-variance trade-off associated with the choice of k in k-fold cross-validation. Typically, given these considerations, one performs k-fold cross-validation using k = 5 or k = 10, as these values have been shown empirically to yield test error rate estimates that suffer neither from excessively high bias nor from very high variance. 5.1.5 Cross-Validation on Classification Problems In this chapter so far, we have illustrated the use of cross-validation in the regression setting where the outcome Y is quantitative, and so have used MSE to quantify test error. But cross-validation can also be a very useful approach in the classification setting when Y is qualitative. In this setting, cross-validation works just as described earlier in this chapter, except that rather than using MSE to quantify test error, we instead use the number of misclassified observations. For instance, in the classification setting, the LOOCV error rate takes the form CV(n) = 1 n n 0 i=1 Erri, (5.4) where Erri = I(yi ̸= ŷi). The k-fold CV error rate and validation set error rates are defined analogously. As an example, we fit various logistic regression models on the two- dimensional classification data displayed in Figure 2.13. In the top-left panel of Figure 5.7, the black solid line shows the estimated decision bound- ary resulting from fitting a standard logistic regression model to this data set. Since this is simulated data, we can compute the true test error rate, which takes a value of 0.201 and so is substantially larger than the Bayes error rate of 0.133. Clearly logistic regression does not have enough flexi- bility to model the Bayes decision boundary in this setting. We can easily extend logistic regression to obtain a non-linear decision boundary by using polynomial functions of the predictors, as we did in the regression setting in Section 3.3.2. For example, we can fit a quadratic logistic regression model, given by log * p 1 − p + = β0 + β1X1 + β2X2 1 + β3X2 + β4X2 2 . (5.5) The top-right panel of Figure 5.7 displays the resulting decision boundary, which is now curved. However, the test error rate has improved only slightly, to 0.197. A much larger improvement is apparent in the bottom-left panel
- 216. 5.1 Cross-Validation 207 Degree=1 o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o Degree=2 o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o Degree=3 o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o Degree=4 o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o FIGURE 5.7. Logistic regression fits on the two-dimensional classification data displayed in Figure 2.13. The Bayes decision boundary is represented using a purple dashed line. Estimated decision boundaries from linear, quadratic, cubic and quartic (degrees 1–4) logistic regressions are displayed in black. The test error rates for the four logistic regression fits are respectively 0.201, 0.197, 0.160, and 0.162, while the Bayes error rate is 0.133. of Figure 5.7, in which we have fit a logistic regression model involving cubic polynomials of the predictors. Now the test error rate has decreased to 0.160. Going to a quartic polynomial (bottom-right) slightly increases the test error. In practice, for real data, the Bayes decision boundary and the test er- ror rates are unknown. So how might we decide between the four logistic regression models displayed in Figure 5.7? We can use cross-validation in order to make this decision. The left-hand panel of Figure 5.8 displays in
- 217. 208 5. Resampling Methods 2 4 6 8 10 0.12 0.14 0.16 0.18 0.20 Order of Polynomials Used Error Rate 0.01 0.02 0.05 0.10 0.20 0.50 1.00 0.12 0.14 0.16 0.18 0.20 1/K Error Rate FIGURE 5.8. Test error (brown), training error (blue), and 10-fold CV error (black) on the two-dimensional classification data displayed in Figure 5.7. Left: Logistic regression using polynomial functions of the predictors. The order of the polynomials used is displayed on the x-axis. Right: The KNN classifier with different values of K, the number of neighbors used in the KNN classifier. black the 10-fold CV error rates that result from fitting ten logistic regres- sion models to the data, using polynomial functions of the predictors up to tenth order. The true test errors are shown in brown, and the training errors are shown in blue. As we have seen previously, the training error tends to decrease as the flexibility of the fit increases. (The figure indicates that though the training error rate doesn’t quite decrease monotonically, it tends to decrease on the whole as the model complexity increases.) In contrast, the test error displays a characteristic U-shape. The 10-fold CV error rate provides a pretty good approximation to the test error rate. While it somewhat underestimates the error rate, it reaches a minimum when fourth-order polynomials are used, which is very close to the min- imum of the test curve, which occurs when third-order polynomials are used. In fact, using fourth-order polynomials would likely lead to good test set performance, as the true test error rate is approximately the same for third, fourth, fifth, and sixth-order polynomials. The right-hand panel of Figure 5.8 displays the same three curves us- ing the KNN approach for classification, as a function of the value of K (which in this context indicates the number of neighbors used in the KNN classifier, rather than the number of CV folds used). Again the training error rate declines as the method becomes more flexible, and so we see that the training error rate cannot be used to select the optimal value for K. Though the cross-validation error curve slightly underestimates the test error rate, it takes on a minimum very close to the best value for K.
- 218. 5.2 The Bootstrap 209 5.2 The Bootstrap The bootstrap is a widely applicable and extremely powerful statistical tool bootstrap that can be used to quantify the uncertainty associated with a given esti- mator or statistical learning method. As a simple example, the bootstrap can be used to estimate the standard errors of the coefficients from a linear regression fit. In the specific case of linear regression, this is not particularly useful, since we saw in Chapter 3 that standard statistical software such as R outputs such standard errors automatically. However, the power of the bootstrap lies in the fact that it can be easily applied to a wide range of statistical learning methods, including some for which a measure of vari- ability is otherwise difficult to obtain and is not automatically output by statistical software. In this section we illustrate the bootstrap on a toy example in which we wish to determine the best investment allocation under a simple model. In Section 5.3 we explore the use of the bootstrap to assess the variability associated with the regression coefficients in a linear model fit. Suppose that we wish to invest a fixed sum of money in two financial assets that yield returns of X and Y , respectively, where X and Y are random quantities. We will invest a fraction α of our money in X, and will invest the remaining 1 − α in Y . Since there is variability associated with the returns on these two assets, we wish to choose α to minimize the total risk, or variance, of our investment. In other words, we want to minimize Var(αX + (1 − α)Y ). One can show that the value that minimizes the risk is given by α = σ2 Y − σXY σ2 X + σ2 Y − 2σXY , (5.6) where σ2 X = Var(X), σ2 Y = Var(Y ), and σXY = Cov(X, Y ). In reality, the quantities σ2 X , σ2 Y , and σXY are unknown. We can compute estimates for these quantities, σ̂2 X, σ̂2 Y , and σ̂XY , using a data set that contains past measurements for X and Y . We can then estimate the value of α that minimizes the variance of our investment using α̂ = σ̂2 Y − σ̂XY σ̂2 X + σ̂2 Y − 2σ̂XY . (5.7) Figure 5.9 illustrates this approach for estimating α on a simulated data set. In each panel, we simulated 100 pairs of returns for the investments X and Y . We used these returns to estimate σ2 X, σ2 Y , and σXY , which we then substituted into (5.7) in order to obtain estimates for α. The value of α̂ resulting from each simulated data set ranges from 0.532 to 0.657. It is natural to wish to quantify the accuracy of our estimate of α. To estimate the standard deviation of α̂, we repeated the process of simu- lating 100 paired observations of X and Y , and estimating α using (5.7), 1,000 times. We thereby obtained 1,000 estimates for α, which we can call
- 219. 210 5. Resampling Methods −2 −1 0 1 2 −2 −1 0 1 2 X Y −2 −1 0 1 2 −2 −1 0 1 2 X Y −3 −2 −1 0 1 2 −3 −2 −1 0 1 2 X Y −2 −1 0 1 2 3 −3 −2 −1 0 1 2 X Y FIGURE 5.9. Each panel displays 100 simulated returns for investments X and Y . From left to right and top to bottom, the resulting estimates for α are 0.576, 0.532, 0.657, and 0.651. α̂1, α̂2, . . . , α̂1,000. The left-hand panel of Figure 5.10 displays a histogram of the resulting estimates. For these simulations the parameters were set to σ2 X = 1, σ2 Y = 1.25, and σXY = 0.5, and so we know that the true value of α is 0.6. We indicated this value using a solid vertical line on the histogram. The mean over all 1,000 estimates for α is ᾱ = 1 1000 1000 0 r=1 α̂r = 0.5996, very close to α = 0.6, and the standard deviation of the estimates is : ; ; < 1 1000 − 1 1000 0 r=1 (α̂r − ᾱ) 2 = 0.083. This gives us a very good idea of the accuracy of α̂: SE(α̂) ≈ 0.083. So roughly speaking, for a random sample from the population, we would expect α̂ to differ from α by approximately 0.08, on average. In practice, however, the procedure for estimating SE(α̂) outlined above cannot be applied, because for real data we cannot generate new samples
- 220. 5.2 The Bootstrap 211 0.4 0.5 0.6 0.7 0.8 0.9 0 50 100 150 200 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0 50 100 150 200 True Bootstrap 0.3 0.4 0.5 0.6 0.7 0.8 0.9 α α α FIGURE 5.10. Left: A histogram of the estimates of α obtained by generating 1,000 simulated data sets from the true population. Center: A histogram of the estimates of α obtained from 1,000 bootstrap samples from a single data set. Right: The estimates of α displayed in the left and center panels are shown as boxplots. In each panel, the pink line indicates the true value of α. from the original population. However, the bootstrap approach allows us to use a computer to emulate the process of obtaining new sample sets, so that we can estimate the variability of α̂ without generating additional samples. Rather than repeatedly obtaining independent data sets from the population, we instead obtain distinct data sets by repeatedly sampling observations from the original data set. This approach is illustrated in Figure 5.11 on a simple data set, which we call Z, that contains only n = 3 observations. We randomly select n observations from the data set in order to produce a bootstrap data set, Z∗1 . The sampling is performed with replacement, which means that the with replacement same observation can occur more than once in the bootstrap data set. In this example, Z∗1 contains the third observation twice, the first observation once, and no instances of the second observation. Note that if an observation is contained in Z∗1 , then both its X and Y values are included. We can use Z∗1 to produce a new bootstrap estimate for α, which we call α̂∗1 . This procedure is repeated B times for some large value of B, in order to produce B different bootstrap data sets, Z∗1 , Z∗2 , . . . , Z∗B , and B corresponding α estimates, α̂∗1 , α̂∗2 , . . . , α̂∗B . We can compute the standard error of these bootstrap estimates using the formula SEB(α̂) = : ; ; < 1 B − 1 B 0 r=1 > α̂∗r − 1 B B 0 r′=1 α̂∗r′ ?2 . (5.8) This serves as an estimate of the standard error of α̂ estimated from the original data set. The bootstrap approach is illustrated in the center panel of Figure 5.10, which displays a histogram of 1,000 bootstrap estimates of α, each com-
- 221. 212 5. Resampling Methods 2.8 5.3 3 1.1 2.1 2 2.4 4.3 1 Y X Obs 2.8 5.3 3 2.4 4.3 1 2.8 5.3 3 Y X Obs 2.4 4.3 1 2.8 5.3 3 1.1 2.1 2 Y X Obs 2.4 4.3 1 1.1 2.1 2 1.1 2.1 2 Y X Obs Original Data (Z) 1 * Z 2 * Z Z*B 1 * α̂ 2 * α̂ α̂*B !! !! !! !! ! !! !! !! !! !! !! !! !! FIGURE 5.11. A graphical illustration of the bootstrap approach on a small sample containing n = 3 observations. Each bootstrap data set contains n obser- vations, sampled with replacement from the original data set. Each bootstrap data set is used to obtain an estimate of α. puted using a distinct bootstrap data set. This panel was constructed on the basis of a single data set, and hence could be created using real data. Note that the histogram looks very similar to the left-hand panel, which displays the idealized histogram of the estimates of α obtained by gener- ating 1,000 simulated data sets from the true population. In particular the bootstrap estimate SE(α̂) from (5.8) is 0.087, very close to the estimate of 0.083 obtained using 1,000 simulated data sets. The right-hand panel displays the information in the center and left panels in a different way, via boxplots of the estimates for α obtained by generating 1,000 simulated data sets from the true population and using the bootstrap approach. Again, the boxplots have similar spreads, indicating that the bootstrap approach can be used to effectively estimate the variability associated with α̂. 5.3 Lab: Cross-Validation and the Bootstrap In this lab, we explore the resampling techniques covered in this chapter. Some of the commands in this lab may take a while to run on your com- puter.
- 222. 5.3 Lab: Cross-Validation and the Bootstrap 213 5.3.1 The Validation Set Approach We explore the use of the validation set approach in order to estimate the test error rates that result from fitting various linear models on the Auto data set. Before we begin, we use the set.seed() function in order to set a seed for seed R’s random number generator, so that the reader of this book will obtain precisely the same results as those shown below. It is generally a good idea to set a random seed when performing an analysis such as cross-validation that contains an element of randomness, so that the results obtained can be reproduced precisely at a later time. We begin by using the sample() function to split the set of observations sample() into two halves, by selecting a random subset of 196 observations out of the original 392 observations. We refer to these observations as the training set. > library(ISLR2) > set.seed (1) > train <- sample (392 , 196) (Here we use a shortcut in the sample command; see ?sample for details.) We then use the subset option in lm() to fit a linear regression using only the observations corresponding to the training set. > lm.fit <- lm(mpg ∼ horsepower , data = Auto , subset = train) We now use the predict() function to estimate the response for all 392 observations, and we use the mean() function to calculate the MSE of the 196 observations in the validation set. Note that the -train index below selects only the observations that are not in the training set. > attach(Auto) > mean (( mpg - predict(lm.fit , Auto))[-train ]^2) [1] 23.27 Therefore, the estimated test MSE for the linear regression fit is 23.27. We can use the poly() function to estimate the test error for the quadratic and cubic regressions. > lm.fit2 <- lm(mpg ∼ poly(horsepower , 2), data = Auto , subset = train) > mean (( mpg - predict(lm.fit2 , Auto))[-train ]^2) [1] 18.72 > lm.fit3 <- lm(mpg ∼ poly(horsepower , 3), data = Auto , subset = train) > mean (( mpg - predict(lm.fit3 , Auto))[-train ]^2) [1] 18.79 These error rates are 18.72 and 18.79, respectively. If we choose a different training set instead, then we will obtain somewhat different errors on the validation set.
- 223. 214 5. Resampling Methods > set.seed (2) > train <- sample (392 , 196) > lm.fit <- lm(mpg ∼ horsepower , subset = train) > mean (( mpg - predict(lm.fit , Auto))[-train ]^2) [1] 25.73 > lm.fit2 <- lm(mpg ∼ poly(horsepower , 2), data = Auto , subset = train) > mean (( mpg - predict(lm.fit2 , Auto))[-train ]^2) [1] 20.43 > lm.fit3 <- lm(mpg ∼ poly(horsepower , 3), data = Auto , subset = train) > mean (( mpg - predict(lm.fit3 , Auto))[-train ]^2) [1] 20.39 Using this split of the observations into a training set and a validation set, we find that the validation set error rates for the models with linear, quadratic, and cubic terms are 25.73, 20.43, and 20.39, respectively. These results are consistent with our previous findings: a model that predicts mpg using a quadratic function of horsepower performs better than a model that involves only a linear function of horsepower, and there is little evidence in favor of a model that uses a cubic function of horsepower. 5.3.2 Leave-One-Out Cross-Validation The LOOCV estimate can be automatically computed for any generalized linear model using the glm() and cv.glm() functions. In the lab for Chap- cv.glm() ter 4, we used the glm() function to perform logistic regression by passing in the family = "binomial" argument. But if we use glm() to fit a model without passing in the family argument, then it performs linear regression, just like the lm() function. So for instance, > glm.fit <- glm(mpg ∼ horsepower , data = Auto) > coef(glm.fit) (Intercept) horsepower 39.936 -0.158 and > lm.fit <- lm(mpg ∼ horsepower , data = Auto) > coef(lm.fit) (Intercept) horsepower 39.936 -0.158 yield identical linear regression models. In this lab, we will perform linear regression using the glm() function rather than the lm() function because the former can be used together with cv.glm(). The cv.glm() function is part of the boot library. > library(boot) > glm.fit <- glm(mpg ∼ horsepower , data = Auto) > cv.err <- cv.glm(Auto , glm.fit) > cv.err$delta
- 224. 5.3 Lab: Cross-Validation and the Bootstrap 215 1 1 24.23 24.23 The cv.glm() function produces a list with several components. The two numbers in the delta vector contain the cross-validation results. In this case the numbers are identical (up to two decimal places) and correspond to the LOOCV statistic given in (5.1). Below, we discuss a situation in which the two numbers differ. Our cross-validation estimate for the test error is approximately 24.23. We can repeat this procedure for increasingly complex polynomial fits. To automate the process, we use the for() function to initiate a for loop for() for loop which iteratively fits polynomial regressions for polynomials of order i = 1 to i = 10, computes the associated cross-validation error, and stores it in the ith element of the vector cv.error. We begin by initializing the vector. > cv.error <- rep(0, 10) > for (i in 1:10) { + glm.fit <- glm(mpg ∼ poly(horsepower , i), data = Auto) + cv.error[i] <- cv.glm(Auto , glm.fit)$delta [1] + } > cv.error [1] 24.23 19.25 19.33 19.42 19.03 18.98 18.83 18.96 19.07 19.49 As in Figure 5.4, we see a sharp drop in the estimated test MSE between the linear and quadratic fits, but then no clear improvement from using higher-order polynomials. 5.3.3 k-Fold Cross-Validation The cv.glm() function can also be used to implement k-fold CV. Below we use k = 10, a common choice for k, on the Auto data set. We once again set a random seed and initialize a vector in which we will store the CV errors corresponding to the polynomial fits of orders one to ten. > set.seed (17) > cv.error .10 <- rep(0, 10) > for (i in 1:10) { + glm.fit <- glm(mpg ∼ poly(horsepower , i), data = Auto) + cv.error .10[i] <- cv.glm(Auto , glm.fit , K = 10)$delta [1] + } > cv.error .10 [1] 24.27 19.27 19.35 19.29 19.03 18.90 19.12 19.15 18.87 20.96 Notice that the computation time is shorter than that of LOOCV. (In principle, the computation time for LOOCV for a least squares linear model should be faster than for k-fold CV, due to the availability of the formula (5.2) for LOOCV; however, unfortunately the cv.glm() function does not make use of this formula.) We still see little evidence that using cubic or higher-order polynomial terms leads to lower test error than simply using a quadratic fit.
- 225. 216 5. Resampling Methods We saw in Section 5.3.2 that the two numbers associated with delta are essentially the same when LOOCV is performed. When we instead perform k-fold CV, then the two numbers associated with delta differ slightly. The first is the standard k-fold CV estimate, as in (5.3). The second is a bias- corrected version. On this data set, the two estimates are very similar to each other. 5.3.4 The Bootstrap We illustrate the use of the bootstrap in the simple example of Section 5.2, as well as on an example involving estimating the accuracy of the linear regression model on the Auto data set. Estimating the Accuracy of a Statistic of Interest One of the great advantages of the bootstrap approach is that it can be applied in almost all situations. No complicated mathematical calculations are required. Performing a bootstrap analysis in R entails only two steps. First, we must create a function that computes the statistic of interest. Second, we use the boot() function, which is part of the boot library, to boot() perform the bootstrap by repeatedly sampling observations from the data set with replacement. The Portfolio data set in the ISLR2 package is simulated data of 100 pairs of returns, generated in the fashion described in Section 5.2. To illus- trate the use of the bootstrap on this data, we must first create a function, alpha.fn(), which takes as input the (X, Y ) data as well as a vector indi- cating which observations should be used to estimate α. The function then outputs the estimate for α based on the selected observations. > alpha.fn <- function (data , index) { + X <- data$X[index] + Y <- data$Y[index] + (var(Y) - cov(X, Y)) / (var(X) + var(Y) - 2 * cov(X, Y)) + } This function returns, or outputs, an estimate for α based on applying (5.7) to the observations indexed by the argument index. For instance, the following command tells R to estimate α using all 100 observations. > alpha.fn(Portfolio , 1:100) [1] 0.576 The next command uses the sample() function to randomly select 100 ob- servations from the range 1 to 100, with replacement. This is equivalent to constructing a new bootstrap data set and recomputing α̂ based on the new data set. > set.seed (7) > alpha.fn(Portfolio , sample (100 , 100, replace = T)) [1] 0.539
- 226. 5.3 Lab: Cross-Validation and the Bootstrap 217 We can implement a bootstrap analysis by performing this command many times, recording all of the corresponding estimates for α, and computing the resulting standard deviation. However, the boot() function automates boot() this approach. Below we produce R = 1, 000 bootstrap estimates for α. > boot(Portfolio , alpha.fn , R = 1000) ORDINARY NONPARAMETRIC BOOTSTRAP Call: boot(data = Portfolio , statistic = alpha.fn , R = 1000) Bootstrap Statistics : original bias std. error t1* 0.5758 0.001 0.0897 The final output shows that using the original data, α̂ = 0.5758, and that the bootstrap estimate for SE(α̂) is 0.0897. Estimating the Accuracy of a Linear Regression Model The bootstrap approach can be used to assess the variability of the coef- ficient estimates and predictions from a statistical learning method. Here we use the bootstrap approach in order to assess the variability of the estimates for β0 and β1, the intercept and slope terms for the linear regres- sion model that uses horsepower to predict mpg in the Auto data set. We will compare the estimates obtained using the bootstrap to those obtained using the formulas for SE(β̂0) and SE(β̂1) described in Section 3.1.2. We first create a simple function, boot.fn(), which takes in the Auto data set as well as a set of indices for the observations, and returns the intercept and slope estimates for the linear regression model. We then apply this function to the full set of 392 observations in order to compute the esti- mates of β0 and β1 on the entire data set using the usual linear regression coefficient estimate formulas from Chapter 3. Note that we do not need the { and } at the beginning and end of the function because it is only one line long. > boot.fn <- function (data , index) + coef(lm(mpg ∼ horsepower , data = data , subset = index)) > boot.fn(Auto , 1:392) (Intercept) horsepower 39.936 -0.158 The boot.fn() function can also be used in order to create bootstrap esti- mates for the intercept and slope terms by randomly sampling from among the observations with replacement. Here we give two examples. > set.seed (1) > boot.fn(Auto , sample (392 , 392, replace = T)) (Intercept) horsepower 40.341 -0.164
- 227. 218 5. Resampling Methods > boot.fn(Auto , sample (392 , 392, replace = T)) (Intercept) horsepower 40.119 -0.158 Next, we use the boot() function to compute the standard errors of 1,000 bootstrap estimates for the intercept and slope terms. > boot(Auto , boot.fn , 1000) ORDINARY NONPARAMETRIC BOOTSTRAP Call: boot(data = Auto , statistic = boot.fn , R = 1000) Bootstrap Statistics : original bias std. error t1* 39.936 0.0545 0.8413 t2* -0.158 -0.0006 0.0073 This indicates that the bootstrap estimate for SE(β̂0) is 0.84, and that the bootstrap estimate for SE(β̂1) is 0.0073. As discussed in Section 3.1.2, standard formulas can be used to compute the standard errors for the regression coefficients in a linear model. These can be obtained using the summary() function. > summary(lm(mpg ∼ horsepower , data = Auto))$coef Estimate Std. Error t value Pr(>|t|) (Intercept) 39.936 0.71750 55.7 1.22e -187 horsepower -0.158 0.00645 -24.5 7.03e -81 The standard error estimates for β̂0 and β̂1 obtained using the formulas from Section 3.1.2 are 0.717 for the intercept and 0.0064 for the slope. Interestingly, these are somewhat different from the estimates obtained using the bootstrap. Does this indicate a problem with the bootstrap? In fact, it suggests the opposite. Recall that the standard formulas given in Equation 3.8 on page 66 rely on certain assumptions. For example, they depend on the unknown parameter σ2 , the noise variance. We then estimate σ2 using the RSS. Now although the formulas for the standard errors do not rely on the linear model being correct, the estimate for σ2 does. We see in Figure 3.8 on page 91 that there is a non-linear relationship in the data, and so the residuals from a linear fit will be inflated, and so will σ̂2 . Secondly, the standard formulas assume (somewhat unrealistically) that the xi are fixed, and all the variability comes from the variation in the errors ϵi. The bootstrap approach does not rely on any of these assumptions, and so it is likely giving a more accurate estimate of the standard errors of β̂0 and β̂1 than is the summary() function. Below we compute the bootstrap standard error estimates and the stan- dard linear regression estimates that result from fitting the quadratic model to the data. Since this model provides a good fit to the data (Figure 3.8),
- 228. 5.4 Exercises 219 there is now a better correspondence between the bootstrap estimates and the standard estimates of SE(β̂0), SE(β̂1) and SE(β̂2). > boot.fn <- function (data , index) + coef( lm(mpg ∼ horsepower + I( horsepower ^2) , data = data , subset = index) ) > set.seed (1) > boot(Auto , boot.fn , 1000) ORDINARY NONPARAMETRIC BOOTSTRAP Call: boot(data = Auto , statistic = boot.fn , R = 1000) Bootstrap Statistics : original bias std. error t1* 56.9001 3.51e -02 2.0300 t2* -0.4661 -7.08e-04 0.0324 t3* 0.0012 2.84e -06 0.0001 > summary( lm(mpg ∼ horsepower + I( horsepower ^2) , data = Auto) )$coef Estimate Std. Error t value Pr(>|t|) (Intercept) 56.9001 1.8004 32 1.7e -109 horsepower -0.4662 0.0311 -15 2.3e -40 I(horsepower ^2) 0.0012 0.0001 10 2.2e -21 5.4 Exercises Conceptual 1. Using basic statistical properties of the variance, as well as single- variable calculus, derive (5.6). In other words, prove that α given by (5.6) does indeed minimize Var(αX + (1 − α)Y ). 2. We will now derive the probability that a given observation is part of a bootstrap sample. Suppose that we obtain a bootstrap sample from a set of n observations. (a) What is the probability that the first bootstrap observation is not the jth observation from the original sample? Justify your answer. (b) What is the probability that the second bootstrap observation is not the jth observation from the original sample? (c) Argue that the probability that the jth observation is not in the bootstrap sample is (1 − 1/n)n .
- 229. 220 5. Resampling Methods (d) When n = 5, what is the probability that the jth observation is in the bootstrap sample? (e) When n = 100, what is the probability that the jth observation is in the bootstrap sample? (f) When n = 10, 000, what is the probability that the jth observa- tion is in the bootstrap sample? (g) Create a plot that displays, for each integer value of n from 1 to 100, 000, the probability that the jth observation is in the bootstrap sample. Comment on what you observe. (h) We will now investigate numerically the probability that a boot- strap sample of size n = 100 contains the jth observation. Here j = 4. We repeatedly create bootstrap samples, and each time we record whether or not the fourth observation is contained in the bootstrap sample. > store <- rep(NA , 10000) > for(i in 1:10000){ store[i] <- sum(sample (1:100 , rep=TRUE) == 4) > 0 } > mean(store) Comment on the results obtained. 3. We now review k-fold cross-validation. (a) Explain how k-fold cross-validation is implemented. (b) What are the advantages and disadvantages of k-fold cross- validation relative to: i. The validation set approach? ii. LOOCV? 4. Suppose that we use some statistical learning method to make a pre- diction for the response Y for a particular value of the predictor X. Carefully describe how we might estimate the standard deviation of our prediction. Applied 5. In Chapter 4, we used logistic regression to predict the probability of default using income and balance on the Default data set. We will now estimate the test error of this logistic regression model using the validation set approach. Do not forget to set a random seed before beginning your analysis. (a) Fit a logistic regression model that uses income and balance to predict default.
- 230. 5.4 Exercises 221 (b) Using the validation set approach, estimate the test error of this model. In order to do this, you must perform the following steps: i. Split the sample set into a training set and a validation set. ii. Fit a multiple logistic regression model using only the train- ing observations. iii. Obtain a prediction of default status for each individual in the validation set by computing the posterior probability of default for that individual, and classifying the individual to the default category if the posterior probability is greater than 0.5. iv. Compute the validation set error, which is the fraction of the observations in the validation set that are misclassified. (c) Repeat the process in (b) three times, using three different splits of the observations into a training set and a validation set. Com- ment on the results obtained. (d) Now consider a logistic regression model that predicts the prob- ability of default using income, balance, and a dummy variable for student. Estimate the test error for this model using the val- idation set approach. Comment on whether or not including a dummy variable for student leads to a reduction in the test error rate. 6. We continue to consider the use of a logistic regression model to predict the probability of default using income and balance on the Default data set. In particular, we will now compute estimates for the standard errors of the income and balance logistic regression co- efficients in two different ways: (1) using the bootstrap, and (2) using the standard formula for computing the standard errors in the glm() function. Do not forget to set a random seed before beginning your analysis. (a) Using the summary() and glm() functions, determine the esti- mated standard errors for the coefficients associated with income and balance in a multiple logistic regression model that uses both predictors. (b) Write a function, boot.fn(), that takes as input the Default data set as well as an index of the observations, and that outputs the coefficient estimates for income and balance in the multiple logistic regression model. (c) Use the boot() function together with your boot.fn() function to estimate the standard errors of the logistic regression coefficients for income and balance. (d) Comment on the estimated standard errors obtained using the glm() function and using your bootstrap function.
- 231. 222 5. Resampling Methods 7. In Sections 5.3.2 and 5.3.3, we saw that the cv.glm() function can be used in order to compute the LOOCV test error estimate. Alterna- tively, one could compute those quantities using just the glm() and predict.glm() functions, and a for loop. You will now take this ap- proach in order to compute the LOOCV error for a simple logistic regression model on the Weekly data set. Recall that in the context of classification problems, the LOOCV error is given in (5.4). (a) Fit a logistic regression model that predicts Direction using Lag1 and Lag2. (b) Fit a logistic regression model that predicts Direction using Lag1 and Lag2 using all but the first observation. (c) Use the model from (b) to predict the direction of the first obser- vation. You can do this by predicting that the first observation will go up if P(Direction = "Up"|Lag1, Lag2) > 0.5. Was this observation correctly classified? (d) Write a for loop from i = 1 to i = n, where n is the number of observations in the data set, that performs each of the following steps: i. Fit a logistic regression model using all but the ith obser- vation to predict Direction using Lag1 and Lag2. ii. Compute the posterior probability of the market moving up for the ith observation. iii. Use the posterior probability for the ith observation in order to predict whether or not the market moves up. iv. Determine whether or not an error was made in predicting the direction for the ith observation. If an error was made, then indicate this as a 1, and otherwise indicate it as a 0. (e) Take the average of the n numbers obtained in (d)iv in order to obtain the LOOCV estimate for the test error. Comment on the results. 8. We will now perform cross-validation on a simulated data set. (a) Generate a simulated data set as follows: > set.seed (1) > x <- rnorm (100) > y <- x - 2 * x^2 + rnorm (100) In this data set, what is n and what is p? Write out the model used to generate the data in equation form. (b) Create a scatterplot of X against Y . Comment on what you find. (c) Set a random seed, and then compute the LOOCV errors that result from fitting the following four models using least squares:
- 232. 5.4 Exercises 223 i. Y = β0 + β1X + ϵ ii. Y = β0 + β1X + β2X2 + ϵ iii. Y = β0 + β1X + β2X2 + β3X3 + ϵ iv. Y = β0 + β1X + β2X2 + β3X3 + β4X4 + ϵ. Note you may find it helpful to use the data.frame() function to create a single data set containing both X and Y . (d) Repeat (c) using another random seed, and report your results. Are your results the same as what you got in (c)? Why? (e) Which of the models in (c) had the smallest LOOCV error? Is this what you expected? Explain your answer. (f) Comment on the statistical significance of the coefficient esti- mates that results from fitting each of the models in (c) using least squares. Do these results agree with the conclusions drawn based on the cross-validation results? 9. We will now consider the Boston housing data set, from the ISLR2 library. (a) Based on this data set, provide an estimate for the population mean of medv. Call this estimate µ̂. (b) Provide an estimate of the standard error of µ̂. Interpret this result. Hint: We can compute the standard error of the sample mean by dividing the sample standard deviation by the square root of the number of observations. (c) Now estimate the standard error of µ̂ using the bootstrap. How does this compare to your answer from (b)? (d) Based on your bootstrap estimate from (c), provide a 95 % con- fidence interval for the mean of medv. Compare it to the results obtained using t.test(Boston$medv). Hint: You can approximate a 95 % confidence interval using the formula [µ̂ − 2SE(µ̂), µ̂ + 2SE(µ̂)]. (e) Based on this data set, provide an estimate, µ̂med, for the median value of medv in the population. (f) We now would like to estimate the standard error of µ̂med. Unfor- tunately, there is no simple formula for computing the standard error of the median. Instead, estimate the standard error of the median using the bootstrap. Comment on your findings. (g) Based on this data set, provide an estimate for the tenth per- centile of medv in Boston census tracts. Call this quantity µ̂0.1. (You can use the quantile() function.) (h) Use the bootstrap to estimate the standard error of µ̂0.1. Com- ment on your findings.
- 233. 6 Linear Model Selection and Regularization In the regression setting, the standard linear model Y = β0 + β1X1 + · · · + βpXp + ϵ (6.1) is commonly used to describe the relationship between a response Y and a set of variables X1, X2, . . . , Xp. We have seen in Chapter 3 that one typically fits this model using least squares. In the chapters that follow, we consider some approaches for extending the linear model framework. In Chapter 7 we generalize (6.1) in order to accommodate non-linear, but still additive, relationships, while in Chap- ters 8 and 10 we consider even more general non-linear models. However, the linear model has distinct advantages in terms of inference and, on real- world problems, is often surprisingly competitive in relation to non-linear methods. Hence, before moving to the non-linear world, we discuss in this chapter some ways in which the simple linear model can be improved, by re- placing plain least squares fitting with some alternative fitting procedures. Why might we want to use another fitting procedure instead of least squares? As we will see, alternative fitting procedures can yield better pre- diction accuracy and model interpretability. • Prediction Accuracy: Provided that the true relationship between the response and the predictors is approximately linear, the least squares estimates will have low bias. If n ≫ p—that is, if n, the number of observations, is much larger than p, the number of variables—then the least squares estimates tend to also have low variance, and hence will perform well on test observations. However, if n is not much larger © Springer Science+Business Media, LLC, part of Springer Nature 2021 G. James et al., An Introduction to Statistical Learning, Springer Texts in Statistics, https://doi.org/10.1007/978-1-0716-1418-1_6 225
- 234. 226 6. Linear Model Selection and Regularization than p, then there can be a lot of variability in the least squares fit, resulting in overfitting and consequently poor predictions on future observations not used in model training. And if p > n, then there is no longer a unique least squares coefficient estimate: the variance is infinite so the method cannot be used at all. By constraining or shrinking the estimated coefficients, we can often substantially reduce the variance at the cost of a negligible increase in bias. This can lead to substantial improvements in the accuracy with which we can predict the response for observations not used in model training. • Model Interpretability: It is often the case that some or many of the variables used in a multiple regression model are in fact not associ- ated with the response. Including such irrelevant variables leads to unnecessary complexity in the resulting model. By removing these variables—that is, by setting the corresponding coefficient estimates to zero—we can obtain a model that is more easily interpreted. Now least squares is extremely unlikely to yield any coefficient estimates that are exactly zero. In this chapter, we see some approaches for au- tomatically performing feature selection or variable selection—that is, feature selection variable selection for excluding irrelevant variables from a multiple regression model. There are many alternatives, both classical and modern, to using least squares to fit (6.1). In this chapter, we discuss three important classes of methods. • Subset Selection. This approach involves identifying a subset of the p predictors that we believe to be related to the response. We then fit a model using least squares on the reduced set of variables. • Shrinkage. This approach involves fitting a model involving all p pre- dictors. However, the estimated coefficients are shrunken towards zero relative to the least squares estimates. This shrinkage (also known as regularization) has the effect of reducing variance. Depending on what type of shrinkage is performed, some of the coefficients may be esti- mated to be exactly zero. Hence, shrinkage methods can also perform variable selection. • Dimension Reduction. This approach involves projecting the p predic- tors into an M-dimensional subspace, where M < p. This is achieved by computing M different linear combinations, or projections, of the variables. Then these M projections are used as predictors to fit a linear regression model by least squares. In the following sections we describe each of these approaches in greater de- tail, along with their advantages and disadvantages. Although this chapter describes extensions and modifications to the linear model for regression seen in Chapter 3, the same concepts apply to other methods, such as the classification models seen in Chapter 4.
- 235. 6.1 Subset Selection 227 6.1 Subset Selection In this section we consider some methods for selecting subsets of predictors. These include best subset and stepwise model selection procedures. 6.1.1 Best Subset Selection To perform best subset selection, we fit a separate least squares regression best subset selection for each possible combination of the p predictors. That is, we fit all p models that contain exactly one predictor, all 'p 2 ( = p(p−1)/2 models that contain exactly two predictors, and so forth. We then look at all of the resulting models, with the goal of identifying the one that is best. The problem of selecting the best model from among the 2p possibilities considered by best subset selection is not trivial. This is usually broken up into two stages, as described in Algorithm 6.1. Algorithm 6.1 Best subset selection 1. Let M0 denote the null model, which contains no predictors. This model simply predicts the sample mean for each observation. 2. For k = 1, 2, . . . p: (a) Fit all 'p k ( models that contain exactly k predictors. (b) Pick the best among these 'p k ( models, and call it Mk. Here best is defined as having the smallest RSS, or equivalently largest R2 . 3. Select a single best model from among M0, . . . , Mp using cross- validated prediction error, Cp (AIC), BIC, or adjusted R2 . In Algorithm 6.1, Step 2 identifies the best model (on the training data) for each subset size, in order to reduce the problem from one of 2p possible models to one of p + 1 possible models. In Figure 6.1, these models form the lower frontier depicted in red. Now in order to select a single best model, we must simply choose among these p + 1 options. This task must be performed with care, because the RSS of these p + 1 models decreases monotonically, and the R2 increases monotonically, as the number of features included in the models increases. Therefore, if we use these statistics to select the best model, then we will always end up with a model involving all of the variables. The problem is that a low RSS or a high R2 indicates a model with a low training error, whereas we wish to choose a model that has a low test error. (As shown in Chapter 2 in Figures 2.9–2.11, training error tends to be quite a bit smaller than test error, and a low training error by no means guarantees a low test error.) Therefore, in Step 3, we use cross-validated prediction
- 236. 228 6. Linear Model Selection and Regularization 2 4 6 8 10 2e+07 4e+07 6e+07 8e+07 Number of Predictors Residual Sum of Squares 2 4 6 8 10 0.0 0.2 0.4 0.6 0.8 1.0 Number of Predictors R 2 FIGURE 6.1. For each possible model containing a subset of the ten predictors in the Credit data set, the RSS and R2 are displayed. The red frontier tracks the best model for a given number of predictors, according to RSS and R2 . Though the data set contains only ten predictors, the x-axis ranges from 1 to 11, since one of the variables is categorical and takes on three values, leading to the creation of two dummy variables. error, Cp, BIC, or adjusted R2 in order to select among M0, M1, . . . , Mp. These approaches are discussed in Section 6.1.3. An application of best subset selection is shown in Figure 6.1. Each plotted point corresponds to a least squares regression model fit using a different subset of the 10 predictors in the Credit data set, discussed in Chapter 3. Here the variable region is a three-level qualitative variable, and so is represented by two dummy variables, which are selected sepa- rately in this case. Hence, there are a total of 11 possible variables which can be included in the model. We have plotted the RSS and R2 statistics for each model, as a function of the number of variables. The red curves connect the best models for each model size, according to RSS or R2 . The figure shows that, as expected, these quantities improve as the number of variables increases; however, from the three-variable model on, there is little improvement in RSS and R2 as a result of including additional predictors. Although we have presented best subset selection here for least squares regression, the same ideas apply to other types of models, such as logistic regression. In the case of logistic regression, instead of ordering models by RSS in Step 2 of Algorithm 6.1, we instead use the deviance, a measure deviance that plays the role of RSS for a broader class of models. The deviance is negative two times the maximized log-likelihood; the smaller the deviance, the better the fit. While best subset selection is a simple and conceptually appealing ap- proach, it suffers from computational limitations. The number of possible models that must be considered grows rapidly as p increases. In general,
- 237. 6.1 Subset Selection 229 there are 2p models that involve subsets of p predictors. So if p = 10, then there are approximately 1,000 possible models to be considered, and if p = 20, then there are over one million possibilities! Consequently, best sub- set selection becomes computationally infeasible for values of p greater than around 40, even with extremely fast modern computers. There are compu- tational shortcuts—so called branch-and-bound techniques—for eliminat- ing some choices, but these have their limitations as p gets large. They also only work for least squares linear regression. We present computationally efficient alternatives to best subset selection next. 6.1.2 Stepwise Selection For computational reasons, best subset selection cannot be applied with very large p. Best subset selection may also suffer from statistical problems when p is large. The larger the search space, the higher the chance of finding models that look good on the training data, even though they might not have any predictive power on future data. Thus an enormous search space can lead to overfitting and high variance of the coefficient estimates. For both of these reasons, stepwise methods, which explore a far more restricted set of models, are attractive alternatives to best subset selection. Forward Stepwise Selection Forward stepwise selection is a computationally efficient alternative to best forward stepwise selection subset selection. While the best subset selection procedure considers all 2p possible models containing subsets of the p predictors, forward step- wise considers a much smaller set of models. Forward stepwise selection begins with a model containing no predictors, and then adds predictors to the model, one-at-a-time, until all of the predictors are in the model. In particular, at each step the variable that gives the greatest additional improvement to the fit is added to the model. More formally, the forward stepwise selection procedure is given in Algorithm 6.2. Unlike best subset selection, which involved fitting 2p models, forward stepwise selection involves fitting one null model, along with p − k models in the kth iteration, for k = 0, . . . , p − 1. This amounts to a total of 1 + )p−1 k=0(p−k) = 1+p(p+1)/2 models. This is a substantial difference: when p = 20, best subset selection requires fitting 1,048,576 models, whereas forward stepwise selection requires fitting only 211 models.1 1Though forward stepwise selection considers p(p + 1)/2 + 1 models, it performs a guided search over model space, and so the effective model space considered contains substantially more than p(p + 1)/2 + 1 models.
- 238. 230 6. Linear Model Selection and Regularization Algorithm 6.2 Forward stepwise selection 1. Let M0 denote the null model, which contains no predictors. 2. For k = 0, . . . , p − 1: (a) Consider all p − k models that augment the predictors in Mk with one additional predictor. (b) Choose the best among these p − k models, and call it Mk+1. Here best is defined as having smallest RSS or highest R2 . 3. Select a single best model from among M0, . . . , Mp using cross- validated prediction error, Cp (AIC), BIC, or adjusted R2 . In Step 2(b) of Algorithm 6.2, we must identify the best model from among those p−k that augment Mk with one additional predictor. We can do this by simply choosing the model with the lowest RSS or the highest R2 . However, in Step 3, we must identify the best model among a set of models with different numbers of variables. This is more challenging, and is discussed in Section 6.1.3. Forward stepwise selection’s computational advantage over best subset selection is clear. Though forward stepwise tends to do well in practice, it is not guaranteed to find the best possible model out of all 2p mod- els containing subsets of the p predictors. For instance, suppose that in a given data set with p = 3 predictors, the best possible one-variable model contains X1, and the best possible two-variable model instead contains X2 and X3. Then forward stepwise selection will fail to select the best possible two-variable model, because M1 will contain X1, so M2 must also contain X1 together with one additional variable. Table 6.1, which shows the first four selected models for best subset and forward stepwise selection on the Credit data set, illustrates this phe- nomenon. Both best subset selection and forward stepwise selection choose rating for the best one-variable model and then include income and student for the two- and three-variable models. However, best subset selection re- places rating by cards in the four-variable model, while forward stepwise selection must maintain rating in its four-variable model. In this example, Figure 6.1 indicates that there is not much difference between the three- and four-variable models in terms of RSS, so either of the four-variable models will likely be adequate. Forward stepwise selection can be applied even in the high-dimensional setting where n < p; however, in this case, it is possible to construct sub- models M0, . . . , Mn−1 only, since each submodel is fit using least squares, which will not yield a unique solution if p ≥ n.
- 239. 6.1 Subset Selection 231 # Variables Best subset Forward stepwise One rating rating Two rating, income rating, income Three rating, income, student rating, income, student Four cards, income rating, income, student, limit student, limit TABLE 6.1. The first four selected models for best subset selection and forward stepwise selection on the Credit data set. The first three models are identical but the fourth models differ. Backward Stepwise Selection Like forward stepwise selection, backward stepwise selection provides an ef- backward stepwise selection ficient alternative to best subset selection. However, unlike forward stepwise selection, it begins with the full least squares model containing all p predic- tors, and then iteratively removes the least useful predictor, one-at-a-time. Details are given in Algorithm 6.3. Algorithm 6.3 Backward stepwise selection 1. Let Mp denote the full model, which contains all p predictors. 2. For k = p, p − 1, . . . , 1: (a) Consider all k models that contain all but one of the predictors in Mk, for a total of k − 1 predictors. (b) Choose the best among these k models, and call it Mk−1. Here best is defined as having smallest RSS or highest R2 . 3. Select a single best model from among M0, . . . , Mp using cross- validated prediction error, Cp (AIC), BIC, or adjusted R2 . Like forward stepwise selection, the backward selection approach searches through only 1+p(p+1)/2 models, and so can be applied in settings where p is too large to apply best subset selection.2 Also like forward stepwise selection, backward stepwise selection is not guaranteed to yield the best model containing a subset of the p predictors. Backward selection requires that the number of samples n is larger than the number of variables p (so that the full model can be fit). In contrast, forward stepwise can be used even when n < p, and so is the only viable subset method when p is very large. 2Like forward stepwise selection, backward stepwise selection performs a guided search over model space, and so effectively considers substantially more than 1+p(p+1)/2 models.
- 240. 232 6. Linear Model Selection and Regularization Hybrid Approaches The best subset, forward stepwise, and backward stepwise selection ap- proaches generally give similar but not identical models. As another al- ternative, hybrid versions of forward and backward stepwise selection are available, in which variables are added to the model sequentially, in analogy to forward selection. However, after adding each new variable, the method may also remove any variables that no longer provide an improvement in the model fit. Such an approach attempts to more closely mimic best sub- set selection while retaining the computational advantages of forward and backward stepwise selection. 6.1.3 Choosing the Optimal Model Best subset selection, forward selection, and backward selection result in the creation of a set of models, each of which contains a subset of the p predictors. To apply these methods, we need a way to determine which of these models is best. As we discussed in Section 6.1.1, the model containing all of the predictors will always have the smallest RSS and the largest R2 , since these quantities are related to the training error. Instead, we wish to choose a model with a low test error. As is evident here, and as we show in Chapter 2, the training error can be a poor estimate of the test error. Therefore, RSS and R2 are not suitable for selecting the best model among a collection of models with different numbers of predictors. In order to select the best model with respect to test error, we need to estimate this test error. There are two common approaches: 1. We can indirectly estimate test error by making an adjustment to the training error to account for the bias due to overfitting. 2. We can directly estimate the test error, using either a validation set approach or a cross-validation approach, as discussed in Chapter 5. We consider both of these approaches below. Cp, AIC, BIC, and Adjusted R2 We show in Chapter 2 that the training set MSE is generally an under- estimate of the test MSE. (Recall that MSE = RSS/n.) This is because when we fit a model to the training data using least squares, we specifi- cally estimate the regression coefficients such that the training RSS (but not the test RSS) is as small as possible. In particular, the training error will decrease as more variables are included in the model, but the test error may not. Therefore, training set RSS and training set R2 cannot be used to select from among a set of models with different numbers of variables. However, a number of techniques for adjusting the training error for the model size are available. These approaches can be used to select among a set
- 241. 6.1 Subset Selection 233 2 4 6 8 10 10000 15000 20000 25000 30000 Number of Predictors C p 2 4 6 8 10 10000 15000 20000 25000 30000 Number of Predictors BIC 2 4 6 8 10 0.86 0.88 0.90 0.92 0.94 0.96 Number of Predictors Adjusted R 2 FIGURE 6.2. Cp, BIC, and adjusted R2 are shown for the best models of each size for the Credit data set (the lower frontier in Figure 6.1). Cp and BIC are estimates of test MSE. In the middle plot we see that the BIC estimate of test error shows an increase after four variables are selected. The other two plots are rather flat after four variables are included. of models with different numbers of variables. We now consider four such approaches: Cp, Akaike information criterion (AIC), Bayesian information Cp Akaike information criterion criterion (BIC), and adjusted R2 . Figure 6.2 displays Cp, BIC, and adjusted Bayesian information criterion adjusted R2 R2 for the best model of each size produced by best subset selection on the Credit data set. For a fitted least squares model containing d predictors, the Cp estimate of test MSE is computed using the equation Cp = 1 n ' RSS + 2dσ̂2 ( , (6.2) where σ̂2 is an estimate of the variance of the error ϵ associated with each response measurement in (6.1).3 Typically σ̂2 is estimated using the full model containing all predictors. Essentially, the Cp statistic adds a penalty of 2dσ̂2 to the training RSS in order to adjust for the fact that the training error tends to underestimate the test error. Clearly, the penalty increases as the number of predictors in the model increases; this is intended to adjust for the corresponding decrease in training RSS. Though it is beyond the scope of this book, one can show that if σ̂2 is an unbiased estimate of σ2 in (6.2), then Cp is an unbiased estimate of test MSE. As a consequence, the Cp statistic tends to take on a small value for models with a low test error, so when determining which of a set of models is best, we choose the model with the lowest Cp value. In Figure 6.2, Cp selects the six-variable model containing the predictors income, limit, rating, cards, age and student. 3Mallow’s Cp is sometimes defined as C′ p = RSS/σ̂2 + 2d − n. This is equivalent to the definition given above in the sense that Cp = 1 n σ̂2(C′ p + n), and so the model with smallest Cp also has smallest C′ p.
- 242. 234 6. Linear Model Selection and Regularization The AIC criterion is defined for a large class of models fit by maximum likelihood. In the case of the model (6.1) with Gaussian errors, maximum likelihood and least squares are the same thing. In this case AIC is given by AIC = 1 n ' RSS + 2dσ̂2 ( , where, for simplicity, we have omitted irrelevant constants.4 Hence for least squares models, Cp and AIC are proportional to each other, and so only Cp is displayed in Figure 6.2. BIC is derived from a Bayesian point of view, but ends up looking similar to Cp (and AIC) as well. For the least squares model with d predictors, the BIC is, up to irrelevant constants, given by BIC = 1 n ' RSS + log(n)dσ̂2 ( . (6.3) Like Cp, the BIC will tend to take on a small value for a model with a low test error, and so generally we select the model that has the lowest BIC value. Notice that BIC replaces the 2dσ̂2 used by Cp with a log(n)dσ̂2 term, where n is the number of observations. Since log n > 2 for any n > 7, the BIC statistic generally places a heavier penalty on models with many variables, and hence results in the selection of smaller models than Cp. In Figure 6.2, we see that this is indeed the case for the Credit data set; BIC chooses a model that contains only the four predictors income, limit, cards, and student. In this case the curves are very flat and so there does not appear to be much difference in accuracy between the four-variable and six-variable models. The adjusted R2 statistic is another popular approach for selecting among a set of models that contain different numbers of variables. Recall from Chapter 3 that the usual R2 is defined as 1 − RSS/TSS, where TSS = ) (yi − y)2 is the total sum of squares for the response. Since RSS always decreases as more variables are added to the model, the R2 always increases as more variables are added. For a least squares model with d variables, the adjusted R2 statistic is calculated as Adjusted R2 = 1 − RSS/(n − d − 1) TSS/(n − 1) . (6.4) Unlike Cp, AIC, and BIC, for which a small value indicates a model with a low test error, a large value of adjusted R2 indicates a model with a 4There are two formulas for AIC for least squares regression. The formula that we provide here requires an expression for σ2, which we obtain using the full model con- taining all predictors. The second formula is appropriate when σ2 is unknown and we do not want to explicitly estimate it; that formula has a log(RSS) term instead of an RSS term. Detailed derivations of these two formulas are outside of the scope of this book.
- 243. 6.1 Subset Selection 235 small test error. Maximizing the adjusted R2 is equivalent to minimizing RSS n−d−1 . While RSS always decreases as the number of variables in the model increases, RSS n−d−1 may increase or decrease, due to the presence of d in the denominator. The intuition behind the adjusted R2 is that once all of the correct variables have been included in the model, adding additional noise variables will lead to only a very small decrease in RSS. Since adding noise variables leads to an increase in d, such variables will lead to an increase in RSS n−d−1 , and consequently a decrease in the adjusted R2 . Therefore, in theory, the model with the largest adjusted R2 will have only correct variables and no noise variables. Unlike the R2 statistic, the adjusted R2 statistic pays a price for the inclusion of unnecessary variables in the model. Figure 6.2 displays the adjusted R2 for the Credit data set. Using this statistic results in the selection of a model that contains seven variables, adding own to the model selected by Cp and AIC. Cp, AIC, and BIC all have rigorous theoretical justifications that are beyond the scope of this book. These justifications rely on asymptotic ar- guments (scenarios where the sample size n is very large). Despite its pop- ularity, and even though it is quite intuitive, the adjusted R2 is not as well motivated in statistical theory as AIC, BIC, and Cp. All of these measures are simple to use and compute. Here we have presented their formulas in the case of a linear model fit using least squares; however, AIC and BIC can also be defined for more general types of models. Validation and Cross-Validation As an alternative to the approaches just discussed, we can directly esti- mate the test error using the validation set and cross-validation methods discussed in Chapter 5. We can compute the validation set error or the cross-validation error for each model under consideration, and then select the model for which the resulting estimated test error is smallest. This pro- cedure has an advantage relative to AIC, BIC, Cp, and adjusted R2 , in that it provides a direct estimate of the test error, and makes fewer assumptions about the true underlying model. It can also be used in a wider range of model selection tasks, even in cases where it is hard to pinpoint the model degrees of freedom (e.g. the number of predictors in the model) or hard to estimate the error variance σ2 . In the past, performing cross-validation was computationally prohibitive for many problems with large p and/or large n, and so AIC, BIC, Cp, and adjusted R2 were more attractive approaches for choosing among a set of models. However, nowadays with fast computers, the computations required to perform cross-validation are hardly ever an issue. Thus, cross- validation is a very attractive approach for selecting from among a number of models under consideration.
- 244. 236 6. Linear Model Selection and Regularization 2 4 6 8 10 100 120 140 160 180 200 220 Number of Predictors Square Root of BIC 2 4 6 8 10 100 120 140 160 180 200 220 Number of Predictors Validation Set Error 2 4 6 8 10 100 120 140 160 180 200 220 Number of Predictors Cross−Validation Error FIGURE 6.3. For the Credit data set, three quantities are displayed for the best model containing d predictors, for d ranging from 1 to 11. The overall best model, based on each of these quantities, is shown as a blue cross. Left: Square root of BIC. Center: Validation set errors. Right: Cross-validation errors. Figure 6.3 displays, as a function of d, the BIC, validation set errors, and cross-validation errors on the Credit data, for the best d-variable model. The validation errors were calculated by randomly selecting three-quarters of the observations as the training set, and the remainder as the valida- tion set. The cross-validation errors were computed using k = 10 folds. In this case, the validation and cross-validation methods both result in a six-variable model. However, all three approaches suggest that the four-, five-, and six-variable models are roughly equivalent in terms of their test errors. In fact, the estimated test error curves displayed in the center and right- hand panels of Figure 6.3 are quite flat. While a three-variable model clearly has lower estimated test error than a two-variable model, the estimated test errors of the 3- to 11-variable models are quite similar. Furthermore, if we repeated the validation set approach using a different split of the data into a training set and a validation set, or if we repeated cross-validation using a different set of cross-validation folds, then the precise model with the lowest estimated test error would surely change. In this setting, we can select a model using the one-standard-error rule. We first calculate the one- standard- error rule standard error of the estimated test MSE for each model size, and then select the smallest model for which the estimated test error is within one standard error of the lowest point on the curve. The rationale here is that if a set of models appear to be more or less equally good, then we might as well choose the simplest model—that is, the model with the smallest number of predictors. In this case, applying the one-standard-error rule to the validation set or cross-validation approach leads to selection of the three-variable model.
- 245. 6.2 Shrinkage Methods 237 6.2 Shrinkage Methods The subset selection methods described in Section 6.1 involve using least squares to fit a linear model that contains a subset of the predictors. As an alternative, we can fit a model containing all p predictors using a technique that constrains or regularizes the coefficient estimates, or equivalently, that shrinks the coefficient estimates towards zero. It may not be immediately obvious why such a constraint should improve the fit, but it turns out that shrinking the coefficient estimates can significantly reduce their variance. The two best-known techniques for shrinking the regression coefficients towards zero are ridge regression and the lasso. 6.2.1 Ridge Regression Recall from Chapter 3 that the least squares fitting procedure estimates β0, β1, . . . , βp using the values that minimize RSS = n 0 i=1 ⎛ ⎝yi − β0 − p 0 j=1 βjxij ⎞ ⎠ 2 . Ridge regression is very similar to least squares, except that the coefficients ridge regression are estimated by minimizing a slightly different quantity. In particular, the ridge regression coefficient estimates β̂R are the values that minimize n 0 i=1 ⎛ ⎝yi − β0 − p 0 j=1 βjxij ⎞ ⎠ 2 + λ p 0 j=1 β2 j = RSS + λ p 0 j=1 β2 j , (6.5) where λ ≥ 0 is a tuning parameter, to be determined separately. Equa- tuning parameter tion 6.5 trades off two different criteria. As with least squares, ridge regres- sion seeks coefficient estimates that fit the data well, by making the RSS small. However, the second term, λ ) j β2 j , called a shrinkage penalty, is shrinkage penalty small when β1, . . . , βp are close to zero, and so it has the effect of shrinking the estimates of βj towards zero. The tuning parameter λ serves to control the relative impact of these two terms on the regression coefficient esti- mates. When λ = 0, the penalty term has no effect, and ridge regression will produce the least squares estimates. However, as λ → ∞, the impact of the shrinkage penalty grows, and the ridge regression coefficient estimates will approach zero. Unlike least squares, which generates only one set of co- efficient estimates, ridge regression will produce a different set of coefficient estimates, β̂R λ , for each value of λ. Selecting a good value for λ is critical; we defer this discussion to Section 6.2.3, where we use cross-validation. Note that in (6.5), the shrinkage penalty is applied to β1, . . . , βp, but not to the intercept β0. We want to shrink the estimated association of
- 246. 238 6. Linear Model Selection and Regularization 1e−02 1e+00 1e+02 1e+04 −300 −100 0 100 200 300 400 Standardized Coefficients Income Limit Rating Student 0.0 0.2 0.4 0.6 0.8 1.0 −300 −100 0 100 200 300 400 Standardized Coefficients λ ∥β̂R λ ∥2/∥β̂∥2 FIGURE 6.4. The standardized ridge regression coefficients are displayed for the Credit data set, as a function of λ and ∥β̂R λ ∥2/∥β̂∥2. each variable with the response; however, we do not want to shrink the intercept, which is simply a measure of the mean value of the response when xi1 = xi2 = . . . = xip = 0. If we assume that the variables—that is, the columns of the data matrix X—have been centered to have mean zero before ridge regression is performed, then the estimated intercept will take the form β̂0 = ȳ = )n i=1 yi/n. An Application to the Credit Data In Figure 6.4, the ridge regression coefficient estimates for the Credit data set are displayed. In the left-hand panel, each curve corresponds to the ridge regression coefficient estimate for one of the ten variables, plotted as a function of λ. For example, the black solid line represents the ridge regression estimate for the income coefficient, as λ is varied. At the extreme left-hand side of the plot, λ is essentially zero, and so the corresponding ridge coefficient estimates are the same as the usual least squares esti- mates. But as λ increases, the ridge coefficient estimates shrink towards zero. When λ is extremely large, then all of the ridge coefficient estimates are basically zero; this corresponds to the null model that contains no pre- dictors. In this plot, the income, limit, rating, and student variables are displayed in distinct colors, since these variables tend to have by far the largest coefficient estimates. While the ridge coefficient estimates tend to decrease in aggregate as λ increases, individual coefficients, such as rating and income, may occasionally increase as λ increases. The right-hand panel of Figure 6.4 displays the same ridge coefficient estimates as the left-hand panel, but instead of displaying λ on the x-axis, we now display ∥β̂R λ ∥2/∥β̂∥2, where β̂ denotes the vector of least squares coefficient estimates. The notation ∥β∥2 denotes the ℓ2 norm (pronounced ℓ2 norm “ell 2”) of a vector, and is defined as ∥β∥2 = G)p j=1 βj 2 . It measures
- 247. 6.2 Shrinkage Methods 239 the distance of β from zero. As λ increases, the ℓ2 norm of β̂R λ will always decrease, and so will ∥β̂R λ ∥2/∥β̂∥2. The latter quantity ranges from 1 (when λ = 0, in which case the ridge regression coefficient estimate is the same as the least squares estimate, and so their ℓ2 norms are the same) to 0 (when λ = ∞, in which case the ridge regression coefficient estimate is a vector of zeros, with ℓ2 norm equal to zero). Therefore, we can think of the x-axis in the right-hand panel of Figure 6.4 as the amount that the ridge regression coefficient estimates have been shrunken towards zero; a small value indicates that they have been shrunken very close to zero. The standard least squares coefficient estimates discussed in Chapter 3 are scale equivariant: multiplying Xj by a constant c simply leads to a scale equivariant scaling of the least squares coefficient estimates by a factor of 1/c. In other words, regardless of how the jth predictor is scaled, Xjβ̂j will remain the same. In contrast, the ridge regression coefficient estimates can change sub- stantially when multiplying a given predictor by a constant. For instance, consider the income variable, which is measured in dollars. One could rea- sonably have measured income in thousands of dollars, which would result in a reduction in the observed values of income by a factor of 1,000. Now due to the sum of squared coefficients term in the ridge regression formulation (6.5), such a change in scale will not simply cause the ridge regression co- efficient estimate for income to change by a factor of 1,000. In other words, Xjβ̂R j,λ will depend not only on the value of λ, but also on the scaling of the jth predictor. In fact, the value of Xjβ̂R j,λ may even depend on the scaling of the other predictors! Therefore, it is best to apply ridge regression after standardizing the predictors, using the formula x̃ij = xij G 1 n )n i=1(xij − xj)2 , (6.6) so that they are all on the same scale. In (6.6), the denominator is the estimated standard deviation of the jth predictor. Consequently, all of the standardized predictors will have a standard deviation of one. As a re- sult the final fit will not depend on the scale on which the predictors are measured. In Figure 6.4, the y-axis displays the standardized ridge regres- sion coefficient estimates—that is, the coefficient estimates that result from performing ridge regression using standardized predictors. Why Does Ridge Regression Improve Over Least Squares? Ridge regression’s advantage over least squares is rooted in the bias-variance trade-off. As λ increases, the flexibility of the ridge regression fit decreases, leading to decreased variance but increased bias. This is illustrated in the left-hand panel of Figure 6.5, using a simulated data set containing p = 45 predictors and n = 50 observations. The green curve in the left-hand panel of Figure 6.5 displays the variance of the ridge regression predictions as a
- 248. 240 6. Linear Model Selection and Regularization 1e−01 1e+01 1e+03 0 10 20 30 40 50 60 Mean Squared Error 0.0 0.2 0.4 0.6 0.8 1.0 0 10 20 30 40 50 60 Mean Squared Error λ ∥β̂R λ ∥2/∥β̂∥2 FIGURE 6.5. Squared bias (black), variance (green), and test mean squared error (purple) for the ridge regression predictions on a simulated data set, as a function of λ and ∥β̂R λ ∥2/∥β̂∥2. The horizontal dashed lines indicate the minimum possible MSE. The purple crosses indicate the ridge regression models for which the MSE is smallest. function of λ. At the least squares coefficient estimates, which correspond to ridge regression with λ = 0, the variance is high but there is no bias. But as λ increases, the shrinkage of the ridge coefficient estimates leads to a substantial reduction in the variance of the predictions, at the expense of a slight increase in bias. Recall that the test mean squared error (MSE), plot- ted in purple, is closely related to the variance plus the squared bias. For values of λ up to about 10, the variance decreases rapidly, with very little increase in bias, plotted in black. Consequently, the MSE drops consider- ably as λ increases from 0 to 10. Beyond this point, the decrease in variance due to increasing λ slows, and the shrinkage on the coefficients causes them to be significantly underestimated, resulting in a large increase in the bias. The minimum MSE is achieved at approximately λ = 30. Interestingly, because of its high variance, the MSE associated with the least squares fit, when λ = 0, is almost as high as that of the null model for which all coefficient estimates are zero, when λ = ∞. However, for an intermediate value of λ, the MSE is considerably lower. The right-hand panel of Figure 6.5 displays the same curves as the left- hand panel, this time plotted against the ℓ2 norm of the ridge regression coefficient estimates divided by the ℓ2 norm of the least squares estimates. Now as we move from left to right, the fits become more flexible, and so the bias decreases and the variance increases. In general, in situations where the relationship between the response and the predictors is close to linear, the least squares estimates will have low bias but may have high variance. This means that a small change in the training data can cause a large change in the least squares coefficient estimates. In particular, when the number of variables p is almost as large as the number of observations n, as in the example in Figure 6.5, the least squares estimates will be extremely variable. And if p > n, then the
- 249. 6.2 Shrinkage Methods 241 least squares estimates do not even have a unique solution, whereas ridge regression can still perform well by trading off a small increase in bias for a large decrease in variance. Hence, ridge regression works best in situations where the least squares estimates have high variance. Ridge regression also has substantial computational advantages over best subset selection, which requires searching through 2p models. As we dis- cussed previously, even for moderate values of p, such a search can be computationally infeasible. In contrast, for any fixed value of λ, ridge re- gression only fits a single model, and the model-fitting procedure can be performed quite quickly. In fact, one can show that the computations re- quired to solve (6.5), simultaneously for all values of λ, are almost identical to those for fitting a model using least squares. 6.2.2 The Lasso Ridge regression does have one obvious disadvantage. Unlike best subset, forward stepwise, and backward stepwise selection, which will generally select models that involve just a subset of the variables, ridge regression will include all p predictors in the final model. The penalty λ ) β2 j in (6.5) will shrink all of the coefficients towards zero, but it will not set any of them exactly to zero (unless λ = ∞). This may not be a problem for prediction accuracy, but it can create a challenge in model interpretation in settings in which the number of variables p is quite large. For example, in the Credit data set, it appears that the most important variables are income, limit, rating, and student. So we might wish to build a model including just these predictors. However, ridge regression will always generate a model involving all ten predictors. Increasing the value of λ will tend to reduce the magnitudes of the coefficients, but will not result in exclusion of any of the variables. The lasso is a relatively recent alternative to ridge regression that over- lasso comes this disadvantage. The lasso coefficients, β̂L λ , minimize the quantity n 0 i=1 ⎛ ⎝yi − β0 − p 0 j=1 βjxij ⎞ ⎠ 2 + λ p 0 j=1 |βj| = RSS + λ p 0 j=1 |βj|. (6.7) Comparing (6.7) to (6.5), we see that the lasso and ridge regression have similar formulations. The only difference is that the β2 j term in the ridge regression penalty (6.5) has been replaced by |βj| in the lasso penalty (6.7). In statistical parlance, the lasso uses an ℓ1 (pronounced “ell 1”) penalty instead of an ℓ2 penalty. The ℓ1 norm of a coefficient vector β is given by ∥β∥1 = ) |βj|. As with ridge regression, the lasso shrinks the coefficient estimates to- wards zero. However, in the case of the lasso, the ℓ1 penalty has the effect of forcing some of the coefficient estimates to be exactly equal to zero when
- 250. 242 6. Linear Model Selection and Regularization the tuning parameter λ is sufficiently large. Hence, much like best subset se- lection, the lasso performs variable selection. As a result, models generated from the lasso are generally much easier to interpret than those produced by ridge regression. We say that the lasso yields sparse models—that is, sparse models that involve only a subset of the variables. As in ridge regression, selecting a good value of λ for the lasso is critical; we defer this discussion to Section 6.2.3, where we use cross-validation. As an example, consider the coefficient plots in Figure 6.6, which are gen- erated from applying the lasso to the Credit data set. When λ = 0, then the lasso simply gives the least squares fit, and when λ becomes sufficiently large, the lasso gives the null model in which all coefficient estimates equal zero. However, in between these two extremes, the ridge regression and lasso models are quite different from each other. Moving from left to right in the right-hand panel of Figure 6.6, we observe that at first the lasso re- sults in a model that contains only the rating predictor. Then student and limit enter the model almost simultaneously, shortly followed by income. Eventually, the remaining variables enter the model. Hence, depending on the value of λ, the lasso can produce a model involving any number of vari- ables. In contrast, ridge regression will always include all of the variables in the model, although the magnitude of the coefficient estimates will depend on λ. 20 50 100 200 500 2000 5000 −200 0 100 200 300 400 Standardized Coefficients 0.0 0.2 0.4 0.6 0.8 1.0 −300 −100 0 100 200 300 400 Standardized Coefficients Income Limit Rating Student λ ∥β̂L λ ∥1/∥β̂∥1 FIGURE 6.6. The standardized lasso coefficients on the Credit data set are shown as a function of λ and ∥β̂L λ ∥1/∥β̂∥1.
- 251. 6.2 Shrinkage Methods 243 Another Formulation for Ridge Regression and the Lasso One can show that the lasso and ridge regression coefficient estimates solve the problems minimize β ⎧ ⎪ ⎨ ⎪ ⎩ n 0 i=1 ⎛ ⎝yi − β0 − p 0 j=1 βjxij ⎞ ⎠ 2 ⎫ ⎪ ⎬ ⎪ ⎭ subject to p 0 j=1 |βj| ≤ s (6.8) and minimize β ⎧ ⎪ ⎨ ⎪ ⎩ n 0 i=1 ⎛ ⎝yi − β0 − p 0 j=1 βjxij ⎞ ⎠ 2 ⎫ ⎪ ⎬ ⎪ ⎭ subject to p 0 j=1 β2 j ≤ s, (6.9) respectively. In other words, for every value of λ, there is some s such that the Equations (6.7) and (6.8) will give the same lasso coefficient estimates. Similarly, for every value of λ there is a corresponding s such that Equa- tions (6.5) and (6.9) will give the same ridge regression coefficient estimates. When p = 2, then (6.8) indicates that the lasso coefficient estimates have the smallest RSS out of all points that lie within the diamond defined by |β1| + |β2| ≤ s. Similarly, the ridge regression estimates have the smallest RSS out of all points that lie within the circle defined by β2 1 + β2 2 ≤ s. We can think of (6.8) as follows. When we perform the lasso we are trying to find the set of coefficient estimates that lead to the smallest RSS, subject to the constraint that there is a budget s for how large )p j=1 |βj| can be. When s is extremely large, then this budget is not very restrictive, and so the coefficient estimates can be large. In fact, if s is large enough that the least squares solution falls within the budget, then (6.8) will simply yield the least squares solution. In contrast, if s is small, then )p j=1 |βj| must be small in order to avoid violating the budget. Similarly, (6.9) indicates that when we perform ridge regression, we seek a set of coefficient estimates such that the RSS is as small as possible, subject to the requirement that )p j=1 β2 j not exceed the budget s. The formulations (6.8) and (6.9) reveal a close connection between the lasso, ridge regression, and best subset selection. Consider the problem minimize β ⎧ ⎪ ⎨ ⎪ ⎩ n 0 i=1 ⎛ ⎝yi − β0 − p 0 j=1 βjxij ⎞ ⎠ 2 ⎫ ⎪ ⎬ ⎪ ⎭ subject to p 0 j=1 I(βj ̸= 0) ≤ s. (6.10) Here I(βj ̸= 0) is an indicator variable: it takes on a value of 1 if βj ̸= 0, and equals zero otherwise. Then (6.10) amounts to finding a set of coefficient estimates such that RSS is as small as possible, subject to the constraint that no more than s coefficients can be nonzero. The problem (6.10) is
- 252. 244 6. Linear Model Selection and Regularization FIGURE 6.7. Contours of the error and constraint functions for the lasso (left) and ridge regression (right). The solid blue areas are the constraint re- gions, |β1| + |β2| ≤ s and β2 1 + β2 2 ≤ s, while the red ellipses are the contours of the RSS. equivalent to best subset selection. Unfortunately, solving (6.10) is com- putationally infeasible when p is large, since it requires considering all 'p s ( models containing s predictors. Therefore, we can interpret ridge regression and the lasso as computationally feasible alternatives to best subset selec- tion that replace the intractable form of the budget in (6.10) with forms that are much easier to solve. Of course, the lasso is much more closely related to best subset selection, since the lasso performs feature selection for s sufficiently small in (6.8), while ridge regression does not. The Variable Selection Property of the Lasso Why is it that the lasso, unlike ridge regression, results in coefficient esti- mates that are exactly equal to zero? The formulations (6.8) and (6.9) can be used to shed light on the issue. Figure 6.7 illustrates the situation. The least squares solution is marked as β̂, while the blue diamond and circle represent the lasso and ridge regression constraints in (6.8) and (6.9), re- spectively. If s is sufficiently large, then the constraint regions will contain β̂, and so the ridge regression and lasso estimates will be the same as the least squares estimates. (Such a large value of s corresponds to λ = 0 in (6.5) and (6.7).) However, in Figure 6.7 the least squares estimates lie out- side of the diamond and the circle, and so the least squares estimates are not the same as the lasso and ridge regression estimates. Each of the ellipses centered around β̂ represents a contour: this means contour that all of the points on a particular ellipse have the same RSS value. As
- 253. 6.2 Shrinkage Methods 245 0.02 0.10 0.50 2.00 10.00 50.00 0 10 20 30 40 50 60 Mean Squared Error 0.0 0.2 0.4 0.6 0.8 1.0 0 10 20 30 40 50 60 R2 on Training Data Mean Squared Error λ FIGURE 6.8. Left: Plots of squared bias (black), variance (green), and test MSE (purple) for the lasso on a simulated data set. Right: Comparison of squared bias, variance, and test MSE between lasso (solid) and ridge (dotted). Both are plotted against their R2 on the training data, as a common form of indexing. The crosses in both plots indicate the lasso model for which the MSE is smallest. the ellipses expand away from the least squares coefficient estimates, the RSS increases. Equations (6.8) and (6.9) indicate that the lasso and ridge regression coefficient estimates are given by the first point at which an ellipse contacts the constraint region. Since ridge regression has a circular constraint with no sharp points, this intersection will not generally occur on an axis, and so the ridge regression coefficient estimates will be exclusively non-zero. However, the lasso constraint has corners at each of the axes, and so the ellipse will often intersect the constraint region at an axis. When this occurs, one of the coefficients will equal zero. In higher dimensions, many of the coefficient estimates may equal zero simultaneously. In Figure 6.7, the intersection occurs at β1 = 0, and so the resulting model will only include β2. In Figure 6.7, we considered the simple case of p = 2. When p = 3, then the constraint region for ridge regression becomes a sphere, and the constraint region for the lasso becomes a polyhedron. When p > 3, the constraint for ridge regression becomes a hypersphere, and the constraint for the lasso becomes a polytope. However, the key ideas depicted in Fig- ure 6.7 still hold. In particular, the lasso leads to feature selection when p > 2 due to the sharp corners of the polyhedron or polytope. Comparing the Lasso and Ridge Regression It is clear that the lasso has a major advantage over ridge regression, in that it produces simpler and more interpretable models that involve only a subset of the predictors. However, which method leads to better prediction accuracy? Figure 6.8 displays the variance, squared bias, and test MSE of the lasso applied to the same simulated data as in Figure 6.5. Clearly the lasso leads to qualitatively similar behavior to ridge regression, in that as λ
- 254. 246 6. Linear Model Selection and Regularization 0.02 0.10 0.50 2.00 10.00 50.00 0 20 40 60 80 100 Mean Squared Error 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0 20 40 60 80 100 R2 on Training Data Mean Squared Error λ FIGURE 6.9. Left: Plots of squared bias (black), variance (green), and test MSE (purple) for the lasso. The simulated data is similar to that in Figure 6.8, except that now only two predictors are related to the response. Right: Comparison of squared bias, variance, and test MSE between lasso (solid) and ridge (dotted). Both are plotted against their R2 on the training data, as a common form of indexing. The crosses in both plots indicate the lasso model for which the MSE is smallest. increases, the variance decreases and the bias increases. In the right-hand panel of Figure 6.8, the dotted lines represent the ridge regression fits. Here we plot both against their R2 on the training data. This is another useful way to index models, and can be used to compare models with different types of regularization, as is the case here. In this example, the lasso and ridge regression result in almost identical biases. However, the variance of ridge regression is slightly lower than the variance of the lasso. Consequently, the minimum MSE of ridge regression is slightly smaller than that of the lasso. However, the data in Figure 6.8 were generated in such a way that all 45 predictors were related to the response—that is, none of the true coefficients β1, . . . , β45 equaled zero. The lasso implicitly assumes that a number of the coefficients truly equal zero. Consequently, it is not surprising that ridge regression outperforms the lasso in terms of prediction error in this setting. Figure 6.9 illustrates a similar situation, except that now the response is a function of only 2 out of 45 predictors. Now the lasso tends to outperform ridge regression in terms of bias, variance, and MSE. These two examples illustrate that neither ridge regression nor the lasso will universally dominate the other. In general, one might expect the lasso to perform better in a setting where a relatively small number of predictors have substantial coefficients, and the remaining predictors have coefficients that are very small or that equal zero. Ridge regression will perform better when the response is a function of many predictors, all with coefficients of roughly equal size. However, the number of predictors that is related to the response is never known a priori for real data sets. A technique such as
- 255. 6.2 Shrinkage Methods 247 cross-validation can be used in order to determine which approach is better on a particular data set. As with ridge regression, when the least squares estimates have exces- sively high variance, the lasso solution can yield a reduction in variance at the expense of a small increase in bias, and consequently can gener- ate more accurate predictions. Unlike ridge regression, the lasso performs variable selection, and hence results in models that are easier to interpret. There are very efficient algorithms for fitting both ridge and lasso models; in both cases the entire coefficient paths can be computed with about the same amount of work as a single least squares fit. We will explore this further in the lab at the end of this chapter. A Simple Special Case for Ridge Regression and the Lasso In order to obtain a better intuition about the behavior of ridge regression and the lasso, consider a simple special case with n = p, and X a diag- onal matrix with 1’s on the diagonal and 0’s in all off-diagonal elements. To simplify the problem further, assume also that we are performing regres- sion without an intercept. With these assumptions, the usual least squares problem simplifies to finding β1, . . . , βp that minimize p 0 j=1 (yj − βj)2 . (6.11) In this case, the least squares solution is given by β̂j = yj. And in this setting, ridge regression amounts to finding β1, . . . , βp such that p 0 j=1 (yj − βj)2 + λ p 0 j=1 β2 j (6.12) is minimized, and the lasso amounts to finding the coefficients such that p 0 j=1 (yj − βj)2 + λ p 0 j=1 |βj| (6.13) is minimized. One can show that in this setting, the ridge regression esti- mates take the form β̂R j = yj/(1 + λ), (6.14) and the lasso estimates take the form β̂L j = ⎧ ⎪ ⎨ ⎪ ⎩ yj − λ/2 if yj > λ/2; yj + λ/2 if yj < −λ/2; 0 if |yj| ≤ λ/2. (6.15)
- 256. 248 6. Linear Model Selection and Regularization −1.5 −0.5 0.0 0.5 1.0 1.5 −1.5 −0.5 0.5 1.5 Coefficient Estimate Ridge Least Squares −1.5 −0.5 0.0 0.5 1.0 1.5 −1.5 −0.5 0.5 1.5 Coefficient Estimate Lasso Least Squares yj yj FIGURE 6.10. The ridge regression and lasso coefficient estimates for a simple setting with n = p and X a diagonal matrix with 1’s on the diagonal. Left: The ridge regression coefficient estimates are shrunken proportionally towards zero, relative to the least squares estimates. Right: The lasso coefficient estimates are soft-thresholded towards zero. Figure 6.10 displays the situation. We can see that ridge regression and the lasso perform two very different types of shrinkage. In ridge regression, each least squares coefficient estimate is shrunken by the same proportion. In contrast, the lasso shrinks each least squares coefficient towards zero by a constant amount, λ/2; the least squares coefficients that are less than λ/2 in absolute value are shrunken entirely to zero. The type of shrink- age performed by the lasso in this simple setting (6.15) is known as soft- thresholding. The fact that some lasso coefficients are shrunken entirely to soft- thresholding zero explains why the lasso performs feature selection. In the case of a more general data matrix X, the story is a little more complicated than what is depicted in Figure 6.10, but the main ideas still hold approximately: ridge regression more or less shrinks every dimension of the data by the same proportion, whereas the lasso more or less shrinks all coefficients toward zero by a similar amount, and sufficiently small co- efficients are shrunken all the way to zero. Bayesian Interpretation for Ridge Regression and the Lasso We now show that one can view ridge regression and the lasso through a Bayesian lens. A Bayesian viewpoint for regression assumes that the coefficient vector β has some prior distribution, say p(β), where β = (β0, β1, . . . , βp)T . The likelihood of the data can be written as f(Y |X, β), where X = (X1, . . . , Xp). Multiplying the prior distribution by the likeli- hood gives us (up to a proportionality constant) the posterior distribution, posterior distribution which takes the form p(β|X, Y ) ∝ f(Y |X, β)p(β|X) = f(Y |X, β)p(β),
- 257. 6.2 Shrinkage Methods 249 −3 −2 −1 0 1 2 3 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 −3 −2 −1 0 1 2 3 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 βj βj g(β j ) g(β j ) FIGURE 6.11. Left: Ridge regression is the posterior mode for β under a Gaus- sian prior. Right: The lasso is the posterior mode for β under a double-exponential prior. where the proportionality above follows from Bayes’ theorem, and the equality above follows from the assumption that X is fixed. We assume the usual linear model, Y = β0 + X1β1 + · · · + Xpβp + ϵ, and suppose that the errors are independent and drawn from a normal dis- tribution. Furthermore, assume that p(β) = Fp j=1 g(βj), for some density function g. It turns out that ridge regression and the lasso follow naturally from two special cases of g: • If g is a Gaussian distribution with mean zero and standard deviation a function of λ, then it follows that the posterior mode for β—that posterior mode is, the most likely value for β, given the data—is given by the ridge regression solution. (In fact, the ridge regression solution is also the posterior mean.) • If g is a double-exponential (Laplace) distribution with mean zero and scale parameter a function of λ, then it follows that the posterior mode for β is the lasso solution. (However, the lasso solution is not the posterior mean, and in fact, the posterior mean does not yield a sparse coefficient vector.) The Gaussian and double-exponential priors are displayed in Figure 6.11. Therefore, from a Bayesian viewpoint, ridge regression and the lasso follow directly from assuming the usual linear model with normal errors, together with a simple prior distribution for β. Notice that the lasso prior is steeply peaked at zero, while the Gaussian is flatter and fatter at zero. Hence, the lasso expects a priori that many of the coefficients are (exactly) zero, while ridge assumes the coefficients are randomly distributed about zero.
- 258. 250 6. Linear Model Selection and Regularization 5e−03 5e−02 5e−01 5e+00 25.0 25.2 25.4 25.6 Cross−Validation Error 5e−03 5e−02 5e−01 5e+00 −300 −100 0 100 300 Standardized Coefficients λ λ FIGURE 6.12. Left: Cross-validation errors that result from applying ridge regression to the Credit data set with various values of λ. Right: The coefficient estimates as a function of λ. The vertical dashed lines indicate the value of λ selected by cross-validation. 6.2.3 Selecting the Tuning Parameter Just as the subset selection approaches considered in Section 6.1 require a method to determine which of the models under consideration is best, implementing ridge regression and the lasso requires a method for selecting a value for the tuning parameter λ in (6.5) and (6.7), or equivalently, the value of the constraint s in (6.9) and (6.8). Cross-validation provides a sim- ple way to tackle this problem. We choose a grid of λ values, and compute the cross-validation error for each value of λ, as described in Chapter 5. We then select the tuning parameter value for which the cross-validation error is smallest. Finally, the model is re-fit using all of the available observations and the selected value of the tuning parameter. Figure 6.12 displays the choice of λ that results from performing leave- one-out cross-validation on the ridge regression fits from the Credit data set. The dashed vertical lines indicate the selected value of λ. In this case the value is relatively small, indicating that the optimal fit only involves a small amount of shrinkage relative to the least squares solution. In addition, the dip is not very pronounced, so there is rather a wide range of values that would give a very similar error. In a case like this we might simply use the least squares solution. Figure 6.13 provides an illustration of ten-fold cross-validation applied to the lasso fits on the sparse simulated data from Figure 6.9. The left-hand panel of Figure 6.13 displays the cross-validation error, while the right-hand panel displays the coefficient estimates. The vertical dashed lines indicate the point at which the cross-validation error is smallest. The two colored lines in the right-hand panel of Figure 6.13 represent the two predictors that are related to the response, while the grey lines represent the unre- lated predictors; these are often referred to as signal and noise variables, signal respectively. Not only has the lasso correctly given much larger coeffi-
- 259. 6.3 Dimension Reduction Methods 251 0.0 0.2 0.4 0.6 0.8 1.0 0 200 600 1000 1400 Cross−Validation Error 0.0 0.2 0.4 0.6 0.8 1.0 −5 0 5 10 15 Standardized Coefficients ∥β̂L λ ∥1/∥β̂∥1 ∥β̂L λ ∥1/∥β̂∥1 FIGURE 6.13. Left: Ten-fold cross-validation MSE for the lasso, applied to the sparse simulated data set from Figure 6.9. Right: The corresponding lasso coefficient estimates are displayed. The two signal variables are shown in color, and the noise variables are in gray. The vertical dashed lines indicate the lasso fit for which the cross-validation error is smallest. cient estimates to the two signal predictors, but also the minimum cross- validation error corresponds to a set of coefficient estimates for which only the signal variables are non-zero. Hence cross-validation together with the lasso has correctly identified the two signal variables in the model, even though this is a challenging setting, with p = 45 variables and only n = 50 observations. In contrast, the least squares solution—displayed on the far right of the right-hand panel of Figure 6.13—assigns a large coefficient estimate to only one of the two signal variables. 6.3 Dimension Reduction Methods The methods that we have discussed so far in this chapter have controlled variance in two different ways, either by using a subset of the original vari- ables, or by shrinking their coefficients toward zero. All of these methods are defined using the original predictors, X1, X2, . . . , Xp. We now explore a class of approaches that transform the predictors and then fit a least squares model using the transformed variables. We will refer to these tech- niques as dimension reduction methods. dimension reduction Let Z1, Z2, . . . , ZM represent M < p linear combinations of our original linear combination p predictors. That is, Zm = p 0 j=1 φjmXj (6.16)
- 260. 252 6. Linear Model Selection and Regularization for some constants φ1m, φ2m . . . , φpm, m = 1, . . . , M. We can then fit the linear regression model yi = θ0 + M 0 m=1 θmzim + ϵi, i = 1, . . . , n, (6.17) using least squares. Note that in (6.17), the regression coefficients are given by θ0, θ1, . . . , θM . If the constants φ1m, φ2m, . . . , φpm are chosen wisely, then such dimension reduction approaches can often outperform least squares regression. In other words, fitting (6.17) using least squares can lead to better results than fitting (6.1) using least squares. The term dimension reduction comes from the fact that this approach reduces the problem of estimating the p+1 coefficients β0, β1, . . . , βp to the simpler problem of estimating the M + 1 coefficients θ0, θ1, . . . , θM , where M < p. In other words, the dimension of the problem has been reduced from p + 1 to M + 1. Notice that from (6.16), M 0 m=1 θmzim = M 0 m=1 θm p 0 j=1 φjmxij = p 0 j=1 M 0 m=1 θmφjmxij = p 0 j=1 βjxij, where βj = M 0 m=1 θmφjm. (6.18) Hence (6.17) can be thought of as a special case of the original linear regression model given by (6.1). Dimension reduction serves to constrain the estimated βj coefficients, since now they must take the form (6.18). This constraint on the form of the coefficients has the potential to bias the coefficient estimates. However, in situations where p is large relative to n, selecting a value of M ≪ p can significantly reduce the variance of the fitted coefficients. If M = p, and all the Zm are linearly independent, then (6.18) poses no constraints. In this case, no dimension reduction occurs, and so fitting (6.17) is equivalent to performing least squares on the original p predictors. All dimension reduction methods work in two steps. First, the trans- formed predictors Z1, Z2, . . . , ZM are obtained. Second, the model is fit using these M predictors. However, the choice of Z1, Z2, . . . , ZM , or equiv- alently, the selection of the φjm’s, can be achieved in different ways. In this chapter, we will consider two approaches for this task: principal components and partial least squares. 6.3.1 Principal Components Regression Principal components analysis (PCA) is a popular approach for deriving principal components analysis
- 261. 6.3 Dimension Reduction Methods 253 10 20 30 40 50 60 70 0 5 10 15 20 25 30 35 Population Ad Spending FIGURE 6.14. The population size (pop) and ad spending (ad) for 100 different cities are shown as purple circles. The green solid line indicates the first principal component, and the blue dashed line indicates the second principal component. a low-dimensional set of features from a large set of variables. PCA is discussed in greater detail as a tool for unsupervised learning in Chapter 12. Here we describe its use as a dimension reduction technique for regression. An Overview of Principal Components Analysis PCA is a technique for reducing the dimension of an n × p data matrix X. The first principal component direction of the data is that along which the observations vary the most. For instance, consider Figure 6.14, which shows population size (pop) in tens of thousands of people, and ad spending for a particular company (ad) in thousands of dollars, for 100 cities5 . The green solid line represents the first principal component direction of the data. We can see by eye that this is the direction along which there is the greatest variability in the data. That is, if we projected the 100 observations onto this line (as shown in the left-hand panel of Figure 6.15), then the resulting projected observations would have the largest possible variance; projecting the observations onto any other line would yield projected observations with lower variance. Projecting a point onto a line simply involves finding the location on the line which is closest to the point. The first principal component is displayed graphically in Figure 6.14, but how can it be summarized mathematically? It is given by the formula Z1 = 0.839 × (pop − pop) + 0.544 × (ad − ad). (6.19) 5This dataset is distinct from the Advertising data discussed in Chapter 3.
- 262. 254 6. Linear Model Selection and Regularization 20 30 40 50 5 10 15 20 25 30 Population Ad Spending −20 −10 0 10 20 −10 −5 0 5 10 1st Principal Component 2nd Principal Component FIGURE 6.15. A subset of the advertising data. The mean pop and ad budgets are indicated with a blue circle. Left: The first principal component direction is shown in green. It is the dimension along which the data vary the most, and it also defines the line that is closest to all n of the observations. The distances from each observation to the principal component are represented using the black dashed line segments. The blue dot represents (pop, ad). Right: The left-hand panel has been rotated so that the first principal component direction coincides with the x-axis. Here φ11 = 0.839 and φ21 = 0.544 are the principal component loadings, which define the direction referred to above. In (6.19), pop indicates the mean of all pop values in this data set, and ad indicates the mean of all ad- vertising spending. The idea is that out of every possible linear combination of pop and ad such that φ2 11 + φ2 21 = 1, this particular linear combination yields the highest variance: i.e. this is the linear combination for which Var(φ11 × (pop − pop) + φ21 × (ad − ad)) is maximized. It is necessary to consider only linear combinations of the form φ2 11 +φ2 21 = 1, since otherwise we could increase φ11 and φ21 arbitrarily in order to blow up the variance. In (6.19), the two loadings are both positive and have similar size, and so Z1 is almost an average of the two variables. Since n = 100, pop and ad are vectors of length 100, and so is Z1 in (6.19). For instance, zi1 = 0.839 × (popi − pop) + 0.544 × (adi − ad). (6.20) The values of z11, . . . , zn1 are known as the principal component scores, and can be seen in the right-hand panel of Figure 6.15. There is also another interpretation for PCA: the first principal compo- nent vector defines the line that is as close as possible to the data. For instance, in Figure 6.14, the first principal component line minimizes the sum of the squared perpendicular distances between each point and the line. These distances are plotted as dashed line segments in the left-hand panel of Figure 6.15, in which the crosses represent the projection of each point onto the first principal component line. The first principal component has been chosen so that the projected observations are as close as possible to the original observations.
- 263. 6.3 Dimension Reduction Methods 255 −3 −2 −1 0 1 2 3 20 30 40 50 60 1st Principal Component Population −3 −2 −1 0 1 2 3 5 10 15 20 25 30 1st Principal Component Ad Spending FIGURE 6.16. Plots of the first principal component scores zi1 versus pop and ad. The relationships are strong. In the right-hand panel of Figure 6.15, the left-hand panel has been rotated so that the first principal component direction coincides with the x-axis. It is possible to show that the first principal component score for the ith observation, given in (6.20), is the distance in the x-direction of the ith cross from zero. So for example, the point in the bottom-left corner of the left-hand panel of Figure 6.15 has a large negative principal component score, zi1 = −26.1, while the point in the top-right corner has a large positive score, zi1 = 18.7. These scores can be computed directly using (6.20). We can think of the values of the principal component Z1 as single- number summaries of the joint pop and ad budgets for each location. In this example, if zi1 = 0.839 × (popi − pop) + 0.544 × (adi − ad) < 0, then this indicates a city with below-average population size and below- average ad spending. A positive score suggests the opposite. How well can a single number represent both pop and ad? In this case, Figure 6.14 indicates that pop and ad have approximately a linear relationship, and so we might expect that a single-number summary will work well. Figure 6.16 displays zi1 versus both pop and ad.6 The plots show a strong relationship between the first principal component and the two features. In other words, the first principal component appears to capture most of the information contained in the pop and ad predictors. So far we have concentrated on the first principal component. In gen- eral, one can construct up to p distinct principal components. The second principal component Z2 is a linear combination of the variables that is un- correlated with Z1, and has largest variance subject to this constraint. The second principal component direction is illustrated as a dashed blue line in 6The principal components were calculated after first standardizing both pop and ad, a common approach. Hence, the x-axes on Figures 6.15 and 6.16 are not on the same scale.
- 264. 256 6. Linear Model Selection and Regularization −1.0 −0.5 0.0 0.5 1.0 20 30 40 50 60 2nd Principal Component Population −1.0 −0.5 0.0 0.5 1.0 5 10 15 20 25 30 2nd Principal Component Ad Spending FIGURE 6.17. Plots of the second principal component scores zi2 versus pop and ad. The relationships are weak. Figure 6.14. It turns out that the zero correlation condition of Z1 with Z2 is equivalent to the condition that the direction must be perpendicular, or perpen- dicular orthogonal, to the first principal component direction. The second principal orthogonal component is given by the formula Z2 = 0.544 × (pop − pop) − 0.839 × (ad − ad). Since the advertising data has two predictors, the first two principal com- ponents contain all of the information that is in pop and ad. However, by construction, the first component will contain the most information. Con- sider, for example, the much larger variability of zi1 (the x-axis) versus zi2 (the y-axis) in the right-hand panel of Figure 6.15. The fact that the second principal component scores are much closer to zero indicates that this component captures far less information. As another illustration, Fig- ure 6.17 displays zi2 versus pop and ad. There is little relationship between the second principal component and these two predictors, again suggesting that in this case, one only needs the first principal component in order to accurately represent the pop and ad budgets. With two-dimensional data, such as in our advertising example, we can construct at most two principal components. However, if we had other predictors, such as population age, income level, education, and so forth, then additional components could be constructed. They would successively maximize variance, subject to the constraint of being uncorrelated with the preceding components. The Principal Components Regression Approach The principal components regression (PCR) approach involves construct- principal components regression ing the first M principal components, Z1, . . . , ZM , and then using these components as the predictors in a linear regression model that is fit us- ing least squares. The key idea is that often a small number of principal components suffice to explain most of the variability in the data, as well
- 265. 6.3 Dimension Reduction Methods 257 0 10 20 30 40 0 10 20 30 40 50 60 70 Number of Components Mean Squared Error 0 10 20 30 40 0 50 100 150 Number of Components Mean Squared Error Squared Bias Test MSE Variance FIGURE 6.18. PCR was applied to two simulated data sets. In each panel, the horizontal dashed line represents the irreducible error. Left: Simulated data from Figure 6.8. Right: Simulated data from Figure 6.9. as the relationship with the response. In other words, we assume that the directions in which X1, . . . , Xp show the most variation are the directions that are associated with Y . While this assumption is not guaranteed to be true, it often turns out to be a reasonable enough approximation to give good results. If the assumption underlying PCR holds, then fitting a least squares model to Z1, . . . , ZM will lead to better results than fitting a least squares model to X1, . . . , Xp, since most or all of the information in the data that relates to the response is contained in Z1, . . . , ZM , and by estimating only M ≪ p coefficients we can mitigate overfitting. In the advertising data, the first principal component explains most of the variance in both pop and ad, so a principal component regression that uses this single variable to predict some response of interest, such as sales, will likely perform quite well. Figure 6.18 displays the PCR fits on the simulated data sets from Fig- ures 6.8 and 6.9. Recall that both data sets were generated using n = 50 observations and p = 45 predictors. However, while the response in the first data set was a function of all the predictors, the response in the sec- ond data set was generated using only two of the predictors. The curves are plotted as a function of M, the number of principal components used as predictors in the regression model. As more principal components are used in the regression model, the bias decreases, but the variance increases. This results in a typical U-shape for the mean squared error. When M = p = 45, then PCR amounts simply to a least squares fit using all of the original predictors. The figure indicates that performing PCR with an appropriate choice of M can result in a substantial improvement over least squares, es- pecially in the left-hand panel. However, by examining the ridge regression and lasso results in Figures 6.5, 6.8, and 6.9, we see that PCR does not perform as well as the two shrinkage methods in this example.
- 266. 258 6. Linear Model Selection and Regularization 0 10 20 30 40 0 10 20 30 40 50 60 70 PCR Number of Components Mean Squared Error Squared Bias Test MSE Variance 0.0 0.2 0.4 0.6 0.8 1.0 0 10 20 30 40 50 60 70 Ridge Regression and Lasso Shrinkage Factor Mean Squared Error FIGURE 6.19. PCR, ridge regression, and the lasso were applied to a simulated data set in which the first five principal components of X contain all the informa- tion about the response Y . In each panel, the irreducible error Var(ϵ) is shown as a horizontal dashed line. Left: Results for PCR. Right: Results for lasso (solid) and ridge regression (dotted). The x-axis displays the shrinkage factor of the co- efficient estimates, defined as the ℓ2 norm of the shrunken coefficient estimates divided by the ℓ2 norm of the least squares estimate. The relatively worse performance of PCR in Figure 6.18 is a consequence of the fact that the data were generated in such a way that many princi- pal components are required in order to adequately model the response. In contrast, PCR will tend to do well in cases when the first few principal components are sufficient to capture most of the variation in the predictors as well as the relationship with the response. The left-hand panel of Fig- ure 6.19 illustrates the results from another simulated data set designed to be more favorable to PCR. Here the response was generated in such a way that it depends exclusively on the first five principal components. Now the bias drops to zero rapidly as M, the number of principal components used in PCR, increases. The mean squared error displays a clear minimum at M = 5. The right-hand panel of Figure 6.19 displays the results on these data using ridge regression and the lasso. All three methods offer a signif- icant improvement over least squares. However, PCR and ridge regression slightly outperform the lasso. We note that even though PCR provides a simple way to perform re- gression using M < p predictors, it is not a feature selection method. This is because each of the M principal components used in the regression is a linear combination of all p of the original features. For instance, in (6.19), Z1 was a linear combination of both pop and ad. Therefore, while PCR of- ten performs quite well in many practical settings, it does not result in the development of a model that relies upon a small set of the original features. In this sense, PCR is more closely related to ridge regression than to the lasso. In fact, one can show that PCR and ridge regression are very closely
- 267. 6.3 Dimension Reduction Methods 259 2 4 6 8 10 −300 −100 0 100 200 300 400 Number of Components Standardized Coefficients Income Limit Rating Student 2 4 6 8 10 20000 40000 60000 80000 Number of Components Cross−Validation MSE FIGURE 6.20. Left: PCR standardized coefficient estimates on the Credit data set for different values of M. Right: The ten-fold cross-validation MSE obtained using PCR, as a function of M. related. One can even think of ridge regression as a continuous version of PCR!7 In PCR, the number of principal components, M, is typically chosen by cross-validation. The results of applying PCR to the Credit data set are shown in Figure 6.20; the right-hand panel displays the cross-validation er- rors obtained, as a function of M. On these data, the lowest cross-validation error occurs when there are M = 10 components; this corresponds to al- most no dimension reduction at all, since PCR with M = 11 is equivalent to simply performing least squares. When performing PCR, we generally recommend standardizing each pre- dictor, using (6.6), prior to generating the principal components. This stan- dardization ensures that all variables are on the same scale. In the absence of standardization, the high-variance variables will tend to play a larger role in the principal components obtained, and the scale on which the vari- ables are measured will ultimately have an effect on the final PCR model. However, if the variables are all measured in the same units (say, kilograms, or inches), then one might choose not to standardize them. 6.3.2 Partial Least Squares The PCR approach that we just described involves identifying linear combi- nations, or directions, that best represent the predictors X1, . . . , Xp. These directions are identified in an unsupervised way, since the response Y is not used to help determine the principal component directions. That is, the response does not supervise the identification of the principal components. 7More details can be found in Section 3.5 of The Elements of Statistical Learning by Hastie, Tibshirani, and Friedman.
- 268. 260 6. Linear Model Selection and Regularization 20 30 40 50 60 5 10 15 20 25 30 Population Ad Spending FIGURE 6.21. For the advertising data, the first PLS direction (solid line) and first PCR direction (dotted line) are shown. Consequently, PCR suffers from a drawback: there is no guarantee that the directions that best explain the predictors will also be the best directions to use for predicting the response. Unsupervised methods are discussed further in Chapter 12. We now present partial least squares (PLS), a supervised alternative to partial least squares PCR. Like PCR, PLS is a dimension reduction method, which first identifies a new set of features Z1, . . . , ZM that are linear combinations of the original features, and then fits a linear model via least squares using these M new features. But unlike PCR, PLS identifies these new features in a supervised way—that is, it makes use of the response Y in order to identify new features that not only approximate the old features well, but also that are related to the response. Roughly speaking, the PLS approach attempts to find directions that help explain both the response and the predictors. We now describe how the first PLS direction is computed. After stan- dardizing the p predictors, PLS computes the first direction Z1 by setting each φj1 in (6.16) equal to the coefficient from the simple linear regression of Y onto Xj. One can show that this coefficient is proportional to the cor- relation between Y and Xj. Hence, in computing Z1 = )p j=1 φj1Xj, PLS places the highest weight on the variables that are most strongly related to the response. Figure 6.21 displays an example of PLS on a synthetic dataset with Sales in each of 100 regions as the response, and two predictors; Population Size and Advertising Spending. The solid green line indicates the first PLS di- rection, while the dotted line shows the first principal component direction. PLS has chosen a direction that has less change in the ad dimension per unit change in the pop dimension, relative to PCA. This suggests that pop is more highly correlated with the response than is ad. The PLS direction
- 269. 6.4 Considerations in High Dimensions 261 does not fit the predictors as closely as does PCA, but it does a better job explaining the response. To identify the second PLS direction we first adjust each of the variables for Z1, by regressing each variable on Z1 and taking residuals. These resid- uals can be interpreted as the remaining information that has not been explained by the first PLS direction. We then compute Z2 using this or- thogonalized data in exactly the same fashion as Z1 was computed based on the original data. This iterative approach can be repeated M times to identify multiple PLS components Z1, . . . , ZM . Finally, at the end of this procedure, we use least squares to fit a linear model to predict Y using Z1, . . . , ZM in exactly the same fashion as for PCR. As with PCR, the number M of partial least squares directions used in PLS is a tuning parameter that is typically chosen by cross-validation. We generally standardize the predictors and response before performing PLS. PLS is popular in the field of chemometrics, where many variables arise from digitized spectrometry signals. In practice it often performs no better than ridge regression or PCR. While the supervised dimension reduction of PLS can reduce bias, it also has the potential to increase variance, so that the overall benefit of PLS relative to PCR is a wash. 6.4 Considerations in High Dimensions 6.4.1 High-Dimensional Data Most traditional statistical techniques for regression and classification are intended for the low-dimensional setting in which n, the number of ob- low- dimensional servations, is much greater than p, the number of features. This is due in part to the fact that throughout most of the field’s history, the bulk of sci- entific problems requiring the use of statistics have been low-dimensional. For instance, consider the task of developing a model to predict a patient’s blood pressure on the basis of his or her age, sex, and body mass index (BMI). There are three predictors, or four if an intercept is included in the model, and perhaps several thousand patients for whom blood pressure and age, sex, and BMI are available. Hence n ≫ p, and so the problem is low-dimensional. (By dimension here we are referring to the size of p.) In the past 20 years, new technologies have changed the way that data are collected in fields as diverse as finance, marketing, and medicine. It is now commonplace to collect an almost unlimited number of feature mea- surements (p very large). While p can be extremely large, the number of observations n is often limited due to cost, sample availability, or other considerations. Two examples are as follows: 1. Rather than predicting blood pressure on the basis of just age, sex, and BMI, one might also collect measurements for half a million sin-
- 270. 262 6. Linear Model Selection and Regularization gle nucleotide polymorphisms (SNPs; these are individual DNA mu- tations that are relatively common in the population) for inclusion in the predictive model. Then n ≈ 200 and p ≈ 500,000. 2. A marketing analyst interested in understanding people’s online shop- ping patterns could treat as features all of the search terms entered by users of a search engine. This is sometimes known as the “bag-of- words” model. The same researcher might have access to the search histories of only a few hundred or a few thousand search engine users who have consented to share their information with the researcher. For a given user, each of the p search terms is scored present (0) or absent (1), creating a large binary feature vector. Then n ≈ 1,000 and p is much larger. Data sets containing more features than observations are often referred to as high-dimensional. Classical approaches such as least squares linear high- dimensional regression are not appropriate in this setting. Many of the issues that arise in the analysis of high-dimensional data were discussed earlier in this book, since they apply also when n > p: these include the role of the bias-variance trade-off and the danger of overfitting. Though these issues are always rele- vant, they can become particularly important when the number of features is very large relative to the number of observations. We have defined the high-dimensional setting as the case where the num- ber of features p is larger than the number of observations n. But the con- siderations that we will now discuss certainly also apply if p is slightly smaller than n, and are best always kept in mind when performing super- vised learning. 6.4.2 What Goes Wrong in High Dimensions? In order to illustrate the need for extra care and specialized techniques for regression and classification when p > n, we begin by examining what can go wrong if we apply a statistical technique not intended for the high- dimensional setting. For this purpose, we examine least squares regression. But the same concepts apply to logistic regression, linear discriminant anal- ysis, and other classical statistical approaches. When the number of features p is as large as, or larger than, the number of observations n, least squares as described in Chapter 3 cannot (or rather, should not) be performed. The reason is simple: regardless of whether or not there truly is a relationship between the features and the response, least squares will yield a set of coefficient estimates that result in a perfect fit to the data, such that the residuals are zero. An example is shown in Figure 6.22 with p = 1 feature (plus an intercept) in two cases: when there are 20 observations, and when there are only two observations. When there are 20 observations, n > p and the least
- 271. 6.4 Considerations in High Dimensions 263 −1.5 −1.0 −0.5 0.0 0.5 1.0 −10 −5 0 5 10 −1.5 −1.0 −0.5 0.0 0.5 1.0 −10 −5 0 5 10 X X Y Y FIGURE 6.22. Left: Least squares regression in the low-dimensional setting. Right: Least squares regression with n = 2 observations and two parameters to be estimated (an intercept and a coefficient). squares regression line does not perfectly fit the data; instead, the regression line seeks to approximate the 20 observations as well as possible. On the other hand, when there are only two observations, then regardless of the values of those observations, the regression line will fit the data exactly. This is problematic because this perfect fit will almost certainly lead to overfitting of the data. In other words, though it is possible to perfectly fit the training data in the high-dimensional setting, the resulting linear model will perform extremely poorly on an independent test set, and therefore does not constitute a useful model. In fact, we can see that this happened in Figure 6.22: the least squares line obtained in the right-hand panel will perform very poorly on a test set comprised of the observations in the left- hand panel. The problem is simple: when p > n or p ≈ n, a simple least squares regression line is too flexible and hence overfits the data. Figure 6.23 further illustrates the risk of carelessly applying least squares when the number of features p is large. Data were simulated with n = 20 observations, and regression was performed with between 1 and 20 features, each of which was completely unrelated to the response. As shown in the figure, the model R2 increases to 1 as the number of features included in the model increases, and correspondingly the training set MSE decreases to 0 as the number of features increases, even though the features are completely unrelated to the response. On the other hand, the MSE on an independent test set becomes extremely large as the number of features included in the model increases, because including the additional predictors leads to a vast increase in the variance of the coefficient estimates. Looking at the test set MSE, it is clear that the best model contains at most a few variables. However, someone who carelessly examines only the R2 or the training set MSE might erroneously conclude that the model with the greatest number
- 272. 264 6. Linear Model Selection and Regularization 5 10 15 0.2 0.4 0.6 0.8 1.0 Number of Variables R 2 5 10 15 0.0 0.2 0.4 0.6 0.8 Number of Variables Training MSE 5 10 15 1 5 50 500 Number of Variables Test MSE FIGURE 6.23. On a simulated example with n = 20 training observations, features that are completely unrelated to the outcome are added to the model. Left: The R2 increases to 1 as more features are included. Center: The training set MSE decreases to 0 as more features are included. Right: The test set MSE increases as more features are included. of variables is best. This indicates the importance of applying extra care when analyzing data sets with a large number of variables, and of always evaluating model performance on an independent test set. In Section 6.1.3, we saw a number of approaches for adjusting the training set RSS or R2 in order to account for the number of variables used to fit a least squares model. Unfortunately, the Cp, AIC, and BIC approaches are not appropriate in the high-dimensional setting, because estimating σ̂2 is problematic. (For instance, the formula for σ̂2 from Chapter 3 yields an estimate σ̂2 = 0 in this setting.) Similarly, problems arise in the application of adjusted R2 in the high-dimensional setting, since one can easily obtain a model with an adjusted R2 value of 1. Clearly, alternative approaches that are better-suited to the high-dimensional setting are required. 6.4.3 Regression in High Dimensions It turns out that many of the methods seen in this chapter for fitting less flexible least squares models, such as forward stepwise selection, ridge regression, the lasso, and principal components regression, are particularly useful for performing regression in the high-dimensional setting. Essentially, these approaches avoid overfitting by using a less flexible fitting approach than least squares. Figure 6.24 illustrates the performance of the lasso in a simple simulated example. There are p = 20, 50, or 2,000 features, of which 20 are truly associated with the outcome. The lasso was performed on n = 100 training observations, and the mean squared error was evaluated on an independent test set. As the number of features increases, the test set error increases. When p = 20, the lowest validation set error was achieved when λ in (6.7) was small; however, when p was larger then the lowest validation
- 273. 6.4 Considerations in High Dimensions 265 1 16 21 0 1 2 3 4 5 1 28 51 0 1 2 3 4 5 1 70 111 0 1 2 3 4 5 p = 20 p = 50 p = 2000 Degrees of Freedom Degrees of Freedom Degrees of Freedom FIGURE 6.24. The lasso was performed with n = 100 observations and three values of p, the number of features. Of the p features, 20 were associated with the response. The boxplots show the test MSEs that result using three different values of the tuning parameter λ in (6.7). For ease of interpretation, rather than reporting λ, the degrees of freedom are reported; for the lasso this turns out to be simply the number of estimated non-zero coefficients. When p = 20, the lowest test MSE was obtained with the smallest amount of regularization. When p = 50, the lowest test MSE was achieved when there is a substantial amount of regularization. When p = 2,000 the lasso performed poorly regardless of the amount of regularization, due to the fact that only 20 of the 2,000 features truly are associated with the outcome. set error was achieved using a larger value of λ. In each boxplot, rather than reporting the values of λ used, the degrees of freedom of the resulting lasso solution is displayed; this is simply the number of non-zero coefficient estimates in the lasso solution, and is a measure of the flexibility of the lasso fit. Figure 6.24 highlights three important points: (1) regularization or shrinkage plays a key role in high-dimensional problems, (2) appropriate tuning parameter selection is crucial for good predictive performance, and (3) the test error tends to increase as the dimensionality of the problem (i.e. the number of features or predictors) increases, unless the additional features are truly associated with the response. The third point above is in fact a key principle in the analysis of high- dimensional data, which is known as the curse of dimensionality. One might curse of di- mensionality think that as the number of features used to fit a model increases, the quality of the fitted model will increase as well. However, comparing the left-hand and right-hand panels in Figure 6.24, we see that this is not necessarily the case: in this example, the test set MSE almost doubles as p increases from 20 to 2,000. In general, adding additional signal features that are truly associated with the response will improve the fitted model, in the sense of leading to a reduction in test set error. However, adding
- 274. 266 6. Linear Model Selection and Regularization noise features that are not truly associated with the response will lead to a deterioration in the fitted model, and consequently an increased test set error. This is because noise features increase the dimensionality of the problem, exacerbating the risk of overfitting (since noise features may be assigned nonzero coefficients due to chance associations with the response on the training set) without any potential upside in terms of improved test set error. Thus, we see that new technologies that allow for the collection of measurements for thousands or millions of features are a double-edged sword: they can lead to improved predictive models if these features are in fact relevant to the problem at hand, but will lead to worse results if the features are not relevant. Even if they are relevant, the variance incurred in fitting their coefficients may outweigh the reduction in bias that they bring. 6.4.4 Interpreting Results in High Dimensions When we perform the lasso, ridge regression, or other regression proce- dures in the high-dimensional setting, we must be quite cautious in the way that we report the results obtained. In Chapter 3, we learned about multi- collinearity, the concept that the variables in a regression might be corre- lated with each other. In the high-dimensional setting, the multicollinearity problem is extreme: any variable in the model can be written as a linear combination of all of the other variables in the model. Essentially, this means that we can never know exactly which variables (if any) truly are predictive of the outcome, and we can never identify the best coefficients for use in the regression. At most, we can hope to assign large regression coefficients to variables that are correlated with the variables that truly are predictive of the outcome. For instance, suppose that we are trying to predict blood pressure on the basis of half a million SNPs, and that forward stepwise selection indicates that 17 of those SNPs lead to a good predictive model on the training data. It would be incorrect to conclude that these 17 SNPs predict blood pressure more effectively than the other SNPs not included in the model. There are likely to be many sets of 17 SNPs that would predict blood pressure just as well as the selected model. If we were to obtain an independent data set and perform forward stepwise selection on that data set, we would likely obtain a model containing a different, and perhaps even non-overlapping, set of SNPs. This does not detract from the value of the model obtained— for instance, the model might turn out to be very effective in predicting blood pressure on an independent set of patients, and might be clinically useful for physicians. But we must be careful not to overstate the results obtained, and to make it clear that what we have identified is simply one of many possible models for predicting blood pressure, and that it must be further validated on independent data sets.
- 275. 6.5 Lab: Linear Models and Regularization Methods 267 It is also important to be particularly careful in reporting errors and measures of model fit in the high-dimensional setting. We have seen that when p > n, it is easy to obtain a useless model that has zero residu- als. Therefore, one should never use sum of squared errors, p-values, R2 statistics, or other traditional measures of model fit on the training data as evidence of a good model fit in the high-dimensional setting. For instance, as we saw in Figure 6.23, one can easily obtain a model with R2 = 1 when p > n. Reporting this fact might mislead others into thinking that a sta- tistically valid and useful model has been obtained, whereas in fact this provides absolutely no evidence of a compelling model. It is important to instead report results on an independent test set, or cross-validation errors. For instance, the MSE or R2 on an independent test set is a valid measure of model fit, but the MSE on the training set certainly is not. 6.5 Lab: Linear Models and Regularization Methods 6.5.1 Subset Selection Methods Best Subset Selection Here we apply the best subset selection approach to the Hitters data. We wish to predict a baseball player’s Salary on the basis of various statistics associated with performance in the previous year. First of all, we note that the Salary variable is missing for some of the players. The is.na() function can be used to identify the missing observa- is.na() tions. It returns a vector of the same length as the input vector, with a TRUE for any elements that are missing, and a FALSE for non-missing elements. The sum() function can then be used to count all of the missing elements. sum() > library(ISLR2) > View(Hitters) > names(Hitters) [1] "AtBat" "Hits" "HmRun" "Runs" "RBI" [6] "Walks" "Years" "CAtBat" "CHits" "CHmRun" [11] "CRuns" "CRBI" "CWalks" "League" "Division" [16] "PutOuts" "Assists" "Errors" "Salary" "NewLeague" > dim(Hitters) [1] 322 20 > sum(is.na(Hitters$Salary)) [1] 59 Hence we see that Salary is missing for 59 players. The na.omit() function removes all of the rows that have missing values in any variable. > Hitters <- na.omit(Hitters) > dim(Hitters) [1] 263 20
- 276. 268 6. Linear Model Selection and Regularization > sum(is.na(Hitters)) [1] 0 The regsubsets() function (part of the leaps library) performs best sub- regsubsets() set selection by identifying the best model that contains a given number of predictors, where best is quantified using RSS. The syntax is the same as for lm(). The summary() command outputs the best set of variables for each model size. > library(leaps) > regfit.full <- regsubsets (Salary ∼ ., Hitters) > summary(regfit.full) Subset selection object Call: regsubsets.formula(Salary ∼ ., Hitters) 19 Variables (and intercept) ... 1 subsets of each size up to 8 Selection Algorithm: exhaustive AtBat Hits HmRun Runs RBI Walks Years CAtBat CHits 1 ( 1 ) " " " " " " " " " " " " " " " " " " 2 ( 1 ) " " "*" " " " " " " " " " " " " " " 3 ( 1 ) " " "*" " " " " " " " " " " " " " " 4 ( 1 ) " " "*" " " " " " " " " " " " " " " 5 ( 1 ) "*" "*" " " " " " " " " " " " " " " 6 ( 1 ) "*" "*" " " " " " " "*" " " " " " " 7 ( 1 ) " " "*" " " " " " " "*" " " "*" "*" 8 ( 1 ) "*" "*" " " " " " " "*" " " " " " " CHmRun CRuns CRBI CWalks LeagueN DivisionW PutOuts 1 ( 1 ) " " " " "*" " " " " " " " " 2 ( 1 ) " " " " "*" " " " " " " " " 3 ( 1 ) " " " " "*" " " " " " " "*" 4 ( 1 ) " " " " "*" " " " " "*" "*" 5 ( 1 ) " " " " "*" " " " " "*" "*" 6 ( 1 ) " " " " "*" " " " " "*" "*" 7 ( 1 ) "*" " " " " " " " " "*" "*" 8 ( 1 ) "*" "*" " " "*" " " "*" "*" Assists Errors NewLeagueN 1 ( 1 ) " " " " " " 2 ( 1 ) " " " " " " 3 ( 1 ) " " " " " " 4 ( 1 ) " " " " " " 5 ( 1 ) " " " " " " 6 ( 1 ) " " " " " " 7 ( 1 ) " " " " " " 8 ( 1 ) " " " " " " An asterisk indicates that a given variable is included in the corresponding model. For instance, this output indicates that the best two-variable model contains only Hits and CRBI. By default, regsubsets() only reports results up to the best eight-variable model. But the nvmax option can be used in order to return as many variables as are desired. Here we fit up to a 19-variable model.
- 277. 6.5 Lab: Linear Models and Regularization Methods 269 > regfit.full <- regsubsets (Salary ∼ ., data = Hitters , nvmax = 19) > reg.summary <- summary(regfit.full) The summary() function also returns R2 , RSS, adjusted R2 , Cp, and BIC. We can examine these to try to select the best overall model. > names(reg.summary) [1] "which" "rsq" "rss" "adjr2" "cp" "bic" [7] "outmat" "obj" For instance, we see that the R2 statistic increases from 32 %, when only one variable is included in the model, to almost 55 %, when all variables are included. As expected, the R2 statistic increases monotonically as more variables are included. > reg.summary$rsq [1] 0.321 0.425 0.451 0.475 0.491 0.509 0.514 0.529 0.535 [10] 0.540 0.543 0.544 0.544 0.545 0.545 0.546 0.546 0.546 [19] 0.546 Plotting RSS, adjusted R2 , Cp, and BIC for all of the models at once will help us decide which model to select. Note the type = "l" option tells R to connect the plotted points with lines. > par(mfrow = c(2, 2)) > plot(reg.summary$rss , xlab = "Number of Variables ", ylab = "RSS", type = "l") > plot(reg.summary$adjr2 , xlab = "Number of Variables ", ylab = "Adjusted RSq", type = "l") The points() command works like the plot() command, except that it points() puts points on a plot that has already been created, instead of creating a new plot. The which.max() function can be used to identify the location of the maximum point of a vector. We will now plot a red dot to indicate the model with the largest adjusted R2 statistic. > which.max(reg.summary$adjr2) [1] 11 > points (11, reg.summary$adjr2 [11] , col = "red", cex = 2, pch = 20) In a similar fashion we can plot the Cp and BIC statistics, and indicate the models with the smallest statistic using which.min(). which.min() > plot(reg.summary$cp , xlab = "Number of Variables ", ylab = "Cp", type = "l") > which.min(reg.summary$cp) [1] 10 > points (10, reg.summary$cp[10] , col = "red", cex = 2, pch = 20) > which.min(reg.summary$bic) [1] 6 > plot(reg.summary$bic , xlab = "Number of Variables ", ylab = "BIC", type = "l")
- 278. 270 6. Linear Model Selection and Regularization > points (6, reg.summary$bic [6], col = "red", cex = 2, pch = 20) The regsubsets() function has a built-in plot() command which can be used to display the selected variables for the best model with a given number of predictors, ranked according to the BIC, Cp, adjusted R2 , or AIC. To find out more about this function, type ?plot.regsubsets. > plot(regfit.full , scale = "r2") > plot(regfit.full , scale = "adjr2") > plot(regfit.full , scale = "Cp") > plot(regfit.full , scale = "bic") The top row of each plot contains a black square for each variable selected according to the optimal model associated with that statistic. For instance, we see that several models share a BIC close to −150. However, the model with the lowest BIC is the six-variable model that contains only AtBat, Hits, Walks, CRBI, DivisionW, and PutOuts. We can use the coef() function to see the coefficient estimates associated with this model. > coef(regfit.full , 6) (Intercept) AtBat Hits Walks CRBI 91.512 -1.869 7.604 3.698 0.643 DivisionW PutOuts -122.952 0.264 Forward and Backward Stepwise Selection We can also use the regsubsets() function to perform forward stepwise or backward stepwise selection, using the argument method = "forward" or method = "backward". > regfit.fwd <- regsubsets (Salary ∼ ., data = Hitters , nvmax = 19, method = "forward") > summary(regfit.fwd) > regfit.bwd <- regsubsets (Salary ∼ ., data = Hitters , nvmax = 19, method = "backward ") > summary(regfit.bwd) For instance, we see that using forward stepwise selection, the best one- variable model contains only CRBI, and the best two-variable model ad- ditionally includes Hits. For this data, the best one-variable through six- variable models are each identical for best subset and forward selection. However, the best seven-variable models identified by forward stepwise se- lection, backward stepwise selection, and best subset selection are different. > coef(regfit.full , 7) (Intercept) Hits Walks CAtBat CHits 79.451 1.283 3.227 -0.375 1.496 CHmRun DivisionW PutOuts 1.442 -129.987 0.237 > coef(regfit.fwd , 7)
- 279. 6.5 Lab: Linear Models and Regularization Methods 271 (Intercept) AtBat Hits Walks CRBI 109.787 -1.959 7.450 4.913 0.854 CWalks DivisionW PutOuts -0.305 -127.122 0.253 > coef(regfit.bwd , 7) (Intercept) AtBat Hits Walks CRuns 105.649 -1.976 6.757 6.056 1.129 CWalks DivisionW PutOuts -0.716 -116.169 0.303 Choosing Among Models Using the Validation-Set Approach and Cross-Validation We just saw that it is possible to choose among a set of models of different sizes using Cp, BIC, and adjusted R2 . We will now consider how to do this using the validation set and cross-validation approaches. In order for these approaches to yield accurate estimates of the test error, we must use only the training observations to perform all aspects of model-fitting—including variable selection. Therefore, the determination of which model of a given size is best must be made using only the training observations. This point is subtle but important. If the full data set is used to perform the best subset selection step, the validation set errors and cross-validation errors that we obtain will not be accurate estimates of the test error. In order to use the validation set approach, we begin by splitting the observations into a training set and a test set. We do this by creating a random vector, train, of elements equal to TRUE if the corresponding observation is in the training set, and FALSE otherwise. The vector test has a TRUE if the observation is in the test set, and a FALSE otherwise. Note the ! in the command to create test causes TRUEs to be switched to FALSEs and vice versa. We also set a random seed so that the user will obtain the same training set/test set split. > set.seed (1) > train <- sample(c(TRUE , FALSE), nrow(Hitters), replace = TRUE) > test <- (! train) Now, we apply regsubsets() to the training set in order to perform best subset selection. > regfit.best <- regsubsets (Salary ∼ ., data = Hitters[train , ], nvmax = 19) Notice that we subset the Hitters data frame directly in the call in or- der to access only the training subset of the data, using the expression Hitters[train, ]. We now compute the validation set error for the best model of each model size. We first make a model matrix from the test data.
- 280. 272 6. Linear Model Selection and Regularization > test.mat <- model.matrix(Salary ∼ ., data = Hitters[test , ]) The model.matrix() function is used in many regression packages for build- model.matrix() ing an “X” matrix from data. Now we run a loop, and for each size i, we extract the coefficients from regfit.best for the best model of that size, multiply them into the appropriate columns of the test model matrix to form the predictions, and compute the test MSE. > val.errors <- rep(NA , 19) > for (i in 1:19) { + coefi <- coef(regfit.best , id = i) + pred <- test.mat[, names(coefi)] %*% coefi + val.errors[i] <- mean (( Hitters$Salary[test] - pred)^2) } We find that the best model is the one that contains seven variables. > val.errors [1] 164377 144405 152176 145198 137902 139176 126849 136191 [9] 132890 135435 136963 140695 140691 141951 141508 142164 [17] 141767 142340 142238 > which.min(val.errors) [1] 7 > coef(regfit.best , 7) (Intercept) AtBat Hits Walks CRuns 67.109 -2.146 7.015 8.072 1.243 CWalks DivisionW PutOuts -0.834 -118.436 0.253 This was a little tedious, partly because there is no predict() method for regsubsets(). Since we will be using this function again, we can capture our steps above and write our own predict method. > predict.regsubsets <- function (object , newdata , id , ...) { + form <- as.formula(object$call [[2]]) + mat <- model.matrix(form , newdata) + coefi <- coef(object , id = id) + xvars <- names(coefi) + mat[, xvars] %*% coefi + } Our function pretty much mimics what we did above. The only complex part is how we extracted the formula used in the call to regsubsets(). We demonstrate how we use this function below, when we do cross-validation. Finally, we perform best subset selection on the full data set, and select the best seven-variable model. It is important that we make use of the full data set in order to obtain more accurate coefficient estimates. Note that we perform best subset selection on the full data set and select the best seven-variable model, rather than simply using the variables that were obtained from the training set, because the best seven-variable model on the full data set may differ from the corresponding model on the training set.
- 281. 6.5 Lab: Linear Models and Regularization Methods 273 > regfit.best <- regsubsets (Salary ∼ ., data = Hitters , nvmax = 19) > coef(regfit.best , 7) (Intercept) Hits Walks CAtBat CHits 79.451 1.283 3.227 -0.375 1.496 CHmRun DivisionW PutOuts 1.442 -129.987 0.237 In fact, we see that the best seven-variable model on the full data set has a different set of variables than the best seven-variable model on the training set. We now try to choose among the models of different sizes using cross- validation. This approach is somewhat involved, as we must perform best subset selection within each of the k training sets. Despite this, we see that with its clever subsetting syntax, R makes this job quite easy. First, we create a vector that allocates each observation to one of k = 10 folds, and we create a matrix in which we will store the results. > k <- 10 > n <- nrow(Hitters) > set.seed (1) > folds <- sample(rep (1:k, length = n)) > cv.errors <- matrix(NA , k, 19, dimnames = list(NULL , paste (1:19))) Now we write a for loop that performs cross-validation. In the jth fold, the elements of folds that equal j are in the test set, and the remainder are in the training set. We make our predictions for each model size (using our new predict() method), compute the test errors on the appropriate subset, and store them in the appropriate slot in the matrix cv.errors. Note that in the following code R will automatically use our predict.regsubsets() function when we call predict() because the best.fit object has class regsubsets. > for (j in 1:k) { + best.fit <- regsubsets (Salary ∼ ., data = Hitters[folds != j, ], nvmax = 19) + for (i in 1:19) { + pred <- predict(best.fit , Hitters[folds == j, ], id = i) + cv.errors[j, i] <- mean (( Hitters$Salary[folds == j] - pred)^2) + } + } This has given us a 10×19 matrix, of which the (j, i)th element corresponds to the test MSE for the jth cross-validation fold for the best i-variable model. We use the apply() function to average over the columns of this apply() matrix in order to obtain a vector for which the ith element is the cross- validation error for the i-variable model. > mean.cv.errors <- apply(cv.errors , 2, mean) > mean.cv.errors
- 282. 274 6. Linear Model Selection and Regularization 1 2 3 4 5 6 7 8 143440 126817 134214 131783 130766 120383 121443 114364 9 10 11 12 13 14 15 16 115163 109366 112738 113617 115558 115853 115631 116050 17 18 19 116117 116419 116299 > par(mfrow = c(1, 1)) > plot(mean.cv.errors , type = "b") We see that cross-validation selects a 10-variable model. We now perform best subset selection on the full data set in order to obtain the 10-variable model. > reg.best <- regsubsets (Salary ∼ ., data = Hitters , nvmax = 19) > coef(reg.best , 10) (Intercept) AtBat Hits Walks CAtBat 162.535 -2.169 6.918 5.773 -0.130 CRuns CRBI CWalks DivisionW PutOuts 1.408 0.774 -0.831 -112.380 0.297 Assists 0.283 6.5.2 Ridge Regression and the Lasso We will use the glmnet package in order to perform ridge regression and the lasso. The main function in this package is glmnet(), which can be used glmnet() to fit ridge regression models, lasso models, and more. This function has slightly different syntax from other model-fitting functions that we have encountered thus far in this book. In particular, we must pass in an x matrix as well as a y vector, and we do not use the y ∼ x syntax. We will now perform ridge regression and the lasso in order to predict Salary on the Hitters data. Before proceeding ensure that the missing values have been removed from the data, as described in Section 6.5.1. > x <- model.matrix(Salary ∼ ., Hitters)[, -1] > y <- Hitters$Salary The model.matrix() function is particularly useful for creating x; not only does it produce a matrix corresponding to the 19 predictors but it also automatically transforms any qualitative variables into dummy variables. The latter property is important because glmnet() can only take numerical, quantitative inputs. Ridge Regression The glmnet() function has an alpha argument that determines what type of model is fit. If alpha=0 then a ridge regression model is fit, and if alpha=1 then a lasso model is fit. We first fit a ridge regression model.
- 283. 6.5 Lab: Linear Models and Regularization Methods 275 > library(glmnet) > grid <- 10^ seq (10, -2, length = 100) > ridge.mod <- glmnet(x, y, alpha = 0, lambda = grid) By default the glmnet() function performs ridge regression for an automati- cally selected range of λ values. However, here we have chosen to implement the function over a grid of values ranging from λ = 1010 to λ = 10−2 , es- sentially covering the full range of scenarios from the null model containing only the intercept, to the least squares fit. As we will see, we can also com- pute model fits for a particular value of λ that is not one of the original grid values. Note that by default, the glmnet() function standardizes the variables so that they are on the same scale. To turn off this default setting, use the argument standardize = FALSE. Associated with each value of λ is a vector of ridge regression coefficients, stored in a matrix that can be accessed by coef(). In this case, it is a 20×100 matrix, with 20 rows (one for each predictor, plus an intercept) and 100 columns (one for each value of λ). > dim(coef(ridge.mod)) [1] 20 100 We expect the coefficient estimates to be much smaller, in terms of ℓ2 norm, when a large value of λ is used, as compared to when a small value of λ is used. These are the coefficients when λ = 11,498, along with their ℓ2 norm: > ridge.mod$lambda [50] [1] 11498 > coef(ridge.mod)[, 50] (Intercept) AtBat Hits HmRun Runs 407.356 0.037 0.138 0.525 0.231 RBI Walks Years CAtBat CHits 0.240 0.290 1.108 0.003 0.012 CHmRun CRuns CRBI CWalks LeagueN 0.088 0.023 0.024 0.025 0.085 DivisionW PutOuts Assists Errors NewLeagueN -6.215 0.016 0.003 -0.021 0.301 > sqrt(sum(coef(ridge.mod)[-1, 50]^2)) [1] 6.36 In contrast, here are the coefficients when λ = 705, along with their ℓ2 norm. Note the much larger ℓ2 norm of the coefficients associated with this smaller value of λ. > ridge.mod$lambda [60] [1] 705 > coef(ridge.mod)[, 60] (Intercept) AtBat Hits HmRun Runs 54.325 0.112 0.656 1.180 0.938 RBI Walks Years CAtBat CHits 0.847 1.320 2.596 0.011 0.047 CHmRun CRuns CRBI CWalks LeagueN 0.338 0.094 0.098 0.072 13.684
- 284. 276 6. Linear Model Selection and Regularization DivisionW PutOuts Assists Errors NewLeagueN -54.659 0.119 0.016 -0.704 8.612 > sqrt(sum(coef(ridge.mod)[-1, 60]^2)) [1] 57.1 We can use the predict() function for a number of purposes. For instance, we can obtain the ridge regression coefficients for a new value of λ, say 50: > predict(ridge.mod , s = 50, type = " coefficients ")[1:20 , ] (Intercept) AtBat Hits HmRun Runs 48.766 -0.358 1.969 -1.278 1.146 RBI Walks Years CAtBat CHits 0.804 2.716 -6.218 0.005 0.106 CHmRun CRuns CRBI CWalks LeagueN 0.624 0.221 0.219 -0.150 45.926 DivisionW PutOuts Assists Errors NewLeagueN -118.201 0.250 0.122 -3.279 -9.497 We now split the samples into a training set and a test set in order to estimate the test error of ridge regression and the lasso. There are two common ways to randomly split a data set. The first is to produce a random vector of TRUE, FALSE elements and select the observations corresponding to TRUE for the training data. The second is to randomly choose a subset of numbers between 1 and n; these can then be used as the indices for the training observations. The two approaches work equally well. We used the former method in Section 6.5.1. Here we demonstrate the latter approach. We first set a random seed so that the results obtained will be repro- ducible. > set.seed (1) > train <- sample (1: nrow(x), nrow(x) / 2) > test <- (-train) > y.test <- y[test] Next we fit a ridge regression model on the training set, and evaluate its MSE on the test set, using λ = 4. Note the use of the predict() function again. This time we get predictions for a test set, by replacing type="coefficients" with the newx argument. > ridge.mod <- glmnet(x[train , ], y[train], alpha = 0, lambda = grid , thresh = 1e -12) > ridge.pred <- predict(ridge.mod , s = 4, newx = x[test , ]) > mean (( ridge.pred - y.test)^2) [1] 142199 The test MSE is 142,199. Note that if we had instead simply fit a model with just an intercept, we would have predicted each test observation using the mean of the training observations. In that case, we could compute the test set MSE like this: > mean (( mean(y[train ]) - y.test)^2) [1] 224670
- 285. 6.5 Lab: Linear Models and Regularization Methods 277 We could also get the same result by fitting a ridge regression model with a very large value of λ. Note that 1e10 means 1010 . > ridge.pred <- predict(ridge.mod , s = 1e10 , newx = x[test , ]) > mean (( ridge.pred - y.test)^2) [1] 224670 So fitting a ridge regression model with λ = 4 leads to a much lower test MSE than fitting a model with just an intercept. We now check whether there is any benefit to performing ridge regression with λ = 4 instead of just performing least squares regression. Recall that least squares is simply ridge regression with λ = 0.8 > ridge.pred <- predict(ridge.mod , s = 0, newx = x[test , ], exact = T, x = x[train , ], y = y[train ]) > mean (( ridge.pred - y.test)^2) [1] 168589 > lm(y ∼ x, subset = train) > predict(ridge.mod , s = 0, exact = T, type = " coefficients ", x = x[train , ], y = y[train ]) [1:20 , ] In general, if we want to fit a (unpenalized) least squares model, then we should use the lm() function, since that function provides more useful outputs, such as standard errors and p-values for the coefficients. In general, instead of arbitrarily choosing λ = 4, it would be better to use cross-validation to choose the tuning parameter λ. We can do this using the built-in cross-validation function, cv.glmnet(). By default, the function cv.glmnet() performs ten-fold cross-validation, though this can be changed using the argument nfolds. Note that we set a random seed first so our results will be reproducible, since the choice of the cross-validation folds is random. > set.seed (1) > cv.out <- cv.glmnet(x[train , ], y[train], alpha = 0) > plot(cv.out) > bestlam <- cv.out$lambda.min > bestlam [1] 326 Therefore, we see that the value of λ that results in the smallest cross- validation error is 326. What is the test MSE associated with this value of λ? > ridge.pred <- predict(ridge.mod , s = bestlam , newx = x[test , ]) 8In order for glmnet() to yield the exact least squares coefficients when λ = 0, we use the argument exact = T when calling the predict() function. Otherwise, the predict() function will interpolate over the grid of λ values used in fitting the glmnet() model, yielding approximate results. When we use exact = T, there remains a slight discrepancy in the third decimal place between the output of glmnet() when λ = 0 and the output of lm(); this is due to numerical approximation on the part of glmnet().
- 286. 278 6. Linear Model Selection and Regularization > mean (( ridge.pred - y.test)^2) [1] 139857 This represents a further improvement over the test MSE that we got using λ = 4. Finally, we refit our ridge regression model on the full data set, using the value of λ chosen by cross-validation, and examine the coefficient estimates. > out <- glmnet(x, y, alpha = 0) > predict(out , type = " coefficients ", s = bestlam)[1:20 , ] (Intercept) AtBat Hits HmRun Runs 15.44 0.08 0.86 0.60 1.06 RBI Walks Years CAtBat CHits 0.88 1.62 1.35 0.01 0.06 CHmRun CRuns CRBI CWalks LeagueN 0.41 0.11 0.12 0.05 22.09 DivisionW PutOuts Assists Errors NewLeagueN -79.04 0.17 0.03 -1.36 9.12 As expected, none of the coefficients are zero—ridge regression does not perform variable selection! The Lasso We saw that ridge regression with a wise choice of λ can outperform least squares as well as the null model on the Hitters data set. We now ask whether the lasso can yield either a more accurate or a more interpretable model than ridge regression. In order to fit a lasso model, we once again use the glmnet() function; however, this time we use the argument alpha=1. Other than that change, we proceed just as we did in fitting a ridge model. > lasso.mod <- glmnet(x[train , ], y[train], alpha = 1, lambda = grid) > plot(lasso.mod) We can see from the coefficient plot that depending on the choice of tuning parameter, some of the coefficients will be exactly equal to zero. We now perform cross-validation and compute the associated test error. > set.seed (1) > cv.out <- cv.glmnet(x[train , ], y[train], alpha = 1) > plot(cv.out) > bestlam <- cv.out$lambda.min > lasso.pred <- predict(lasso.mod , s = bestlam , newx = x[test , ]) > mean (( lasso.pred - y.test)^2) [1] 143674 This is substantially lower than the test set MSE of the null model and of least squares, and very similar to the test MSE of ridge regression with λ chosen by cross-validation. However, the lasso has a substantial advantage over ridge regression in that the resulting coefficient estimates are sparse. Here we see that 8 of the
- 287. 6.5 Lab: Linear Models and Regularization Methods 279 19 coefficient estimates are exactly zero. So the lasso model with λ chosen by cross-validation contains only eleven variables. > out <- glmnet(x, y, alpha = 1, lambda = grid) > lasso.coef <- predict(out , type = " coefficients ", s = bestlam)[1:20 , ] > lasso.coef (Intercept) AtBat Hits HmRun Runs 1.27 -0.05 2.18 0.00 0.00 RBI Walks Years CAtBat CHits 0.00 2.29 -0.34 0.00 0.00 CHmRun CRuns CRBI CWalks LeagueN 0.03 0.22 0.42 0.00 20.29 DivisionW PutOuts Assists Errors NewLeagueN -116.17 0.24 0.00 -0.86 0.00 > lasso.coef[lasso.coef != 0] (Intercept) AtBat Hits Walks Years 1.27 -0.05 2.18 2.29 -0.34 CHmRun CRuns CRBI LeagueN DivisionW 0.03 0.22 0.42 20.29 -116.17 PutOuts Errors 0.24 -0.86 6.5.3 PCR and PLS Regression Principal Components Regression Principal components regression (PCR) can be performed using the pcr() pcr() function, which is part of the pls library. We now apply PCR to the Hitters data, in order to predict Salary. Again, we ensure that the missing values have been removed from the data, as described in Section 6.5.1. > library(pls) > set.seed (2) > pcr.fit <- pcr(Salary ∼ ., data = Hitters , scale = TRUE , validation = "CV") The syntax for the pcr() function is similar to that for lm(), with a few additional options. Setting scale = TRUE has the effect of standardizing each predictor, using (6.6), prior to generating the principal components, so that the scale on which each variable is measured will not have an effect. Setting validation = "CV" causes pcr() to compute the ten-fold cross-validation error for each possible value of M, the number of principal components used. The resulting fit can be examined using summary(). > summary(pcr.fit) Data: X dimension: 263 19 Y dimension: 263 1 Fit method: svdpc Number of components considered: 19 VALIDATION : RMSEP
- 288. 280 6. Linear Model Selection and Regularization Cross -validated using 10 random segments. (Intercept) 1 comps 2 comps 3 comps 4 comps CV 452 351.9 353.2 355.0 352.8 adjCV 452 351.6 352.7 354.4 352.1 ... TRAINING: % variance explained 1 comps 2 comps 3 comps 4 comps 5 comps X 38.31 60.16 70.84 79.03 84.29 Salary 40.63 41.58 42.17 43.22 44.90 ... The CV score is provided for each possible number of components, ranging from M = 0 onwards. (We have printed the CV output only up to M = 4.) Note that pcr() reports the root mean squared error; in order to obtain the usual MSE, we must square this quantity. For instance, a root mean squared error of 352.8 corresponds to an MSE of 352.82 = 124,468. One can also plot the cross-validation scores using the validationplot() validationplot() function. Using val.type = "MSEP" will cause the cross-validation MSE to be plotted. > validationplot (pcr.fit , val.type = "MSEP") We see that the smallest cross-validation error occurs when M = 18 com- ponents are used. This is barely fewer than M = 19, which amounts to simply performing least squares, because when all of the components are used in PCR no dimension reduction occurs. However, from the plot we also see that the cross-validation error is roughly the same when only one component is included in the model. This suggests that a model that uses just a small number of components might suffice. The summary() function also provides the percentage of variance explained in the predictors and in the response using different numbers of compo- nents. This concept is discussed in greater detail in Chapter 12. Briefly, we can think of this as the amount of information about the predictors or the response that is captured using M principal components. For example, setting M = 1 only captures 38.31 % of all the variance, or information, in the predictors. In contrast, using M = 5 increases the value to 84.29 %. If we were to use all M = p = 19 components, this would increase to 100 %. We now perform PCR on the training data and evaluate its test set performance. > set.seed (1) > pcr.fit <- pcr(Salary ∼ ., data = Hitters , subset = train , scale = TRUE , validation = "CV") > validationplot (pcr.fit , val.type = "MSEP") Now we find that the lowest cross-validation error occurs when M = 5 components are used. We compute the test MSE as follows. > pcr.pred <- predict(pcr.fit , x[test , ], ncomp = 5) > mean (( pcr.pred - y.test)^2)
- 289. 6.5 Lab: Linear Models and Regularization Methods 281 [1] 142812 This test set MSE is competitive with the results obtained using ridge re- gression and the lasso. However, as a result of the way PCR is implemented, the final model is more difficult to interpret because it does not perform any kind of variable selection or even directly produce coefficient estimates. Finally, we fit PCR on the full data set, using M = 5, the number of components identified by cross-validation. > pcr.fit <- pcr(y ∼ x, scale = TRUE , ncomp = 5) > summary(pcr.fit) Data: X dimension: 263 19 Y dimension: 263 1 Fit method: svdpc Number of components considered: 5 TRAINING: % variance explained 1 comps 2 comps 3 comps 4 comps 5 comps X 38.31 60.16 70.84 79.03 84.29 y 40.63 41.58 42.17 43.22 44.90 Partial Least Squares We implement partial least squares (PLS) using the plsr() function, also plsr() in the pls library. The syntax is just like that of the pcr() function. > set.seed (1) > pls.fit <- plsr(Salary ∼ ., data = Hitters , subset = train , scale = TRUE , validation = "CV") > summary(pls.fit) Data: X dimension: 131 19 Y dimension: 131 1 Fit method: kernelpls Number of components considered: 19 VALIDATION : RMSEP Cross -validated using 10 random segments. (Intercept) 1 comps 2 comps 3 comps 4 comps CV 428.3 325.5 329.9 328.8 339.0 adjCV 428.3 325.0 328.2 327.2 336.6 ... TRAINING: % variance explained 1 comps 2 comps 3 comps 4 comps 5 comps X 39.13 48.80 60.09 75.07 78.58 Salary 46.36 50.72 52.23 53.03 54.07 ... > validationplot (pls.fit , val.type = "MSEP") The lowest cross-validation error occurs when only M = 1 partial least squares directions are used. We now evaluate the corresponding test set MSE. > pls.pred <- predict(pls.fit , x[test , ], ncomp = 1)
- 290. 282 6. Linear Model Selection and Regularization > mean (( pls.pred - y.test)^2) [1] 151995 The test MSE is comparable to, but slightly higher than, the test MSE obtained using ridge regression, the lasso, and PCR. Finally, we perform PLS using the full data set, using M = 1, the number of components identified by cross-validation. > pls.fit <- plsr(Salary ∼ ., data = Hitters , scale = TRUE , ncomp = 1) > summary(pls.fit) Data: X dimension: 263 19 Y dimension: 263 1 Fit method: kernelpls Number of components considered: 1 TRAINING: % variance explained 1 comps X 38.08 Salary 43.05 Notice that the percentage of variance in Salary that the one-component PLS fit explains, 43.05 %, is almost as much as that explained using the final five-component model PCR fit, 44.90 %. This is because PCR only attempts to maximize the amount of variance explained in the predictors, while PLS searches for directions that explain variance in both the predictors and the response. 6.6 Exercises Conceptual 1. We perform best subset, forward stepwise, and backward stepwise selection on a single data set. For each approach, we obtain p + 1 models, containing 0, 1, 2, . . . , p predictors. Explain your answers: (a) Which of the three models with k predictors has the smallest training RSS? (b) Which of the three models with k predictors has the smallest test RSS? (c) True or False: i. The predictors in the k-variable model identified by forward stepwise are a subset of the predictors in the (k+1)-variable model identified by forward stepwise selection. ii. The predictors in the k-variable model identified by back- ward stepwise are a subset of the predictors in the (k + 1)- variable model identified by backward stepwise selection.
- 291. 6.6 Exercises 283 iii. The predictors in the k-variable model identified by back- ward stepwise are a subset of the predictors in the (k + 1)- variable model identified by forward stepwise selection. iv. The predictors in the k-variable model identified by forward stepwise are a subset of the predictors in the (k+1)-variable model identified by backward stepwise selection. v. The predictors in the k-variable model identified by best subset are a subset of the predictors in the (k + 1)-variable model identified by best subset selection. 2. For parts (a) through (c), indicate which of i. through iv. is correct. Justify your answer. (a) The lasso, relative to least squares, is: i. More flexible and hence will give improved prediction ac- curacy when its increase in bias is less than its decrease in variance. ii. More flexible and hence will give improved prediction accu- racy when its increase in variance is less than its decrease in bias. iii. Less flexible and hence will give improved prediction accu- racy when its increase in bias is less than its decrease in variance. iv. Less flexible and hence will give improved prediction accu- racy when its increase in variance is less than its decrease in bias. (b) Repeat (a) for ridge regression relative to least squares. (c) Repeat (a) for non-linear methods relative to least squares. 3. Suppose we estimate the regression coefficients in a linear regression model by minimizing n 0 i=1 ⎛ ⎝yi − β0 − p 0 j=1 βjxij ⎞ ⎠ 2 subject to p 0 j=1 |βj| ≤ s for a particular value of s. For parts (a) through (e), indicate which of i. through v. is correct. Justify your answer. (a) As we increase s from 0, the training RSS will: i. Increase initially, and then eventually start decreasing in an inverted U shape. ii. Decrease initially, and then eventually start increasing in a U shape.
- 292. 284 6. Linear Model Selection and Regularization iii. Steadily increase. iv. Steadily decrease. v. Remain constant. (b) Repeat (a) for test RSS. (c) Repeat (a) for variance. (d) Repeat (a) for (squared) bias. (e) Repeat (a) for the irreducible error. 4. Suppose we estimate the regression coefficients in a linear regression model by minimizing n 0 i=1 ⎛ ⎝yi − β0 − p 0 j=1 βjxij ⎞ ⎠ 2 + λ p 0 j=1 β2 j for a particular value of λ. For parts (a) through (e), indicate which of i. through v. is correct. Justify your answer. (a) As we increase λ from 0, the training RSS will: i. Increase initially, and then eventually start decreasing in an inverted U shape. ii. Decrease initially, and then eventually start increasing in a U shape. iii. Steadily increase. iv. Steadily decrease. v. Remain constant. (b) Repeat (a) for test RSS. (c) Repeat (a) for variance. (d) Repeat (a) for (squared) bias. (e) Repeat (a) for the irreducible error. 5. It is well-known that ridge regression tends to give similar coefficient values to correlated variables, whereas the lasso may give quite dif- ferent coefficient values to correlated variables. We will now explore this property in a very simple setting. Suppose that n = 2, p = 2, x11 = x12, x21 = x22. Furthermore, suppose that y1 +y2 = 0 and x11 +x21 = 0 and x12 +x22 = 0, so that the estimate for the intercept in a least squares, ridge regression, or lasso model is zero: β̂0 = 0. (a) Write out the ridge regression optimization problem in this set- ting.
- 293. 6.6 Exercises 285 (b) Argue that in this setting, the ridge coefficient estimates satisfy β̂1 = β̂2. (c) Write out the lasso optimization problem in this setting. (d) Argue that in this setting, the lasso coefficients β̂1 and β̂2 are not unique—in other words, there are many possible solutions to the optimization problem in (c). Describe these solutions. 6. We will now explore (6.12) and (6.13) further. (a) Consider (6.12) with p = 1. For some choice of y1 and λ > 0, plot (6.12) as a function of β1. Your plot should confirm that (6.12) is solved by (6.14). (b) Consider (6.13) with p = 1. For some choice of y1 and λ > 0, plot (6.13) as a function of β1. Your plot should confirm that (6.13) is solved by (6.15). 7. We will now derive the Bayesian connection to the lasso and ridge regression discussed in Section 6.2.2. (a) Suppose that yi = β0 + )p j=1 xijβj +ϵi where ϵ1, . . . , ϵn are inde- pendent and identically distributed from a N(0, σ2 ) distribution. Write out the likelihood for the data. (b) Assume the following prior for β: β1, . . . , βp are independent and identically distributed according to a double-exponential distribution with mean 0 and common scale parameter b: i.e. p(β) = 1 2b exp(−|β|/b). Write out the posterior for β in this setting. (c) Argue that the lasso estimate is the mode for β under this pos- terior distribution. (d) Now assume the following prior for β: β1, . . . , βp are independent and identically distributed according to a normal distribution with mean zero and variance c. Write out the posterior for β in this setting. (e) Argue that the ridge regression estimate is both the mode and the mean for β under this posterior distribution. Applied 8. In this exercise, we will generate simulated data, and will then use this data to perform best subset selection. (a) Use the rnorm() function to generate a predictor X of length n = 100, as well as a noise vector ϵ of length n = 100.
- 294. 286 6. Linear Model Selection and Regularization (b) Generate a response vector Y of length n = 100 according to the model Y = β0 + β1X + β2X2 + β3X3 + ϵ, where β0, β1, β2, and β3 are constants of your choice. (c) Use the regsubsets() function to perform best subset selection in order to choose the best model containing the predictors X, X2 , . . . , X10 . What is the best model obtained according to Cp, BIC, and adjusted R2 ? Show some plots to provide evidence for your answer, and report the coefficients of the best model ob- tained. Note you will need to use the data.frame() function to create a single data set containing both X and Y . (d) Repeat (c), using forward stepwise selection and also using back- wards stepwise selection. How does your answer compare to the results in (c)? (e) Now fit a lasso model to the simulated data, again using X, X2 , . . . , X10 as predictors. Use cross-validation to select the optimal value of λ. Create plots of the cross-validation error as a function of λ. Report the resulting coefficient estimates, and discuss the results obtained. (f) Now generate a response vector Y according to the model Y = β0 + β7X7 + ϵ, and perform best subset selection and the lasso. Discuss the results obtained. 9. In this exercise, we will predict the number of applications received using the other variables in the College data set. (a) Split the data set into a training set and a test set. (b) Fit a linear model using least squares on the training set, and report the test error obtained. (c) Fit a ridge regression model on the training set, with λ chosen by cross-validation. Report the test error obtained. (d) Fit a lasso model on the training set, with λ chosen by cross- validation. Report the test error obtained, along with the num- ber of non-zero coefficient estimates. (e) Fit a PCR model on the training set, with M chosen by cross- validation. Report the test error obtained, along with the value of M selected by cross-validation. (f) Fit a PLS model on the training set, with M chosen by cross- validation. Report the test error obtained, along with the value of M selected by cross-validation.
- 295. 6.6 Exercises 287 (g) Comment on the results obtained. How accurately can we pre- dict the number of college applications received? Is there much difference among the test errors resulting from these five ap- proaches? 10. We have seen that as the number of features used in a model increases, the training error will necessarily decrease, but the test error may not. We will now explore this in a simulated data set. (a) Generate a data set with p = 20 features, n = 1,000 observa- tions, and an associated quantitative response vector generated according to the model Y = Xβ + ϵ, where β has some elements that are exactly equal to zero. (b) Split your data set into a training set containing 100 observations and a test set containing 900 observations. (c) Perform best subset selection on the training set, and plot the training set MSE associated with the best model of each size. (d) Plot the test set MSE associated with the best model of each size. (e) For which model size does the test set MSE take on its minimum value? Comment on your results. If it takes on its minimum value for a model containing only an intercept or a model containing all of the features, then play around with the way that you are generating the data in (a) until you come up with a scenario in which the test set MSE is minimized for an intermediate model size. (f) How does the model at which the test set MSE is minimized compare to the true model used to generate the data? Comment on the coefficient values. (g) Create a plot displaying G)p j=1(βj − β̂r j )2 for a range of values of r, where β̂r j is the jth coefficient estimate for the best model containing r coefficients. Comment on what you observe. How does this compare to the test MSE plot from (d)? 11. We will now try to predict per capita crime rate in the Boston data set. (a) Try out some of the regression methods explored in this chapter, such as best subset selection, the lasso, ridge regression, and PCR. Present and discuss results for the approaches that you consider.
- 296. 288 6. Linear Model Selection and Regularization (b) Propose a model (or set of models) that seem to perform well on this data set, and justify your answer. Make sure that you are evaluating model performance using validation set error, cross- validation, or some other reasonable alternative, as opposed to using training error. (c) Does your chosen model involve all of the features in the data set? Why or why not?
- 297. 7 Moving Beyond Linearity So far in this book, we have mostly focused on linear models. Linear models are relatively simple to describe and implement, and have advantages over other approaches in terms of interpretation and inference. However, stan- dard linear regression can have significant limitations in terms of predic- tive power. This is because the linearity assumption is almost always an approximation, and sometimes a poor one. In Chapter 6 we see that we can improve upon least squares using ridge regression, the lasso, principal com- ponents regression, and other techniques. In that setting, the improvement is obtained by reducing the complexity of the linear model, and hence the variance of the estimates. But we are still using a linear model, which can only be improved so far! In this chapter we relax the linearity assumption while still attempting to maintain as much interpretability as possible. We do this by examining very simple extensions of linear models like polyno- mial regression and step functions, as well as more sophisticated approaches such as splines, local regression, and generalized additive models. • Polynomial regression extends the linear model by adding extra pre- dictors, obtained by raising each of the original predictors to a power. For example, a cubic regression uses three variables, X, X2 , and X3 , as predictors. This approach provides a simple way to provide a non- linear fit to data. • Step functions cut the range of a variable into K distinct regions in order to produce a qualitative variable. This has the effect of fitting a piecewise constant function. © Springer Science+Business Media, LLC, part of Springer Nature 2021 G. James et al., An Introduction to Statistical Learning, Springer Texts in Statistics, https://doi.org/10.1007/978-1-0716-1418-1_7 289
- 298. 290 7. Moving Beyond Linearity • Regression splines are more flexible than polynomials and step func- tions, and in fact are an extension of the two. They involve dividing the range of X into K distinct regions. Within each region, a poly- nomial function is fit to the data. However, these polynomials are constrained so that they join smoothly at the region boundaries, or knots. Provided that the interval is divided into enough regions, this can produce an extremely flexible fit. • Smoothing splines are similar to regression splines, but arise in a slightly different situation. Smoothing splines result from minimizing a residual sum of squares criterion subject to a smoothness penalty. • Local regression is similar to splines, but differs in an important way. The regions are allowed to overlap, and indeed they do so in a very smooth way. • Generalized additive models allow us to extend the methods above to deal with multiple predictors. In Sections 7.1–7.6, we present a number of approaches for modeling the relationship between a response Y and a single predictor X in a flexible way. In Section 7.7, we show that these approaches can be seamlessly inte- grated in order to model a response Y as a function of several predictors X1, . . . , Xp. 7.1 Polynomial Regression Historically, the standard way to extend linear regression to settings in which the relationship between the predictors and the response is non- linear has been to replace the standard linear model yi = β0 + β1xi + ϵi with a polynomial function yi = β0 + β1xi + β2x2 i + β3x3 i + · · · + βdxd i + ϵi, (7.1) where ϵi is the error term. This approach is known as polynomial regression, polynomial regression and in fact we saw an example of this method in Section 3.3.2. For large enough degree d, a polynomial regression allows us to produce an extremely non-linear curve. Notice that the coefficients in (7.1) can be easily estimated using least squares linear regression because this is just a standard linear model with predictors xi, x2 i , x3 i , . . . , xd i . Generally speaking, it is unusual to use d greater than 3 or 4 because for large values of d, the polynomial curve can become overly flexible and can take on some very strange shapes. This is especially true near the boundary of the X variable.
- 299. 7.1 Polynomial Regression 291 20 30 40 50 60 70 80 50 100 150 200 250 300 Age Wage Degree−4 Polynomial 20 30 40 50 60 70 80 0.00 0.05 0.10 0.15 0.20 Age | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | || | Pr(Wage>250 | Age) FIGURE 7.1. The Wage data. Left: The solid blue curve is a degree-4 polynomial of wage (in thousands of dollars) as a function of age, fit by least squares. The dashed curves indicate an estimated 95 % confidence interval. Right: We model the binary event wage>250 using logistic regression, again with a degree-4 polynomial. The fitted posterior probability of wage exceeding $250,000 is shown in blue, along with an estimated 95 % confidence interval. The left-hand panel in Figure 7.1 is a plot of wage against age for the Wage data set, which contains income and demographic information for males who reside in the central Atlantic region of the United States. We see the results of fitting a degree-4 polynomial using least squares (solid blue curve). Even though this is a linear regression model like any other, the individual coefficients are not of particular interest. Instead, we look at the entire fitted function across a grid of 63 values for age from 18 to 80 in order to understand the relationship between age and wage. In Figure 7.1, a pair of dashed curves accompanies the fit; these are (2×) standard error curves. Let’s see how these arise. Suppose we have computed the fit at a particular value of age, x0: ˆ f(x0) = β̂0 + β̂1x0 + β̂2x2 0 + β̂3x3 0 + β̂4x4 0. (7.2) What is the variance of the fit, i.e. Var ˆ f(x0)? Least squares returns variance estimates for each of the fitted coefficients β̂j, as well as the covariances between pairs of coefficient estimates. We can use these to compute the estimated variance of ˆ f(x0).1 The estimated pointwise standard error of 1If Ĉ is the 5 × 5 covariance matrix of the β̂j, and if ℓT 0 = (1, x0, x2 0, x3 0, x4 0), then Var[ ˆ f(x0)] = ℓT 0 Ĉℓ0.
- 300. 292 7. Moving Beyond Linearity ˆ f(x0) is the square-root of this variance. This computation is repeated at each reference point x0, and we plot the fitted curve, as well as twice the standard error on either side of the fitted curve. We plot twice the standard error because, for normally distributed error terms, this quantity corresponds to an approximate 95 % confidence interval. It seems like the wages in Figure 7.1 are from two distinct populations: there appears to be a high earners group earning more than $250,000 per annum, as well as a low earners group. We can treat wage as a binary variable by splitting it into these two groups. Logistic regression can then be used to predict this binary response, using polynomial functions of age as predictors. In other words, we fit the model Pr(yi > 250|xi) = exp(β0 + β1xi + β2x2 i + · · · + βdxd i ) 1 + exp(β0 + β1xi + β2x2 i + · · · + βdxd i ) . (7.3) The result is shown in the right-hand panel of Figure 7.1. The gray marks on the top and bottom of the panel indicate the ages of the high earners and the low earners. The solid blue curve indicates the fitted probabilities of being a high earner, as a function of age. The estimated 95 % confidence interval is shown as well. We see that here the confidence intervals are fairly wide, especially on the right-hand side. Although the sample size for this data set is substantial (n = 3,000), there are only 79 high earners, which results in a high variance in the estimated coefficients and consequently wide confidence intervals. 7.2 Step Functions Using polynomial functions of the features as predictors in a linear model imposes a global structure on the non-linear function of X. We can instead use step functions in order to avoid imposing such a global structure. Here step function we break the range of X into bins, and fit a different constant in each bin. This amounts to converting a continuous variable into an ordered categorical variable. ordered categorical variable In greater detail, we create cutpoints c1, c2, . . . , cK in the range of X, and then construct K + 1 new variables C0(X) = I(X < c1), C1(X) = I(c1 ≤ X < c2), C2(X) = I(c2 ≤ X < c3), . . . CK−1(X) = I(cK−1 ≤ X < cK), CK(X) = I(cK ≤ X), (7.4) where I(·) is an indicator function that returns a 1 if the condition is true, indicator function
- 301. 7.2 Step Functions 293 20 30 40 50 60 70 80 50 100 150 200 250 300 Age Wage Piecewise Constant 20 30 40 50 60 70 80 0.00 0.05 0.10 0.15 0.20 Age | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pr(Wage>250 | Age) FIGURE 7.2. The Wage data. Left: The solid curve displays the fitted value from a least squares regression of wage (in thousands of dollars) using step functions of age. The dashed curves indicate an estimated 95 % confidence interval. Right: We model the binary event wage>250 using logistic regression, again using step functions of age. The fitted posterior probability of wage exceeding $250,000 is shown, along with an estimated 95 % confidence interval. and returns a 0 otherwise. For example, I(cK ≤ X) equals 1 if cK ≤ X, and equals 0 otherwise. These are sometimes called dummy variables. Notice that for any value of X, C0(X) + C1(X) + · · · + CK(X) = 1, since X must be in exactly one of the K + 1 intervals. We then use least squares to fit a linear model using C1(X), C2(X), . . . , CK(X) as predictors2 : yi = β0 + β1C1(xi) + β2C2(xi) + · · · + βKCK(xi) + ϵi. (7.5) For a given value of X, at most one of C1, C2, . . . , CK can be non-zero. Note that when X < c1, all of the predictors in (7.5) are zero, so β0 can be interpreted as the mean value of Y for X < c1. By comparison, (7.5) predicts a response of β0+βj for cj ≤ X < cj+1, so βj represents the average increase in the response for X in cj ≤ X < cj+1 relative to X < c1. An example of fitting step functions to the Wage data from Figure 7.1 is shown in the left-hand panel of Figure 7.2. We also fit the logistic regression model 2We exclude C0(X) as a predictor in (7.5) because it is redundant with the intercept. This is similar to the fact that we need only two dummy variables to code a qualitative variable with three levels, provided that the model will contain an intercept. The decision to exclude C0(X) instead of some other Ck(X) in (7.5) is arbitrary. Alternatively, we could include C0(X), C1(X), . . . , CK (X), and exclude the intercept.
- 302. 294 7. Moving Beyond Linearity Pr(yi > 250|xi) = exp(β0 + β1C1(xi) + · · · + βKCK(xi)) 1 + exp(β0 + β1C1(xi) + · · · + βKCK(xi)) (7.6) in order to predict the probability that an individual is a high earner on the basis of age. The right-hand panel of Figure 7.2 displays the fitted posterior probabilities obtained using this approach. Unfortunately, unless there are natural breakpoints in the predictors, piecewise-constant functions can miss the action. For example, in the left- hand panel of Figure 7.2, the first bin clearly misses the increasing trend of wage with age. Nevertheless, step function approaches are very popular in biostatistics and epidemiology, among other disciplines. For example, 5-year age groups are often used to define the bins. 7.3 Basis Functions Polynomial and piecewise-constant regression models are in fact special cases of a basis function approach. The idea is to have at hand a fam- basis function ily of functions or transformations that can be applied to a variable X: b1(X), b2(X), . . . , bK(X). Instead of fitting a linear model in X, we fit the model yi = β0 + β1b1(xi) + β2b2(xi) + β3b3(xi) + · · · + βKbK(xi) + ϵi. (7.7) Note that the basis functions b1(·), b2(·), . . . , bK(·) are fixed and known. (In other words, we choose the functions ahead of time.) For polynomial regression, the basis functions are bj(xi) = xj i , and for piecewise constant functions they are bj(xi) = I(cj ≤ xi < cj+1). We can think of (7.7) as a standard linear model with predictors b1(xi), b2(xi), . . . , bK(xi). Hence, we can use least squares to estimate the unknown regression coefficients in (7.7). Importantly, this means that all of the inference tools for linear models that are discussed in Chapter 3, such as standard errors for the coefficient estimates and F-statistics for the model’s overall significance, are available in this setting. Thus far we have considered the use of polynomial functions and piece- wise constant functions for our basis functions; however, many alternatives are possible. For instance, we can use wavelets or Fourier series to construct basis functions. In the next section, we investigate a very common choice for a basis function: regression splines. regression spline
- 303. 7.4 Regression Splines 295 7.4 Regression Splines Now we discuss a flexible class of basis functions that extends upon the polynomial regression and piecewise constant regression approaches that we have just seen. 7.4.1 Piecewise Polynomials Instead of fitting a high-degree polynomial over the entire range of X, piece- wise polynomial regression involves fitting separate low-degree polynomials piecewise polynomial regression over different regions of X. For example, a piecewise cubic polynomial works by fitting a cubic regression model of the form yi = β0 + β1xi + β2x2 i + β3x3 i + ϵi, (7.8) where the coefficients β0, β1, β2, and β3 differ in different parts of the range of X. The points where the coefficients change are called knots. knot For example, a piecewise cubic with no knots is just a standard cubic polynomial, as in (7.1) with d = 3. A piecewise cubic polynomial with a single knot at a point c takes the form yi = = β01 + β11xi + β21x2 i + β31x3 i + ϵi if xi < c β02 + β12xi + β22x2 i + β32x3 i + ϵi if xi ≥ c. In other words, we fit two different polynomial functions to the data, one on the subset of the observations with xi < c, and one on the subset of the observations with xi ≥ c. The first polynomial function has coefficients β01, β11, β21, and β31, and the second has coefficients β02, β12, β22, and β32. Each of these polynomial functions can be fit using least squares applied to simple functions of the original predictor. Using more knots leads to a more flexible piecewise polynomial. In gen- eral, if we place K different knots throughout the range of X, then we will end up fitting K + 1 different cubic polynomials. Note that we do not need to use a cubic polynomial. For example, we can instead fit piecewise linear functions. In fact, our piecewise constant functions of Section 7.2 are piecewise polynomials of degree 0! The top left panel of Figure 7.3 shows a piecewise cubic polynomial fit to a subset of the Wage data, with a single knot at age=50. We immediately see a problem: the function is discontinuous and looks ridiculous! Since each polynomial has four parameters, we are using a total of eight degrees of freedom in fitting this piecewise polynomial model. degrees of freedom 7.4.2 Constraints and Splines The top left panel of Figure 7.3 looks wrong because the fitted curve is just too flexible. To remedy this problem, we can fit a piecewise polynomial
- 304. 296 7. Moving Beyond Linearity 20 30 40 50 60 70 50 100 150 200 250 Age Wage Piecewise Cubic 20 30 40 50 60 70 50 100 150 200 250 Age Wage Continuous Piecewise Cubic 20 30 40 50 60 70 50 100 150 200 250 Age Wage Cubic Spline 20 30 40 50 60 70 50 100 150 200 250 Age Wage Linear Spline FIGURE 7.3. Various piecewise polynomials are fit to a subset of the Wage data, with a knot at age=50. Top Left: The cubic polynomials are unconstrained. Top Right: The cubic polynomials are constrained to be continuous at age=50. Bottom Left: The cubic polynomials are constrained to be continuous, and to have continuous first and second derivatives. Bottom Right: A linear spline is shown, which is constrained to be continuous. under the constraint that the fitted curve must be continuous. In other words, there cannot be a jump when age=50. The top right plot in Figure 7.3 shows the resulting fit. This looks better than the top left plot, but the V- shaped join looks unnatural. In the lower left plot, we have added two additional constraints: now both the first and second derivatives of the piecewise polynomials are continuous derivative at age=50. In other words, we are requiring that the piecewise polynomial be not only continuous when age=50, but also very smooth. Each constraint that we impose on the piecewise cubic polynomials effectively frees up one degree of freedom, by reducing the complexity of the resulting piecewise polynomial fit. So in the top left plot, we are using eight degrees of free- dom, but in the bottom left plot we imposed three constraints (continuity, continuity of the first derivative, and continuity of the second derivative)
- 305. 7.4 Regression Splines 297 and so are left with five degrees of freedom. The curve in the bottom left plot is called a cubic spline.3 In general, a cubic spline with K knots uses cubic spline a total of 4 + K degrees of freedom. In Figure 7.3, the lower right plot is a linear spline, which is continuous linear spline at age=50. The general definition of a degree-d spline is that it is a piecewise degree-d polynomial, with continuity in derivatives up to degree d − 1 at each knot. Therefore, a linear spline is obtained by fitting a line in each region of the predictor space defined by the knots, requiring continuity at each knot. In Figure 7.3, there is a single knot at age=50. Of course, we could add more knots, and impose continuity at each. 7.4.3 The Spline Basis Representation The regression splines that we just saw in the previous section may have seemed somewhat complex: how can we fit a piecewise degree-d polynomial under the constraint that it (and possibly its first d − 1 derivatives) be continuous? It turns out that we can use the basis model (7.7) to represent a regression spline. A cubic spline with K knots can be modeled as yi = β0 + β1b1(xi) + β2b2(xi) + · · · + βK+3bK+3(xi) + ϵi, (7.9) for an appropriate choice of basis functions b1, b2, . . . , bK+3. The model (7.9) can then be fit using least squares. Just as there were several ways to represent polynomials, there are also many equivalent ways to represent cubic splines using different choices of basis functions in (7.9). The most direct way to represent a cubic spline using (7.9) is to start off with a basis for a cubic polynomial—namely, x, x2 , and x3 —and then add one truncated power basis function per knot. truncated power basis A truncated power basis function is defined as h(x, ξ) = (x − ξ)3 + = K (x − ξ)3 if x > ξ 0 otherwise, (7.10) where ξ is the knot. One can show that adding a term of the form β4h(x, ξ) to the model (7.8) for a cubic polynomial will lead to a discontinuity in only the third derivative at ξ; the function will remain continuous, with continuous first and second derivatives, at each of the knots. In other words, in order to fit a cubic spline to a data set with K knots, we perform least squares regression with an intercept and 3 + K predictors, of the form X, X2 , X3 , h(X, ξ1), h(X, ξ2), . . . , h(X, ξK), where ξ1, . . . , ξK are the knots. This amounts to estimating a total of K + 4 regression coeffi- cients; for this reason, fitting a cubic spline with K knots uses K+4 degrees of freedom. 3Cubic splines are popular because most human eyes cannot detect the discontinuity at the knots.
- 306. 298 7. Moving Beyond Linearity 20 30 40 50 60 70 50 100 150 200 250 Age Wage Natural Cubic Spline Cubic Spline FIGURE 7.4. A cubic spline and a natural cubic spline, with three knots, fit to a subset of the Wage data. The dashed lines denote the knot locations. Unfortunately, splines can have high variance at the outer range of the predictors—that is, when X takes on either a very small or very large value. Figure 7.4 shows a fit to the Wage data with three knots. We see that the confidence bands in the boundary region appear fairly wild. A natu- ral spline is a regression spline with additional boundary constraints: the natural spline function is required to be linear at the boundary (in the region where X is smaller than the smallest knot, or larger than the largest knot). This addi- tional constraint means that natural splines generally produce more stable estimates at the boundaries. In Figure 7.4, a natural cubic spline is also displayed as a red line. Note that the corresponding confidence intervals are narrower. 7.4.4 Choosing the Number and Locations of the Knots When we fit a spline, where should we place the knots? The regression spline is most flexible in regions that contain a lot of knots, because in those regions the polynomial coefficients can change rapidly. Hence, one option is to place more knots in places where we feel the function might vary most rapidly, and to place fewer knots where it seems more stable. While this option can work well, in practice it is common to place knots in a uniform fashion. One way to do this is to specify the desired degrees of freedom, and then have the software automatically place the corresponding number of knots at uniform quantiles of the data. Figure 7.5 shows an example on the Wage data. As in Figure 7.4, we have fit a natural cubic spline with three knots, except this time the knot locations were chosen automatically as the 25th, 50th, and 75th percentiles of age. This was specified by requesting four degrees of freedom. The ar-
- 307. 7.4 Regression Splines 299 20 30 40 50 60 70 80 50 100 150 200 250 300 Age Wage Natural Cubic Spline 20 30 40 50 60 70 80 0.00 0.05 0.10 0.15 0.20 Age | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | || | | || | Pr(Wage>250 | Age) FIGURE 7.5. A natural cubic spline function with four degrees of freedom is fit to the Wage data. Left: A spline is fit to wage (in thousands of dollars) as a function of age. Right: Logistic regression is used to model the binary event wage>250 as a function of age. The fitted posterior probability of wage exceeding $250,000 is shown. The dashed lines denote the knot locations. gument by which four degrees of freedom leads to three interior knots is somewhat technical.4 How many knots should we use, or equivalently how many degrees of freedom should our spline contain? One option is to try out different num- bers of knots and see which produces the best looking curve. A somewhat more objective approach is to use cross-validation, as discussed in Chap- ters 5 and 6. With this method, we remove a portion of the data (say 10 %), fit a spline with a certain number of knots to the remaining data, and then use the spline to make predictions for the held-out portion. We repeat this process multiple times until each observation has been left out once, and then compute the overall cross-validated RSS. This procedure can be re- peated for different numbers of knots K. Then the value of K giving the smallest RSS is chosen. Figure 7.6 shows ten-fold cross-validated mean squared errors for splines with various degrees of freedom fit to the Wage data. The left-hand panel 4There are actually five knots, including the two boundary knots. A cubic spline with five knots has nine degrees of freedom. But natural cubic splines have two additional natural constraints at each boundary to enforce linearity, resulting in 9 − 4 = 5 degrees of freedom. Since this includes a constant, which is absorbed in the intercept, we count it as four degrees of freedom.
- 308. 300 7. Moving Beyond Linearity 2 4 6 8 10 1600 1620 1640 1660 1680 Degrees of Freedom of Natural Spline Mean Squared Error 2 4 6 8 10 1600 1620 1640 1660 1680 Degrees of Freedom of Cubic Spline Mean Squared Error FIGURE 7.6. Ten-fold cross-validated mean squared errors for selecting the degrees of freedom when fitting splines to the Wage data. The response is wage and the predictor age. Left: A natural cubic spline. Right: A cubic spline. corresponds to a natural cubic spline and the right-hand panel to a cu- bic spline. The two methods produce almost identical results, with clear evidence that a one-degree fit (a linear regression) is not adequate. Both curves flatten out quickly, and it seems that three degrees of freedom for the natural spline and four degrees of freedom for the cubic spline are quite adequate. In Section 7.7 we fit additive spline models simultaneously on several variables at a time. This could potentially require the selection of degrees of freedom for each variable. In cases like this we typically adopt a more pragmatic approach and set the degrees of freedom to a fixed number, say four, for all terms. 7.4.5 Comparison to Polynomial Regression Figure 7.7 compares a natural cubic spline with 15 degrees of freedom to a degree-15 polynomial on the Wage data set. The extra flexibility in the poly- nomial produces undesirable results at the boundaries, while the natural cubic spline still provides a reasonable fit to the data. Regression splines often give superior results to polynomial regression. This is because unlike polynomials, which must use a high degree (exponent in the highest mono- mial term, e.g. X15 ) to produce flexible fits, splines introduce flexibility by increasing the number of knots but keeping the degree fixed. Generally, this approach produces more stable estimates. Splines also allow us to place more knots, and hence flexibility, over regions where the function f seems to be changing rapidly, and fewer knots where f appears more stable.
- 309. 7.5 Smoothing Splines 301 20 30 40 50 60 70 80 50 100 150 200 250 300 Age Wage Natural Cubic Spline Polynomial FIGURE 7.7. On the Wage data set, a natural cubic spline with 15 degrees of freedom is compared to a degree-15 polynomial. Polynomials can show wild behavior, especially near the tails. 7.5 Smoothing Splines In the last section we discussed regression splines, which we create by spec- ifying a set of knots, producing a sequence of basis functions, and then using least squares to estimate the spline coefficients. We now introduce a somewhat different approach that also produces a spline. 7.5.1 An Overview of Smoothing Splines In fitting a smooth curve to a set of data, what we really want to do is find some function, say g(x), that fits the observed data well: that is, we want RSS = )n i=1(yi − g(xi))2 to be small. However, there is a problem with this approach. If we don’t put any constraints on g(xi), then we can always make RSS zero simply by choosing g such that it interpolates all of the yi. Such a function would woefully overfit the data—it would be far too flexible. What we really want is a function g that makes RSS small, but that is also smooth. How might we ensure that g is smooth? There are a number of ways to do this. A natural approach is to find the function g that minimizes n 0 i=1 (yi − g(xi))2 + λ L g′′ (t)2 dt (7.11) where λ is a nonnegative tuning parameter. The function g that minimizes (7.11) is known as a smoothing spline. smoothing spline What does (7.11) mean? Equation 7.11 takes the “Loss+Penalty” for- mulation that we encounter in the context of ridge regression and the lasso
- 310. 302 7. Moving Beyond Linearity in Chapter 6. The term )n i=1(yi − g(xi))2 is a loss function that encour- loss function ages g to fit the data well, and the term λ M g′′ (t)2 dt is a penalty term that penalizes the variability in g. The notation g′′ (t) indicates the second derivative of the function g. The first derivative g′ (t) measures the slope of a function at t, and the second derivative corresponds to the amount by which the slope is changing. Hence, broadly speaking, the second derivative of a function is a measure of its roughness: it is large in absolute value if g(t) is very wiggly near t, and it is close to zero otherwise. (The second derivative of a straight line is zero; note that a line is perfectly smooth.) The M notation is an integral, which we can think of as a summation over the range of t. In other words, M g′′ (t)2 dt is simply a measure of the total change in the function g′ (t), over its entire range. If g is very smooth, then g′ (t) will be close to constant and M g′′ (t)2 dt will take on a small value. Conversely, if g is jumpy and variable then g′ (t) will vary significantly and M g′′ (t)2 dt will take on a large value. Therefore, in (7.11), λ M g′′ (t)2 dt en- courages g to be smooth. The larger the value of λ, the smoother g will be. When λ = 0, then the penalty term in (7.11) has no effect, and so the function g will be very jumpy and will exactly interpolate the training observations. When λ → ∞, g will be perfectly smooth—it will just be a straight line that passes as closely as possible to the training points. In fact, in this case, g will be the linear least squares line, since the loss function in (7.11) amounts to minimizing the residual sum of squares. For an intermediate value of λ, g will approximate the training observations but will be somewhat smooth. We see that λ controls the bias-variance trade-off of the smoothing spline. The function g(x) that minimizes (7.11) can be shown to have some spe- cial properties: it is a piecewise cubic polynomial with knots at the unique values of x1, . . . , xn, and continuous first and second derivatives at each knot. Furthermore, it is linear in the region outside of the extreme knots. In other words, the function g(x) that minimizes (7.11) is a natural cubic spline with knots at x1, . . . , xn! However, it is not the same natural cubic spline that one would get if one applied the basis function approach de- scribed in Section 7.4.3 with knots at x1, . . . , xn—rather, it is a shrunken version of such a natural cubic spline, where the value of the tuning pa- rameter λ in (7.11) controls the level of shrinkage. 7.5.2 Choosing the Smoothing Parameter λ We have seen that a smoothing spline is simply a natural cubic spline with knots at every unique value of xi. It might seem that a smoothing spline will have far too many degrees of freedom, since a knot at each data point allows a great deal of flexibility. But the tuning parameter λ controls the roughness of the smoothing spline, and hence the effective degrees of freedom. It is possible to show that as λ increases from 0 to ∞, the effective effective degrees of freedom degrees of freedom, which we write dfλ, decrease from n to 2.
- 311. 7.5 Smoothing Splines 303 In the context of smoothing splines, why do we discuss effective degrees of freedom instead of degrees of freedom? Usually degrees of freedom refer to the number of free parameters, such as the number of coefficients fit in a polynomial or cubic spline. Although a smoothing spline has n parameters and hence n nominal degrees of freedom, these n parameters are heavily constrained or shrunk down. Hence dfλ is a measure of the flexibility of the smoothing spline—the higher it is, the more flexible (and the lower-bias but higher-variance) the smoothing spline. The definition of effective degrees of freedom is somewhat technical. We can write ĝλ = Sλy, (7.12) where ĝλ is the solution to (7.11) for a particular choice of λ—that is, it is an n-vector containing the fitted values of the smoothing spline at the training points x1, . . . , xn. Equation 7.12 indicates that the vector of fitted values when applying a smoothing spline to the data can be written as a n × n matrix Sλ (for which there is a formula) times the response vector y. Then the effective degrees of freedom is defined to be dfλ = n 0 i=1 {Sλ}ii, (7.13) the sum of the diagonal elements of the matrix Sλ. In fitting a smoothing spline, we do not need to select the number or location of the knots—there will be a knot at each training observation, x1, . . . , xn. Instead, we have another problem: we need to choose the value of λ. It should come as no surprise that one possible solution to this problem is cross-validation. In other words, we can find the value of λ that makes the cross-validated RSS as small as possible. It turns out that the leave- one-out cross-validation error (LOOCV) can be computed very efficiently for smoothing splines, with essentially the same cost as computing a single fit, using the following formula: RSScv(λ) = n 0 i=1 (yi − ĝ (−i) λ (xi))2 = n 0 i=1 3 yi − ĝλ(xi) 1 − {Sλ}ii 42 . The notation ĝ (−i) λ (xi) indicates the fitted value for this smoothing spline evaluated at xi, where the fit uses all of the training observations except for the ith observation (xi, yi). In contrast, ĝλ(xi) indicates the smoothing spline function fit to all of the training observations and evaluated at xi. This remarkable formula says that we can compute each of these leave- one-out fits using only ĝλ, the original fit to all of the data!5 We have 5The exact formulas for computing ĝ(xi) and Sλ are very technical; however, efficient algorithms are available for computing these quantities.
- 312. 304 7. Moving Beyond Linearity 20 30 40 50 60 70 80 0 50 100 200 300 Age Wage Smoothing Spline 16 Degrees of Freedom 6.8 Degrees of Freedom (LOOCV) FIGURE 7.8. Smoothing spline fits to the Wage data. The red curve results from specifying 16 effective degrees of freedom. For the blue curve, λ was found automatically by leave-one-out cross-validation, which resulted in 6.8 effective degrees of freedom. a very similar formula (5.2) on page 202 in Chapter 5 for least squares linear regression. Using (5.2), we can very quickly perform LOOCV for the regression splines discussed earlier in this chapter, as well as for least squares regression using arbitrary basis functions. Figure 7.8 shows the results from fitting a smoothing spline to the Wage data. The red curve indicates the fit obtained from pre-specifying that we would like a smoothing spline with 16 effective degrees of freedom. The blue curve is the smoothing spline obtained when λ is chosen using LOOCV; in this case, the value of λ chosen results in 6.8 effective degrees of freedom (computed using (7.13)). For this data, there is little discernible difference between the two smoothing splines, beyond the fact that the one with 16 degrees of freedom seems slightly wigglier. Since there is little difference between the two fits, the smoothing spline fit with 6.8 degrees of freedom is preferable, since in general simpler models are better unless the data provides evidence in support of a more complex model. 7.6 Local Regression Local regression is a different approach for fitting flexible non-linear func- local regression tions, which involves computing the fit at a target point x0 using only the nearby training observations. Figure 7.9 illustrates the idea on some simu- lated data, with one target point near 0.4, and another near the boundary
- 313. 7.6 Local Regression 305 0.0 0.2 0.4 0.6 0.8 1.0 −1.0 −0.5 0.0 0.5 1.0 1.5 O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O 0.0 0.2 0.4 0.6 0.8 1.0 −1.0 −0.5 0.0 0.5 1.0 1.5 O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O Local Regression FIGURE 7.9. Local regression illustrated on some simulated data, where the blue curve represents f(x) from which the data were generated, and the light orange curve corresponds to the local regression estimate ˆ f(x). The orange colored points are local to the target point x0, represented by the orange vertical line. The yellow bell-shape superimposed on the plot indicates weights assigned to each point, decreasing to zero with distance from the target point. The fit ˆ f(x0) at x0 is obtained by fitting a weighted linear regression (orange line segment), and using the fitted value at x0 (orange solid dot) as the estimate ˆ f(x0). at 0.05. In this figure the blue line represents the function f(x) from which the data were generated, and the light orange line corresponds to the local regression estimate ˆ f(x). Local regression is described in Algorithm 7.1. Note that in Step 3 of Algorithm 7.1, the weights Ki0 will differ for each value of x0. In other words, in order to obtain the local regression fit at a new point, we need to fit a new weighted least squares regression model by minimizing (7.14) for a new set of weights. Local regression is sometimes referred to as a memory-based procedure, because like nearest-neighbors, we need all the training data each time we wish to compute a prediction. We will avoid getting into the technical details of local regression here—there are books written on the topic. In order to perform local regression, there are a number of choices to be made, such as how to define the weighting function K, and whether to fit a linear, constant, or quadratic regression in Step 3. (Equation 7.14 corresponds to a linear regression.) While all of these choices make some difference, the most important choice is the span s, which is the proportion of points used to compute the local regression at x0, as defined in Step 1 above. The span plays a role like that of the tuning parameter λ in smooth- ing splines: it controls the flexibility of the non-linear fit. The smaller the value of s, the more local and wiggly will be our fit; alternatively, a very large value of s will lead to a global fit to the data using all of the train- ing observations. We can again use cross-validation to choose s, or we can
- 314. 306 7. Moving Beyond Linearity Algorithm 7.1 Local Regression At X = x0 1. Gather the fraction s = k/n of training points whose xi are closest to x0. 2. Assign a weight Ki0 = K(xi, x0) to each point in this neighborhood, so that the point furthest from x0 has weight zero, and the closest has the highest weight. All but these k nearest neighbors get weight zero. 3. Fit a weighted least squares regression of the yi on the xi using the aforementioned weights, by finding β̂0 and β̂1 that minimize n 0 i=1 Ki0(yi − β0 − β1xi)2 . (7.14) 4. The fitted value at x0 is given by ˆ f(x0) = β̂0 + β̂1x0. specify it directly. Figure 7.10 displays local linear regression fits on the Wage data, using two values of s: 0.7 and 0.2. As expected, the fit obtained using s = 0.7 is smoother than that obtained using s = 0.2. The idea of local regression can be generalized in many different ways. In a setting with multiple features X1, X2, . . . , Xp, one very useful general- ization involves fitting a multiple linear regression model that is global in some variables, but local in another, such as time. Such varying coefficient models are a useful way of adapting a model to the most recently gathered varying coefficient model data. Local regression also generalizes very naturally when we want to fit models that are local in a pair of variables X1 and X2, rather than one. We can simply use two-dimensional neighborhoods, and fit bivariate linear regression models using the observations that are near each target point in two-dimensional space. Theoretically the same approach can be imple- mented in higher dimensions, using linear regressions fit to p-dimensional neighborhoods. However, local regression can perform poorly if p is much larger than about 3 or 4 because there will generally be very few training observations close to x0. Nearest-neighbors regression, discussed in Chap- ter 3, suffers from a similar problem in high dimensions. 7.7 Generalized Additive Models In Sections 7.1–7.6, we present a number of approaches for flexibly predict- ing a response Y on the basis of a single predictor X. These approaches can be seen as extensions of simple linear regression. Here we explore the prob-
- 315. 7.7 Generalized Additive Models 307 20 30 40 50 60 70 80 0 50 100 200 300 Age Wage Local Linear Regression Span is 0.2 (16.4 Degrees of Freedom) Span is 0.7 (5.3 Degrees of Freedom) FIGURE 7.10. Local linear fits to the Wage data. The span specifies the fraction of the data used to compute the fit at each target point. lem of flexibly predicting Y on the basis of several predictors, X1, . . . , Xp. This amounts to an extension of multiple linear regression. Generalized additive models (GAMs) provide a general framework for generalized additive model extending a standard linear model by allowing non-linear functions of each of the variables, while maintaining additivity. Just like linear models, GAMs additivity can be applied with both quantitative and qualitative responses. We first examine GAMs for a quantitative response in Section 7.7.1, and then for a qualitative response in Section 7.7.2. 7.7.1 GAMs for Regression Problems A natural way to extend the multiple linear regression model yi = β0 + β1xi1 + β2xi2 + · · · + βpxip + ϵi in order to allow for non-linear relationships between each feature and the response is to replace each linear component βjxij with a (smooth) non- linear function fj(xij). We would then write the model as yi = β0 + p 0 j=1 fj(xij) + ϵi = β0 + f1(xi1) + f2(xi2) + · · · + fp(xip) + ϵi. (7.15) This is an example of a GAM. It is called an additive model because we calculate a separate fj for each Xj, and then add together all of their contributions.
- 316. 308 7. Moving Beyond Linearity 2003 2005 2007 2009 −30 −20 −10 0 10 20 30 20 30 40 50 60 70 80 −50 −40 −30 −20 −10 0 10 20 −30 −20 −10 0 10 20 30 40 <HS HS <Coll Coll >Coll f 1 (year) f 2 (age) f 3 (education) year age education FIGURE 7.11. For the Wage data, plots of the relationship between each feature and the response, wage, in the fitted model (7.16). Each plot displays the fitted function and pointwise standard errors. The first two functions are natural splines in year and age, with four and five degrees of freedom, respectively. The third function is a step function, fit to the qualitative variable education. In Sections 7.1–7.6, we discuss many methods for fitting functions to a single variable. The beauty of GAMs is that we can use these methods as building blocks for fitting an additive model. In fact, for most of the methods that we have seen so far in this chapter, this can be done fairly trivially. Take, for example, natural splines, and consider the task of fitting the model wage = β0 + f1(year) + f2(age) + f3(education) + ϵ (7.16) on the Wage data. Here year and age are quantitative variables, and education is a qualitative variable with five levels: <HS, HS, <Coll, Coll, >Coll, refer- ring to the amount of high school or college education that an individual has completed. We fit the first two functions using natural splines. We fit the third function using a separate constant for each level, via the usual dummy variable approach of Section 3.3.1. Figure 7.11 shows the results of fitting the model (7.16) using least squares. This is easy to do, since as discussed in Section 7.4, natural splines can be constructed using an appropriately chosen set of basis functions. Hence the entire model is just a big regression onto spline basis variables and dummy variables, all packed into one big regression matrix. Figure 7.11 can be easily interpreted. The left-hand panel indicates that holding age and education fixed, wage tends to increase slightly with year; this may be due to inflation. The center panel indicates that holding education and year fixed, wage tends to be highest for intermediate values of age, and lowest for the very young and very old. The right-hand panel indicates that holding year and age fixed, wage tends to increase with education: the more educated a person is, the higher their salary, on average. All of these findings are intuitive.
- 317. 7.7 Generalized Additive Models 309 2003 2005 2007 2009 −30 −20 −10 0 10 20 30 20 30 40 50 60 70 80 −50 −40 −30 −20 −10 0 10 20 −30 −20 −10 0 10 20 30 40 <HS HS <Coll Coll >Coll f 1 (year) f 2 (age) f 3 (education) year age education FIGURE 7.12. Details are as in Figure 7.11, but now f1 and f2 are smoothing splines with four and five degrees of freedom, respectively. Figure 7.12 shows a similar triple of plots, but this time f1 and f2 are smoothing splines with four and five degrees of freedom, respectively. Fit- ting a GAM with a smoothing spline is not quite as simple as fitting a GAM with a natural spline, since in the case of smoothing splines, least squares cannot be used. However, standard software such as the gam() function in R can be used to fit GAMs using smoothing splines, via an approach known as backfitting. This method fits a model involving multiple predictors by backfitting repeatedly updating the fit for each predictor in turn, holding the others fixed. The beauty of this approach is that each time we update a function, we simply apply the fitting method for that variable to a partial residual.6 The fitted functions in Figures 7.11 and 7.12 look rather similar. In most situations, the differences in the GAMs obtained using smoothing splines versus natural splines are small. We do not have to use splines as the building blocks for GAMs: we can just as well use local regression, polynomial regression, or any combination of the approaches seen earlier in this chapter in order to create a GAM. GAMs are investigated in further detail in the lab at the end of this chapter. Pros and Cons of GAMs Before we move on, let us summarize the advantages and limitations of a GAM. ▲ GAMs allow us to fit a non-linear fj to each Xj, so that we can automatically model non-linear relationships that standard linear re- gression will miss. This means that we do not need to manually try out many different transformations on each variable individually. 6A partial residual for X3, for example, has the form ri = yi − f1(xi1) − f2(xi2). If we know f1 and f2, then we can fit f3 by treating this residual as a response in a non-linear regression on X3.
- 318. 310 7. Moving Beyond Linearity ▲ The non-linear fits can potentially make more accurate predictions for the response Y . ▲ Because the model is additive, we can examine the effect of each Xj on Y individually while holding all of the other variables fixed. ▲ The smoothness of the function fj for the variable Xj can be sum- marized via degrees of freedom. ◆ The main limitation of GAMs is that the model is restricted to be additive. With many variables, important interactions can be missed. However, as with linear regression, we can manually add interaction terms to the GAM model by including additional predictors of the form Xj × Xk. In addition we can add low-dimensional interaction functions of the form fjk(Xj, Xk) into the model; such terms can be fit using two-dimensional smoothers such as local regression, or two-dimensional splines (not covered here). For fully general models, we have to look for even more flexible approaches such as random forests and boosting, described in Chapter 8. GAMs provide a useful compromise between linear and fully nonparametric models. 7.7.2 GAMs for Classification Problems GAMs can also be used in situations where Y is qualitative. For simplicity, here we will assume Y takes on values zero or one, and let p(X) = Pr(Y = 1|X) be the conditional probability (given the predictors) that the response equals one. Recall the logistic regression model (4.6): log * p(X) 1 − p(X) + = β0 + β1X1 + β2X2 + · · · + βpXp. (7.17) The left-hand side is the log of the odds of P(Y = 1|X) versus P(Y = 0|X), which (7.17) represents as a linear function of the predictors. A natural way to extend (7.17) to allow for non-linear relationships is to use the model log * p(X) 1 − p(X) + = β0 + f1(X1) + f2(X2) + · · · + fp(Xp). (7.18) Equation 7.18 is a logistic regression GAM. It has all the same pros and cons as discussed in the previous section for quantitative responses. We fit a GAM to the Wage data in order to predict the probability that an individual’s income exceeds $250,000 per year. The GAM that we fit takes the form log * p(X) 1 − p(X) + = β0 + β1 × year + f2(age) + f3(education), (7.19) where p(X) = Pr(wage > 250|year, age, education).
- 319. 7.8 Lab: Non-linear Modeling 311 2003 2005 2007 2009 −4 −2 0 2 4 20 30 40 50 60 70 80 −8 −6 −4 −2 0 2 −400 −200 0 200 400 <HS HS <Coll Coll >Coll f 1 (year) f 2 (age) f 3 (education) year age education FIGURE 7.13. For the Wage data, the logistic regression GAM given in (7.19) is fit to the binary response I(wage>250). Each plot displays the fitted function and pointwise standard errors. The first function is linear in year, the second function a smoothing spline with five degrees of freedom in age, and the third a step function for education. There are very wide standard errors for the first level <HS of education. Once again f2 is fit using a smoothing spline with five degrees of freedom, and f3 is fit as a step function, by creating dummy variables for each of the levels of education. The resulting fit is shown in Figure 7.13. The last panel looks suspicious, with very wide confidence intervals for level <HS. In fact, no response values equal one for that category: no individuals with less than a high school education make more than $250,000 per year. Hence we refit the GAM, excluding the individuals with less than a high school education. The resulting model is shown in Figure 7.14. As in Figures 7.11 and 7.12, all three panels have similar vertical scales. This allows us to visually assess the relative contributions of each of the variables. We observe that age and education have a much larger effect than year on the probability of being a high earner. 7.8 Lab: Non-linear Modeling In this lab, we re-analyze the Wage data considered in the examples through- out this chapter, in order to illustrate the fact that many of the complex non-linear fitting procedures discussed can be easily implemented in R. We begin by loading the ISLR2 library, which contains the data. > library(ISLR2) > attach(Wage)
- 320. 312 7. Moving Beyond Linearity 2003 2005 2007 2009 −4 −2 0 2 4 20 30 40 50 60 70 80 −8 −6 −4 −2 0 2 −4 −2 0 2 4 HS <Coll Coll >Coll f 1 (year) f 2 (age) f 3 (education) year age education FIGURE 7.14. The same model is fit as in Figure 7.13, this time excluding the observations for which education is <HS. Now we see that increased education tends to be associated with higher salaries. 7.8.1 Polynomial Regression and Step Functions We now examine how Figure 7.1 was produced. We first fit the model using the following command: > fit <- lm(wage ∼ poly(age , 4), data = Wage) > coef(summary(fit)) Estimate Std. Error t value Pr(>|t|) (Intercept) 111.704 0.729 153.28 <2e -16 poly(age , 4)1 447.068 39.915 11.20 <2e -16 poly(age , 4)2 -478.316 39.915 -11.98 <2e-16 poly(age , 4)3 125.522 39.915 3.14 0.0017 poly(age , 4)4 -77.911 39.915 -1.95 0.0510 This syntax fits a linear model, using the lm() function, in order to predict wage using a fourth-degree polynomial in age: poly(age, 4). The poly() command allows us to avoid having to write out a long formula with pow- ers of age. The function returns a matrix whose columns are a basis of orthogonal polynomials, which essentially means that each column is a lin- orthogonal polynomial ear combination of the variables age, age^2, age^3 and age^4. However, we can also use poly() to obtain age, age^2, age^3 and age^4 directly, if we prefer. We can do this by using the raw = TRUE argument to the poly() function. Later we see that this does not affect the model in a meaningful way—though the choice of basis clearly affects the coefficient estimates, it does not affect the fitted values obtained. > fit2 <- lm(wage ∼ poly(age , 4, raw = T), data = Wage) > coef(summary(fit2)) Estimate Std. Error t value Pr(>|t|) (Intercept) -1.84e+02 6.00e+01 -3.07 0.002180 poly(age , 4, raw = T)1 2.12e+01 5.89e+00 3.61 0.000312 poly(age , 4, raw = T)2 -5.64e -01 2.06e -01 -2.74 0.006261 poly(age , 4, raw = T)3 6.81e -03 3.07e -03 2.22 0.026398 poly(age , 4, raw = T)4 -3.20e -05 1.64e -05 -1.95 0.051039
- 321. 7.8 Lab: Non-linear Modeling 313 There are several other equivalent ways of fitting this model, which show- case the flexibility of the formula language in R. For example > fit2a <- lm(wage ∼ age + I(age ^2) + I(age ^3) + I(age ^4) , data = Wage) > coef(fit2a) (Intercept) age I(age ^2) I(age ^3) I(age ^4) -1.84e+02 2.12e+01 -5.64e-01 6.81e-03 -3.20e -05 This simply creates the polynomial basis functions on the fly, taking care to protect terms like age^2 via the wrapper function I() (the ^ symbol has wrapper a special meaning in formulas). > fit2b <- lm(wage ∼ cbind(age , age^2, age^3, age ^4) , data = Wage) This does the same more compactly, using the cbind() function for building a matrix from a collection of vectors; any function call such as cbind() inside a formula also serves as a wrapper. We now create a grid of values for age at which we want predictions, and then call the generic predict() function, specifying that we want standard errors as well. > agelims <- range(age) > age.grid <- seq(from = agelims [1], to = agelims [2]) > preds <- predict(fit , newdata = list(age = age.grid), se = TRUE) > se.bands <- cbind(preds$fit + 2 * preds$se.fit , preds$fit - 2 * preds$se.fit) Finally, we plot the data and add the fit from the degree-4 polynomial. > par(mfrow = c(1, 2), mar = c(4.5 , 4.5, 1, 1), oma = c(0, 0, 4, 0)) > plot(age , wage , xlim = agelims , cex = .5, col = "darkgrey") > title("Degree -4 Polynomial ", outer = T) > lines(age.grid , preds$fit , lwd = 2, col = "blue") > matlines (age.grid , se.bands , lwd = 1, col = "blue", lty = 3) Here the mar and oma arguments to par() allow us to control the margins of the plot, and the title() function creates a figure title that spans both title() subplots. We mentioned earlier that whether or not an orthogonal set of basis func- tions is produced in the poly() function will not affect the model obtained in a meaningful way. What do we mean by this? The fitted values obtained in either case are identical: > preds2 <- predict(fit2 , newdata = list(age = age.grid), se = TRUE) > max(abs(preds$fit - preds2$fit)) [1] 7.82e-11 In performing a polynomial regression we must decide on the degree of the polynomial to use. One way to do this is by using hypothesis tests. We
- 322. 314 7. Moving Beyond Linearity now fit models ranging from linear to a degree-5 polynomial and seek to determine the simplest model which is sufficient to explain the relationship between wage and age. We use the anova() function, which performs an anova() analysis of variance (ANOVA, using an F-test) in order to test the null analysis of variance hypothesis that a model M1 is sufficient to explain the data against the alternative hypothesis that a more complex model M2 is required. In order to use the anova() function, M1 and M2 must be nested models: the predictors in M1 must be a subset of the predictors in M2. In this case, we fit five different models and sequentially compare the simpler model to the more complex model. > fit.1 <- lm(wage ∼ age , data = Wage) > fit.2 <- lm(wage ∼ poly(age , 2), data = Wage) > fit.3 <- lm(wage ∼ poly(age , 3), data = Wage) > fit.4 <- lm(wage ∼ poly(age , 4), data = Wage) > fit.5 <- lm(wage ∼ poly(age , 5), data = Wage) > anova(fit.1, fit.2, fit.3, fit.4, fit .5) Analysis of Variance Table Model 1: wage ∼ age Model 2: wage ∼ poly(age , 2) Model 3: wage ∼ poly(age , 3) Model 4: wage ∼ poly(age , 4) Model 5: wage ∼ poly(age , 5) Res.Df RSS Df Sum of Sq F Pr(>F) 1 2998 5022216 2 2997 4793430 1 228786 143.59 <2e -16 *** 3 2996 4777674 1 15756 9.89 0.0017 ** 4 2995 4771604 1 6070 3.81 0.0510 . 5 2994 4770322 1 1283 0.80 0.3697 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 The p-value comparing the linear Model 1 to the quadratic Model 2 is essentially zero (<10−15 ), indicating that a linear fit is not sufficient. Sim- ilarly the p-value comparing the quadratic Model 2 to the cubic Model 3 is very low (0.0017), so the quadratic fit is also insufficient. The p-value comparing the cubic and degree-4 polynomials, Model 3 and Model 4, is ap- proximately 5 % while the degree-5 polynomial Model 5 seems unnecessary because its p-value is 0.37. Hence, either a cubic or a quartic polynomial appear to provide a reasonable fit to the data, but lower- or higher-order models are not justified. In this case, instead of using the anova() function, we could have obtained these p-values more succinctly by exploiting the fact that poly() creates orthogonal polynomials. > coef(summary(fit .5)) Estimate Std. Error t value Pr(>|t|) (Intercept) 111.70 0.7288 153.2780 0.000e+00 poly(age , 5)1 447.07 39.9161 11.2002 1.491e-28
- 323. 7.8 Lab: Non-linear Modeling 315 poly(age , 5)2 -478.32 39.9161 -11.9830 2.368e -32 poly(age , 5)3 125.52 39.9161 3.1446 1.679e -03 poly(age , 5)4 -77.91 39.9161 -1.9519 5.105e-02 poly(age , 5)5 -35.81 39.9161 -0.8972 3.697e-01 Notice that the p-values are the same, and in fact the square of the t-statistics are equal to the F-statistics from the anova() function; for example: > ( -11.983) ^2 [1] 143.6 However, the ANOVA method works whether or not we used orthogonal polynomials; it also works when we have other terms in the model as well. For example, we can use anova() to compare these three models: > fit.1 <- lm(wage ∼ education + age , data = Wage) > fit.2 <- lm(wage ∼ education + poly(age , 2), data = Wage) > fit.3 <- lm(wage ∼ education + poly(age , 3), data = Wage) > anova(fit.1, fit.2, fit .3) As an alternative to using hypothesis tests and ANOVA, we could choose the polynomial degree using cross-validation, as discussed in Chapter 5. Next we consider the task of predicting whether an individual earns more than $250,000 per year. We proceed much as before, except that first we create the appropriate response vector, and then apply the glm() function using family = "binomial" in order to fit a polynomial logistic regression model. > fit <- glm(I(wage > 250) ∼ poly(age , 4), data = Wage , family = binomial) Note that we again use the wrapper I() to create this binary response variable on the fly. The expression wage > 250 evaluates to a logical variable containing TRUEs and FALSEs, which glm() coerces to binary by setting the TRUEs to 1 and the FALSEs to 0. Once again, we make predictions using the predict() function. > preds <- predict(fit , newdata = list(age = age.grid), se = T) However, calculating the confidence intervals is slightly more involved than in the linear regression case. The default prediction type for a glm() model is type = "link", which is what we use here. This means we get predictions for the logit, or log-odds: that is, we have fit a model of the form log * Pr(Y = 1|X) 1 − Pr(Y = 1|X) + = Xβ, and the predictions given are of the form Xβ̂. The standard errors given are also for Xβ̂. In order to obtain confidence intervals for Pr(Y = 1|X), we use the transformation Pr(Y = 1|X) = exp(Xβ) 1 + exp(Xβ) .
- 324. 316 7. Moving Beyond Linearity > pfit <- exp(preds$fit) / (1 + exp(preds$fit)) > se.bands.logit <- cbind(preds$fit + 2 * preds$se.fit , preds$fit - 2 * preds$se.fit) > se.bands <- exp(se.bands.logit) / (1 + exp(se.bands.logit)) Note that we could have directly computed the probabilities by selecting the type = "response" option in the predict() function. > preds <- predict(fit , newdata = list(age = age.grid), type = "response", se = T) However, the corresponding confidence intervals would not have been sen- sible because we would end up with negative probabilities! Finally, the right-hand plot from Figure 7.1 was made as follows: > plot(age , I(wage > 250) , xlim = agelims , type = "n", ylim = c(0, .2)) > points(jitter(age), I(( wage > 250) / 5), cex = .5, pch = "|", col = "darkgrey") > lines(age.grid , pfit , lwd = 2, col = "blue") > matlines (age.grid , se.bands , lwd = 1, col = "blue", lty = 3) We have drawn the age values corresponding to the observations with wage values above 250 as gray marks on the top of the plot, and those with wage values below 250 are shown as gray marks on the bottom of the plot. We used the jitter() function to jitter the age values a bit so that observations jitter() with the same age value do not cover each other up. This is often called a rug plot. rug plot In order to fit a step function, as discussed in Section 7.2, we use the cut() function. cut() > table(cut(age , 4)) (17.9 ,33.5] (33.5 ,49] (49 ,64.5] (64.5 ,80.1] 750 1399 779 72 > fit <- lm(wage ∼ cut(age , 4), data = Wage) > coef(summary(fit)) Estimate Std. Error t value Pr(>|t|) (Intercept) 94.16 1.48 63.79 0.00e+00 cut(age , 4) (33.5 ,49] 24.05 1.83 13.15 1.98e-38 cut(age , 4) (49 ,64.5] 23.66 2.07 11.44 1.04e-29 cut(age , 4) (64.5 ,80.1] 7.64 4.99 1.53 1.26e -01 Here cut() automatically picked the cutpoints at 33.5, 49, and 64.5 years of age. We could also have specified our own cutpoints directly using the breaks option. The function cut() returns an ordered categorical variable; the lm() function then creates a set of dummy variables for use in the re- gression. The age < 33.5 category is left out, so the intercept coefficient of $94,160 can be interpreted as the average salary for those under 33.5 years of age, and the other coefficients can be interpreted as the average addi- tional salary for those in the other age groups. We can produce predictions and plots just as we did in the case of the polynomial fit.
- 325. 7.8 Lab: Non-linear Modeling 317 7.8.2 Splines In order to fit regression splines in R, we use the splines library. In Section 7.4, we saw that regression splines can be fit by constructing an appropriate matrix of basis functions. The bs() function generates the entire matrix of bs() basis functions for splines with the specified set of knots. By default, cubic splines are produced. Fitting wage to age using a regression spline is simple: > library(splines) > fit <- lm(wage ∼ bs(age , knots = c(25, 40, 60)), data = Wage) > pred <- predict(fit , newdata = list(age = age.grid), se = T) > plot(age , wage , col = "gray") > lines(age.grid , pred$fit , lwd = 2) > lines(age.grid , pred$fit + 2 * pred$se, lty = "dashed") > lines(age.grid , pred$fit - 2 * pred$se, lty = "dashed") Here we have prespecified knots at ages 25, 40, and 60. This produces a spline with six basis functions. (Recall that a cubic spline with three knots has seven degrees of freedom; these degrees of freedom are used up by an intercept, plus six basis functions.) We could also use the df option to produce a spline with knots at uniform quantiles of the data. > dim(bs(age , knots = c(25, 40, 60))) [1] 3000 6 > dim(bs(age , df = 6)) [1] 3000 6 > attr(bs(age , df = 6), "knots") 25% 50% 75% 33.8 42.0 51.0 In this case R chooses knots at ages 33.8, 42.0, and 51.0, which correspond to the 25th, 50th, and 75th percentiles of age. The function bs() also has a degree argument, so we can fit splines of any degree, rather than the default degree of 3 (which yields a cubic spline). In order to instead fit a natural spline, we use the ns() function. Here ns() we fit a natural spline with four degrees of freedom. > fit2 <- lm(wage ∼ ns(age , df = 4), data = Wage) > pred2 <- predict(fit2 , newdata = list(age = age.grid), se = T) > lines(age.grid , pred2$fit , col = "red", lwd = 2) As with the bs() function, we could instead specify the knots directly using the knots option. In order to fit a smoothing spline, we use the smooth.spline() function. smooth. spline() Figure 7.8 was produced with the following code: > plot(age , wage , xlim = agelims , cex = .5, col = "darkgrey") > title("Smoothing Spline") > fit <- smooth.spline(age , wage , df = 16) > fit2 <- smooth.spline(age , wage , cv = TRUE) > fit2$df
- 326. 318 7. Moving Beyond Linearity [1] 6.8 > lines(fit , col = "red", lwd = 2) > lines(fit2 , col = "blue", lwd = 2) > legend("topright", legend = c("16 DF", "6.8 DF"), col = c("red", "blue"), lty = 1, lwd = 2, cex = .8) Notice that in the first call to smooth.spline(), we specified df = 16. The function then determines which value of λ leads to 16 degrees of freedom. In the second call to smooth.spline(), we select the smoothness level by cross- validation; this results in a value of λ that yields 6.8 degrees of freedom. In order to perform local regression, we use the loess() function. loess() > plot(age , wage , xlim = agelims , cex = .5, col = "darkgrey") > title("Local Regression ") > fit <- loess(wage ∼ age , span = .2, data = Wage) > fit2 <- loess(wage ∼ age , span = .5, data = Wage) > lines(age.grid , predict(fit , data.frame(age = age.grid)), col = "red", lwd = 2) > lines(age.grid , predict(fit2 , data.frame(age = age.grid)), col = "blue", lwd = 2) > legend("topright", legend = c("Span = 0.2", "Span = 0.5"), col = c("red", "blue"), lty = 1, lwd = 2, cex = .8) Here we have performed local linear regression using spans of 0.2 and 0.5: that is, each neighborhood consists of 20 % or 50 % of the observations. The larger the span, the smoother the fit. The locfit library can also be used for fitting local regression models in R. 7.8.3 GAMs We now fit a GAM to predict wage using natural spline functions of year and age, treating education as a qualitative predictor, as in (7.16). Since this is just a big linear regression model using an appropriate choice of basis functions, we can simply do this using the lm() function. > gam1 <- lm(wage ∼ ns(year , 4) + ns(age , 5) + education , data = Wage) We now fit the model (7.16) using smoothing splines rather than natural splines. In order to fit more general sorts of GAMs, using smoothing splines or other components that cannot be expressed in terms of basis functions and then fit using least squares regression, we will need to use the gam library in R. The s() function, which is part of the gam library, is used to indicate that s() we would like to use a smoothing spline. We specify that the function of year should have 4 degrees of freedom, and that the function of age will have 5 degrees of freedom. Since education is qualitative, we leave it as is, and it is converted into four dummy variables. We use the gam() function in gam() order to fit a GAM using these components. All of the terms in (7.16) are fit simultaneously, taking each other into account to explain the response.
- 327. 7.8 Lab: Non-linear Modeling 319 > library(gam) > gam.m3 <- gam(wage ∼ s(year , 4) + s(age , 5) + education , data = Wage) In order to produce Figure 7.12, we simply call the plot() function: > par(mfrow = c(1, 3)) > plot(gam.m3 , se = TRUE , col = "blue") The generic plot() function recognizes that gam.m3 is an object of class Gam, and invokes the appropriate plot.Gam() method. Conveniently, even though plot.Gam() gam1 is not of class Gam but rather of class lm, we can still use plot.Gam() on it. Figure 7.11 was produced using the following expression: > plot.Gam(gam1 , se = TRUE , col = "red") Notice here we had to use plot.Gam() rather than the generic plot() function. In these plots, the function of year looks rather linear. We can perform a series of ANOVA tests in order to determine which of these three models is best: a GAM that excludes year (M1), a GAM that uses a linear function of year (M2), or a GAM that uses a spline function of year (M3). > gam.m1 <- gam(wage ∼ s(age , 5) + education , data = Wage) > gam.m2 <- gam(wage ∼ year + s(age , 5) + education , data = Wage) > anova(gam.m1 , gam.m2 , gam.m3 , test = "F") Analysis of Deviance Table Model 1: wage ∼ s(age , 5) + education Model 2: wage ∼ year + s(age , 5) + education Model 3: wage ∼ s(year , 4) + s(age , 5) + education Resid. Df Resid. Dev Df Deviance F Pr(>F) 1 2990 3711730 2 2989 3693841 1 17889 14.5 0.00014 *** 3 2986 3689770 3 4071 1.1 0.34857 --- Signif.codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 We find that there is compelling evidence that a GAM with a linear func- tion of year is better than a GAM that does not include year at all (p-value = 0.00014). However, there is no evidence that a non-linear func- tion of year is needed (p-value = 0.349). In other words, based on the results of this ANOVA, M2 is preferred. The summary() function produces a summary of the gam fit. > summary(gam.m3) Call: gam(formula = wage ∼ s(year , 4) + s(age , 5) + education , data = Wage) Deviance Residuals: Min 1Q Median 3Q Max -119.43 -19.70 -3.33 14.17 213.48
- 328. 320 7. Moving Beyond Linearity (Dispersion Parameter for gaussian family taken to be 1236) Null Deviance: 5222086 on 2999 degrees of freedom Residual Deviance: 3689770 on 2986 degrees of freedom AIC: 29888 Number of Local Scoring Iterations: 2 Anova for Parametric Effects Df Sum Sq Mean Sq F value Pr(>F) s(year , 4) 1 27162 27162 22 2.9e-06 *** s(age , 5) 1 195338 195338 158 < 2e-16 *** education 4 1069726 267432 216 < 2e -16 *** Residuals 2986 3689770 1236 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Anova for Nonparametric Effects Npar Df Npar F Pr(F) (Intercept) s(year , 4) 3 1.1 0.35 s(age , 5) 4 32.4 <2e-16 *** education --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 The “Anova for Parametric Effects” p-values clearly demonstrate that year, age, and education are all highly statistically significant, even when only assuming a linear relationship. Alternatively, the “Anova for Nonparamet- ric Effects” p-values for year and age correspond to a null hypothesis of a linear relationship versus the alternative of a non-linear relationship. The large p-value for year reinforces our conclusion from the ANOVA test that a linear function is adequate for this term. However, there is very clear evidence that a non-linear term is required for age. We can make predictions using the predict() method for the class Gam. Here we make predictions on the training set. > preds <- predict(gam.m2 , newdata = Wage) We can also use local regression fits as building blocks in a GAM, using the lo() function. lo() > gam.lo <- gam( wage ∼ s(year , df = 4) + lo(age , span = 0.7) + education , data = Wage ) > plot.Gam(gam.lo , se = TRUE , col = "green") Here we have used local regression for the age term, with a span of 0.7. We can also use the lo() function to create interactions before calling the gam() function. For example,
- 329. 7.9 Exercises 321 > gam.lo.i <- gam(wage ∼ lo(year , age , span = 0.5) + education , data = Wage) fits a two-term model, in which the first term is an interaction between year and age, fit by a local regression surface. We can plot the resulting two-dimensional surface if we first install the akima package. > library(akima) > plot(gam.lo.i) In order to fit a logistic regression GAM, we once again use the I() func- tion in constructing the binary response variable, and set family=binomial. > gam.lr <- gam( I(wage > 250) ∼ year + s(age , df = 5) + education , family = binomial , data = Wage ) > par(mfrow = c(1, 3)) > plot(gam.lr , se = T, col = "green") It is easy to see that there are no high earners in the < HS category: > table(education , I(wage > 250)) education FALSE TRUE 1. < HS Grad 268 0 2. HS Grad 966 5 3. Some College 643 7 4. College Grad 663 22 5. Advanced Degree 381 45 Hence, we fit a logistic regression GAM using all but this category. This provides more sensible results. > gam.lr.s <- gam( I(wage > 250) ∼ year + s(age , df = 5) + education , family = binomial , data = Wage , subset = (education != "1. < HS Grad") ) > plot(gam.lr.s, se = T, col = "green") 7.9 Exercises Conceptual 1. It was mentioned in the chapter that a cubic regression spline with one knot at ξ can be obtained using a basis of the form x, x2 , x3 , (x − ξ)3 +, where (x − ξ)3 + = (x − ξ)3 if x > ξ and equals 0 otherwise. We will now show that a function of the form f(x) = β0 + β1x + β2x2 + β3x3 + β4(x − ξ)3 +
- 330. 322 7. Moving Beyond Linearity is indeed a cubic regression spline, regardless of the values of β0, β1, β2, β3, β4. (a) Find a cubic polynomial f1(x) = a1 + b1x + c1x2 + d1x3 such that f(x) = f1(x) for all x ≤ ξ. Express a1, b1, c1, d1 in terms of β0, β1, β2, β3, β4. (b) Find a cubic polynomial f2(x) = a2 + b2x + c2x2 + d2x3 such that f(x) = f2(x) for all x > ξ. Express a2, b2, c2, d2 in terms of β0, β1, β2, β3, β4. We have now established that f(x) is a piecewise polynomial. (c) Show that f1(ξ) = f2(ξ). That is, f(x) is continuous at ξ. (d) Show that f′ 1(ξ) = f′ 2(ξ). That is, f′ (x) is continuous at ξ. (e) Show that f′′ 1 (ξ) = f′′ 2 (ξ). That is, f′′ (x) is continuous at ξ. Therefore, f(x) is indeed a cubic spline. Hint: Parts (d) and (e) of this problem require knowledge of single- variable calculus. As a reminder, given a cubic polynomial f1(x) = a1 + b1x + c1x2 + d1x3 , the first derivative takes the form f′ 1(x) = b1 + 2c1x + 3d1x2 and the second derivative takes the form f′′ 1 (x) = 2c1 + 6d1x. 2. Suppose that a curve ĝ is computed to smoothly fit a set of n points using the following formula: ĝ = arg min g > n 0 i=1 (yi − g(xi))2 + λ L 7 g(m) (x) 82 dx ? , where g(m) represents the mth derivative of g (and g(0) = g). Provide example sketches of ĝ in each of the following scenarios. (a) λ = ∞, m = 0. (b) λ = ∞, m = 1.
- 331. 7.9 Exercises 323 (c) λ = ∞, m = 2. (d) λ = ∞, m = 3. (e) λ = 0, m = 3. 3. Suppose we fit a curve with basis functions b1(X) = X, b2(X) = (X − 1)2 I(X ≥ 1). (Note that I(X ≥ 1) equals 1 for X ≥ 1 and 0 otherwise.) We fit the linear regression model Y = β0 + β1b1(X) + β2b2(X) + ϵ, and obtain coefficient estimates β̂0 = 1, β̂1 = 1, β̂2 = −2. Sketch the estimated curve between X = −2 and X = 2. Note the intercepts, slopes, and other relevant information. 4. Suppose we fit a curve with basis functions b1(X) = I(0 ≤ X ≤ 2) − (X −1)I(1 ≤ X ≤ 2), b2(X) = (X −3)I(3 ≤ X ≤ 4)+I(4 < X ≤ 5). We fit the linear regression model Y = β0 + β1b1(X) + β2b2(X) + ϵ, and obtain coefficient estimates β̂0 = 1, β̂1 = 1, β̂2 = 3. Sketch the estimated curve between X = −2 and X = 6. Note the intercepts, slopes, and other relevant information. 5. Consider two curves, ĝ1 and ĝ2, defined by ĝ1 = arg min g > n 0 i=1 (yi − g(xi))2 + λ L 7 g(3) (x) 82 dx ? , ĝ2 = arg min g > n 0 i=1 (yi − g(xi))2 + λ L 7 g(4) (x) 82 dx ? , where g(m) represents the mth derivative of g. (a) As λ → ∞, will ĝ1 or ĝ2 have the smaller training RSS? (b) As λ → ∞, will ĝ1 or ĝ2 have the smaller test RSS? (c) For λ = 0, will ĝ1 or ĝ2 have the smaller training and test RSS? Applied 6. In this exercise, you will further analyze the Wage data set considered throughout this chapter. (a) Perform polynomial regression to predict wage using age. Use cross-validation to select the optimal degree d for the polyno- mial. What degree was chosen, and how does this compare to the results of hypothesis testing using ANOVA? Make a plot of the resulting polynomial fit to the data.
- 332. 324 7. Moving Beyond Linearity (b) Fit a step function to predict wage using age, and perform cross- validation to choose the optimal number of cuts. Make a plot of the fit obtained. 7. The Wage data set contains a number of other features not explored in this chapter, such as marital status (maritl), job class (jobclass), and others. Explore the relationships between some of these other predictors and wage, and use non-linear fitting techniques in order to fit flexible models to the data. Create plots of the results obtained, and write a summary of your findings. 8. Fit some of the non-linear models investigated in this chapter to the Auto data set. Is there evidence for non-linear relationships in this data set? Create some informative plots to justify your answer. 9. This question uses the variables dis (the weighted mean of distances to five Boston employment centers) and nox (nitrogen oxides concen- tration in parts per 10 million) from the Boston data. We will treat dis as the predictor and nox as the response. (a) Use the poly() function to fit a cubic polynomial regression to predict nox using dis. Report the regression output, and plot the resulting data and polynomial fits. (b) Plot the polynomial fits for a range of different polynomial degrees (say, from 1 to 10), and report the associated residual sum of squares. (c) Perform cross-validation or another approach to select the opti- mal degree for the polynomial, and explain your results. (d) Use the bs() function to fit a regression spline to predict nox using dis. Report the output for the fit using four degrees of freedom. How did you choose the knots? Plot the resulting fit. (e) Now fit a regression spline for a range of degrees of freedom, and plot the resulting fits and report the resulting RSS. Describe the results obtained. (f) Perform cross-validation or another approach in order to select the best degrees of freedom for a regression spline on this data. Describe your results. 10. This question relates to the College data set. (a) Split the data into a training set and a test set. Using out-of-state tuition as the response and the other variables as the predictors, perform forward stepwise selection on the training set in order to identify a satisfactory model that uses just a subset of the predictors.
- 333. 7.9 Exercises 325 (b) Fit a GAM on the training data, using out-of-state tuition as the response and the features selected in the previous step as the predictors. Plot the results, and explain your findings. (c) Evaluate the model obtained on the test set, and explain the results obtained. (d) For which variables, if any, is there evidence of a non-linear relationship with the response? 11. In Section 7.7, it was mentioned that GAMs are generally fit using a backfitting approach. The idea behind backfitting is actually quite simple. We will now explore backfitting in the context of multiple linear regression. Suppose that we would like to perform multiple linear regression, but we do not have software to do so. Instead, we only have software to perform simple linear regression. Therefore, we take the following iterative approach: we repeatedly hold all but one coefficient esti- mate fixed at its current value, and update only that coefficient estimate using a simple linear regression. The process is continued un- til convergence—that is, until the coefficient estimates stop changing. We now try this out on a toy example. (a) Generate a response Y and two predictors X1 and X2, with n = 100. (b) Initialize β̂1 to take on a value of your choice. It does not matter what value you choose. (c) Keeping β̂1 fixed, fit the model Y − β̂1X1 = β0 + β2X2 + ϵ. You can do this as follows: > a <- y - beta1 * x1 > beta2 <- lm(a ∼ x2)$coef [2] (d) Keeping β̂2 fixed, fit the model Y − β̂2X2 = β0 + β1X1 + ϵ. You can do this as follows: > a <- y - beta2 * x2 > beta1 <- lm(a ∼ x1)$coef [2] (e) Write a for loop to repeat (c) and (d) 1,000 times. Report the estimates of β̂0, β̂1, and β̂2 at each iteration of the for loop. Create a plot in which each of these values is displayed, with β̂0, β̂1, and β̂2 each shown in a different color.
- 334. 326 7. Moving Beyond Linearity (f) Compare your answer in (e) to the results of simply performing multiple linear regression to predict Y using X1 and X2. Use the abline() function to overlay those multiple linear regression coefficient estimates on the plot obtained in (e). (g) On this data set, how many backfitting iterations were required in order to obtain a “good” approximation to the multiple re- gression coefficient estimates? 12. This problem is a continuation of the previous exercise. In a toy example with p = 100, show that one can approximate the multiple linear regression coefficient estimates by repeatedly performing simple linear regression in a backfitting procedure. How many backfitting iterations are required in order to obtain a “good” approximation to the multiple regression coefficient estimates? Create a plot to justify your answer.
- 335. 8 Tree-Based Methods In this chapter, we describe tree-based methods for regression and classi- fication. These involve stratifying or segmenting the predictor space into a number of simple regions. In order to make a prediction for a given ob- servation, we typically use the mean or the mode response value for the training observations in the region to which it belongs. Since the set of splitting rules used to segment the predictor space can be summarized in a tree, these types of approaches are known as decision tree methods. decision tree Tree-based methods are simple and useful for interpretation. However, they typically are not competitive with the best supervised learning ap- proaches, such as those seen in Chapters 6 and 7, in terms of prediction accuracy. Hence in this chapter we also introduce bagging, random forests, boosting, and Bayesian additive regression trees. Each of these approaches involves producing multiple trees which are then combined to yield a single consensus prediction. We will see that combining a large number of trees can often result in dramatic improvements in prediction accuracy, at the expense of some loss in interpretation. 8.1 The Basics of Decision Trees Decision trees can be applied to both regression and classification problems. We first consider regression problems, and then move on to classification. © Springer Science+Business Media, LLC, part of Springer Nature 2021 G. James et al., An Introduction to Statistical Learning, Springer Texts in Statistics, https://doi.org/10.1007/978-1-0716-1418-1_8 327
- 336. 328 8. Tree-Based Methods | Years < 4.5 Hits < 117.5 5.11 6.00 6.74 FIGURE 8.1. For the Hitters data, a regression tree for predicting the log salary of a baseball player, based on the number of years that he has played in the major leagues and the number of hits that he made in the previous year. At a given internal node, the label (of the form Xj < tk) indicates the left-hand branch emanating from that split, and the right-hand branch corresponds to Xj ≥ tk. For instance, the split at the top of the tree results in two large branches. The left-hand branch corresponds to Years<4.5, and the right-hand branch corresponds to Years>=4.5. The tree has two internal nodes and three terminal nodes, or leaves. The number in each leaf is the mean of the response for the observations that fall there. 8.1.1 Regression Trees In order to motivate regression trees, we begin with a simple example. regression tree Predicting Baseball Players’ Salaries Using Regression Trees We use the Hitters data set to predict a baseball player’s Salary based on Years (the number of years that he has played in the major leagues) and Hits (the number of hits that he made in the previous year). We first remove observations that are missing Salary values, and log-transform Salary so that its distribution has more of a typical bell-shape. (Recall that Salary is measured in thousands of dollars.) Figure 8.1 shows a regression tree fit to this data. It consists of a series of splitting rules, starting at the top of the tree. The top split assigns observations having Years<4.5 to the left branch.1 The predicted salary for these players is given by the mean response value for the players in the data set with Years<4.5. For such players, the mean log salary is 5.107, and so we make a prediction of e5.107 thousands of dollars, i.e. $165,174, for 1Both Years and Hits are integers in these data; the tree() function in R labels the splits at the midpoint between two adjacent values.
- 337. 8.1 The Basics of Decision Trees 329 Years Hits 1 117.5 238 1 4.5 24 R1 R3 R2 FIGURE 8.2. The three-region partition for the Hitters data set from the re- gression tree illustrated in Figure 8.1. these players. Players with Years>=4.5 are assigned to the right branch, and then that group is further subdivided by Hits. Overall, the tree stratifies or segments the players into three regions of predictor space: players who have played for four or fewer years, players who have played for five or more years and who made fewer than 118 hits last year, and players who have played for five or more years and who made at least 118 hits last year. These three regions can be written as R1 ={X | Years<4.5}, R2 ={X | Years>=4.5, Hits<117.5}, and R3 ={X | Years>=4.5, Hits>=117.5}. Figure 8.2 illustrates the regions as a function of Years and Hits. The predicted salaries for these three groups are $1,000×e5.107 =$165,174, $1,000×e5.999 =$402,834, and $1,000×e6.740 =$845,346 respectively. In keeping with the tree analogy, the regions R1, R2, and R3 are known as terminal nodes or leaves of the tree. As is the case for Figure 8.1, decision terminal node leaf trees are typically drawn upside down, in the sense that the leaves are at the bottom of the tree. The points along the tree where the predictor space is split are referred to as internal nodes. In Figure 8.1, the two internal internal node nodes are indicated by the text Years<4.5 and Hits<117.5. We refer to the segments of the trees that connect the nodes as branches. branch We might interpret the regression tree displayed in Figure 8.1 as follows: Years is the most important factor in determining Salary, and players with less experience earn lower salaries than more experienced players. Given that a player is less experienced, the number of hits that he made in the previous year seems to play little role in his salary. But among players who have been in the major leagues for five or more years, the number of hits made in the previous year does affect salary, and players who made more hits last year tend to have higher salaries. The regression tree shown in
- 338. 330 8. Tree-Based Methods Figure 8.1 is likely an over-simplification of the true relationship between Hits, Years, and Salary. However, it has advantages over other types of regression models (such as those seen in Chapters 3 and 6): it is easier to interpret, and has a nice graphical representation. Prediction via Stratification of the Feature Space We now discuss the process of building a regression tree. Roughly speaking, there are two steps. 1. We divide the predictor space — that is, the set of possible values for X1, X2, . . . , Xp — into J distinct and non-overlapping regions, R1, R2, . . . , RJ . 2. For every observation that falls into the region Rj, we make the same prediction, which is simply the mean of the response values for the training observations in Rj. For instance, suppose that in Step 1 we obtain two regions, R1 and R2, and that the response mean of the training observations in the first region is 10, while the response mean of the training observations in the second region is 20. Then for a given observation X = x, if x ∈ R1 we will predict a value of 10, and if x ∈ R2 we will predict a value of 20. We now elaborate on Step 1 above. How do we construct the regions R1, . . . , RJ ? In theory, the regions could have any shape. However, we choose to divide the predictor space into high-dimensional rectangles, or boxes, for simplicity and for ease of interpretation of the resulting predic- tive model. The goal is to find boxes R1, . . . , RJ that minimize the RSS, given by J 0 j=1 0 i∈Rj (yi − ŷRj )2 , (8.1) where ŷRj is the mean response for the training observations within the jth box. Unfortunately, it is computationally infeasible to consider every possible partition of the feature space into J boxes. For this reason, we take a top-down, greedy approach that is known as recursive binary splitting. The recursive binary splitting approach is top-down because it begins at the top of the tree (at which point all observations belong to a single region) and then successively splits the predictor space; each split is indicated via two new branches further down on the tree. It is greedy because at each step of the tree-building process, the best split is made at that particular step, rather than looking ahead and picking a split that will lead to a better tree in some future step. In order to perform recursive binary splitting, we first select the pre- dictor Xj and the cutpoint s such that splitting the predictor space into the regions {X|Xj < s} and {X|Xj ≥ s} leads to the greatest possible reduction in RSS. (The notation {X|Xj < s} means the region of predictor
- 339. 8.1 The Basics of Decision Trees 331 space in which Xj takes on a value less than s.) That is, we consider all predictors X1, . . . , Xp, and all possible values of the cutpoint s for each of the predictors, and then choose the predictor and cutpoint such that the resulting tree has the lowest RSS. In greater detail, for any j and s, we define the pair of half-planes R1(j, s) = {X|Xj < s} and R2(j, s) = {X|Xj ≥ s}, (8.2) and we seek the value of j and s that minimize the equation 0 i: xi∈R1(j,s) (yi − ŷR1 )2 + 0 i: xi∈R2(j,s) (yi − ŷR2 )2 , (8.3) where ŷR1 is the mean response for the training observations in R1(j, s), and ŷR2 is the mean response for the training observations in R2(j, s). Finding the values of j and s that minimize (8.3) can be done quite quickly, especially when the number of features p is not too large. Next, we repeat the process, looking for the best predictor and best cutpoint in order to split the data further so as to minimize the RSS within each of the resulting regions. However, this time, instead of splitting the entire predictor space, we split one of the two previously identified regions. We now have three regions. Again, we look to split one of these three regions further, so as to minimize the RSS. The process continues until a stopping criterion is reached; for instance, we may continue until no region contains more than five observations. Once the regions R1, . . . , RJ have been created, we predict the response for a given test observation using the mean of the training observations in the region to which that test observation belongs. A five-region example of this approach is shown in Figure 8.3. Tree Pruning The process described above may produce good predictions on the training set, but is likely to overfit the data, leading to poor test set performance. This is because the resulting tree might be too complex. A smaller tree with fewer splits (that is, fewer regions R1, . . . , RJ ) might lead to lower variance and better interpretation at the cost of a little bias. One possible alternative to the process described above is to build the tree only so long as the decrease in the RSS due to each split exceeds some (high) threshold. This strategy will result in smaller trees, but is too short-sighted since a seemingly worthless split early on in the tree might be followed by a very good split—that is, a split that leads to a large reduction in RSS later on. Therefore, a better strategy is to grow a very large tree T0, and then prune it back in order to obtain a subtree. How do we determine the best prune subtree way to prune the tree? Intuitively, our goal is to select a subtree that leads to the lowest test error rate. Given a subtree, we can estimate its
- 340. 332 8. Tree-Based Methods | t1 t2 t3 t4 R1 R1 R2 R2 R3 R3 R4 R4 R5 R5 X1 X1 X1 X2 X 2 X 2 X1 ≤ t1 X2 ≤ t2 X1 ≤ t3 X2 ≤ t4 FIGURE 8.3. Top Left: A partition of two-dimensional feature space that could not result from recursive binary splitting. Top Right: The output of recursive binary splitting on a two-dimensional example. Bottom Left: A tree corresponding to the partition in the top right panel. Bottom Right: A perspective plot of the prediction surface corresponding to that tree. test error using cross-validation or the validation set approach. However, estimating the cross-validation error for every possible subtree would be too cumbersome, since there is an extremely large number of possible subtrees. Instead, we need a way to select a small set of subtrees for consideration. Cost complexity pruning—also known as weakest link pruning—gives us cost complexity pruning weakest link pruning a way to do just this. Rather than considering every possible subtree, we consider a sequence of trees indexed by a nonnegative tuning parameter α. For each value of α there corresponds a subtree T ⊂ T0 such that |T | 0 m=1 0 i: xi∈Rm (yi − ŷRm )2 + α|T| (8.4)
- 341. 8.1 The Basics of Decision Trees 333 Algorithm 8.1 Building a Regression Tree 1. Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations. 2. Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of α. 3. Use K-fold cross-validation to choose α. That is, divide the training observations into K folds. For each k = 1, . . . , K: (a) Repeat Steps 1 and 2 on all but the kth fold of the training data. (b) Evaluate the mean squared prediction error on the data in the left-out kth fold, as a function of α. Average the results for each value of α, and pick α to minimize the average error. 4. Return the subtree from Step 2 that corresponds to the chosen value of α. is as small as possible. Here |T| indicates the number of terminal nodes of the tree T, Rm is the rectangle (i.e. the subset of predictor space) cor- responding to the mth terminal node, and ŷRm is the predicted response associated with Rm—that is, the mean of the training observations in Rm. The tuning parameter α controls a trade-off between the subtree’s com- plexity and its fit to the training data. When α = 0, then the subtree T will simply equal T0, because then (8.4) just measures the training error. However, as α increases, there is a price to pay for having a tree with many terminal nodes, and so the quantity (8.4) will tend to be minimized for a smaller subtree. Equation 8.4 is reminiscent of the lasso (6.7) from Chapter 6, in which a similar formulation was used in order to control the complexity of a linear model. It turns out that as we increase α from zero in (8.4), branches get pruned from the tree in a nested and predictable fashion, so obtaining the whole sequence of subtrees as a function of α is easy. We can select a value of α using a validation set or using cross-validation. We then return to the full data set and obtain the subtree corresponding to α. This process is summarized in Algorithm 8.1. Figures 8.4 and 8.5 display the results of fitting and pruning a regression tree on the Hitters data, using nine of the features. First, we randomly divided the data set in half, yielding 132 observations in the training set and 131 observations in the test set. We then built a large regression tree on the training data and varied α in (8.4) in order to create subtrees with different numbers of terminal nodes. Finally, we performed six-fold cross-
- 342. 334 8. Tree-Based Methods | Years < 4.5 RBI < 60.5 Putouts < 82 Years < 3.5 Years < 3.5 Hits < 117.5 Walks < 43.5 Runs < 47.5 Walks < 52.5 RBI < 80.5 Years < 6.5 5.487 4.622 5.183 5.394 6.189 6.015 5.571 6.407 6.549 6.459 7.007 7.289 FIGURE 8.4. Regression tree analysis for the Hitters data. The unpruned tree that results from top-down greedy splitting on the training data is shown. validation in order to estimate the cross-validated MSE of the trees as a function of α. (We chose to perform six-fold cross-validation because 132 is an exact multiple of six.) The unpruned regression tree is shown in Figure 8.4. The green curve in Figure 8.5 shows the CV error as a function of the number of leaves,2 while the orange curve indicates the test error. Also shown are standard error bars around the estimated errors. For reference, the training error curve is shown in black. The CV error is a reasonable approximation of the test error: the CV error takes on its minimum for a three-node tree, while the test error also dips down at the three-node tree (though it takes on its lowest value at the ten-node tree). The pruned tree containing three terminal nodes is shown in Figure 8.1. 2Although CV error is computed as a function of α, it is convenient to display the result as a function of |T|, the number of leaves; this is based on the relationship between α and |T| in the original tree grown to all the training data.
- 343. 8.1 The Basics of Decision Trees 335 2 4 6 8 10 0.0 0.2 0.4 0.6 0.8 1.0 Tree Size Mean Squared Error Training Cross−Validation Test FIGURE 8.5. Regression tree analysis for the Hitters data. The training, cross-validation, and test MSE are shown as a function of the number of termi- nal nodes in the pruned tree. Standard error bands are displayed. The minimum cross-validation error occurs at a tree size of three. 8.1.2 Classification Trees A classification tree is very similar to a regression tree, except that it is classification tree used to predict a qualitative response rather than a quantitative one. Re- call that for a regression tree, the predicted response for an observation is given by the mean response of the training observations that belong to the same terminal node. In contrast, for a classification tree, we predict that each observation belongs to the most commonly occurring class of training observations in the region to which it belongs. In interpreting the results of a classification tree, we are often interested not only in the class prediction corresponding to a particular terminal node region, but also in the class proportions among the training observations that fall into that region. The task of growing a classification tree is quite similar to the task of growing a regression tree. Just as in the regression setting, we use recursive binary splitting to grow a classification tree. However, in the classification setting, RSS cannot be used as a criterion for making the binary splits. A natural alternative to RSS is the classification error rate. Since we plan classification error rate to assign an observation in a given region to the most commonly occurring class of training observations in that region, the classification error rate is simply the fraction of the training observations in that region that do not belong to the most common class: E = 1 − max k (p̂mk). (8.5)
- 344. 336 8. Tree-Based Methods Here p̂mk represents the proportion of training observations in the mth region that are from the kth class. However, it turns out that classification error is not sufficiently sensitive for tree-growing, and in practice two other measures are preferable. The Gini index is defined by Gini index G = K 0 k=1 p̂mk(1 − p̂mk), (8.6) a measure of total variance across the K classes. It is not hard to see that the Gini index takes on a small value if all of the p̂mk’s are close to zero or one. For this reason the Gini index is referred to as a measure of node purity—a small value indicates that a node contains predominantly observations from a single class. An alternative to the Gini index is entropy, given by entropy D = − K 0 k=1 p̂mk log p̂mk. (8.7) Since 0 ≤ p̂mk ≤ 1, it follows that 0 ≤ −p̂mk log p̂mk. One can show that the entropy will take on a value near zero if the p̂mk’s are all near zero or near one. Therefore, like the Gini index, the entropy will take on a small value if the mth node is pure. In fact, it turns out that the Gini index and the entropy are quite similar numerically. When building a classification tree, either the Gini index or the entropy are typically used to evaluate the quality of a particular split, since these two approaches are more sensitive to node purity than is the classification error rate. Any of these three approaches might be used when pruning the tree, but the classification error rate is preferable if prediction accuracy of the final pruned tree is the goal. Figure 8.6 shows an example on the Heart data set. These data con- tain a binary outcome HD for 303 patients who presented with chest pain. An outcome value of Yes indicates the presence of heart disease based on an angiographic test, while No means no heart disease. There are 13 predic- tors including Age, Sex, Chol (a cholesterol measurement), and other heart and lung function measurements. Cross-validation results in a tree with six terminal nodes. In our discussion thus far, we have assumed that the predictor vari- ables take on continuous values. However, decision trees can be constructed even in the presence of qualitative predictor variables. For instance, in the Heart data, some of the predictors, such as Sex, Thal (Thallium stress test), and ChestPain, are qualitative. Therefore, a split on one of these variables amounts to assigning some of the qualitative values to one branch and assigning the remaining to the other branch. In Figure 8.6, some of the in- ternal nodes correspond to splitting qualitative variables. For instance, the
- 345. 8.1 The Basics of Decision Trees 337 | Thal:a Ca < 0.5 MaxHR < 161.5 RestBP < 157 Chol < 244 MaxHR < 156 MaxHR < 145.5 ChestPain:bc Chol < 244 Sex < 0.5 Ca < 0.5 Slope < 1.5 Age < 52 Thal:b ChestPain:a Oldpeak < 1.1 RestECG < 1 No Yes No No Yes No No No No Yes Yes No No No Yes Yes Yes Yes 5 10 15 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Tree Size Error Training Cross−Validation Test | Thal:a Ca < 0.5 MaxHR < 161.5 ChestPain:bc Ca < 0.5 No No No Yes Yes Yes FIGURE 8.6. Heart data. Top: The unpruned tree. Bottom Left: Cross-valida- tion error, training, and test error, for different sizes of the pruned tree. Bottom Right: The pruned tree corresponding to the minimal cross-validation error. top internal node corresponds to splitting Thal. The text Thal:a indicates that the left-hand branch coming out of that node consists of observations with the first value of the Thal variable (normal), and the right-hand node consists of the remaining observations (fixed or reversible defects). The text ChestPain:bc two splits down the tree on the left indicates that the left-hand branch coming out of that node consists of observations with the second and third values of the ChestPain variable, where the possible values are typical angina, atypical angina, non-anginal pain, and asymptomatic. Figure 8.6 has a surprising characteristic: some of the splits yield two terminal nodes that have the same predicted value. For instance, consider the split RestECG<1 near the bottom right of the unpruned tree. Regardless
- 346. 338 8. Tree-Based Methods of the value of RestECG, a response value of Yes is predicted for those ob- servations. Why, then, is the split performed at all? The split is performed because it leads to increased node purity. That is, all 9 of the observations corresponding to the right-hand leaf have a response value of Yes, whereas 7/11 of those corresponding to the left-hand leaf have a response value of Yes. Why is node purity important? Suppose that we have a test obser- vation that belongs to the region given by that right-hand leaf. Then we can be pretty certain that its response value is Yes. In contrast, if a test observation belongs to the region given by the left-hand leaf, then its re- sponse value is probably Yes, but we are much less certain. Even though the split RestECG<1 does not reduce the classification error, it improves the Gini index and the entropy, which are more sensitive to node purity. 8.1.3 Trees Versus Linear Models Regression and classification trees have a very different flavor from the more classical approaches for regression and classification presented in Chapters 3 and 4. In particular, linear regression assumes a model of the form f(X) = β0 + p 0 j=1 Xjβj, (8.8) whereas regression trees assume a model of the form f(X) = M 0 m=1 cm · 1(X∈Rm) (8.9) where R1, . . . , RM represent a partition of feature space, as in Figure 8.3. Which model is better? It depends on the problem at hand. If the re- lationship between the features and the response is well approximated by a linear model as in (8.8), then an approach such as linear regression will likely work well, and will outperform a method such as a regression tree that does not exploit this linear structure. If instead there is a highly non- linear and complex relationship between the features and the response as indicated by model (8.9), then decision trees may outperform classical ap- proaches. An illustrative example is displayed in Figure 8.7. The relative performances of tree-based and classical approaches can be assessed by es- timating the test error, using either cross-validation or the validation set approach (Chapter 5). Of course, other considerations beyond simply test error may come into play in selecting a statistical learning method; for instance, in certain set- tings, prediction using a tree may be preferred for the sake of interpretabil- ity and visualization.
- 347. 8.1 The Basics of Decision Trees 339 −2 −1 0 1 2 −2 −1 0 1 2 X1 X 2 −2 −1 0 1 2 −2 −1 0 1 2 X1 X 2 −2 −1 0 1 2 −2 −1 0 1 2 X1 X 2 −2 −1 0 1 2 −2 −1 0 1 2 X1 X 2 FIGURE 8.7. Top Row: A two-dimensional classification example in which the true decision boundary is linear, and is indicated by the shaded regions. A classical approach that assumes a linear boundary (left) will outperform a decision tree that performs splits parallel to the axes (right). Bottom Row: Here the true de- cision boundary is non-linear. Here a linear model is unable to capture the true decision boundary (left), whereas a decision tree is successful (right). 8.1.4 Advantages and Disadvantages of Trees Decision trees for regression and classification have a number of advantages over the more classical approaches seen in Chapters 3 and 4: ▲ Trees are very easy to explain to people. In fact, they are even easier to explain than linear regression! ▲ Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches seen in previous chapters. ▲ Trees can be displayed graphically, and are easily interpreted even by a non-expert (especially if they are small). ▲ Trees can easily handle qualitative predictors without the need to create dummy variables.
- 348. 340 8. Tree-Based Methods ▼ Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches seen in this book. ▼ Additionally, trees can be very non-robust. In other words, a small change in the data can cause a large change in the final estimated tree. However, by aggregating many decision trees, using methods like bagging, random forests, and boosting, the predictive performance of trees can be substantially improved. We introduce these concepts in the next section. 8.2 Bagging, Random Forests, Boosting, and Bayesian Additive Regression Trees An ensemble method is an approach that combines many simple “building ensemble block” models in order to obtain a single and potentially very powerful model. These simple building block models are sometimes known as weak learners, since they may lead to mediocre predictions on their own. weak learners We will now discuss bagging, random forests, boosting, and Bayesian additive regression trees. These are ensemble methods for which the simple building block is a regression or a classification tree. 8.2.1 Bagging The bootstrap, introduced in Chapter 5, is an extremely powerful idea. It is used in many situations in which it is hard or even impossible to directly compute the standard deviation of a quantity of interest. We see here that the bootstrap can be used in a completely different context, in order to improve statistical learning methods such as decision trees. The decision trees discussed in Section 8.1 suffer from high variance. This means that if we split the training data into two parts at random, and fit a decision tree to both halves, the results that we get could be quite different. In contrast, a procedure with low variance will yield similar results if applied repeatedly to distinct data sets; linear regression tends to have low variance, if the ratio of n to p is moderately large. Bootstrap aggregation, or bagging, is a general-purpose procedure for reducing the bagging variance of a statistical learning method; we introduce it here because it is particularly useful and frequently used in the context of decision trees. Recall that given a set of n independent observations Z1, . . . , Zn, each with variance σ2 , the variance of the mean Z̄ of the observations is given by σ2 /n. In other words, averaging a set of observations reduces variance. Hence a natural way to reduce the variance and increase the test set ac- curacy of a statistical learning method is to take many training sets from
- 349. 8.2 Bagging, Random Forests, Boosting, and Bayesian Additive Regression Trees 341 the population, build a separate prediction model using each training set, and average the resulting predictions. In other words, we could calculate ˆ f1 (x), ˆ f2 (x), . . . , ˆ fB (x) using B separate training sets, and average them in order to obtain a single low-variance statistical learning model, given by ˆ favg(x) = 1 B B 0 b=1 ˆ fb (x). Of course, this is not practical because we generally do not have access to multiple training sets. Instead, we can bootstrap, by taking repeated samples from the (single) training data set. In this approach we generate B different bootstrapped training data sets. We then train our method on the bth bootstrapped training set in order to get ˆ f∗b (x), and finally average all the predictions, to obtain ˆ fbag(x) = 1 B B 0 b=1 ˆ f∗b (x). This is called bagging. While bagging can improve predictions for many regression methods, it is particularly useful for decision trees. To apply bagging to regression trees, we simply construct B regression trees using B bootstrapped training sets, and average the resulting predictions. These trees are grown deep, and are not pruned. Hence each individual tree has high variance, but low bias. Averaging these B trees reduces the variance. Bagging has been demonstrated to give impressive improvements in accuracy by combining together hundreds or even thousands of trees into a single procedure. Thus far, we have described the bagging procedure in the regression context, to predict a quantitative outcome Y . How can bagging be extended to a classification problem where Y is qualitative? In that situation, there are a few possible approaches, but the simplest is as follows. For a given test observation, we can record the class predicted by each of the B trees, and take a majority vote: the overall prediction is the most commonly occurring majority vote class among the B predictions. Figure 8.8 shows the results from bagging trees on the Heart data. The test error rate is shown as a function of B, the number of trees constructed using bootstrapped training data sets. We see that the bagging test error rate is slightly lower in this case than the test error rate obtained from a single tree. The number of trees B is not a critical parameter with bagging; using a very large value of B will not lead to overfitting. In practice we use a value of B sufficiently large that the error has settled down. Using B = 100 is sufficient to achieve good performance in this example.
- 350. 342 8. Tree-Based Methods 0 50 100 150 200 250 300 0.10 0.15 0.20 0.25 0.30 Number of Trees Error Test: Bagging Test: RandomForest OOB: Bagging OOB: RandomForest FIGURE 8.8. Bagging and random forest results for the Heart data. The test error (black and orange) is shown as a function of B, the number of bootstrapped training sets used. Random forests were applied with m = √ p. The dashed line indicates the test error resulting from a single classification tree. The green and blue traces show the OOB error, which in this case is — by chance — considerably lower. Out-of-Bag Error Estimation It turns out that there is a very straightforward way to estimate the test error of a bagged model, without the need to perform cross-validation or the validation set approach. Recall that the key to bagging is that trees are repeatedly fit to bootstrapped subsets of the observations. One can show that on average, each bagged tree makes use of around two-thirds of the observations.3 The remaining one-third of the observations not used to fit a given bagged tree are referred to as the out-of-bag (OOB) observations. We out-of-bag can predict the response for the ith observation using each of the trees in which that observation was OOB. This will yield around B/3 predictions for the ith observation. In order to obtain a single prediction for the ith observation, we can average these predicted responses (if regression is the goal) or can take a majority vote (if classification is the goal). This leads to a single OOB prediction for the ith observation. An OOB prediction can be obtained in this way for each of the n observations, from which the 3This relates to Exercise 2 of Chapter 5.
- 351. 8.2 Bagging, Random Forests, Boosting, and Bayesian Additive Regression Trees 343 overall OOB MSE (for a regression problem) or classification error (for a classification problem) can be computed. The resulting OOB error is a valid estimate of the test error for the bagged model, since the response for each observation is predicted using only the trees that were not fit using that observation. Figure 8.8 displays the OOB error on the Heart data. It can be shown that with B sufficiently large, OOB error is virtually equivalent to leave-one-out cross-validation error. The OOB approach for estimating the test error is particularly convenient when performing bagging on large data sets for which cross-validation would be computationally onerous. Variable Importance Measures As we have discussed, bagging typically results in improved accuracy over prediction using a single tree. Unfortunately, however, it can be difficult to interpret the resulting model. Recall that one of the advantages of decision trees is the attractive and easily interpreted diagram that results, such as the one displayed in Figure 8.1. However, when we bag a large number of trees, it is no longer possible to represent the resulting statistical learning procedure using a single tree, and it is no longer clear which variables are most important to the procedure. Thus, bagging improves prediction accuracy at the expense of interpretability. Although the collection of bagged trees is much more difficult to interpret than a single tree, one can obtain an overall summary of the importance of each predictor using the RSS (for bagging regression trees) or the Gini index (for bagging classification trees). In the case of bagging regression trees, we can record the total amount that the RSS (8.1) is decreased due to splits over a given predictor, averaged over all B trees. A large value indicates an important predictor. Similarly, in the context of bagging classification trees, we can add up the total amount that the Gini index (8.6) is decreased by splits over a given predictor, averaged over all B trees. A graphical representation of the variable importances in the Heart data variable importance is shown in Figure 8.9. We see the mean decrease in Gini index for each vari- able, relative to the largest. The variables with the largest mean decrease in Gini index are Thal, Ca, and ChestPain. 8.2.2 Random Forests Random forests provide an improvement over bagged trees by way of a random forest small tweak that decorrelates the trees. As in bagging, we build a number of decision trees on bootstrapped training samples. But when building these decision trees, each time a split in a tree is considered, a random sample of m predictors is chosen as split candidates from the full set of p predictors. The split is allowed to use only one of those m predictors. A fresh sample of m predictors is taken at each split, and typically we choose m ≈ √ p—that is, the number of predictors considered at each split is approximately equal
- 352. 344 8. Tree-Based Methods Thal Ca ChestPain Oldpeak MaxHR RestBP Age Chol Slope Sex ExAng RestECG Fbs 0 20 40 60 80 100 Variable Importance FIGURE 8.9. A variable importance plot for the Heart data. Variable impor- tance is computed using the mean decrease in Gini index, and expressed relative to the maximum. to the square root of the total number of predictors (4 out of the 13 for the Heart data). In other words, in building a random forest, at each split in the tree, the algorithm is not even allowed to consider a majority of the available predictors. This may sound crazy, but it has a clever rationale. Suppose that there is one very strong predictor in the data set, along with a num- ber of other moderately strong predictors. Then in the collection of bagged trees, most or all of the trees will use this strong predictor in the top split. Consequently, all of the bagged trees will look quite similar to each other. Hence the predictions from the bagged trees will be highly correlated. Un- fortunately, averaging many highly correlated quantities does not lead to as large of a reduction in variance as averaging many uncorrelated quanti- ties. In particular, this means that bagging will not lead to a substantial reduction in variance over a single tree in this setting. Random forests overcome this problem by forcing each split to consider only a subset of the predictors. Therefore, on average (p − m)/p of the splits will not even consider the strong predictor, and so other predictors will have more of a chance. We can think of this process as decorrelating the trees, thereby making the average of the resulting trees less variable and hence more reliable.
- 353. 8.2 Bagging, Random Forests, Boosting, and Bayesian Additive Regression Trees 345 The main difference between bagging and random forests is the choice of predictor subset size m. For instance, if a random forest is built using m = p, then this amounts simply to bagging. On the Heart data, random forests using m = √ p leads to a reduction in both test error and OOB error over bagging (Figure 8.8). Using a small value of m in building a random forest will typically be helpful when we have a large number of correlated predictors. We applied random forests to a high-dimensional biological data set consisting of ex- pression measurements of 4,718 genes measured on tissue samples from 349 patients. There are around 20,000 genes in humans, and individual genes have different levels of activity, or expression, in particular cells, tissues, and biological conditions. In this data set, each of the patient samples has a qualitative label with 15 different levels: either normal or 1 of 14 different types of cancer. Our goal was to use random forests to predict cancer type based on the 500 genes that have the largest variance in the training set. We randomly divided the observations into a training and a test set, and applied random forests to the training set for three different values of the number of splitting variables m. The results are shown in Figure 8.10. The error rate of a single tree is 45.7 %, and the null rate is 75.4 %.4 We see that using 400 trees is sufficient to give good performance, and that the choice m = √ p gave a small improvement in test error over bagging (m = p) in this example. As with bagging, random forests will not overfit if we increase B, so in practice we use a value of B sufficiently large for the error rate to have settled down. 8.2.3 Boosting We now discuss boosting, yet another approach for improving the predic- boosting tions resulting from a decision tree. Like bagging, boosting is a general approach that can be applied to many statistical learning methods for re- gression or classification. Here we restrict our discussion of boosting to the context of decision trees. Recall that bagging involves creating multiple copies of the original train- ing data set using the bootstrap, fitting a separate decision tree to each copy, and then combining all of the trees in order to create a single predic- tive model. Notably, each tree is built on a bootstrap data set, independent of the other trees. Boosting works in a similar way, except that the trees are grown sequentially: each tree is grown using information from previously grown trees. Boosting does not involve bootstrap sampling; instead each tree is fit on a modified version of the original data set. 4The null rate results from simply classifying each observation to the dominant class overall, which is in this case the normal class.
- 354. 346 8. Tree-Based Methods 0 100 200 300 400 500 0.2 0.3 0.4 0.5 Number of Trees Test Classification Error m=p m=p/2 m= p FIGURE 8.10. Results from random forests for the 15-class gene expression data set with p = 500 predictors. The test error is displayed as a function of the number of trees. Each colored line corresponds to a different value of m, the number of predictors available for splitting at each interior tree node. Random forests (m < p) lead to a slight improvement over bagging (m = p). A single classification tree has an error rate of 45.7 %. Consider first the regression setting. Like bagging, boosting involves com- bining a large number of decision trees, ˆ f1 , . . . , ˆ fB . Boosting is described in Algorithm 8.2. What is the idea behind this procedure? Unlike fitting a single large deci- sion tree to the data, which amounts to fitting the data hard and potentially overfitting, the boosting approach instead learns slowly. Given the current model, we fit a decision tree to the residuals from the model. That is, we fit a tree using the current residuals, rather than the outcome Y , as the re- sponse. We then add this new decision tree into the fitted function in order to update the residuals. Each of these trees can be rather small, with just a few terminal nodes, determined by the parameter d in the algorithm. By fitting small trees to the residuals, we slowly improve ˆ f in areas where it does not perform well. The shrinkage parameter λ slows the process down even further, allowing more and different shaped trees to attack the resid- uals. In general, statistical learning approaches that learn slowly tend to perform well. Note that in boosting, unlike in bagging, the construction of each tree depends strongly on the trees that have already been grown. We have just described the process of boosting regression trees. Boosting classification trees proceeds in a similar but slightly more complex way, and the details are omitted here. Boosting has three tuning parameters:
- 355. 8.2 Bagging, Random Forests, Boosting, and Bayesian Additive Regression Trees 347 Algorithm 8.2 Boosting for Regression Trees 1. Set ˆ f(x) = 0 and ri = yi for all i in the training set. 2. For b = 1, 2, . . . , B, repeat: (a) Fit a tree ˆ fb with d splits (d + 1 terminal nodes) to the training data (X, r). (b) Update ˆ f by adding in a shrunken version of the new tree: ˆ f(x) ← ˆ f(x) + λ ˆ fb (x). (8.10) (c) Update the residuals, ri ← ri − λ ˆ fb (xi). (8.11) 3. Output the boosted model, ˆ f(x) = B 0 b=1 λ ˆ fb (x). (8.12) 1. The number of trees B. Unlike bagging and random forests, boosting can overfit if B is too large, although this overfitting tends to occur slowly if at all. We use cross-validation to select B. 2. The shrinkage parameter λ, a small positive number. This controls the rate at which boosting learns. Typical values are 0.01 or 0.001, and the right choice can depend on the problem. Very small λ can require using a very large value of B in order to achieve good performance. 3. The number d of splits in each tree, which controls the complexity of the boosted ensemble. Often d = 1 works well, in which case each tree is a stump, consisting of a single split. In this case, the boosted stump ensemble is fitting an additive model, since each term involves only a single variable. More generally d is the interaction depth, and controls interaction depth the interaction order of the boosted model, since d splits can involve at most d variables. In Figure 8.11, we applied boosting to the 15-class cancer gene expression data set, in order to develop a classifier that can distinguish the normal class from the 14 cancer classes. We display the test error as a function of the total number of trees and the interaction depth d. We see that simple stumps with an interaction depth of one perform well if enough of them are included. This model outperforms the depth-two model, and both out- perform a random forest. This highlights one difference between boosting
- 356. 348 8. Tree-Based Methods 0 1000 2000 3000 4000 5000 0.05 0.10 0.15 0.20 0.25 Number of Trees Test Classification Error Boosting: depth=1 Boosting: depth=2 RandomForest: m= p FIGURE 8.11. Results from performing boosting and random forests on the 15-class gene expression data set in order to predict cancer versus normal. The test error is displayed as a function of the number of trees. For the two boosted models, λ = 0.01. Depth-1 trees slightly outperform depth-2 trees, and both out- perform the random forest, although the standard errors are around 0.02, making none of these differences significant. The test error rate for a single tree is 24 %. and random forests: in boosting, because the growth of a particular tree takes into account the other trees that have already been grown, smaller trees are typically sufficient. Using smaller trees can aid in interpretability as well; for instance, using stumps leads to an additive model. 8.2.4 Bayesian Additive Regression Trees Finally, we discuss Bayesian additive regression trees (BART), another en- Bayesian additive regression trees semble method that uses decision trees as its building blocks. For simplicity, we present BART for regression (as opposed to classification). Recall that bagging and random forests make predictions from an aver- age of regression trees, each of which is built using a random sample of data and/or predictors. Each tree is built separately from the others. By con- trast, boosting uses a weighted sum of trees, each of which is constructed by fitting a tree to the residual of the current fit. Thus, each new tree at- tempts to capture signal that is not yet accounted for by the current set of trees. BART is related to both approaches: each tree is constructed in a random manner as in bagging and random forests, and each tree tries to capture signal not yet accounted for by the current model, as in boosting. The main novelty in BART is the way in which new trees are generated. Before we introduce the BART algorithm, we define some notation. We let K denote the number of regression trees, and B the number of iterations for which the BART algorithm will be run. The notation ˆ fb k(x) represents
- 357. 8.2 Bagging, Random Forests, Boosting, and Bayesian Additive Regression Trees 349 (a): ˆ fb−1 k (X) (b): Possibility #1 for ˆ fb k(X) | X < 169.17 X < 114.305 X < 140.35 −0.5031 0.2667 −0.2470 0.4079 | X < 169.17 X < 114.305 X < 140.35 −0.5110 0.2693 −0.2649 0.4221 (c): Possibility #2 for ˆ fb k(X) (d): Possibility #3 for ˆ fb k(X) | X < 169.17 −0.1218 0.4079 | X < 169.17 X < 114.305 X < 106.755 X < 140.35 −0.05089 −1.03100 0.26670 −0.24700 0.40790 FIGURE 8.12. A schematic of perturbed trees from the BART algorithm. (a): The kth tree at the (b−1)st iteration, ˆ fb−1 k (X), is displayed. Panels (b)–(d) dis- play three of many possibilities for ˆ fb k(X), given the form of ˆ fb−1 k (X). (b): One possibility is that ˆ fb k(X) has the same structure as ˆ fb−1 k (X), but with different predictions at the terminal nodes. (c): Another possibility is that ˆ fb k(X) results from pruning ˆ fb−1 k (X). (d): Alternatively, ˆ fb k(X) may have more terminal nodes than ˆ fb−1 k (X). the prediction at x for the kth regression tree used in the bth iteration. At the end of each iteration, the K trees from that iteration will be summed, i.e. ˆ fb (x) = )K k=1 ˆ fb k(x) for b = 1, . . . , B. In the first iteration of the BART algorithm, all trees are initialized to have a single root node, with ˆ f1 k (x) = 1 nK )n i=1 yi, the mean of the response values divided by the total number of trees. Thus, ˆ f1 (x) = )K k=1 ˆ f1 k (x) = 1 n )n i=1 yi. In subsequent iterations, BART updates each of the K trees, one at a time. In the bth iteration, to update the kth tree, we subtract from each response value the predictions from all but the kth tree, in order to obtain a partial residual ri = yi − 0 k′<k ˆ fb k′ (xi) − 0 k′>k ˆ fb−1 k′ (xi) for the ith observation, i = 1, . . . , n. Rather than fitting a fresh tree to this partial residual, BART randomly chooses a perturbation to the tree from the previous iteration ( ˆ fb−1 k ) from a set of possible perturbations, favoring ones that improve the fit to the partial residual. There are two components to this perturbation:
- 358. 350 8. Tree-Based Methods 1. We may change the structure of the tree by adding or pruning branches. 2. We may change the prediction in each terminal node of the tree. Figure 8.12 illustrates examples of possible perturbations to a tree. The output of BART is a collection of prediction models, ˆ fb (x) = K 0 k=1 ˆ fb k(x), for b = 1, 2, . . . , B. We typically throw away the first few of these prediction models, since models obtained in the earlier iterations — known as the burn-in period burn-in — tend not to provide very good results. We can let L denote the num- ber of burn-in iterations; for instance, we might take L = 200. Then, to obtain a single prediction, we simply take the average after the burn-in iterations, ˆ f(x) = 1 B−L )B b=L+1 ˆ fb (x). However, it is also possible to com- pute quantities other than the average: for instance, the percentiles of ˆ fL+1 (x), . . . , ˆ fB (x) provide a measure of uncertainty in the final predic- tion. The overall BART procedure is summarized in Algorithm 8.3. A key element of the BART approach is that in Step 3(a)ii., we do not fit a fresh tree to the current partial residual: instead, we try to improve the fit to the current partial residual by slightly modifying the tree obtained in the previous iteration (see Figure 8.12). Roughly speaking, this guards against overfitting since it limits how “hard” we fit the data in each iteration. Furthermore, the individual trees are typically quite small. We limit the tree size in order to avoid overfitting the data, which would be more likely to occur if we grew very large trees. Figure 8.13 shows the result of applying BART to the Heart data, using K = 200 trees, as the number of iterations is increased to 10, 000. During the initial iterations, the test and training errors jump around a bit. After this initial burn-in period, the error rates settle down. We note that there is only a small difference between the training error and the test error, indicating that the tree perturbation process largely avoids overfitting. The training and test errors for boosting are also displayed in Figure 8.13. We see that the test error for boosting approaches that of BART, but then begins to increase as the number of iterations increases. Furthermore, the training error for boosting decreases as the number of iterations increases, indicating that boosting has overfit the data. Though the details are outside of the scope of this book, it turns out that the BART method can be viewed as a Bayesian approach to fitting an ensemble of trees: each time we randomly perturb a tree in order to fit the residuals, we are in fact drawing a new tree from a posterior distribution. (Of course, this Bayesian connection is the motivation for BART’s name.) Furthermore, Algorithm 8.3 can be viewed as a Markov chain Monte Carlo Markov chain Monte Carlo algorithm for fitting the BART model.
- 359. 8.2 Bagging, Random Forests, Boosting, and Bayesian Additive Regression Trees 351 Algorithm 8.3 Bayesian Additive Regression Trees 1. Let ˆ f1 1 (x) = ˆ f1 2 (x) = · · · = ˆ f1 K(x) = 1 nK )n i=1 yi. 2. Compute ˆ f1 (x) = )K k=1 ˆ f1 k (x) = 1 n )n i=1 yi. 3. For b = 2, . . . , B: (a) For k = 1, 2, . . . , K: i. For i = 1, . . . , n, compute the current partial residual ri = yi − 0 k′<k ˆ fb k′ (xi) − 0 k′>k ˆ fb−1 k′ (xi). ii. Fit a new tree, ˆ fb k(x), to ri, by randomly perturbing the kth tree from the previous iteration, ˆ fb−1 k (x). Perturbations that improve the fit are favored. (b) Compute ˆ fb (x) = )K k=1 ˆ fb k(x). 4. Compute the mean after L burn-in samples, ˆ f(x) = 1 B − L B 0 b=L+1 ˆ fb (x). When we apply BART, we must select the number of trees K, the number of iterations B, and the number of burn-in iterations L. We typically choose large values for B and K, and a moderate value for L: for instance, K = 200, B = 1,000, and L = 100 is a reasonable choice. BART has been shown to have very impressive out-of-box performance — that is, it performs well with minimal tuning. 8.2.5 Summary of Tree Ensemble Methods Trees are an attractive choice of weak learner for an ensemble method for a number of reasons, including their flexibility and ability to handle predictors of mixed types (i.e. qualitative as well as quantitative). We have now seen four approaches for fitting an ensemble of trees: bagging, random forests, boosting, and BART. • In bagging, the trees are grown independently on random samples of the observations. Consequently, the trees tend to be quite similar to each other. Thus, bagging can get caught in local optima and can fail to thoroughly explore the model space.
- 360. 352 8. Tree-Based Methods 5 10 50 100 500 5000 0.0 0.1 0.2 0.3 0.4 0.5 Number of Iterations Error BART Training Error BART Test Error Boosting Training Error Boosting Test Error FIGURE 8.13. BART and boosting results for the Heart data. Both training and test errors are displayed. After a burn-in period of 100 iterations (shown in gray), the error rates for BART settle down. Boosting begins to overfit after a few hundred iterations. • In random forests, the trees are once again grown independently on random samples of the observations. However, each split on each tree is performed using a random subset of the features, thereby decorre- lating the trees, and leading to a more thorough exploration of model space relative to bagging. • In boosting, we only use the original data, and do not draw any ran- dom samples. The trees are grown successively, using a “slow” learn- ing approach: each new tree is fit to the signal that is left over from the earlier trees, and shrunken down before it is used. • In BART, we once again only make use of the original data, and we grow the trees successively. However, each tree is perturbed in order to avoid local minima and achieve a more thorough exploration of the model space.
- 361. 8.3 Lab: Decision Trees 353 8.3 Lab: Decision Trees 8.3.1 Fitting Classification Trees The tree library is used to construct classification and regression trees. > library(tree) We first use classification trees to analyze the Carseats data set. In these data, Sales is a continuous variable, and so we begin by recoding it as a binary variable. We use the ifelse() function to create a variable, called ifelse() High, which takes on a value of Yes if the Sales variable exceeds 8, and takes on a value of No otherwise. > library(ISLR2) > attach(Carseats) > High <- factor(ifelse(Sales <= 8, "No", "Yes")) Finally, we use the data.frame() function to merge High with the rest of the Carseats data. > Carseats <- data.frame(Carseats , High) We now use the tree() function to fit a classification tree in order to predict tree() High using all variables but Sales. The syntax of the tree() function is quite similar to that of the lm() function. > tree.carseats <- tree(High ∼ . - Sales , Carseats) The summary() function lists the variables that are used as internal nodes in the tree, the number of terminal nodes, and the (training) error rate. > summary(tree.carseats) Classification tree: tree(formula = High ∼ . - Sales , data = Carseats) Variables actually used in tree construction : [1] "ShelveLoc" "Price" "Income" "CompPrice" [5] "Population" " Advertising " "Age" "US" Number of terminal nodes: 27 Residual mean deviance: 0.4575 = 170.7 / 373 Misclassification error rate: 0.09 = 36 / 400 We see that the training error rate is 9 %. For classification trees, the de- viance reported in the output of summary() is given by −2 0 m 0 k nmk log p̂mk, where nmk is the number of observations in the mth terminal node that belong to the kth class. This is closely related to the entropy, defined in (8.7). A small deviance indicates a tree that provides a good fit to the (training) data. The residual mean deviance reported is simply the deviance divided by n − |T0|, which in this case is 400 − 27 = 373.
- 362. 354 8. Tree-Based Methods One of the most attractive properties of trees is that they can be graphically displayed. We use the plot() function to display the tree struc- ture, and the text() function to display the node labels. The argument pretty = 0 instructs R to include the category names for any qualitative predictors, rather than simply displaying a letter for each category. > plot(tree.carseats) > text(tree.carseats , pretty = 0) The most important indicator of Sales appears to be shelving location, since the first branch differentiates Good locations from Bad and Medium locations. If we just type the name of the tree object, R prints output corresponding to each branch of the tree. R displays the split criterion (e.g. Price < 92.5), the number of observations in that branch, the deviance, the overall pre- diction for the branch (Yes or No), and the fraction of observations in that branch that take on values of Yes and No. Branches that lead to terminal nodes are indicated using asterisks. > tree.carseats node), split , n, deviance , yval , (yprob) * denotes terminal node 1) root 400 541.5 No ( 0.590 0.410 ) 2) ShelveLoc: Bad ,Medium 315 390.6 No ( 0.689 0.311 ) 4) Price < 92.5 46 56.53 Yes ( 0.304 0.696 ) 8) Income < 57 10 12.22 No ( 0.700 0.300 ) In order to properly evaluate the performance of a classification tree on these data, we must estimate the test error rather than simply computing the training error. We split the observations into a training set and a test set, build the tree using the training set, and evaluate its performance on the test data. The predict() function can be used for this purpose. In the case of a classification tree, the argument type = "class" instructs R to return the actual class prediction. This approach leads to correct predictions for around 77 % of the locations in the test data set. > set.seed (2) > train <- sample (1: nrow(Carseats), 200) > Carseats.test <- Carseats[-train , ] > High.test <- High[-train] > tree.carseats <- tree(High ∼ . - Sales , Carseats , subset = train) > tree.pred <- predict(tree.carseats , Carseats.test , type = "class") > table(tree.pred , High.test) High.test tree.pred No Yes No 104 33 Yes 13 50 > (104 + 50) / 200 [1] 0.77
- 363. 8.3 Lab: Decision Trees 355 (If you re-run the predict() function then you might get slightly different results, due to “ties”: for instance, this can happen when the training ob- servations corresponding to a terminal node are evenly split between Yes and No response values.) Next, we consider whether pruning the tree might lead to improved results. The function cv.tree() performs cross-validation in order to cv.tree() determine the optimal level of tree complexity; cost complexity pruning is used in order to select a sequence of trees for consideration. We use the argument FUN = prune.misclass in order to indicate that we want the classification error rate to guide the cross-validation and pruning process, rather than the default for the cv.tree() function, which is deviance. The cv.tree() function reports the number of terminal nodes of each tree con- sidered (size) as well as the corresponding error rate and the value of the cost-complexity parameter used (k, which corresponds to α in (8.4)). > set.seed (7) > cv.carseats <- cv.tree(tree.carseats , FUN = prune.misclass) > names(cv.carseats) [1] "size" "dev" "k" "method" > cv.carseats $size [1] 21 19 14 9 8 5 3 2 1 $dev [1] 75 75 75 74 82 83 83 85 82 $k [1] -Inf 0.0 1.0 1.4 2.0 3.0 4.0 9.0 18.0 $method [1] "misclass" attr(," class ") [1] "prune" "tree.sequence" Despite its name, dev corresponds to the number of cross-validation errors. The tree with 9 terminal nodes results in only 74 cross-validation errors. We plot the error rate as a function of both size and k. > par(mfrow = c(1, 2)) > plot(cv.carseats$size , cv.carseats$dev , type = "b") > plot(cv.carseats$k, cv.carseats$dev , type = "b") We now apply the prune.misclass() function in order to prune the tree to obtain the nine-node tree. > prune.carseats <- prune.misclass(tree.carseats , best = 9) > plot(prune.carseats) > text(prune.carseats , pretty = 0) How well does this pruned tree perform on the test data set? Once again, we apply the predict() function. > tree.pred <- predict(prune.carseats , Carseats.test , type = "class") > table(tree.pred , High.test) High.test prune.misclass()
- 364. 356 8. Tree-Based Methods tree.pred No Yes No 97 25 Yes 20 58 > (97 + 58) / 200 [1] 0.775 Now 77.5 % of the test observations are correctly classified, so not only has the pruning process produced a more interpretable tree, but it has also slightly improved the classification accuracy. If we increase the value of best, we obtain a larger pruned tree with lower classification accuracy: > prune.carseats <- prune.misclass(tree.carseats , best = 14) > plot(prune.carseats) > text(prune.carseats , pretty = 0) > tree.pred <- predict(prune.carseats , Carseats.test , type = "class") > table(tree.pred , High.test) High.test tree.pred No Yes No 102 31 Yes 15 52 > (102 + 52) / 200 [1] 0.77 8.3.2 Fitting Regression Trees Here we fit a regression tree to the Boston data set. First, we create a training set, and fit the tree to the training data. > set.seed (1) > train <- sample (1: nrow(Boston), nrow(Boston) / 2) > tree.boston <- tree(medv ∼ ., Boston , subset = train) > summary(tree.boston) Regression tree: tree(formula = medv ∼ ., data = Boston , subset = train) Variables actually used in tree construction : [1] "rm" "lstat" "crim" "age" Number of terminal nodes: 7 Residual mean deviance: 10.4 = 2550 / 246 Distribution of residuals: Min. 1st Qu. Median Mean 3rd Qu. Max. -10.200 -1.780 -0.177 0.000 1.920 16.600 Notice that the output of summary() indicates that only four of the variables have been used in constructing the tree. In the context of a regression tree, the deviance is simply the sum of squared errors for the tree. We now plot the tree. > plot(tree.boston) > text(tree.boston , pretty = 0)
- 365. 8.3 Lab: Decision Trees 357 The variable lstat measures the percentage of individuals with lower socioeconomic status, while the variable rm corresponds to the average num- ber of rooms. The tree indicates that larger values of rm, or lower values of lstat, correspond to more expensive houses. For example, the tree predicts a median house price of $45,400 for homes in census tracts in which rm >= 7.553. It is worth noting that we could have fit a much bigger tree, by pass- ing control = tree.control(nobs = length(train), mindev = 0) into the tree() function. Now we use the cv.tree() function to see whether pruning the tree will improve performance. > cv.boston <- cv.tree(tree.boston) > plot(cv.boston$size , cv.boston$dev , type = "b") In this case, the most complex tree under consideration is selected by cross- validation. However, if we wish to prune the tree, we could do so as follows, using the prune.tree() function: prune.tree() > prune.boston <- prune.tree(tree.boston , best = 5) > plot(prune.boston) > text(prune.boston , pretty = 0) In keeping with the cross-validation results, we use the unpruned tree to make predictions on the test set. > yhat <- predict(tree.boston , newdata = Boston[-train , ]) > boston.test <- Boston[-train , "medv"] > plot(yhat , boston.test) > abline (0, 1) > mean (( yhat - boston.test)^2) [1] 35.29 In other words, the test set MSE associated with the regression tree is 35.29. The square root of the MSE is therefore around 5.941, indicating that this model leads to test predictions that are (on average) within approximately $5,941 of the true median home value for the census tract. 8.3.3 Bagging and Random Forests Here we apply bagging and random forests to the Boston data, using the randomForest package in R. The exact results obtained in this section may depend on the version of R and the version of the randomForest package installed on your computer. Recall that bagging is simply a special case of a random forest with m = p. Therefore, the randomForest() function can randomForest() be used to perform both random forests and bagging. We perform bagging as follows: > library( randomForest ) > set.seed (1) > bag.boston <- randomForest (medv ∼ ., data = Boston ,
- 366. 358 8. Tree-Based Methods subset = train , mtry = 12, importance = TRUE) > bag.boston Call: randomForest (formula = medv ∼ ., data = Boston , mtry = 12, importance = TRUE , subset = train) Type of random forest: regression Number of trees: 500 No. of variables tried at each split: 12 Mean of squared residuals: 11.40 % Var explained: 85.17 The argument mtry = 12 indicates that all 12 predictors should be consid- ered for each split of the tree—in other words, that bagging should be done. How well does this bagged model perform on the test set? > yhat.bag <- predict(bag.boston , newdata = Boston[-train , ]) > plot(yhat.bag , boston.test) > abline (0, 1) > mean (( yhat.bag - boston.test)^2) [1] 23.42 The test set MSE associated with the bagged regression tree is 23.42, about two-thirds of that obtained using an optimally-pruned single tree. We could change the number of trees grown by randomForest() using the ntree argu- ment: > bag.boston <- randomForest (medv ∼ ., data = Boston , subset = train , mtry = 12, ntree = 25) > yhat.bag <- predict(bag.boston , newdata = Boston[-train , ]) > mean (( yhat.bag - boston.test)^2) [1] 25.75 Growing a random forest proceeds in exactly the same way, except that we use a smaller value of the mtry argument. By default, randomForest() uses p/3 variables when building a random forest of regression trees, and √ p variables when building a random forest of classification trees. Here we use mtry = 6. > set.seed (1) > rf.boston <- randomForest (medv ∼ ., data = Boston , subset = train , mtry = 6, importance = TRUE) > yhat.rf <- predict(rf.boston , newdata = Boston[-train , ]) > mean (( yhat.rf - boston.test)^2) [1] 20.07 The test set MSE is 20.07; this indicates that random forests yielded an improvement over bagging in this case. Using the importance() function, we can view the importance of each importance() variable. > importance (rf.boston) %IncMSE IncNodePurity crim 19.436 1070.42
- 367. 8.3 Lab: Decision Trees 359 zn 3.092 82.19 indus 6.141 590.10 chas 1.370 36.70 nox 13.263 859.97 rm 35.095 8270.34 age 15.145 634.31 dis 9.164 684.88 rad 4.794 83.19 tax 4.411 292.21 ptratio 8.613 902.20 lstat 28.725 5813.05 Two measures of variable importance are reported. The first is based upon the mean decrease of accuracy in predictions on the out of bag samples when a given variable is permuted. The second is a measure of the total decrease in node impurity that results from splits over that variable, averaged over all trees (this was plotted in Figure 8.9). In the case of regression trees, the node impurity is measured by the training RSS, and for classification trees by the deviance. Plots of these importance measures can be produced using the varImpPlot() function. varImpPlot() > varImpPlot (rf.boston) The results indicate that across all of the trees considered in the random forest, the wealth of the community (lstat) and the house size (rm) are by far the two most important variables. 8.3.4 Boosting Here we use the gbm package, and within it the gbm() function, to fit boosted gbm() regression trees to the Boston data set. We run gbm() with the option distribution = "gaussian" since this is a regression problem; if it were a binary classification problem, we would use distribution = "bernoulli". The argument n.trees = 5000 indicates that we want 5000 trees, and the option interaction.depth = 4 limits the depth of each tree. > library(gbm) > set.seed (1) > boost.boston <- gbm(medv ∼ ., data = Boston[train , ], distribution = "gaussian ", n.trees = 5000 , interaction .depth = 4) The summary() function produces a relative influence plot and also outputs the relative influence statistics. > summary(boost.boston) var rel.inf rm rm 44.482 lstat lstat 32.703 crim crim 4.851 dis dis 4.487 nox nox 3.752
- 368. 360 8. Tree-Based Methods age age 3.198 ptratio ptratio 2.814 tax tax 1.544 indus indus 1.034 rad rad 0.876 zn zn 0.162 chas chas 0.097 We see that lstat and rm are by far the most important variables. We can also produce partial dependence plots for these two variables. These plots partial dependence plot illustrate the marginal effect of the selected variables on the response after integrating out the other variables. In this case, as we might expect, median house prices are increasing with rm and decreasing with lstat. > plot(boost.boston , i = "rm") > plot(boost.boston , i = "lstat") We now use the boosted model to predict medv on the test set: > yhat.boost <- predict(boost.boston , newdata = Boston[-train , ], n.trees = 5000) > mean (( yhat.boost - boston.test)^2) [1] 18.39 The test MSE obtained is 18.39: this is superior to the test MSE of random forests and bagging. If we want to, we can perform boosting with a different value of the shrinkage parameter λ in (8.10). The default value is 0.001, but this is easily modified. Here we take λ = 0.2. > boost.boston <- gbm(medv ∼ ., data = Boston[train , ], distribution = "gaussian ", n.trees = 5000 , interaction .depth = 4, shrinkage = 0.2, verbose = F) > yhat.boost <- predict(boost.boston , newdata = Boston[-train , ], n.trees = 5000) > mean (( yhat.boost - boston.test)^2) [1] 16.55 In this case, using λ = 0.2 leads to a lower test MSE than λ = 0.001. 8.3.5 Bayesian Additive Regression Trees In this section we use the BART package, and within it the gbart() function, gbart() to fit a Bayesian additive regression tree model to the Boston housing data set. The gbart() function is designed for quantitative outcome variables. For binary outcomes, lbart() and pbart() are available. lbart() pbart() To run the gbart() function, we must first create matrices of predictors for the training and test data. We run BART with default settings. > library(BART) > x <- Boston[, 1:12] > y <- Boston[, "medv"] > xtrain <- x[train , ] > ytrain <- y[train]
- 369. 8.4 Exercises 361 > xtest <- x[-train , ] > ytest <- y[-train] > set.seed (1) > bartfit <- gbart(xtrain , ytrain , x.test = xtest) Next we compute the test error. > yhat.bart <- bartfit$yhat.test.mean > mean (( ytest - yhat.bart)^2) [1] 15.95 On this data set, the test error of BART is lower than the test error of random forests and boosting. Now we can check how many times each variable appeared in the collec- tion of trees. > ord <- order(bartfit$varcount.mean , decreasing = T) > bartfit$varcount.mean[ord] nox lstat tax rad rm indus chas ptratio 22.95 21.33 21.25 20.78 19.89 19.82 19.05 18.98 age zn dis crim 18.27 15.95 14.46 11.01 8.4 Exercises Conceptual 1. Draw an example (of your own invention) of a partition of two- dimensional feature space that could result from recursive binary splitting. Your example should contain at least six regions. Draw a decision tree corresponding to this partition. Be sure to label all as- pects of your figures, including the regions R1, R2, . . ., the cutpoints t1, t2, . . ., and so forth. Hint: Your result should look something like Figures 8.1 and 8.2. 2. It is mentioned in Section 8.2.3 that boosting using depth-one trees (or stumps) leads to an additive model: that is, a model of the form f(X) = p 0 j=1 fj(Xj). Explain why this is the case. You can begin with (8.12) in Algorithm 8.2. 3. Consider the Gini index, classification error, and entropy in a simple classification setting with two classes. Create a single plot that dis- plays each of these quantities as a function of p̂m1. The x-axis should
- 370. 362 8. Tree-Based Methods | X2 < 1 X1 < 1 X1 < 0 X2 < 2 -1.80 -1.06 0.21 0.63 2.49 5 15 10 0 3 0 1 X2 X1 0 1 FIGURE 8.14. Left: A partition of the predictor space corresponding to Exer- cise 4a. Right: A tree corresponding to Exercise 4b. display p̂m1, ranging from 0 to 1, and the y-axis should display the value of the Gini index, classification error, and entropy. Hint: In a setting with two classes, p̂m1 = 1 − p̂m2. You could make this plot by hand, but it will be much easier to make in R. 4. This question relates to the plots in Figure 8.14. (a) Sketch the tree corresponding to the partition of the predictor space illustrated in the left-hand panel of Figure 8.14. The num- bers inside the boxes indicate the mean of Y within each region. (b) Create a diagram similar to the left-hand panel of Figure 8.14, using the tree illustrated in the right-hand panel of the same figure. You should divide up the predictor space into the correct regions, and indicate the mean for each region. 5. Suppose we produce ten bootstrapped samples from a data set containing red and green classes. We then apply a classification tree to each bootstrapped sample and, for a specific value of X, produce 10 estimates of P(Class is Red|X): 0.1, 0.15, 0.2, 0.2, 0.55, 0.6, 0.6, 0.65, 0.7, and 0.75. There are two common ways to combine these results together into a single class prediction. One is the majority vote approach discussed in this chapter. The second approach is to classify based on the average probability. In this example, what is the final classification under each of these two approaches? 6. Provide a detailed explanation of the algorithm that is used to fit a regression tree.
- 371. 8.4 Exercises 363 Applied 7. In the lab, we applied random forests to the Boston data using mtry = 6 and using ntree = 25 and ntree = 500. Create a plot displaying the test error resulting from random forests on this data set for a more comprehensive range of values for mtry and ntree. You can model your plot after Figure 8.10. Describe the results obtained. 8. In the lab, a classification tree was applied to the Carseats data set af- ter converting Sales into a qualitative response variable. Now we will seek to predict Sales using regression trees and related approaches, treating the response as a quantitative variable. (a) Split the data set into a training set and a test set. (b) Fit a regression tree to the training set. Plot the tree, and inter- pret the results. What test MSE do you obtain? (c) Use cross-validation in order to determine the optimal level of tree complexity. Does pruning the tree improve the test MSE? (d) Use the bagging approach in order to analyze this data. What test MSE do you obtain? Use the importance() function to de- termine which variables are most important. (e) Use random forests to analyze this data. What test MSE do you obtain? Use the importance() function to determine which vari- ables are most important. Describe the effect of m, the number of variables considered at each split, on the error rate obtained. (f) Now analyze the data using BART, and report your results. 9. This problem involves the OJ data set which is part of the ISLR2 package. (a) Create a training set containing a random sample of 800 obser- vations, and a test set containing the remaining observations. (b) Fit a tree to the training data, with Purchase as the response and the other variables as predictors. Use the summary() function to produce summary statistics about the tree, and describe the results obtained. What is the training error rate? How many terminal nodes does the tree have? (c) Type in the name of the tree object in order to get a detailed text output. Pick one of the terminal nodes, and interpret the information displayed. (d) Create a plot of the tree, and interpret the results. (e) Predict the response on the test data, and produce a confusion matrix comparing the test labels to the predicted test labels. What is the test error rate?
- 372. 364 8. Tree-Based Methods (f) Apply the cv.tree() function to the training set in order to determine the optimal tree size. (g) Produce a plot with tree size on the x-axis and cross-validated classification error rate on the y-axis. (h) Which tree size corresponds to the lowest cross-validated classi- fication error rate? (i) Produce a pruned tree corresponding to the optimal tree size obtained using cross-validation. If cross-validation does not lead to selection of a pruned tree, then create a pruned tree with five terminal nodes. (j) Compare the training error rates between the pruned and un- pruned trees. Which is higher? (k) Compare the test error rates between the pruned and unpruned trees. Which is higher? 10. We now use boosting to predict Salary in the Hitters data set. (a) Remove the observations for whom the salary information is unknown, and then log-transform the salaries. (b) Create a training set consisting of the first 200 observations, and a test set consisting of the remaining observations. (c) Perform boosting on the training set with 1,000 trees for a range of values of the shrinkage parameter λ. Produce a plot with different shrinkage values on the x-axis and the corresponding training set MSE on the y-axis. (d) Produce a plot with different shrinkage values on the x-axis and the corresponding test set MSE on the y-axis. (e) Compare the test MSE of boosting to the test MSE that results from applying two of the regression approaches seen in Chapters 3 and 6. (f) Which variables appear to be the most important predictors in the boosted model? (g) Now apply bagging to the training set. What is the test set MSE for this approach? 11. This question uses the Caravan data set. (a) Create a training set consisting of the first 1,000 observations, and a test set consisting of the remaining observations. (b) Fit a boosting model to the training set with Purchase as the response and the other variables as predictors. Use 1,000 trees, and a shrinkage value of 0.01. Which predictors appear to be the most important?
- 373. 8.4 Exercises 365 (c) Use the boosting model to predict the response on the test data. Predict that a person will make a purchase if the estimated prob- ability of purchase is greater than 20 %. Form a confusion ma- trix. What fraction of the people predicted to make a purchase do in fact make one? How does this compare with the results obtained from applying KNN or logistic regression to this data set? 12. Apply boosting, bagging, random forests, and BART to a data set of your choice. Be sure to fit the models on a training set and to evaluate their performance on a test set. How accurate are the results compared to simple methods like linear or logistic regression? Which of these approaches yields the best performance?
- 374. 9 Support Vector Machines In this chapter, we discuss the support vector machine (SVM), an approach for classification that was developed in the computer science community in the 1990s and that has grown in popularity since then. SVMs have been shown to perform well in a variety of settings, and are often considered one of the best “out of the box” classifiers. The support vector machine is a generalization of a simple and intu- itive classifier called the maximal margin classifier, which we introduce in Section 9.1. Though it is elegant and simple, we will see that this classifier unfortunately cannot be applied to most data sets, since it requires that the classes be separable by a linear boundary. In Section 9.2, we introduce the support vector classifier, an extension of the maximal margin classifier that can be applied in a broader range of cases. Section 9.3 introduces the support vector machine, which is a further extension of the support vec- tor classifier in order to accommodate non-linear class boundaries. Support vector machines are intended for the binary classification setting in which there are two classes; in Section 9.4 we discuss extensions of support vector machines to the case of more than two classes. In Section 9.5 we discuss the close connections between support vector machines and other statistical methods such as logistic regression. People often loosely refer to the maximal margin classifier, the support vector classifier, and the support vector machine as “support vector machines”. To avoid confusion, we will carefully distinguish between these three notions in this chapter. © Springer Science+Business Media, LLC, part of Springer Nature 2021 G. James et al., An Introduction to Statistical Learning, Springer Texts in Statistics, https://doi.org/10.1007/978-1-0716-1418-1_9 367
- 375. 368 9. Support Vector Machines 9.1 Maximal Margin Classifier In this section, we define a hyperplane and introduce the concept of an optimal separating hyperplane. 9.1.1 What Is a Hyperplane? In a p-dimensional space, a hyperplane is a flat affine subspace of hyperplane dimension p − 1.1 For instance, in two dimensions, a hyperplane is a flat one-dimensional subspace—in other words, a line. In three dimensions, a hyperplane is a flat two-dimensional subspace—that is, a plane. In p > 3 dimensions, it can be hard to visualize a hyperplane, but the notion of a (p − 1)-dimensional flat subspace still applies. The mathematical definition of a hyperplane is quite simple. In two di- mensions, a hyperplane is defined by the equation β0 + β1X1 + β2X2 = 0 (9.1) for parameters β0, β1, and β2. When we say that (9.1) “defines” the hyper- plane, we mean that any X = (X1, X2)T for which (9.1) holds is a point on the hyperplane. Note that (9.1) is simply the equation of a line, since indeed in two dimensions a hyperplane is a line. Equation 9.1 can be easily extended to the p-dimensional setting: β0 + β1X1 + β2X2 + · · · + βpXp = 0 (9.2) defines a p-dimensional hyperplane, again in the sense that if a point X = (X1, X2, . . . , Xp)T in p-dimensional space (i.e. a vector of length p) satisfies (9.2), then X lies on the hyperplane. Now, suppose that X does not satisfy (9.2); rather, β0 + β1X1 + β2X2 + · · · + βpXp > 0. (9.3) Then this tells us that X lies to one side of the hyperplane. On the other hand, if β0 + β1X1 + β2X2 + · · · + βpXp < 0, (9.4) then X lies on the other side of the hyperplane. So we can think of the hyperplane as dividing p-dimensional space into two halves. One can easily determine on which side of the hyperplane a point lies by simply calculating the sign of the left hand side of (9.2). A hyperplane in two-dimensional space is shown in Figure 9.1. 1The word affine indicates that the subspace need not pass through the origin.
- 376. 9.1 Maximal Margin Classifier 369 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 X1 X 2 FIGURE 9.1. The hyperplane 1 + 2X1 + 3X2 = 0 is shown. The blue region is the set of points for which 1 + 2X1 + 3X2 > 0, and the purple region is the set of points for which 1 + 2X1 + 3X2 < 0. 9.1.2 Classification Using a Separating Hyperplane Now suppose that we have a n×p data matrix X that consists of n training observations in p-dimensional space, x1 = ⎛ ⎜ ⎝ x11 . . . x1p ⎞ ⎟ ⎠ , . . . , xn = ⎛ ⎜ ⎝ xn1 . . . xnp ⎞ ⎟ ⎠ , (9.5) and that these observations fall into two classes—that is, y1, . . . , yn ∈ {−1, 1} where −1 represents one class and 1 the other class. We also have a test observation, a p-vector of observed features x∗ = ' x∗ 1 . . . x∗ p (T . Our goal is to develop a classifier based on the training data that will correctly classify the test observation using its feature measurements. We have seen a number of approaches for this task, such as linear discriminant analysis and logistic regression in Chapter 4, and classification trees, bagging, and boosting in Chapter 8. We will now see a new approach that is based upon the concept of a separating hyperplane. separating hyperplane Suppose that it is possible to construct a hyperplane that separates the training observations perfectly according to their class labels. Examples of three such separating hyperplanes are shown in the left-hand panel of Figure 9.2. We can label the observations from the blue class as yi = 1 and those from the purple class as yi = −1. Then a separating hyperplane has the property that β0 + β1xi1 + β2xi2 + · · · + βpxip > 0 if yi = 1, (9.6)
- 377. 370 9. Support Vector Machines −1 0 1 2 3 −1 0 1 2 3 −1 0 1 2 3 −1 0 1 2 3 X1 X1 X 2 X 2 FIGURE 9.2. Left: There are two classes of observations, shown in blue and in purple, each of which has measurements on two variables. Three separating hyperplanes, out of many possible, are shown in black. Right: A separating hy- perplane is shown in black. The blue and purple grid indicates the decision rule made by a classifier based on this separating hyperplane: a test observation that falls in the blue portion of the grid will be assigned to the blue class, and a test observation that falls into the purple portion of the grid will be assigned to the purple class. and β0 + β1xi1 + β2xi2 + · · · + βpxip < 0 if yi = −1. (9.7) Equivalently, a separating hyperplane has the property that yi(β0 + β1xi1 + β2xi2 + · · · + βpxip) > 0 (9.8) for all i = 1, . . . , n. If a separating hyperplane exists, we can use it to construct a very natural classifier: a test observation is assigned a class depending on which side of the hyperplane it is located. The right-hand panel of Figure 9.2 shows an example of such a classifier. That is, we classify the test observation x∗ based on the sign of f(x∗ ) = β0+β1x∗ 1+β2x∗ 2+· · ·+βpx∗ p. If f(x∗ ) is positive, then we assign the test observation to class 1, and if f(x∗ ) is negative, then we assign it to class −1. We can also make use of the magnitude of f(x∗ ). If f(x∗ ) is far from zero, then this means that x∗ lies far from the hyperplane, and so we can be confident about our class assignment for x∗ . On the other hand, if f(x∗ ) is close to zero, then x∗ is located near the hyperplane, and so we are less certain about the class assignment for x∗ . Not surprisingly, and as we see in Figure 9.2, a classifier that is based on a separating hyperplane leads to a linear decision boundary.
- 378. 9.1 Maximal Margin Classifier 371 9.1.3 The Maximal Margin Classifier In general, if our data can be perfectly separated using a hyperplane, then there will in fact exist an infinite number of such hyperplanes. This is because a given separating hyperplane can usually be shifted a tiny bit up or down, or rotated, without coming into contact with any of the observations. Three possible separating hyperplanes are shown in the left-hand panel of Figure 9.2. In order to construct a classifier based upon a separating hyperplane, we must have a reasonable way to decide which of the infinite possible separating hyperplanes to use. A natural choice is the maximal margin hyperplane (also known as the maximal margin hyperplane optimal separating hyperplane), which is the separating hyperplane that optimal separating hyperplane is farthest from the training observations. That is, we can compute the (perpendicular) distance from each training observation to a given separat- ing hyperplane; the smallest such distance is the minimal distance from the observations to the hyperplane, and is known as the margin. The maximal margin margin hyperplane is the separating hyperplane for which the margin is largest—that is, it is the hyperplane that has the farthest minimum dis- tance to the training observations. We can then classify a test observation based on which side of the maximal margin hyperplane it lies. This is known as the maximal margin classifier. We hope that a classifier that has a large maximal margin classifier margin on the training data will also have a large margin on the test data, and hence will classify the test observations correctly. Although the maxi- mal margin classifier is often successful, it can also lead to overfitting when p is large. If β0, β1, . . . , βp are the coefficients of the maximal margin hyperplane, then the maximal margin classifier classifies the test observation x∗ based on the sign of f(x∗ ) = β0 + β1x∗ 1 + β2x∗ 2 + · · · + βpx∗ p. Figure 9.3 shows the maximal margin hyperplane on the data set of Figure 9.2. Comparing the right-hand panel of Figure 9.2 to Figure 9.3, we see that the maximal margin hyperplane shown in Figure 9.3 does in- deed result in a greater minimal distance between the observations and the separating hyperplane—that is, a larger margin. In a sense, the maximal margin hyperplane represents the mid-line of the widest “slab” that we can insert between the two classes. Examining Figure 9.3, we see that three training observations are equidis- tant from the maximal margin hyperplane and lie along the dashed lines indicating the width of the margin. These three observations are known as support vectors, since they are vectors in p-dimensional space (in Figure 9.3, support vector p = 2) and they “support” the maximal margin hyperplane in the sense that if these points were moved slightly then the maximal margin hyper- plane would move as well. Interestingly, the maximal margin hyperplane depends directly on the support vectors, but not on the other observations: a movement to any of the other observations would not affect the separating hyperplane, provided that the observation’s movement does not cause it to
- 379. 372 9. Support Vector Machines −1 0 1 2 3 −1 0 1 2 3 X1 X 2 FIGURE 9.3. There are two classes of observations, shown in blue and in pur- ple. The maximal margin hyperplane is shown as a solid line. The margin is the distance from the solid line to either of the dashed lines. The two blue points and the purple point that lie on the dashed lines are the support vectors, and the dis- tance from those points to the hyperplane is indicated by arrows. The purple and blue grid indicates the decision rule made by a classifier based on this separating hyperplane. cross the boundary set by the margin. The fact that the maximal margin hyperplane depends directly on only a small subset of the observations is an important property that will arise later in this chapter when we discuss the support vector classifier and support vector machines. 9.1.4 Construction of the Maximal Margin Classifier We now consider the task of constructing the maximal margin hyperplane based on a set of n training observations x1, . . . , xn ∈ Rp and associated class labels y1, . . . , yn ∈ {−1, 1}. Briefly, the maximal margin hyperplane is the solution to the optimization problem maximize β0,β1,...,βp,M M (9.9) subject to p 0 j=1 β2 j = 1, (9.10) yi(β0 + β1xi1 + β2xi2 + · · · + βpxip) ≥ M ∀ i = 1, . . . , n. (9.11) This optimization problem (9.9)–(9.11) is actually simpler than it looks. First of all, the constraint in (9.11) that yi(β0 + β1xi1 + β2xi2 + · · · + βpxip) ≥ M ∀ i = 1, . . . , n
- 380. 9.2 Support Vector Classifiers 373 guarantees that each observation will be on the correct side of the hyper- plane, provided that M is positive. (Actually, for each observation to be on the correct side of the hyperplane we would simply need yi(β0 + β1xi1 + β2xi2 +· · ·+βpxip) > 0, so the constraint in (9.11) in fact requires that each observation be on the correct side of the hyperplane, with some cushion, provided that M is positive.) Second, note that (9.10) is not really a constraint on the hyperplane, since if β0 + β1xi1 + β2xi2 + · · · + βpxip = 0 defines a hyperplane, then so does k(β0 + β1xi1 + β2xi2 + · · · + βpxip) = 0 for any k ̸= 0. However, (9.10) adds meaning to (9.11); one can show that with this constraint the perpendicular distance from the ith observation to the hyperplane is given by yi(β0 + β1xi1 + β2xi2 + · · · + βpxip). Therefore, the constraints (9.10) and (9.11) ensure that each observation is on the correct side of the hyperplane and at least a distance M from the hyperplane. Hence, M represents the margin of our hyperplane, and the optimization problem chooses β0, β1, . . . , βp to maximize M. This is exactly the definition of the maximal margin hyperplane! The problem (9.9)–(9.11) can be solved efficiently, but details of this optimization are outside of the scope of this book. 9.1.5 The Non-separable Case The maximal margin classifier is a very natural way to perform classifi- cation, if a separating hyperplane exists. However, as we have hinted, in many cases no separating hyperplane exists, and so there is no maximal margin classifier. In this case, the optimization problem (9.9)–(9.11) has no solution with M > 0. An example is shown in Figure 9.4. In this case, we cannot exactly separate the two classes. However, as we will see in the next section, we can extend the concept of a separating hyperplane in order to develop a hyperplane that almost separates the classes, using a so-called soft margin. The generalization of the maximal margin classifier to the non-separable case is known as the support vector classifier. 9.2 Support Vector Classifiers 9.2.1 Overview of the Support Vector Classifier In Figure 9.4, we see that observations that belong to two classes are not necessarily separable by a hyperplane. In fact, even if a separating hyper- plane does exist, then there are instances in which a classifier based on a separating hyperplane might not be desirable. A classifier based on a separating hyperplane will necessarily perfectly classify all of the training
- 381. 374 9. Support Vector Machines 0 1 2 3 −1.0 −0.5 0.0 0.5 1.0 1.5 2.0 X1 X 2 FIGURE 9.4. There are two classes of observations, shown in blue and in pur- ple. In this case, the two classes are not separable by a hyperplane, and so the maximal margin classifier cannot be used. observations; this can lead to sensitivity to individual observations. An ex- ample is shown in Figure 9.5. The addition of a single observation in the right-hand panel of Figure 9.5 leads to a dramatic change in the maxi- mal margin hyperplane. The resulting maximal margin hyperplane is not satisfactory—for one thing, it has only a tiny margin. This is problematic because as discussed previously, the distance of an observation from the hyperplane can be seen as a measure of our confidence that the obser- vation was correctly classified. Moreover, the fact that the maximal mar- gin hyperplane is extremely sensitive to a change in a single observation suggests that it may have overfit the training data. In this case, we might be willing to consider a classifier based on a hy- perplane that does not perfectly separate the two classes, in the interest of • Greater robustness to individual observations, and • Better classification of most of the training observations. That is, it could be worthwhile to misclassify a few training observations in order to do a better job in classifying the remaining observations. The support vector classifier, sometimes called a soft margin classifier, support vector classifier soft margin classifier does exactly this. Rather than seeking the largest possible margin so that every observation is not only on the correct side of the hyperplane but also on the correct side of the margin, we instead allow some observations to be on the incorrect side of the margin, or even the incorrect side of the hyperplane. (The margin is soft because it can be violated by some of the training observations.) An example is shown in the left-hand panel
- 382. 9.2 Support Vector Classifiers 375 −1 0 1 2 3 −1 0 1 2 3 −1 0 1 2 3 −1 0 1 2 3 X1 X1 X 2 X 2 FIGURE 9.5. Left: Two classes of observations are shown in blue and in purple, along with the maximal margin hyperplane. Right: An additional blue observation has been added, leading to a dramatic shift in the maximal margin hyperplane shown as a solid line. The dashed line indicates the maximal margin hyperplane that was obtained in the absence of this additional point. of Figure 9.6. Most of the observations are on the correct side of the margin. However, a small subset of the observations are on the wrong side of the margin. An observation can be not only on the wrong side of the margin, but also on the wrong side of the hyperplane. In fact, when there is no separating hyperplane, such a situation is inevitable. Observations on the wrong side of the hyperplane correspond to training observations that are misclassified by the support vector classifier. The right-hand panel of Figure 9.6 illustrates such a scenario. 9.2.2 Details of the Support Vector Classifier The support vector classifier classifies a test observation depending on which side of a hyperplane it lies. The hyperplane is chosen to correctly separate most of the training observations into the two classes, but may misclassify a few observations. It is the solution to the optimization problem maximize β0,β1,...,βp,ϵ1,...,ϵn, M M (9.12) subject to p 0 j=1 β2 j = 1, (9.13) yi(β0 + β1xi1 + β2xi2 + · · · + βpxip) ≥ M(1 − ϵi), (9.14) ϵi ≥ 0, n 0 i=1 ϵi ≤ C, (9.15)
- 383. 376 9. Support Vector Machines −0.5 0.0 0.5 1.0 1.5 2.0 2.5 −1 0 1 2 3 4 1 2 3 4 5 6 7 8 9 10 −0.5 0.0 0.5 1.0 1.5 2.0 2.5 −1 0 1 2 3 4 1 2 3 4 5 6 7 8 9 10 11 12 X1 X1 X 2 X 2 FIGURE 9.6. Left: A support vector classifier was fit to a small data set. The hyperplane is shown as a solid line and the margins are shown as dashed lines. Purple observations: Observations 3, 4, 5, and 6 are on the correct side of the margin, observation 2 is on the margin, and observation 1 is on the wrong side of the margin. Blue observations: Observations 7 and 10 are on the correct side of the margin, observation 9 is on the margin, and observation 8 is on the wrong side of the margin. No observations are on the wrong side of the hyperplane. Right: Same as left panel with two additional points, 11 and 12. These two observations are on the wrong side of the hyperplane and the wrong side of the margin. where C is a nonnegative tuning parameter. As in (9.11), M is the width of the margin; we seek to make this quantity as large as possible. In (9.14), ϵ1, . . . , ϵn are slack variables that allow individual observations to be on slack variable the wrong side of the margin or the hyperplane; we will explain them in greater detail momentarily. Once we have solved (9.12)–(9.15), we classify a test observation x∗ as before, by simply determining on which side of the hyperplane it lies. That is, we classify the test observation based on the sign of f(x∗ ) = β0 + β1x∗ 1 + · · · + βpx∗ p. The problem (9.12)–(9.15) seems complex, but insight into its behavior can be made through a series of simple observations presented below. First of all, the slack variable ϵi tells us where the ith observation is located, relative to the hyperplane and relative to the margin. If ϵi = 0 then the ith observation is on the correct side of the margin, as we saw in Section 9.1.4. If ϵi > 0 then the ith observation is on the wrong side of the margin, and we say that the ith observation has violated the margin. If ϵi > 1 then it is on the wrong side of the hyperplane. We now consider the role of the tuning parameter C. In (9.15), C bounds the sum of the ϵi’s, and so it determines the number and severity of the vio- lations to the margin (and to the hyperplane) that we will tolerate. We can think of C as a budget for the amount that the margin can be violated by the n observations. If C = 0 then there is no budget for violations to the margin, and it must be the case that ϵ1 = · · · = ϵn = 0, in which case (9.12)–(9.15) simply amounts to the maximal margin hyperplane optimiza-
- 384. 9.2 Support Vector Classifiers 377 tion problem (9.9)–(9.11). (Of course, a maximal margin hyperplane exists only if the two classes are separable.) For C > 0 no more than C observa- tions can be on the wrong side of the hyperplane, because if an observation is on the wrong side of the hyperplane then ϵi > 1, and (9.15) requires that )n i=1 ϵi ≤ C. As the budget C increases, we become more tolerant of violations to the margin, and so the margin will widen. Conversely, as C decreases, we become less tolerant of violations to the margin and so the margin narrows. An example is shown in Figure 9.7. In practice, C is treated as a tuning parameter that is generally chosen via cross-validation. As with the tuning parameters that we have seen through- out this book, C controls the bias-variance trade-off of the statistical learn- ing technique. When C is small, we seek narrow margins that are rarely violated; this amounts to a classifier that is highly fit to the data, which may have low bias but high variance. On the other hand, when C is larger, the margin is wider and we allow more violations to it; this amounts to fitting the data less hard and obtaining a classifier that is potentially more biased but may have lower variance. The optimization problem (9.12)–(9.15) has a very interesting property: it turns out that only observations that either lie on the margin or that violate the margin will affect the hyperplane, and hence the classifier ob- tained. In other words, an observation that lies strictly on the correct side of the margin does not affect the support vector classifier! Changing the position of that observation would not change the classifier at all, provided that its position remains on the correct side of the margin. Observations that lie directly on the margin, or on the wrong side of the margin for their class, are known as support vectors. These observations do affect the support vector classifier. The fact that only support vectors affect the classifier is in line with our previous assertion that C controls the bias-variance trade-off of the support vector classifier. When the tuning parameter C is large, then the margin is wide, many observations violate the margin, and so there are many support vectors. In this case, many observations are involved in determining the hyperplane. The top left panel in Figure 9.7 illustrates this setting: this classifier has low variance (since many observations are support vectors) but potentially high bias. In contrast, if C is small, then there will be fewer support vectors and hence the resulting classifier will have low bias but high variance. The bottom right panel in Figure 9.7 illustrates this setting, with only eight support vectors. The fact that the support vector classifier’s decision rule is based only on a potentially small subset of the training observations (the support vec- tors) means that it is quite robust to the behavior of observations that are far away from the hyperplane. This property is distinct from some of the other classification methods that we have seen in preceding chapters, such as linear discriminant analysis. Recall that the LDA classification rule depends on the mean of all of the observations within each class, as well as
- 385. 378 9. Support Vector Machines −1 0 1 2 −3 −2 −1 0 1 2 3 −1 0 1 2 −3 −2 −1 0 1 2 3 −1 0 1 2 −3 −2 −1 0 1 2 3 −1 0 1 2 −3 −2 −1 0 1 2 3 X1 X1 X1 X1 X 2 X 2 X 2 X 2 FIGURE 9.7. A support vector classifier was fit using four different values of the tuning parameter C in (9.12)–(9.15). The largest value of C was used in the top left panel, and smaller values were used in the top right, bottom left, and bottom right panels. When C is large, then there is a high tolerance for observations being on the wrong side of the margin, and so the margin will be large. As C decreases, the tolerance for observations being on the wrong side of the margin decreases, and the margin narrows. the within-class covariance matrix computed using all of the observations. In contrast, logistic regression, unlike LDA, has very low sensitivity to ob- servations far from the decision boundary. In fact we will see in Section 9.5 that the support vector classifier and logistic regression are closely related.
- 386. 9.3 Support Vector Machines 379 −4 −2 0 2 4 −4 −2 0 2 4 −4 −2 0 2 4 −4 −2 0 2 4 X1 X1 X 2 X 2 FIGURE 9.8. Left: The observations fall into two classes, with a non-linear boundary between them. Right: The support vector classifier seeks a linear bound- ary, and consequently performs very poorly. 9.3 Support Vector Machines We first discuss a general mechanism for converting a linear classifier into one that produces non-linear decision boundaries. We then introduce the support vector machine, which does this in an automatic way. 9.3.1 Classification with Non-Linear Decision Boundaries The support vector classifier is a natural approach for classification in the two-class setting, if the boundary between the two classes is linear. How- ever, in practice we are sometimes faced with non-linear class boundaries. For instance, consider the data in the left-hand panel of Figure 9.8. It is clear that a support vector classifier or any linear classifier will perform poorly here. Indeed, the support vector classifier shown in the right-hand panel of Figure 9.8 is useless here. In Chapter 7, we are faced with an analogous situation. We see there that the performance of linear regression can suffer when there is a non- linear relationship between the predictors and the outcome. In that case, we consider enlarging the feature space using functions of the predictors, such as quadratic and cubic terms, in order to address this non-linearity. In the case of the support vector classifier, we could address the prob- lem of possibly non-linear boundaries between classes in a similar way, by enlarging the feature space using quadratic, cubic, and even higher-order polynomial functions of the predictors. For instance, rather than fitting a support vector classifier using p features X1, X2, . . . , Xp,
- 387. 380 9. Support Vector Machines we could instead fit a support vector classifier using 2p features X1, X2 1 , X2, X2 2 , . . . , Xp, X2 p . Then (9.12)–(9.15) would become maximize β0,β11,β12,...,βp1,βp2,ϵ1,...,ϵn, M M (9.16) subject to yi ⎛ ⎝β0 + p 0 j=1 βj1xij + p 0 j=1 βj2x2 ij ⎞ ⎠ ≥ M(1 − ϵi), n 0 i=1 ϵi ≤ C, ϵi ≥ 0, p 0 j=1 2 0 k=1 β2 jk = 1. Why does this lead to a non-linear decision boundary? In the enlarged feature space, the decision boundary that results from (9.16) is in fact lin- ear. But in the original feature space, the decision boundary is of the form q(x) = 0, where q is a quadratic polynomial, and its solutions are gener- ally non-linear. One might additionally want to enlarge the feature space with higher-order polynomial terms, or with interaction terms of the form XjXj′ for j ̸= j′ . Alternatively, other functions of the predictors could be considered rather than polynomials. It is not hard to see that there are many possible ways to enlarge the feature space, and that unless we are careful, we could end up with a huge number of features. Then compu- tations would become unmanageable. The support vector machine, which we present next, allows us to enlarge the feature space used by the support vector classifier in a way that leads to efficient computations. 9.3.2 The Support Vector Machine The support vector machine (SVM) is an extension of the support vector support vector machine classifier that results from enlarging the feature space in a specific way, using kernels. We will now discuss this extension, the details of which are kernel somewhat complex and beyond the scope of this book. However, the main idea is described in Section 9.3.1: we may want to enlarge our feature space in order to accommodate a non-linear boundary between the classes. The kernel approach that we describe here is simply an efficient computational approach for enacting this idea. We have not discussed exactly how the support vector classifier is com- puted because the details become somewhat technical. However, it turns out that the solution to the support vector classifier problem (9.12)–(9.15) involves only the inner products of the observations (as opposed to the observations themselves). The inner product of two r-vectors a and b is defined as ⟨a, b⟩ = )r i=1 aibi. Thus the inner product of two observations
- 388. 9.3 Support Vector Machines 381 xi, xi′ is given by ⟨xi, xi′ ⟩ = p 0 j=1 xijxi′j. (9.17) It can be shown that • The linear support vector classifier can be represented as f(x) = β0 + n 0 i=1 αi⟨x, xi⟩, (9.18) where there are n parameters αi, i = 1, . . . , n, one per training observation. • To estimate the parameters α1, . . . , αn and β0, all we need are the 'n 2 ( inner products ⟨xi, xi′ ⟩ between all pairs of training observations. (The notation 'n 2 ( means n(n − 1)/2, and gives the number of pairs among a set of n items.) Notice that in (9.18), in order to evaluate the function f(x), we need to compute the inner product between the new point x and each of the training points xi. However, it turns out that αi is nonzero only for the support vectors in the solution—that is, if a training observation is not a support vector, then its αi equals zero. So if S is the collection of indices of these support points, we can rewrite any solution function of the form (9.18) as f(x) = β0 + 0 i∈S αi⟨x, xi⟩, (9.19) which typically involves far fewer terms than in (9.18).2 To summarize, in representing the linear classifier f(x), and in computing its coefficients, all we need are inner products. Now suppose that every time the inner product (9.17) appears in the representation (9.18), or in a calculation of the solution for the support vector classifier, we replace it with a generalization of the inner product of the form K(xi, xi′ ), (9.20) where K is some function that we will refer to as a kernel. A kernel is a kernel function that quantifies the similarity of two observations. For instance, we could simply take K(xi, xi′ ) = p 0 j=1 xijxi′j, (9.21) 2By expanding each of the inner products in (9.19), it is easy to see that f(x) is a linear function of the coordinates of x. Doing so also establishes the correspondence between the αi and the original parameters βj.
- 389. 382 9. Support Vector Machines which would just give us back the support vector classifier. Equation 9.21 is known as a linear kernel because the support vector classifier is linear in the features; the linear kernel essentially quantifies the similarity of a pair of observations using Pearson (standard) correlation. But one could instead choose another form for (9.20). For instance, one could replace every instance of )p j=1 xijxi′j with the quantity K(xi, xi′ ) = (1 + p 0 j=1 xijxi′j)d . (9.22) This is known as a polynomial kernel of degree d, where d is a positive polynomial kernel integer. Using such a kernel with d > 1, instead of the standard linear kernel (9.21), in the support vector classifier algorithm leads to a much more flexible decision boundary. It essentially amounts to fitting a support vector classifier in a higher-dimensional space involving polynomials of degree d, rather than in the original feature space. When the support vector classifier is combined with a non-linear kernel such as (9.22), the resulting classifier is known as a support vector machine. Note that in this case the (non-linear) function has the form f(x) = β0 + 0 i∈S αiK(x, xi). (9.23) The left-hand panel of Figure 9.9 shows an example of an SVM with a polynomial kernel applied to the non-linear data from Figure 9.8. The fit is a substantial improvement over the linear support vector classifier. When d = 1, then the SVM reduces to the support vector classifier seen earlier in this chapter. The polynomial kernel shown in (9.22) is one example of a possible non-linear kernel, but alternatives abound. Another popular choice is the radial kernel, which takes the form radial kernel K(xi, xi′ ) = exp(−γ p 0 j=1 (xij − xi′j)2 ). (9.24) In (9.24), γ is a positive constant. The right-hand panel of Figure 9.9 shows an example of an SVM with a radial kernel on this non-linear data; it also does a good job in separating the two classes. How does the radial kernel (9.24) actually work? If a given test obser- vation x∗ = (x∗ 1, . . . , x∗ p)T is far from a training observation xi in terms of Euclidean distance, then )p j=1(x∗ j −xij)2 will be large, and so K(x∗ , xi) = exp(−γ )p j=1(x∗ j − xij)2 ) will be tiny. This means that in (9.23), xi will play virtually no role in f(x∗ ). Recall that the predicted class label for the test observation x∗ is based on the sign of f(x∗ ). In other words, training observations that are far from x∗ will play essentially no role in the pre- dicted class label for x∗ . This means that the radial kernel has very local
- 390. 9.3 Support Vector Machines 383 −4 −2 0 2 4 −4 −2 0 2 4 −4 −2 0 2 4 −4 −2 0 2 4 X1 X1 X 2 X 2 FIGURE 9.9. Left: An SVM with a polynomial kernel of degree 3 is applied to the non-linear data from Figure 9.8, resulting in a far more appropriate decision rule. Right: An SVM with a radial kernel is applied. In this example, either kernel is capable of capturing the decision boundary. behavior, in the sense that only nearby training observations have an effect on the class label of a test observation. What is the advantage of using a kernel rather than simply enlarging the feature space using functions of the original features, as in (9.16)? One advantage is computational, and it amounts to the fact that using kernels, one need only compute K(xi, x′ i) for all 'n 2 ( distinct pairs i, i′ . This can be done without explicitly working in the enlarged feature space. This is im- portant because in many applications of SVMs, the enlarged feature space is so large that computations are intractable. For some kernels, such as the radial kernel (9.24), the feature space is implicit and infinite-dimensional, so we could never do the computations there anyway! 9.3.3 An Application to the Heart Disease Data In Chapter 8 we apply decision trees and related methods to the Heart data. The aim is to use 13 predictors such as Age, Sex, and Chol in order to predict whether an individual has heart disease. We now investigate how an SVM compares to LDA on this data. After removing 6 missing observations, the data consist of 297 subjects, which we randomly split into 207 training and 90 test observations. We first fit LDA and the support vector classifier to the training data. Note that the support vector classifier is equivalent to a SVM using a poly- nomial kernel of degree d = 1. The left-hand panel of Figure 9.10 displays ROC curves (described in Section 4.4.2) for the training set predictions for both LDA and the support vector classifier. Both classifiers compute scores of the form ˆ f(X) = β̂0 + β̂1X1 + β̂2X2 + · · · + β̂pXp for each observation.
- 391. 384 9. Support Vector Machines False positive rate True positive rate 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Support Vector Classifier LDA False positive rate True positive rate 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Support Vector Classifier SVM: γ=10−3 SVM: γ=10−2 SVM: γ=10−1 FIGURE 9.10. ROC curves for the Heart data training set. Left: The support vector classifier and LDA are compared. Right: The support vector classifier is compared to an SVM using a radial basis kernel with γ = 10−3 , 10−2 , and 10−1 . For any given cutoff t, we classify observations into the heart disease or no heart disease categories depending on whether ˆ f(X) < t or ˆ f(X) ≥ t. The ROC curve is obtained by forming these predictions and computing the false positive and true positive rates for a range of values of t. An opti- mal classifier will hug the top left corner of the ROC plot. In this instance LDA and the support vector classifier both perform well, though there is a suggestion that the support vector classifier may be slightly superior. The right-hand panel of Figure 9.10 displays ROC curves for SVMs using a radial kernel, with various values of γ. As γ increases and the fit becomes more non-linear, the ROC curves improve. Using γ = 10−1 appears to give an almost perfect ROC curve. However, these curves represent training error rates, which can be misleading in terms of performance on new test data. Figure 9.11 displays ROC curves computed on the 90 test observa- tions. We observe some differences from the training ROC curves. In the left-hand panel of Figure 9.11, the support vector classifier appears to have a small advantage over LDA (although these differences are not statisti- cally significant). In the right-hand panel, the SVM using γ = 10−1 , which showed the best results on the training data, produces the worst estimates on the test data. This is once again evidence that while a more flexible method will often produce lower training error rates, this does not neces- sarily lead to improved performance on test data. The SVMs with γ = 10−2 and γ = 10−3 perform comparably to the support vector classifier, and all three outperform the SVM with γ = 10−1 .
- 392. 9.4 SVMs with More than Two Classes 385 False positive rate True positive rate 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Support Vector Classifier LDA False positive rate True positive rate 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Support Vector Classifier SVM: γ=10−3 SVM: γ=10−2 SVM: γ=10−1 FIGURE 9.11. ROC curves for the test set of the Heart data. Left: The support vector classifier and LDA are compared. Right: The support vector classifier is compared to an SVM using a radial basis kernel with γ = 10−3 , 10−2 , and 10−1 . 9.4 SVMs with More than Two Classes So far, our discussion has been limited to the case of binary classification: that is, classification in the two-class setting. How can we extend SVMs to the more general case where we have some arbitrary number of classes? It turns out that the concept of separating hyperplanes upon which SVMs are based does not lend itself naturally to more than two classes. Though a number of proposals for extending SVMs to the K-class case have been made, the two most popular are the one-versus-one and one-versus-all approaches. We briefly discuss those two approaches here. 9.4.1 One-Versus-One Classification Suppose that we would like to perform classification using SVMs, and there are K > 2 classes. A one-versus-one or all-pairs approach constructs 'K 2 ( one-versus- one SVMs, each of which compares a pair of classes. For example, one such SVM might compare the kth class, coded as +1, to the k′ th class, coded as −1. We classify a test observation using each of the 'K 2 ( classifiers, and we tally the number of times that the test observation is assigned to each of the K classes. The final classification is performed by assigning the test observation to the class to which it was most frequently assigned in these 'K 2 ( pairwise classifications. 9.4.2 One-Versus-All Classification The one-versus-all approach is an alternative procedure for applying SVMs one-versus- all in the case of K > 2 classes. We fit K SVMs, each time comparing one of
- 393. 386 9. Support Vector Machines the K classes to the remaining K − 1 classes. Let β0k, β1k, . . . , βpk denote the parameters that result from fitting an SVM comparing the kth class (coded as +1) to the others (coded as −1). Let x∗ denote a test observation. We assign the observation to the class for which β0k +β1kx∗ 1 +β2kx∗ 2 +· · ·+ βpkx∗ p is largest, as this amounts to a high level of confidence that the test observation belongs to the kth class rather than to any of the other classes. 9.5 Relationship to Logistic Regression When SVMs were first introduced in the mid-1990s, they made quite a splash in the statistical and machine learning communities. This was due in part to their good performance, good marketing, and also to the fact that the underlying approach seemed both novel and mysterious. The idea of finding a hyperplane that separates the data as well as possible, while al- lowing some violations to this separation, seemed distinctly different from classical approaches for classification, such as logistic regression and lin- ear discriminant analysis. Moreover, the idea of using a kernel to expand the feature space in order to accommodate non-linear class boundaries ap- peared to be a unique and valuable characteristic. However, since that time, deep connections between SVMs and other more classical statistical methods have emerged. It turns out that one can rewrite the criterion (9.12)–(9.15) for fitting the support vector classifier f(X) = β0 + β1X1 + · · · + βpXp as minimize β0,β1,...,βp ⎧ ⎨ ⎩ n 0 i=1 max [0, 1 − yif(xi)] + λ p 0 j=1 β2 j ⎫ ⎬ ⎭ , (9.25) where λ is a nonnegative tuning parameter. When λ is large then β1, . . . , βp are small, more violations to the margin are tolerated, and a low-variance but high-bias classifier will result. When λ is small then few violations to the margin will occur; this amounts to a high-variance but low-bias classifier. Thus, a small value of λ in (9.25) amounts to a small value of C in (9.15). Note that the λ )p j=1 β2 j term in (9.25) is the ridge penalty term from Section 6.2.1, and plays a similar role in controlling the bias-variance trade-off for the support vector classifier. Now (9.25) takes the “Loss + Penalty” form that we have seen repeatedly throughout this book: minimize β0,β1,...,βp {L(X, y, β) + λP(β)} . (9.26) In (9.26), L(X, y, β) is some loss function quantifying the extent to which the model, parametrized by β, fits the data (X, y), and P(β) is a penalty
- 394. 9.5 Relationship to Logistic Regression 387 function on the parameter vector β whose effect is controlled by a nonneg- ative tuning parameter λ. For instance, ridge regression and the lasso both take this form with L(X, y, β) = n 0 i=1 ⎛ ⎝yi − β0 − p 0 j=1 xijβj ⎞ ⎠ 2 and with P(β) = )p j=1 β2 j for ridge regression and P(β) = )p j=1 |βj| for the lasso. In the case of (9.25) the loss function instead takes the form L(X, y, β) = n 0 i=1 max [0, 1 − yi(β0 + β1xi1 + · · · + βpxip)] . This is known as hinge loss, and is depicted in Figure 9.12. However, it hinge loss turns out that the hinge loss function is closely related to the loss function used in logistic regression, also shown in Figure 9.12. An interesting characteristic of the support vector classifier is that only support vectors play a role in the classifier obtained; observations on the correct side of the margin do not affect it. This is due to the fact that the loss function shown in Figure 9.12 is exactly zero for observations for which yi(β0 + β1xi1 + · · · + βpxip) ≥ 1; these correspond to observations that are on the correct side of the margin.3 In contrast, the loss function for logistic regression shown in Figure 9.12 is not exactly zero anywhere. But it is very small for observations that are far from the decision boundary. Due to the similarities between their loss functions, logistic regression and the support vector classifier often give very similar results. When the classes are well separated, SVMs tend to behave better than logistic regression; in more overlapping regimes, logistic regression is often preferred. When the support vector classifier and SVM were first introduced, it was thought that the tuning parameter C in (9.15) was an unimportant “nui- sance” parameter that could be set to some default value, like 1. However, the “Loss + Penalty” formulation (9.25) for the support vector classifier indicates that this is not the case. The choice of tuning parameter is very important and determines the extent to which the model underfits or over- fits the data, as illustrated, for example, in Figure 9.7. We have established that the support vector classifier is closely related to logistic regression and other preexisting statistical methods. Is the SVM unique in its use of kernels to enlarge the feature space to accommodate non-linear class boundaries? The answer to this question is “no”. We could just as well perform logistic regression or many of the other classification methods seen in this book using non-linear kernels; this is closely related to 3With this hinge-loss + penalty representation, the margin corresponds to the value one, and the width of the margin is determined by ! β2 j .
- 395. 388 9. Support Vector Machines −6 −4 −2 0 2 0 2 4 6 8 Loss SVM Loss Logistic Regression Loss yi(β0 + β1xi1 + . . . + βpxip) FIGURE 9.12. The SVM and logistic regression loss functions are compared, as a function of yi(β0 + β1xi1 + · · · + βpxip). When yi(β0 + β1xi1 + · · · + βpxip) is greater than 1, then the SVM loss is zero, since this corresponds to an observation that is on the correct side of the margin. Overall, the two loss functions have quite similar behavior. some of the non-linear approaches seen in Chapter 7. However, for histor- ical reasons, the use of non-linear kernels is much more widespread in the context of SVMs than in the context of logistic regression or other methods. Though we have not addressed it here, there is in fact an extension of the SVM for regression (i.e. for a quantitative rather than a qualita- tive response), called support vector regression. In Chapter 3, we saw that support vector regression least squares regression seeks coefficients β0, β1, . . . , βp such that the sum of squared residuals is as small as possible. (Recall from Chapter 3 that residuals are defined as yi − β0 − β1xi1 − · · · − βpxip.) Support vector regression instead seeks coefficients that minimize a different type of loss, where only residuals larger in absolute value than some positive constant contribute to the loss function. This is an extension of the margin used in support vector classifiers to the regression setting. 9.6 Lab: Support Vector Machines We use the e1071 library in R to demonstrate the support vector classifier and the SVM. Another option is the LiblineaR library, which is useful for very large linear problems.
- 396. 9.6 Lab: Support Vector Machines 389 9.6.1 Support Vector Classifier The e1071 library contains implementations for a number of statistical learning methods. In particular, the svm() function can be used to fit a svm() support vector classifier when the argument kernel = "linear" is used. This function uses a slightly different formulation from (9.14) and (9.25) for the support vector classifier. A cost argument allows us to specify the cost of a violation to the margin. When the cost argument is small, then the margins will be wide and many support vectors will be on the margin or will violate the margin. When the cost argument is large, then the mar- gins will be narrow and there will be few support vectors on the margin or violating the margin. We now use the svm() function to fit the support vector classifier for a given value of the cost parameter. Here we demonstrate the use of this function on a two-dimensional example so that we can plot the resulting decision boundary. We begin by generating the observations, which belong to two classes, and checking whether the classes are linearly separable. > set.seed (1) > x <- matrix(rnorm (20 * 2), ncol = 2) > y <- c(rep(-1, 10) , rep(1, 10)) > x[y == 1, ] <- x[y == 1, ] + 1 > plot(x, col = (3 - y)) They are not. Next, we fit the support vector classifier. Note that in order for the svm() function to perform classification (as opposed to SVM-based regression), we must encode the response as a factor variable. We now create a data frame with the response coded as a factor. > dat <- data.frame(x = x, y = as.factor(y)) > library(e1071) > svmfit <- svm(y ∼ ., data = dat , kernel = "linear", cost = 10, scale = FALSE) The argument scale = FALSE tells the svm() function not to scale each feature to have mean zero or standard deviation one; depending on the application, one might prefer to use scale = TRUE. We can now plot the support vector classifier obtained: > plot(svmfit , dat) Note that the two arguments to the SVM plot() function are the output of the call to svm(), as well as the data used in the call to svm(). The region of feature space that will be assigned to the −1 class is shown in light yellow, and the region that will be assigned to the +1 class is shown in red. The decision boundary between the two classes is linear (because we used the argument kernel = "linear"), though due to the way in which the plotting function is implemented in this library the decision boundary looks somewhat jagged in the plot. (Note that here the second feature is plotted on the x-axis and the first feature is plotted on the y-axis, in contrast to
- 397. 390 9. Support Vector Machines the behavior of the usual plot() function in R.) The support vectors are plotted as crosses and the remaining observations are plotted as circles; we see here that there are seven support vectors. We can determine their identities as follows: > svmfit$index [1] 1 2 5 7 14 16 17 We can obtain some basic information about the support vector classifier fit using the summary() command: > summary(svmfit) Call: svm(formula = y ∼ ., data = dat , kernel = "linear", cost = 10, scale = FALSE) Parameters : SVM -Type: C- classification SVM -Kernel: linear cost: 10 Number of Support Vectors: 7 ( 4 3 ) Number of Classes: 2 Levels: -1 1 This tells us, for instance, that a linear kernel was used with cost = 10, and that there were seven support vectors, four in one class and three in the other. What if we instead used a smaller value of the cost parameter? > svmfit <- svm(y ∼ ., data = dat , kernel = "linear", cost = 0.1, scale = FALSE) > plot(svmfit , dat) > svmfit$index [1] 1 2 3 4 5 7 9 10 12 13 14 15 16 17 18 20 Now that a smaller value of the cost parameter is being used, we obtain a larger number of support vectors, because the margin is now wider. Unfor- tunately, the svm() function does not explicitly output the coefficients of the linear decision boundary obtained when the support vector classifier is fit, nor does it output the width of the margin. The e1071 library includes a built-in function, tune(), to perform cross- tune() validation. By default, tune() performs ten-fold cross-validation on a set of models of interest. In order to use this function, we pass in relevant information about the set of models that are under consideration. The following command indicates that we want to compare SVMs with a linear kernel, using a range of values of the cost parameter. > set.seed (1) > tune.out <- tune(svm , y ∼ ., data = dat , kernel = "linear", ranges = list(cost = c(0.001 , 0.01 , 0.1, 1, 5, 10, 100)))
- 398. 9.6 Lab: Support Vector Machines 391 We can easily access the cross-validation errors for each of these models using the summary() command: > summary(tune.out) Parameter tuning of ‘svm ’: - sampling method: 10-fold cross validation - best parameters: cost 0.1 - best performance : 0.05 - Detailed performance results: cost error dispersion 1 1e -03 0.55 0.438 2 1e -02 0.55 0.438 3 1e -01 0.05 0.158 4 1e+00 0.15 0.242 5 5e+00 0.15 0.242 6 1e+01 0.15 0.242 7 1e+02 0.15 0.242 We see that cost = 0.1 results in the lowest cross-validation error rate. The tune() function stores the best model obtained, which can be accessed as follows: > bestmod <- tune.out$best.model > summary(bestmod) The predict() function can be used to predict the class label on a set of test observations, at any given value of the cost parameter. We begin by generating a test data set. > xtest <- matrix(rnorm (20 * 2), ncol = 2) > ytest <- sample(c(-1, 1), 20, rep = TRUE) > xtest[ytest == 1, ] <- xtest[ytest == 1, ] + 1 > testdat <- data.frame(x = xtest , y = as.factor(ytest)) Now we predict the class labels of these test observations. Here we use the best model obtained through cross-validation in order to make predictions. > ypred <- predict(bestmod , testdat) > table(predict = ypred , truth = testdat$y) truth predict -1 1 -1 9 1 1 2 8 Thus, with this value of cost, 17 of the test observations are correctly classified. What if we had instead used cost = 0.01? > svmfit <- svm(y ∼ ., data = dat , kernel = "linear", cost = .01, scale = FALSE) > ypred <- predict(svmfit , testdat) > table(predict = ypred , truth = testdat$y) truth predict -1 1
- 399. 392 9. Support Vector Machines -1 11 6 1 0 3 In this case three additional observations are misclassified. Now consider a situation in which the two classes are linearly separable. Then we can find a separating hyperplane using the svm() function. We first further separate the two classes in our simulated data so that they are linearly separable: > x[y == 1, ] <- x[y == 1, ] + 0.5 > plot(x, col = (y + 5) / 2, pch = 19) Now the observations are just barely linearly separable. We fit the support vector classifier and plot the resulting hyperplane, using a very large value of cost so that no observations are misclassified. > dat <- data.frame(x = x, y = as.factor(y)) > svmfit <- svm(y ∼ ., data = dat , kernel = "linear", cost = 1e5) > summary(svmfit) Call: svm(formula = y ∼ ., data = dat , kernel = "linear", cost = 1e+05) Parameters : SVM -Type: C- classification SVM -Kernel: linear cost: 1e+05 Number of Support Vectors: 3 ( 1 2 ) Number of Classes: 2 Levels: -1 1 > plot(svmfit , dat) No training errors were made and only three support vectors were used. However, we can see from the figure that the margin is very narrow (because the observations that are not support vectors, indicated as circles, are very close to the decision boundary). It seems likely that this model will perform poorly on test data. We now try a smaller value of cost: > svmfit <- svm(y ∼ ., data = dat , kernel = "linear", cost = 1) > summary(svmfit) > plot(svmfit , dat) Using cost = 1, we misclassify a training observation, but we also obtain a much wider margin and make use of seven support vectors. It seems likely that this model will perform better on test data than the model with cost = 1e5. 9.6.2 Support Vector Machine In order to fit an SVM using a non-linear kernel, we once again use the svm() function. However, now we use a different value of the parameter kernel.
- 400. 9.6 Lab: Support Vector Machines 393 To fit an SVM with a polynomial kernel we use kernel = "polynomial", and to fit an SVM with a radial kernel we use kernel = "radial". In the former case we also use the degree argument to specify a degree for the polynomial kernel (this is d in (9.22)), and in the latter case we use gamma to specify a value of γ for the radial basis kernel (9.24). We first generate some data with a non-linear class boundary, as follows: > set.seed (1) > x <- matrix(rnorm (200 * 2), ncol = 2) > x[1:100 , ] <- x[1:100 , ] + 2 > x[101:150 , ] <- x[101:150 , ] - 2 > y <- c(rep(1, 150) , rep(2, 50)) > dat <- data.frame(x = x, y = as.factor(y)) Plotting the data makes it clear that the class boundary is indeed non- linear: > plot(x, col = y) The data is randomly split into training and testing groups. We then fit the training data using the svm() function with a radial kernel and γ = 1: > train <- sample (200 , 100) > svmfit <- svm(y ∼ ., data = dat[train , ], kernel = "radial", gamma = 1, cost = 1) > plot(svmfit , dat[train , ]) The plot shows that the resulting SVM has a decidedly non-linear boundary. The summary() function can be used to obtain some information about the SVM fit: > summary(svmfit) Call: svm(formula = y ∼ ., data = dat[train , ], kernel = "radial", gamma = 1, cost = 1) Parameters : SVM -Type: C-classification SVM -Kernel: radial cost: 1 Number of Support Vectors: 31 ( 16 15 ) Number of Classes: 2 Levels: 1 2 We can see from the figure that there are a fair number of training errors in this SVM fit. If we increase the value of cost, we can reduce the number of training errors. However, this comes at the price of a more irregular decision boundary that seems to be at risk of overfitting the data. > svmfit <- svm(y ∼ ., data = dat[train , ], kernel = "radial", gamma = 1, cost = 1e5) > plot(svmfit , dat[train , ])
- 401. 394 9. Support Vector Machines We can perform cross-validation using tune() to select the best choice of γ and cost for an SVM with a radial kernel: > set.seed (1) > tune.out <- tune(svm , y ∼ ., data = dat[train , ], kernel = "radial", ranges = list( cost = c(0.1 , 1, 10, 100, 1000) , gamma = c(0.5 , 1, 2, 3, 4) ) ) > summary(tune.out) Parameter tuning of ‘svm ’: - sampling method: 10-fold cross validation - best parameters: cost gamma 1 0.5 - best performance : 0.07 - Detailed performance results: cost gamma error dispersion 1 1e-01 0.5 0.26 0.158 2 1e+00 0.5 0.07 0.082 3 1e+01 0.5 0.07 0.082 4 1e+02 0.5 0.14 0.151 5 1e+03 0.5 0.11 0.074 6 1e-01 1.0 0.22 0.162 7 1e+00 1.0 0.07 0.082 . . . Therefore, the best choice of parameters involves cost = 1 and gamma = 0.5. We can view the test set predictions for this model by applying the predict() function to the data. Notice that to do this we subset the dataframe dat using -train as an index set. > table( true = dat[-train , "y"], pred = predict( tune.out$best.model , newdata = dat[-train , ] ) ) 12 % of test observations are misclassified by this SVM. 9.6.3 ROC Curves The ROCR package can be used to produce ROC curves such as those in Figures 9.10 and 9.11. We first write a short function to plot an ROC curve given a vector containing a numerical score for each observation, pred, and a vector containing the class label for each observation, truth. > library(ROCR) > rocplot <- function (pred , truth , ...) { + predob <- prediction (pred , truth)
- 402. 9.6 Lab: Support Vector Machines 395 + perf <- performance (predob , "tpr", "fpr") + plot(perf , ...) + } SVMs and support vector classifiers output class labels for each observa- tion. However, it is also possible to obtain fitted values for each observation, which are the numerical scores used to obtain the class labels. For instance, in the case of a support vector classifier, the fitted value for an observation X = (X1, X2, . . . , Xp)T takes the form β̂0 + β̂1X1 + β̂2X2 + · · · + β̂pXp. For an SVM with a non-linear kernel, the equation that yields the fitted value is given in (9.23). In essence, the sign of the fitted value determines on which side of the decision boundary the observation lies. Therefore, the relationship between the fitted value and the class prediction for a given observation is simple: if the fitted value exceeds zero then the observation is assigned to one class, and if it is less than zero then it is assigned to the other. In order to obtain the fitted values for a given SVM model fit, we use decision.values = TRUE when fitting svm(). Then the predict() function will output the fitted values. > svmfit.opt <- svm(y ∼ ., data = dat[train , ], kernel = "radial", gamma = 2, cost = 1, decision.values = T) > fitted <- attributes ( predict(svmfit.opt , dat[train , ], decision.values = TRUE) )$decision.values Now we can produce the ROC plot. Note we use the negative of the fitted values so that negative values correspond to class 1 and positive values to class 2. > par(mfrow = c(1, 2)) > rocplot(-fitted , dat[train , "y"], main = "Training Data") SVM appears to be producing accurate predictions. By increasing γ we can produce a more flexible fit and generate further improvements in accuracy. > svmfit.flex <- svm(y ∼ ., data = dat[train , ], kernel = "radial", gamma = 50, cost = 1, decision.values = T) > fitted <- attributes ( predict(svmfit.flex , dat[train , ], decision.values = T) )$decision.values > rocplot(-fitted , dat[train , "y"], add = T, col = "red") However, these ROC curves are all on the training data. We are really more interested in the level of prediction accuracy on the test data. When we compute the ROC curves on the test data, the model with γ = 2 appears to provide the most accurate results. > fitted <- attributes ( predict(svmfit.opt , dat[-train , ], decision.values = T) )$decision.values
- 403. 396 9. Support Vector Machines > rocplot(-fitted , dat[-train , "y"], main = "Test Data") > fitted <- attributes ( predict(svmfit.flex , dat[-train , ], decision.values = T) )$decision.values > rocplot(-fitted , dat[-train , "y"], add = T, col = "red") 9.6.4 SVM with Multiple Classes If the response is a factor containing more than two levels, then the svm() function will perform multi-class classification using the one-versus-one ap- proach. We explore that setting here by generating a third class of obser- vations. > set.seed (1) > x <- rbind(x, matrix(rnorm (50 * 2), ncol = 2)) > y <- c(y, rep(0, 50)) > x[y == 0, 2] <- x[y == 0, 2] + 2 > dat <- data.frame(x = x, y = as.factor(y)) > par(mfrow = c(1, 1)) > plot(x, col = (y + 1)) We now fit an SVM to the data: > svmfit <- svm(y ∼ ., data = dat , kernel = "radial", cost = 10, gamma = 1) > plot(svmfit , dat) The e1071 library can also be used to perform support vector regression, if the response vector that is passed in to svm() is numerical rather than a factor. 9.6.5 Application to Gene Expression Data We now examine the Khan data set, which consists of a number of tissue samples corresponding to four distinct types of small round blue cell tu- mors. For each tissue sample, gene expression measurements are available. The data set consists of training data, xtrain and ytrain, and testing data, xtest and ytest. We examine the dimension of the data: > library(ISLR2) > names(Khan) [1] "xtrain" "xtest" "ytrain" "ytest" > dim(Khan$xtrain) [1] 63 2308 > dim(Khan$xtest) [1] 20 2308 > length(Khan$ytrain) [1] 63 > length(Khan$ytest) [1] 20
- 404. 9.6 Lab: Support Vector Machines 397 This data set consists of expression measurements for 2,308 genes. The training and test sets consist of 63 and 20 observations respectively. > table(Khan$ytrain) 1 2 3 4 8 23 12 20 > table(Khan$ytest) 1 2 3 4 3 6 6 5 We will use a support vector approach to predict cancer subtype using gene expression measurements. In this data set, there are a very large number of features relative to the number of observations. This suggests that we should use a linear kernel, because the additional flexibility that will result from using a polynomial or radial kernel is unnecessary. > dat <- data.frame( x = Khan$xtrain , y = as.factor(Khan$ytrain) ) > out <- svm(y ∼ ., data = dat , kernel = "linear", cost = 10) > summary(out) Call: svm(formula = y ∼ ., data = dat , kernel = "linear", cost = 10) Parameters : SVM -Type: C- classification SVM -Kernel: linear cost: 10 Number of Support Vectors: 58 ( 20 20 11 7 ) Number of Classes: 4 Levels: 1 2 3 4 > table(out$fitted , dat$y) 1 2 3 4 1 8 0 0 0 2 0 23 0 0 3 0 0 12 0 4 0 0 0 20 We see that there are no training errors. In fact, this is not surprising, because the large number of variables relative to the number of observations implies that it is easy to find hyperplanes that fully separate the classes. We are most interested not in the support vector classifier’s performance on the training observations, but rather its performance on the test observations. > dat.te <- data.frame( x = Khan$xtest , y = as.factor(Khan$ytest)) > pred.te <- predict(out , newdata = dat.te) > table(pred.te , dat.te$y) pred.te 1 2 3 4 1 3 0 0 0
- 405. 398 9. Support Vector Machines 2 0 6 2 0 3 0 0 4 0 4 0 0 0 5 We see that using cost = 10 yields two test set errors on this data. 9.7 Exercises Conceptual 1. This problem involves hyperplanes in two dimensions. (a) Sketch the hyperplane 1 + 3X1 − X2 = 0. Indicate the set of points for which 1 + 3X1 − X2 > 0, as well as the set of points for which 1 + 3X1 − X2 < 0. (b) On the same plot, sketch the hyperplane −2 + X1 + 2X2 = 0. Indicate the set of points for which −2 + X1 + 2X2 > 0, as well as the set of points for which −2 + X1 + 2X2 < 0. 2. We have seen that in p = 2 dimensions, a linear decision boundary takes the form β0 +β1X1 +β2X2 = 0. We now investigate a non-linear decision boundary. (a) Sketch the curve (1 + X1)2 + (2 − X2)2 = 4. (b) On your sketch, indicate the set of points for which (1 + X1)2 + (2 − X2)2 > 4, as well as the set of points for which (1 + X1)2 + (2 − X2)2 ≤ 4. (c) Suppose that a classifier assigns an observation to the blue class if (1 + X1)2 + (2 − X2)2 > 4, and to the red class otherwise. To what class is the observation (0, 0) classified? (−1, 1)? (2, 2)? (3, 8)? (d) Argue that while the decision boundary in (c) is not linear in terms of X1 and X2, it is linear in terms of X1, X2 1 , X2, and X2 2 . 3. Here we explore the maximal margin classifier on a toy data set.
- 406. 9.7 Exercises 399 (a) We are given n = 7 observations in p = 2 dimensions. For each observation, there is an associated class label. Obs. X1 X2 Y 1 3 4 Red 2 2 2 Red 3 4 4 Red 4 1 4 Red 5 2 1 Blue 6 4 3 Blue 7 4 1 Blue Sketch the observations. (b) Sketch the optimal separating hyperplane, and provide the equa- tion for this hyperplane (of the form (9.1)). (c) Describe the classification rule for the maximal margin classifier. It should be something along the lines of “Classify to Red if β0 + β1X1 + β2X2 > 0, and classify to Blue otherwise.” Provide the values for β0, β1, and β2. (d) On your sketch, indicate the margin for the maximal margin hyperplane. (e) Indicate the support vectors for the maximal margin classifier. (f) Argue that a slight movement of the seventh observation would not affect the maximal margin hyperplane. (g) Sketch a hyperplane that is not the optimal separating hyper- plane, and provide the equation for this hyperplane. (h) Draw an additional observation on the plot so that the two classes are no longer separable by a hyperplane. Applied 4. Generate a simulated two-class data set with 100 observations and two features in which there is a visible but non-linear separation be- tween the two classes. Show that in this setting, a support vector machine with a polynomial kernel (with degree greater than 1) or a radial kernel will outperform a support vector classifier on the train- ing data. Which technique performs best on the test data? Make plots and report training and test error rates in order to back up your assertions. 5. We have seen that we can fit an SVM with a non-linear kernel in order to perform classification using a non-linear decision boundary. We will now see that we can also obtain a non-linear decision boundary by performing logistic regression using non-linear transformations of the features.
- 407. 400 9. Support Vector Machines (a) Generate a data set with n = 500 and p = 2, such that the obser- vations belong to two classes with a quadratic decision boundary between them. For instance, you can do this as follows: > x1 <- runif (500) - 0.5 > x2 <- runif (500) - 0.5 > y <- 1 * (x1^2 - x2^2 > 0) (b) Plot the observations, colored according to their class labels. Your plot should display X1 on the x-axis, and X2 on the y- axis. (c) Fit a logistic regression model to the data, using X1 and X2 as predictors. (d) Apply this model to the training data in order to obtain a pre- dicted class label for each training observation. Plot the ob- servations, colored according to the predicted class labels. The decision boundary should be linear. (e) Now fit a logistic regression model to the data using non-linear functions of X1 and X2 as predictors (e.g. X2 1 , X1 ×X2, log(X2), and so forth). (f) Apply this model to the training data in order to obtain a pre- dicted class label for each training observation. Plot the ob- servations, colored according to the predicted class labels. The decision boundary should be obviously non-linear. If it is not, then repeat (a)-(e) until you come up with an example in which the predicted class labels are obviously non-linear. (g) Fit a support vector classifier to the data with X1 and X2 as predictors. Obtain a class prediction for each training observa- tion. Plot the observations, colored according to the predicted class labels. (h) Fit a SVM using a non-linear kernel to the data. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels. (i) Comment on your results. 6. At the end of Section 9.6.1, it is claimed that in the case of data that is just barely linearly separable, a support vector classifier with a small value of cost that misclassifies a couple of training observations may perform better on test data than one with a huge value of cost that does not misclassify any training observations. You will now investigate this claim. (a) Generate two-class data with p = 2 in such a way that the classes are just barely linearly separable.
- 408. 9.7 Exercises 401 (b) Compute the cross-validation error rates for support vector classifiers with a range of cost values. How many training er- rors are misclassified for each value of cost considered, and how does this relate to the cross-validation errors obtained? (c) Generate an appropriate test data set, and compute the test errors corresponding to each of the values of cost considered. Which value of cost leads to the fewest test errors, and how does this compare to the values of cost that yield the fewest training errors and the fewest cross-validation errors? (d) Discuss your results. 7. In this problem, you will use support vector approaches in order to predict whether a given car gets high or low gas mileage based on the Auto data set. (a) Create a binary variable that takes on a 1 for cars with gas mileage above the median, and a 0 for cars with gas mileage below the median. (b) Fit a support vector classifier to the data with various values of cost, in order to predict whether a car gets high or low gas mileage. Report the cross-validation errors associated with dif- ferent values of this parameter. Comment on your results. Note you will need to fit the classifier without the gas mileage variable to produce sensible results. (c) Now repeat (b), this time using SVMs with radial and polyno- mial basis kernels, with different values of gamma and degree and cost. Comment on your results. (d) Make some plots to back up your assertions in (b) and (c). Hint: In the lab, we used the plot() function for svm objects only in cases with p = 2. When p > 2, you can use the plot() function to create plots displaying pairs of variables at a time. Essentially, instead of typing > plot(svmfit , dat) where svmfit contains your fitted model and dat is a data frame containing your data, you can type > plot(svmfit , dat , x1 ∼ x4) in order to plot just the first and fourth variables. However, you must replace x1 and x4 with the correct variable names. To find out more, type ?plot.svm. 8. This problem involves the OJ data set which is part of the ISLR2 package.
- 409. 402 9. Support Vector Machines (a) Create a training set containing a random sample of 800 observations, and a test set containing the remaining observations. (b) Fit a support vector classifier to the training data using cost = 0.01, with Purchase as the response and the other vari- ables as predictors. Use the summary() function to produce sum- mary statistics, and describe the results obtained. (c) What are the training and test error rates? (d) Use the tune() function to select an optimal cost. Consider val- ues in the range 0.01 to 10. (e) Compute the training and test error rates using this new value for cost. (f) Repeat parts (b) through (e) using a support vector machine with a radial kernel. Use the default value for gamma. (g) Repeat parts (b) through (e) using a support vector machine with a polynomial kernel. Set degree = 2. (h) Overall, which approach seems to give the best results on this data?
- 410. 10 Deep Learning This chapter covers the important topic of deep learning. At the time of deep learning writing (2020), deep learning is a very active area of research in the machine learning and artificial intelligence communities. The cornerstone of deep learning is the neural network. neural network Neural networks rose to fame in the late 1980s. There was a lot of excite- ment and a certain amount of hype associated with this approach, and they were the impetus for the popular Neural Information Processing Systems meetings (NeurIPS, formerly NIPS) held every year, typically in exotic places like ski resorts. This was followed by a synthesis stage, where the properties of neural networks were analyzed by machine learners, math- ematicians and statisticians; algorithms were improved, and the method- ology stabilized. Then along came SVMs, boosting, and random forests, and neural networks fell somewhat from favor. Part of the reason was that neural networks required a lot of tinkering, while the new methods were more automatic. Also, on many problems the new methods outperformed poorly-trained neural networks. This was the status quo for the first decade in the new millennium. All the while, though, a core group of neural-network enthusiasts were pushing their technology harder on ever-larger computing architectures and data sets. Neural networks resurfaced after 2010 with the new name deep learning, with new architectures, additional bells and whistles, and a string of success stories on some niche problems such as image and video classifi- cation, speech and text modeling. Many in the field believe that the major reason for these successes is the availability of ever-larger training datasets, made possible by the wide-scale use of digitization in science and industry. © Springer Science+Business Media, LLC, part of Springer Nature 2021 G. James et al., An Introduction to Statistical Learning, Springer Texts in Statistics, https://doi.org/10.1007/978-1-0716-1418-1_10 403
- 411. 404 10. Deep Learning In this chapter we discuss the basics of neural networks and deep learning, and then go into some of the specializations for specific problems, such as convolutional neural networks (CNNs) for image classification, and recur- rent neural networks (RNNs) for time series and other sequences. We will also demonstrate these models using the R package keras, which interfaces with the tensorflow deep-learning software developed at Google.1 The material in this chapter is slightly more challenging than elsewhere in this book. 10.1 Single Layer Neural Networks A neural network takes an input vector of p variables X = (X1, X2, . . . , Xp) and builds a nonlinear function f(X) to predict the response Y . We have built nonlinear prediction models in earlier chapters, using trees, boosting and generalized additive models. What distinguishes neural networks from these methods is the particular structure of the model. Figure 10.1 shows a simple feed-forward neural network for modeling a quantitative response feed-forward neural network using p = 4 predictors. In the terminology of neural networks, the four fea- tures X1, . . . , X4 make up the units in the input layer. The arrows indicate input layer that each of the inputs from the input layer feeds into each of the K hidden units (we get to pick K; here we chose 5). The neural network model has hidden units the form f(X) = β0 + )K k=1 βkhk(X) = β0 + )K k=1 βkg(wk0 + )p j=1 wkjXj). (10.1) It is built up here in two steps. First the K activations Ak, k = 1, . . . , K, in activations the hidden layer are computed as functions of the input features X1, . . . , Xp, Ak = hk(X) = g(wk0 + )p j=1 wkjXj), (10.2) where g(z) is a nonlinear activation function that is specified in advance. activation function We can think of each Ak as a different transformation hk(X) of the original features, much like the basis functions of Chapter 7. These K activations from the hidden layer then feed into the output layer, resulting in f(X) = β0 + K 0 k=1 βkAk, (10.3) a linear regression model in the K = 5 activations. All the parameters β0, . . . , βK and w10, . . . , wKp need to be estimated from data. In the early 1For more information about keras, see Chollet et al. (2015) “Keras”, available at https://keras.io. For more information about tensorflow, see Abadi et al. (2015) “TensorFlow: Large-scale machine learning on heterogeneous distributed systems”, avail- able at https://www.tensorflow.org/.
- 412. 10.1 Single Layer Neural Networks 405 X1 X2 X3 X4 A1 A2 A3 A4 A5 f(X) Y Hidden Layer Input Layer Output Layer FIGURE 10.1. Neural network with a single hidden layer. The hidden layer computes activations Ak = hk(X) that are nonlinear transformations of linear combinations of the inputs X1, X2, . . . , Xp. Hence these Ak are not directly ob- served. The functions hk(·) are not fixed in advance, but are learned during the training of the network. The output layer is a linear model that uses these acti- vations Ak as inputs, resulting in a function f(X). instances of neural networks, the sigmoid activation function was favored, sigmoid g(z) = ez 1 + ez = 1 1 + e−z , (10.4) which is the same function used in logistic regression to convert a linear function into probabilities between zero and one (see Figure 10.2). The preferred choice in modern neural networks is the ReLU (rectified linear ReLU unit) activation function, which takes the form rectified linear unit g(z) = (z)+ = K 0 if z < 0 z otherwise. (10.5) A ReLU activation can be computed and stored more efficiently than a sigmoid activation. Although it thresholds at zero, because we apply it to a linear function (10.2) the constant term wk0 will shift this inflection point. So in words, the model depicted in Figure 10.1 derives five new features by computing five different linear combinations of X, and then squashes each through an activation function g(·) to transform it. The final model is linear in these derived variables. The name neural network originally derived from thinking of these hidden units as analogous to neurons in the brain — values of the activations Ak = hk(X) close to one are firing, while those close to zero are silent (using the sigmoid activation function).
- 413. 406 10. Deep Learning −4 −2 0 2 4 0.0 0.2 0.4 0.6 0.8 1.0 z g(z) sigmoid ReLU FIGURE 10.2. Activation functions. The piecewise-linear ReLU function is pop- ular for its efficiency and computability. We have scaled it down by a factor of five for ease of comparison. The nonlinearity in the activation function g(·) is essential, since without it the model f(X) in (10.1) would collapse into a simple linear model in X1, . . . , Xp. Moreover, having a nonlinear activation function allows the model to capture complex nonlinearities and interaction effects. Consider a very simple example with p = 2 input variables X = (X1, X2), and K = 2 hidden units h1(X) and h2(X) with g(z) = z2 . We specify the other parameters as β0 = 0, β1 = 1 4 , β2 = −1 4 , w10 = 0, w11 = 1, w12 = 1, w20 = 0, w21 = 1, w22 = −1. (10.6) From (10.2), this means that h1(X) = (0 + X1 + X2)2 , h2(X) = (0 + X1 − X2)2 . (10.7) Then plugging (10.7) into (10.1), we get f(X) = 0 + 1 4 · (0 + X1 + X2)2 − 1 4 · (0 + X1 − X2)2 = 1 4 N (X1 + X2)2 − (X1 − X2)2 O = X1X2. (10.8) So the sum of two nonlinear transformations of linear functions can give us an interaction! In practice we would not use a quadratic function for g(z), since we would always get a second-degree polynomial in the original coordinates X1, . . . , Xp. The sigmoid or ReLU activations do not have such a limitation. Fitting a neural network requires estimating the unknown parameters in (10.1). For a quantitative response, typically squared-error loss is used, so that the parameters are chosen to minimize n 0 i=1 (yi − f(xi)) 2 . (10.9)
- 414. 10.2 Multilayer Neural Networks 407 FIGURE 10.3. Examples of handwritten digits from the MNIST corpus. Each grayscale image has 28 × 28 pixels, each of which is an eight-bit number (0–255) which represents how dark that pixel is. The first 3, 5, and 8 are enlarged to show their 784 individual pixel values. Details about how to perform this minimization are provided in Section 10.7. 10.2 Multilayer Neural Networks Modern neural networks typically have more than one hidden layer, and often many units per layer. In theory a single hidden layer with a large number of units has the ability to approximate most functions. However, the learning task of discovering a good solution is made much easier with multiple layers each of modest size. We will illustrate a large dense network on the famous and publicly available MNIST handwritten digit dataset.2 Figure 10.3 shows examples of these digits. The idea is to build a model to classify the images into their correct digit class 0–9. Every image has p = 28 × 28 = 784 pixels, each of which is an eight-bit grayscale value between 0 and 255 representing the relative amount of the written digit in that tiny square.3 These pixels are stored in the input vector X (in, say, column order). The output is the class label, represented by a vector Y = (Y0, Y1, . . . , Y9) of 10 dummy variables, with a one in the position corresponding to the label, and zeros elsewhere. In the machine learning community, this is known as one-hot encoding. There are 60,000 training images, and 10,000 test images. one-hot encoding On a historical note, digit recognition problems were the catalyst that accelerated the development of neural network technology in the late 1980s at AT&T Bell Laboratories and elsewhere. Pattern recognition tasks of this 2See LeCun, Cortes, and Burges (2010) “The MNIST database of handwritten digits”, available at http://yann.lecun.com/exdb/mnist. 3In the analog-to-digital conversion process, only part of the written numeral may fall in the square representing a particular pixel.
- 415. 408 10. Deep Learning kind are relatively simple for humans. Our visual system occupies a large fraction of our brains, and good recognition is an evolutionary force for survival. These tasks are not so simple for machines, and it has taken more than 30 years to refine the neural-network architectures to match human performance. Figure 10.4 shows a multilayer network architecture that works well for solving the digit-classification task. It differs from Figure 10.1 in several ways: • It has two hidden layers L1 (256 units) and L2 (128 units) rather than one. Later we will see a network with seven hidden layers. • It has ten output variables, rather than one. In this case the ten vari- ables really represent a single qualitative variable and so are quite dependent. (We have indexed them by the digit class 0–9 rather than 1–10, for clarity.) More generally, in multi-task learning one can pre- multi-task learning dict different responses simultaneously with a single network; they all have a say in the formation of the hidden layers. • The loss function used for training the network is tailored for the multiclass classification task. The first hidden layer is as in (10.2), with A (1) k = h (1) k (X) = g(w (1) k0 + )p j=1 w (1) kj Xj) (10.10) for k = 1, . . . , K1. The second hidden layer treats the activations A (1) k of the first hidden layer as inputs and computes new activations A (2) ℓ = h (2) ℓ (X) = g(w (2) ℓ0 + )K1 k=1 w (2) ℓk A (1) k ) (10.11) for ℓ = 1, . . . , K2. Notice that each of the activations in the second layer A (2) ℓ = h (2) ℓ (X) is a function of the input vector X. This is the case because while they are explicitly a function of the activations A (1) k from layer L1, these in turn are functions of X. This would also be the case with more hidden layers. Thus, through a chain of transformations, the network is able to build up fairly complex transformations of X that ultimately feed into the output layer as features. We have introduced additional superscript notation such as h (2) ℓ (X) and w (2) ℓj in (10.10) and (10.11) to indicate to which layer the activations and weights (coefficients) belong, in this case layer 2. The notation W1 in Fig- weights ure 10.4 represents the entire matrix of weights that feed from the input layer to the first hidden layer L1. This matrix will have 785×256 = 200,960
- 416. 10.2 Multilayer Neural Networks 409 X1 X2 X3 X4 X5 X6 . . . Xp A (1) 1 A (1) 2 A (1) 3 A (1) 4 . . . A (1) K1 A (2) 1 A (2) 2 A (2) 3 . . . A (2) K2 f0(X) Y0 f1(X) Y1 . . . . . . f9(X) Y9 Hidden layer L2 Hidden layer L1 Input layer Output layer W1 W2 B FIGURE 10.4. Neural network diagram with two hidden layers and multiple outputs, suitable for the MNIST handwritten-digit problem. The input layer has p = 784 units, the two hidden layers K1 = 256 and K2 = 128 units respectively, and the output layer 10 units. Along with intercepts (referred to as biases in the deep-learning community) this network has 235,146 parameters (referred to as weights). elements; there are 785 rather than 784 because we must account for the intercept or bias term.4 bias Each element A (1) k feeds to the second hidden layer L2 via the matrix of weights W2 of dimension 257 × 128 = 32,896. We now get to the output layer, where we now have ten responses rather than one. The first step is to compute ten different linear models similar to our single model (10.1), Zm = βm0 + )K2 ℓ=1 βmℓh (2) ℓ (X) = βm0 + )K2 ℓ=1 βmℓA (2) ℓ , (10.12) for m = 0, 1, . . . , 9. The matrix B stores all 129 × 10 = 1,290 of these weights. 4The use of “weights” for coefficients and “bias” for the intercepts wk0 in (10.2) is popular in the machine learning community; this use of bias is not to be confused with the “bias-variance” usage elsewhere in this book.
- 417. 410 10. Deep Learning Method Test Error Neural Network + Ridge Regularization 2.3% Neural Network + Dropout Regularization 1.8% Multinomial Logistic Regression 7.2% Linear Discriminant Analysis 12.7% TABLE 10.1. Test error rate on the MNIST data, for neural networks with two forms of regularization, as well as multinomial logistic regression and linear dis- criminant analysis. In this example, the extra complexity of the neural network leads to a marked improvement in test error. If these were all separate quantitative responses, we would simply set each fm(X) = Zm and be done. However, we would like our estimates to represent class probabilities fm(X) = Pr(Y = m|X), just like in multi- nomial logistic regression in Section 4.3.5. So we use the special softmax softmax activation function (see (4.13) on page 141), fm(X) = Pr(Y = m|X) = eZm )9 ℓ=0 eZℓ , (10.13) for m = 0, 1, . . . , 9. This ensures that the 10 numbers behave like proba- bilities (non-negative and sum to one). Even though the goal is to build a classifier, our model actually estimates a probability for each of the 10 classes. The classifier then assigns the image to the class with the highest probability. To train this network, since the response is qualitative, we look for coef- ficient estimates that minimize the negative multinomial log-likelihood − n 0 i=1 9 0 m=0 yim log(fm(xi)), (10.14) also known as the cross-entropy. This is a generalization of the crite- cross- entropy rion (4.5) for two-class logistic regression. Details on how to minimize this objective are given in Section 10.7. If the response were quantitative, we would instead minimize squared-error loss as in (10.9). Table 10.1 compares the test performance of the neural network with two simple models presented in Chapter 4 that make use of linear decision boundaries: multinomial logistic regression and linear discriminant analysis. The improvement of neural networks over both of these linear methods is dramatic: the network with dropout regularization achieves a test error rate below 2% on the 10,000 test images. (We describe dropout regularization in Section 10.7.3.) In Section 10.9.2 of the lab, we present the code for fitting this model, which runs in just over two minutes on a laptop computer. Adding the number of coefficients in W1, W2 and B, we get 235,146 in all, more than 33 times the number 785 × 9 = 7,065 needed for multino- mial logistic regression. Recall that there are 60,000 images in the training
- 418. 10.3 Convolutional Neural Networks 411 FIGURE 10.5. A sample of images from the CIFAR100 database: a collection of natural images from everyday life, with 100 different classes represented. set. While this might seem like a large training set, there are almost four times as many coefficients in the neural network model as there are ob- servations in the training set! To avoid overfitting, some regularization is needed. In this example, we used two forms of regularization: ridge regu- larization, which is similar to ridge regression from Chapter 6, and dropout dropout regularization. We discuss both forms of regularization in Section 10.7. 10.3 Convolutional Neural Networks Neural networks rebounded around 2010 with big successes in image classi- fication. Around that time, massive databases of labeled images were being accumulated, with ever-increasing numbers of classes. Figure 10.5 shows 75 images drawn from the CIFAR100 database.5 This database consists of 60,000 images labeled according to 20 superclasses (e.g. aquatic mammals), with five classes per superclass (beaver, dolphin, otter, seal, whale). Each image has a resolution of 32 × 32 pixels, with three eight-bit numbers per pixel representing red, green and blue. The numbers for each image are organized in a three-dimensional array called a feature map. The first two feature map axes are spatial (both are 32-dimensional), and the third is the channel channel axis,6 representing the three colors. There is a designated training set of 50,000 images, and a test set of 10,000. A special family of convolutional neural networks (CNNs) has evolved for convolutional neural networks classifying images such as these, and has shown spectacular success on a wide range of problems. CNNs mimic to some degree how humans classify images, by recognizing specific features or patterns anywhere in the image 5See Chapter 3 of Krizhevsky (2009) “Learning multiple layers of fea- tures from tiny images”, available at https://www.cs.toronto.edu/~kriz/ learning-features-2009-TR.pdf. 6The term channel is taken from the signal-processing literature. Each channel is a distinct source of information.
- 419. 412 10. Deep Learning FIGURE 10.6. Schematic showing how a convolutional neural network classifies an image of a tiger. The network takes in the image and identifies local features. It then combines the local features in order to create compound features, which in this example include eyes and ears. These compound features are used to output the label “tiger”. that distinguish each particular object class. In this section we give a brief overview of how they work. Figure 10.6 illustrates the idea behind a convolutional neural network on a cartoon image of a tiger.7 The network first identifies low-level features in the input image, such as small edges, patches of color, and the like. These low-level features are then combined to form higher-level features, such as parts of ears, eyes, and so on. Eventually, the presence or absence of these higher-level features contributes to the probability of any given output class. How does a convolutional neural network build up this hierarchy? It com- bines two specialized types of hidden layers, called convolution layers and pooling layers. Convolution layers search for instances of small patterns in the image, whereas pooling layers downsample these to select a prominent subset. In order to achieve state-of-the-art results, contemporary neural- network architectures make use of many convolution and pooling layers. We describe convolution and pooling layers next. 10.3.1 Convolution Layers A convolution layer is made up of a large number of convolution filters, each convolution layer convolution filter of which is a template that determines whether a particular local feature is present in an image. A convolution filter relies on a very simple operation, called a convolution, which basically amounts to repeatedly multiplying matrix elements and then adding the results. 7Thanks to Elena Tuzhilina for producing the diagram and https://www. cartooning4kids.com/ for permission to use the cartoon tiger.
- 420. 10.3 Convolutional Neural Networks 413 To understand how a convolution filter works, consider a very simple example of a 4 × 3 image: Original Image = ⎡ ⎢ ⎢ ⎣ a b c d e f g h i j k l ⎤ ⎥ ⎥ ⎦ . Now consider a 2 × 2 filter of the form Convolution Filter = 3 α β γ δ 4 . When we convolve the image with the filter, we get the result8 Convolved Image = ⎡ ⎣ aα + bβ + dγ + eδ bα + cβ + eγ + fδ dα + eβ + gγ + hδ eα + fβ + hγ + iδ gα + hβ + jγ + kδ hα + iβ + kγ + lδ ⎤ ⎦ . For instance, the top-left element comes from multiplying each element in the 2 × 2 filter by the corresponding element in the top left 2 × 2 portion of the image, and adding the results. The other elements are obtained in a similar way: the convolution filter is applied to every 2 × 2 submatrix of the original image in order to obtain the convolved image. If a 2 × 2 submatrix of the original image resembles the convolution filter, then it will have a large value in the convolved image; otherwise, it will have a small value. Thus, the convolved image highlights regions of the original image that resemble the convolution filter. We have used 2 × 2 as an example; in general convolution filters are small ℓ1 × ℓ2 arrays, with ℓ1 and ℓ2 small positive integers that are not necessarily equal. Figure 10.7 illustrates the application of two convolution filters to a 192× 179 image of a tiger, shown on the left-hand side.9 Each convolution filter is a 15 × 15 image containing mostly zeros (black), with a narrow strip of ones (white) oriented either vertically or horizontally within the image. When each filter is convolved with the image of the tiger, areas of the tiger that resemble the filter (i.e. that have either horizontal or vertical stripes or edges) are given large values, and areas of the tiger that do not resemble the feature are given small values. The convolved images are displayed on the right-hand side. We see that the horizontal stripe filter picks out horizontal stripes and edges in the original image, whereas the vertical stripe filter picks out vertical stripes and edges in the original image. 8The convolved image is smaller than the original image because its dimension is given by the number of 2 × 2 submatrices in the original image. Note that 2 × 2 is the dimension of the convolution filter. If we want the convolved image to have the same dimension as the original image, then padding can be applied. 9The tiger image used in Figures 10.7–10.9 was obtained from the public domain image resource https://www.needpix.com/.
- 421. 414 10. Deep Learning FIGURE 10.7. Convolution filters find local features in an image, such as edges and small shapes. We begin with the image of the tiger shown on the left, and apply the two small convolution filters in the middle. The convolved images high- light areas in the original image where details similar to the filters are found. Specifically, the top convolved image highlights the tiger’s vertical stripes, whereas the bottom convolved image highlights the tiger’s horizontal stripes. We can think of the original image as the input layer in a convolutional neural network, and the convolved images as the units in the first hidden layer. We have used a large image and two large filters in Figure 10.7 for illus- tration. For the CIFAR100 database there are 32×32 color pixels per image, and we use 3 × 3 convolution filters. In a convolution layer, we use a whole bank of filters to pick out a variety of differently-oriented edges and shapes in the image. Using predefined filters in this way is standard practice in image processing. By contrast, with CNNs the filters are learned for the specific classification task. We can think of the filter weights as the parameters going from an input layer to a hidden layer, with one hidden unit for each pixel in the convolved image. This is in fact the case, though the parameters are highly structured and constrained (see Exercise 4 for more details). They operate on localized patches in the input image (so there are many structural zeros), and the same weights in a given filter are reused for all possible patches in the image (so the weights are constrained).10 We now give some additional details. • Since the input image is in color, it has three channels represented by a three-dimensional feature map (array). Each channel is a two- dimensional (32 × 32) feature map — one for red, one for green, and one for blue. A single convolution filter will also have three channels, one per color, each of dimension 3×3, with potentially different filter weights. The results of the three convolutions are summed to form 10This used to be called weight sharing in the early years of neural networks.
- 422. 10.3 Convolutional Neural Networks 415 a two-dimensional output feature map. Note that at this point the color information has been used, and is not passed on to subsequent layers except through its role in the convolution. • If we use K different convolution filters at this first hidden layer, we get K two-dimensional output feature maps, which together are treated as a single three-dimensional feature map. We view each of the K output feature maps as a separate channel of information, so now we have K channels in contrast to the three color channels of the original input feature map. The three-dimensional feature map is just like the activations in a hidden layer of a simple neural network, except organized and produced in a spatially structured way. • We typically apply the ReLU activation function (10.5) to the con- volved image. This step is sometimes viewed as a separate layer in the convolutional neural network, in which case it is referred to as a detector layer. detector layer 10.3.2 Pooling Layers A pooling layer provides a way to condense a large image into a smaller pooling summary image. While there are a number of possible ways to perform pooling, the max pooling operation summarizes each non-overlapping 2 × 2 block of pixels in an image using the maximum value in the block. This reduces the size of the image by a factor of two in each direction, and it also provides some location invariance: i.e. as long as there is a large value in one of the four pixels in the block, the whole block registers as a large value in the reduced image. Here is a simple example of max pooling: Max pool ⎡ ⎢ ⎢ ⎣ 1 2 5 3 3 0 1 2 2 1 3 4 1 1 2 0 ⎤ ⎥ ⎥ ⎦ → 3 3 5 2 4 4 . 10.3.3 Architecture of a Convolutional Neural Network So far we have defined a single convolution layer — each filter produces a new two-dimensional feature map. The number of convolution filters in a convolution layer is akin to the number of units at a particular hidden layer in a fully-connected neural network of the type we saw in Section 10.2. This number also defines the number of channels in the resulting three- dimensional feature map. We have also described a pooling layer, which reduces the first two dimensions of each three-dimensional feature map. Deep CNNs have many such layers. Figure 10.8 shows a typical architecture for a CNN for the CIFAR100 image classification task.
- 423. 416 10. Deep Learning 32 16 32 8 16 4 32 2 5 0 0 1 0 0 convolve convolve convolve pool pool pool flatten 8 FIGURE 10.8. Architecture of a deep CNN for the CIFAR100 classification task. Convolution layers are interspersed with 2 × 2 max-pool layers, which reduce the size by a factor of 2 in both dimensions. At the input layer, we see the three-dimensional feature map of a color image, where the channel axis represents each color by a 32 × 32 two- dimensional feature map of pixels. Each convolution filter produces a new channel at the first hidden layer, each of which is a 32 × 32 feature map (after some padding at the edges). After this first round of convolutions, we now have a new “image”; a feature map with considerably more channels than the three color input channels (six in the figure, since we used six convolution filters). This is followed by a max-pool layer, which reduces the size of the feature map in each channel by a factor of four: two in each dimension. This convolve-then-pool sequence is now repeated for the next two layers. Some details are as follows: • Each subsequent convolve layer is similar to the first. It takes as input the three-dimensional feature map from the previous layer and treats it like a single multi-channel image. Each convolution filter learned has as many channels as this feature map. • Since the channel feature maps are reduced in size after each pool layer, we usually increase the number of filters in the next convolve layer to compensate. • Sometimes we repeat several convolve layers before a pool layer. This effectively increases the dimension of the filter. These operations are repeated until the pooling has reduced each channel feature map down to just a few pixels in each dimension. At this point the three-dimensional feature maps are flattened — the pixels are treated as separate units — and fed into one or more fully-connected layers before reaching the output layer, which is a softmax activation for the 100 classes (as in (10.13)). There are many tuning parameters to be selected in constructing such a network, apart from the number, nature, and sizes of each layer. Dropout learning can be used at each layer, as well as lasso or ridge regularization (see Section 10.7). The details of constructing a convolutional neural net- work can seem daunting. Fortunately, terrific software is available, with
- 424. 10.3 Convolutional Neural Networks 417 FIGURE 10.9. Data augmentation. The original image (leftmost) is distorted in natural ways to produce different images with the same class label. These dis- tortions do not fool humans, and act as a form of regularization when fitting the CNN. extensive examples and vignettes that provide guidance on sensible choices for the parameters. For the CIFAR100 official test set, the best accuracy as of this writing is just above 75%, but undoubtedly this performance will continue to improve. 10.3.4 Data Augmentation An additional important trick used with image modeling is data augment- data aug- mentation ation. Essentially, each training image is replicated many times, with each replicate randomly distorted in a natural way such that human recognition is unaffected. Figure 10.9 shows some examples. Typical distortions are zoom, horizontal and vertical shift, shear, small rotations, and in this case horizontal flips. At face value this is a way of increasing the training set considerably with somewhat different examples, and thus protects against overfitting. In fact we can see this as a form of regularization: we build a cloud of images around each original image, all with the same label. This kind of fattening of the data is similar in spirit to ridge regularization. We will see in Section 10.7.2 that the stochastic gradient descent al- gorithms for fitting deep learning models repeatedly process randomly- selected batches of, say, 128 training images at a time. This works hand-in- glove with augmentation, because we can distort each image in the batch on the fly, and hence do not have to store all the new images. 10.3.5 Results Using a Pretrained Classifier Here we use an industry-level pretrained classifier to predict the class of some new images. The resnet50 classifier is a convolutional neural network that was trained using the imagenet data set, which consists of millions of images that belong to an ever-growing number of categories.11 Figure 10.10 11For more information about resnet50, see He, Zhang, Ren, and Sun (2015) “Deep residual learning for image recognition”, https://arxiv.org/abs/1512.03385. For de- tails about imagenet, see Russakovsky, Deng, et al. (2015) “ImageNet Large Scale Visual Recognition Challenge”, in International Journal of Computer Vision.
- 425. 418 10. Deep Learning flamingo Cooper’s hawk Cooper’s hawk flamingo 0.83 kite 0.60 fountain 0.35 spoonbill 0.17 great grey owl 0.09 nail 0.12 white stork 0.00 robin 0.06 hook 0.07 Lhasa Apso cat Cape weaver Tibetan terrier 0.56 Old English sheepdog 0.82 jacamar 0.28 Lhasa 0.32 Shih-Tzu 0.04 macaw 0.12 cocker spaniel 0.03 Persian cat 0.04 robin 0.12 FIGURE 10.10. Classification of six photographs using the resnet50 CNN trained on the imagenet corpus. The table below the images displays the true (intended) label at the top of each panel, and the top three choices of the classifier (out of 100). The numbers are the estimated probabilities for each choice. (A kite is a raptor, but not a hawk.) demonstrates the performance of resnet50 on six photographs (private col- lection of one of the authors).12 The CNN does a reasonable job classifying the hawk in the second image. If we zoom out as in the third image, it gets confused and chooses the fountain rather than the hawk. In the final image a “jacamar” is a tropical bird from South and Central America with similar coloring to the South African Cape Weaver. We give more details on this example in Section 10.9.4. Much of the work in fitting a CNN is in learning the convolution filters at the hidden layers; these are the coefficients of a CNN. For models fit to massive corpora such as imagenet with many classes, the output of these 12These resnet results can change with time, since the publicly-trained model gets updated periodically.
- 426. 10.4 Document Classification 419 filters can serve as features for general natural-image classification prob- lems. One can use these pretrained hidden layers for new problems with much smaller training sets (a process referred to as weight freezing), and weight freezing just train the last few layers of the network, which requires much less data. The vignettes and book13 that accompany the keras package give more details on such applications. 10.4 Document Classification In this section we introduce a new type of example that has important applications in industry and science: predicting attributes of documents. Examples of documents include articles in medical journals, Reuters news feeds, emails, tweets, and so on. Our example will be IMDb (Internet Movie Database) ratings — short documents where viewers have written critiques of movies.14 The response in this case is the sentiment of the review, which will be positive or negative. Here is the beginning of a rather amusing negative review: This has to be one of the worst films of the 1990s. When my friends & I were watching this film (being the target audience it was aimed at) we just sat & watched the first half an hour with our jaws touching the floor at how bad it really was. The rest of the time, everyone else in the theater just started talking to each other, leaving or generally crying into their popcorn . . . Each review can be a different length, include slang or non-words, have spelling errors, etc. We need to find a way to featurize such a document. featurize This is modern parlance for defining a set of predictors. The simplest and most common featurization is the bag-of-words model. bag-of-words We score each document for the presence or absence of each of the words in a language dictionary — in this case an English dictionary. If the dictionary contains M words, that means for each document we create a binary feature vector of length M, and score a 1 for every word present, and 0 otherwise. That can be a very wide feature vector, so we limit the dictionary — in this case to the 10,000 most frequently occurring words in the training corpus of 25,000 reviews. Fortunately there are nice tools for doing this automatically. Here is the beginning of a positive review that has been redacted in this way: ⟨START ⟩ this film was just brilliant casting location scenery story direction everyone’s really suited the part they played and 13Deep Learning with R by F. Chollet and J.J. Allaire, 2018, Manning Publications. 14For details, see Maas et al. (2011) “Learning word vectors for sentiment analysis”, in Proceedings of the 49th Annual Meeting of the Association for Computational Lin- guistics: Human Language Technologies, pages 142–150.
- 427. 420 10. Deep Learning ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 4 6 8 10 12 0.6 0.7 0.8 0.9 1.0 Lasso − log(λ) Accuracy ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● train validation test ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5 10 15 20 0.6 0.7 0.8 0.9 1.0 Neural Net Epochs Accuracy ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● FIGURE 10.11. Accuracy of the lasso and a two-hidden-layer neural network on the IMDb data. For the lasso, the x-axis displays − log(λ), while for the neural network it displays epochs (number of times the fitting algorithm passes through the training set). Both show a tendency to overfit, and achieve approximately the same test accuracy. you could just imagine being there robert ⟨UNK ⟩ is an amazing actor and now the same being director ⟨UNK ⟩ father came from the same scottish island as myself so i loved . . . Here we can see many words have been omitted, and some unknown words (UNK) have been marked as such. With this reduction the binary feature vector has length 10,000, and consists mostly of 0’s and a smattering of 1’s in the positions corresponding to words that are present in the document. We have a training set and test set, each with 25,000 examples, and each balanced with regard to sentiment. The resulting training feature matrix X has dimension 25,000×10,000, but only 1.3% of the binary entries are non- zero. We call such a matrix sparse, because most of the values are the same (zero in this case); it can be stored efficiently in sparse matrix format.15 sparse matrix format There are a variety of ways to account for the document length; here we only score a word as in or out of the document, but for example one could instead record the relative frequency of words. We split off a validation set of size 2,000 from the 25,000 training observations (for model tuning), and fit two model sequences: 15Rather than store the whole matrix, we can store instead the location and values for the nonzero entries. In this case, since the nonzero entries are all 1, just the locations are stored.
- 428. 10.5 Recurrent Neural Networks 421 • A lasso logistic regression using the glmnet package; • A two-class neural network with two hidden layers, each with 16 ReLU units. Both methods produce a sequence of solutions. The lasso sequence is in- dexed by the regularization parameter λ. The neural-net sequence is in- dexed by the number of gradient-descent iterations used in the fitting, as measured by training epochs or passes through the training set (Sec- tion 10.7). Notice that the training accuracy in Figure 10.11 (black points) increases monotonically in both cases. We can use the validation error to pick a good solution from each sequence (blue points in the plots), which would then be used to make predictions on the test data set. Note that a two-class neural network amounts to a nonlinear logistic regression model. From (10.12) and (10.13) we can see that log * Pr(Y = 1|X) Pr(Y = 0|X) + = Z1 − Z0 (10.15) = (β10 − β00) + K2 0 ℓ=1 (β1ℓ − β0ℓ)A (2) ℓ . (This shows the redundancy in the softmax function; for K classes we really only need to estimate K −1 sets of coefficients. See Section 4.3.5.) In Figure 10.11 we show accuracy (fraction correct) rather than classification accuracy error (fraction incorrect), the former being more popular in the machine learning community. Both models achieve a test-set accuracy of about 88%. The bag-of-words model summarizes a document by the words present, and ignores their context. There are at least two popular ways to take the context into account: • The bag-of-n-grams model. For example, a bag of 2-grams records bag-of-n- grams the consecutive co-occurrence of every distinct pair of words. “Bliss- fully long” can be seen as a positive phrase in a movie review, while “blissfully short” a negative. • Treat the document as a sequence, taking account of all the words in the context of those that preceded and those that follow. In the next section we explore models for sequences of data, which have applications in weather forecasting, speech recognition, language transla- tion, and time-series prediction, to name a few. We continue with this IMDb example there. 10.5 Recurrent Neural Networks Many data sources are sequential in nature, and call for special treatment when building predictive models. Examples include:
- 429. 422 10. Deep Learning A1 A2 A3 AL-1 AL = Aℓ Oℓ Y Xℓ O1 X1 O2 X2 O3 X3 OL-1 XL-1 OL Y XL . . . W U B W B W B W B W B W B U U U U FIGURE 10.12. Schematic of a simple recurrent neural network. The input is a sequence of vectors {Xℓ}L 1 , and here the target is a single response. The network processes the input sequence X sequentially; each Xℓ feeds into the hidden layer, which also has as input the activation vector Aℓ−1 from the previous element in the sequence, and produces the current activation vector Aℓ. The same collections of weights W, U and B are used as each element of the sequence is processed. The output layer produces a sequence of predictions Oℓ from the current activation Aℓ, but typically only the last of these, OL, is of relevance. To the left of the equal sign is a concise representation of the network, which is unrolled into a more explicit version on the right. • Documents such as book and movie reviews, newspaper articles, and tweets. The sequence and relative positions of words in a document capture the narrative, theme and tone, and can be exploited in tasks such as topic classification, sentiment analysis, and language transla- tion. • Time series of temperature, rainfall, wind speed, air quality, and so on. We may want to forecast the weather several days ahead, or cli- mate several decades ahead. • Financial time series, where we track market indices, trading volumes, stock and bond prices, and exchange rates. Here prediction is often difficult, but as we will see, certain indices can be predicted with reasonable accuracy. • Recorded speech, musical recordings, and other sound recordings. We may want to give a text transcription of a speech, or perhaps a lan- guage translation. We may want to assess the quality of a piece of music, or assign certain attributes. • Handwriting, such as doctor’s notes, and handwritten digits such as zip codes. Here we want to turn the handwriting into digital text, or read the digits (optical character recognition). In a recurrent neural network (RNN), the input object X is a sequence. recurrent neural network
- 430. 10.5 Recurrent Neural Networks 423 Consider a corpus of documents, such as the collection of IMDb movie re- views. Each document can be represented as a sequence of L words, so X = {X1, X2, . . . , XL}, where each Xℓ represents a word. The order of the words, and closeness of certain words in a sentence, convey semantic meaning. RNNs are designed to accommodate and take advantage of the sequential nature of such input objects, much like convolutional neural net- works accommodate the spatial structure of image inputs. The output Y can also be a sequence (such as in language translation), but often is a scalar, like the binary sentiment label of a movie review document. Figure 10.12 illustrates the structure of a very basic RNN with a sequence X = {X1, X2, . . . , XL} as input, a simple output Y , and a hidden-layer sequence {Aℓ}L 1 = {A1, A2, . . . , AL}. Each Xℓ is a vector; in the document example Xℓ could represent a one-hot encoding for the ℓth word based on the language dictionary for the corpus (see the top panel in Figure 10.13 for a simple example). As the sequence is processed one vector Xℓ at a time, the network updates the activations Aℓ in the hidden layer, taking as input the vector Xℓ and the activation vector Aℓ−1 from the previous step in the sequence. Each Aℓ feeds into the output layer and produces a prediction Oℓ for Y . OL, the last of these, is the most relevant. In detail, suppose each vector Xℓ of the input sequence has p components XT ℓ = (Xℓ1, Xℓ2, . . . , Xℓp), and the hidden layer consists of K units AT ℓ = (Aℓ1, Aℓ2, . . . , AℓK). As in Figure 10.4, we represent the collection of K × (p+1) shared weights wkj for the input layer by a matrix W, and similarly U is a K × K matrix of the weights uks for the hidden-to-hidden layers, and B is a K + 1 vector of weights βk for the output layer. Then Aℓk = g 1 wk0 + p 0 j=1 wkjXℓj + K 0 s=1 uksAℓ−1,s 2 , (10.16) and the output Oℓ is computed as Oℓ = β0 + K 0 k=1 βkAℓk (10.17) for a quantitative response, or with an additional sigmoid activation func- tion for a binary response, for example. Here g(·) is an activation function such as ReLU. Notice that the same weights W, U and B are used as we process each element in the sequence, i.e. they are not functions of ℓ. This is a form of weight sharing used by RNNs, and similar to the use of filters weight sharing in convolutional neural networks (Section 10.3.1.) As we proceed from be- ginning to end, the activations Aℓ accumulate a history of what has been seen before, so that the learned context can be used for prediction. For regression problems the loss function for an observation (X, Y ) is (Y − OL)2 , (10.18)
- 431. 424 10. Deep Learning which only references the final output OL = β0+ )K k=1 βkALk. Thus O1, O2, . . . , OL−1 are not used. When we fit the model, each element Xℓ of the input sequence X contributes to OL via the chain (10.16), and hence contributes indirectly to learning the shared parameters W, U and B via the loss (10.18). With n input sequence/response pairs (xi, yi), the parameters are found by minimizing the sum of squares n 0 i=1 (yi−oiL)2 = n 0 i=1 1 yi− ' β0+ K 0 k=1 βkg ' wk0+ p 0 j=1 wkjxiLj+ K 0 s=1 uksai,L−1,s ((22 . (10.19) Here we use lowercase letters for the observed yi and vector sequences xi = {xi1, xi2, . . . , xiL},16 as well as the derived activations. Since the intermediate outputs Oℓ are not used, one may well ask why they are there at all. First of all, they come for free, since they use the same output weights B needed to produce OL, and provide an evolving prediction for the output. Furthermore, for some learning tasks the response is also a sequence, and so the output sequence {O1, O2, . . . , OL} is explicitly needed. When used at full strength, recurrent neural networks can be quite com- plex. We illustrate their use in two simple applications. In the first, we continue with the IMDb sentiment analysis of the previous section, where we process the words in the reviews sequentially. In the second application, we illustrate their use in a financial time series forecasting problem. 10.5.1 Sequential Models for Document Classification Here we return to our classification task with the IMDb reviews. Our ap- proach in Section 10.4 was to use the bag-of-words model. Here the plan is to use instead the sequence of words occurring in a document to make predictions about the label for the entire document. We have, however, a dimensionality problem: each word in our document is represented by a one-hot-encoded vector (dummy variable) with 10,000 elements (one per word in the dictionary)! An approach that has become popular is to represent each word in a much lower-dimensional embedding embedding space. This means that rather than representing each word by a binary vector with 9,999 zeros and a single one in some position, we will represent it instead by a set of m real numbers, none of which are typically zero. Here m is the embedding dimension, and can be in the low 100s, or even less. This means (in our case) that we need a matrix E of dimension m×10,000, where each column is indexed by one of the 10,000 words in our dictionary, and the values in that column give the m coordinates for that word in the embedding space. 16This is a sequence of vectors; each element xiℓ is a p-vector.
- 432. 10.5 Recurrent Neural Networks 425 this is one of the best films actually the best I have ever seen the film starts one fall day One−hot Embed FIGURE 10.13. Depiction of a sequence of 20 words representing a single doc- ument: one-hot encoded using a dictionary of 16 words (top panel) and embedded in an m-dimensional space with m = 5 (bottom panel). Figure 10.13 illustrates the idea (with a dictionary of 16 rather than 10,000, and m = 5). Where does E come from? If we have a large corpus of labeled documents, we can have the neural network learn E as part of the optimization. In this case E is referred to as an embedding layer, embedding layer and a specialized E is learned for the task at hand. Otherwise we can insert a precomputed matrix E in the embedding layer, a process known as weight freezing. Two pretrained embeddings, word2vec and GloVe, are weight freezing word2vec GloVe widely used.17 These are built from a very large corpus of documents by a variant of principal components analysis (Section 12.2). The idea is that the positions of words in the embedding space preserve semantic meaning; e.g. synonyms should appear near each other. So far, so good. Each document is now represented as a sequence of m- vectors that represents the sequence of words. The next step is to limit each document to the last L words. Documents that are shorter than L get padded with zeros upfront. So now each document is represented by a series consisting of L vectors X = {X1, X2, . . . , XL}, and each Xℓ in the sequence has m components. We now use the RNN structure in Figure 10.12. The training corpus consists of n separate series (documents) of length L, each of which gets processed sequentially from left to right. In the process, a parallel series of hidden activation vectors Aℓ, ℓ = 1, . . . , L is created as in (10.16) for each document. Aℓ feeds into the output layer to produce the evolving prediction 17word2vec is described in Mikolov, Chen, Corrado, and Dean (2013), available at https://code.google.com/archive/p/word2vec. GloVe is described in Pennington, Socher, and Manning (2014), available at https://nlp.stanford.edu/projects/glove.
- 433. 426 10. Deep Learning Oℓ. We use the final value OL to predict the response: the sentiment of the review. This is a simple RNN, and has relatively few parameters. If there are K hidden units, the common weight matrix W has K × (m + 1) parameters, the matrix U has K × K parameters, and B has 2(K + 1) for the two-class logistic regression as in (10.15). These are used repeatedly as we process the sequence X = {Xℓ}L 1 from left to right, much like we use a single convolution filter to process each patch in an image (Section 10.3.1). If the embedding layer E is learned, that adds an additional m × D parameters (D = 10,000 here), and is by far the biggest cost. We fit the RNN as described in Figure 10.12 and the accompaying text to the IMDb data. The model had an embedding matrix E with m = 32 (which was learned in training as opposed to precomputed), followed by a single recurrent layer with K = 32 hidden units. The model was trained with dropout regularization on the 25,000 reviews in the designated training set, and achieved a disappointing 76% accuracy on the IMDb test data. A network using the GloVe pretrained embedding matrix E performed slightly worse. For ease of exposition we have presented a very simple RNN. More elab- orate versions use long term and short term memory (LSTM). Two tracks of hidden-layer activations are maintained, so that when the activation Aℓ is computed, it gets input from hidden units both further back in time, and closer in time — a so-called LSTM RNN. With long sequences, this LSTM RNN overcomes the problem of early signals being washed out by the time they get propagated through the chain to the final activation vector AL. When we refit our model using the LSTM architecture for the hidden layer, the performance improved to 87% on the IMDb test data. This is com- parable with the 88% achieved by the bag-of-words model in Section 10.4. We give details on fitting these models in Section 10.9.6. Despite this added LSTM complexity, our RNN is still somewhat “entry level”. We could probably achieve slightly better results by changing the size of the model, changing the regularization, and including additional hidden layers. However, LSTM models take a long time to train, which makes exploring many architectures and parameter optimization tedious. RNNs provide a rich framework for modeling data sequences, and they continue to evolve. There have been many advances in the development of RNNs — in architecture, data augmentation, and in the learning algo- rithms. At the time of this writing (early 2020) the leading RNN configura- tions report accuracy above 95% on the IMDb data. The details are beyond the scope of this book.18 18An IMDb leaderboard can be found at https://paperswithcode.com/sota/ sentiment-analysis-on-imdb.
- 434. 10.5 Recurrent Neural Networks 427 10.5.2 Time Series Forecasting Figure 10.14 shows historical trading statistics from the New York Stock Exchange. Shown are three daily time series covering the period December 3, 1962 to December 31, 1986:19 • Log trading volume. This is the fraction of all outstanding shares that are traded on that day, relative to a 100-day moving average of past turnover, on the log scale. • Dow Jones return. This is the difference between the log of the Dow Jones Industrial Index on consecutive trading days. • Log volatility. This is based on the absolute values of daily price movements. Predicting stock prices is a notoriously hard problem, but it turns out that predicting trading volume based on recent past history is more manageable (and is useful for planning trading strategies). An observation here consists of the measurements (vt, rt, zt) on day t, in this case the values for log volume, DJ return and log volatility. There are a total of T = 6,051 such triples, each of which is plotted as a time series in Figure 10.14. One feature that strikes us immediately is that the day- to-day observations are not independent of each other. The series exhibit auto-correlation — in this case values nearby in time tend to be similar auto- correlation to each other. This distinguishes time series from other data sets we have encountered, in which observations can be assumed to be independent of each other. To be clear, consider pairs of observations (vt, vt−ℓ), a lag of ℓ lag days apart. If we take all such pairs in the vt series and compute their corre- lation coefficient, this gives the autocorrelation at lag ℓ. Figure 10.15 shows the autocorrelation function for all lags up to 37, and we see considerable correlation. Another interesting characteristic of this forecasting problem is that the response variable vt — log volume — is also a predictor! In particular, we will use the past values of log volume to predict values in the future. RNN forecaster We wish to predict a value vt from past values vt−1, vt−2, . . ., and also to make use of past values of the other series rt−1, rt−2, . . . and zt−1, zt−2, . . .. Although our combined data is quite a long series with 6,051 trading days, the structure of the problem is different from the previous document- classification example. • We only have one series of data, not 25,000. 19These data were assembled by LeBaron and Weigend (1998) IEEE Transactions on Neural Networks, 9(1): 213–220.
- 435. 428 10. Deep Learning Log(Trading Volume) −1.0 0.0 0.5 1.0 Dow Jones Return −0.04 0.00 0.04 1965 1970 1975 1980 1985 −13 −11 −9 −8 Log(Volatility) FIGURE 10.14. Historical trading statistics from the New York Stock Exchange. Daily values of the normalized log trading volume, DJIA return, and log volatility are shown for a 24-year period from 1962–1986. We wish to predict trading volume on any day, given the history on all earlier days. To the left of the red bar (January 2, 1980) is training data, and to the right test data. • We have an entire series of targets vt, and the inputs include past values of this series. How do we represent this problem in terms of the structure displayed in Figure 10.12? The idea is to extract many short mini-series of input se- quences X = {X1, X2, . . . , XL} with a predefined length L (called the lag lag in this context), and a corresponding target Y . They have the form X1 = ⎛ ⎝ vt−L rt−L zt−L ⎞ ⎠ , X2 = ⎛ ⎝ vt−L+1 rt−L+1 zt−L+1 ⎞ ⎠ , · · · , XL = ⎛ ⎝ vt−1 rt−1 zt−1 ⎞ ⎠ , and Y = vt. (10.20) So here the target Y is the value of log volume vt at a single timepoint t, and the input sequence X is the series of 3-vectors {Xℓ}L 1 each consisting of the three measurements log volume, DJ return and log volatility from day t − L, t − L + 1, up to t − 1. Each value of t makes a separate (X, Y )
- 436. 10.5 Recurrent Neural Networks 429 0 5 10 15 20 25 30 35 0.0 0.4 0.8 Log( Trading Volume) Lag Autocorrelation Function FIGURE 10.15. The autocorrelation function for log volume. We see that nearby values are fairly strongly correlated, with correlations above 0.2 as far as 20 days apart. 1980 1982 1984 1986 −1.0 0.0 0.5 1.0 Test Period: Observed and Predicted Year log(Trading Volume) FIGURE 10.16. RNN forecast of log volume on the NYSE test data. The black lines are the true volumes, and the superimposed orange the forecasts. The fore- casted series accounts for 42% of the variance of log volume. pair, for t running from L + 1 to T. For the NYSE data we will use the past five trading days to predict the next day’s trading volume. Hence, we use L = 5. Since T = 6,051, we can create 6,046 such (X, Y ) pairs. Clearly L is a parameter that should be chosen with care, perhaps using validation data. We fit this model with K = 12 hidden units using the 4,281 training sequences derived from the data before January 2, 1980 (see Figure 10.14), and then used it to forecast the 1,770 values of log volume after this date. We achieve an R2 = 0.42 on the test data. Details are given in Sec- tion 10.9.6. As a straw man,20 using yesterday’s value for log volume as the prediction for today has R2 = 0.18. Figure 10.16 shows the forecast results. We have plotted the observed values of the daily log volume for the 20A straw man here refers to a simple and sensible prediction that can be used as a baseline for comparison.
- 437. 430 10. Deep Learning test period 1980–1986 in black, and superimposed the predicted series in orange. The correspondence seems rather good. In forecasting the value of log volume in the test period, we have to use the test data itself in forming the input sequences X. This may feel like cheating, but in fact it is not; we are always using past data to predict the future. Autoregression The RNN we just fit has much in common with a traditional autoregression auto- regression (AR) linear model, which we present now for comparison. We first consider the response sequence vt alone, and construct a response vector y and a matrix M of predictors for least squares regression as follows: y = ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ vL+1 vL+2 vL+3 . . . vT ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ M = ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 1 vL vL−1 · · · v1 1 vL+1 vL · · · v2 1 vL+2 vL+1 · · · v3 . . . . . . . . . ... . . . 1 vT −1 vT −2 · · · vT −L ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ . (10.21) M and y each have T − L rows, one per observation. We see that the predictors for any given response vt on day t are the previous L values of the same series. Fitting a regression of y on M amounts to fitting the model v̂t = β̂0 + β̂1vt−1 + β̂2vt−2 + · · · + β̂Lvt−L, (10.22) and is called an order-L autoregressive model, or simply AR(L). For the NYSE data we can include lagged versions of DJ return and log volatility, rt and zt, in the predictor matrix M, resulting in 3L + 1 columns. An AR model with L = 5 achieves a test R2 of 0.41, slightly inferior to the 0.42 achieved by the RNN. Of course the RNN and AR models are very similar. They both use the same response Y and input sequences X of length L = 5 and dimen- sion p = 3 in this case. The RNN processes this sequence from left to right with the same weights W (for the input layer), while the AR model simply treats all L elements of the sequence equally as a vector of L × p predictors — a process called flattening in the neural network literature. flattening Of course the RNN also includes the hidden layer activations Aℓ which transfer information along the sequence, and introduces additional nonlin- earity. From (10.19) with K = 12 hidden units, we see that the RNN has 13 + 12 × (1 + 3 + 12) = 205 parameters, compared to the 16 for the AR(5) model. An obvious extension of the AR model is to use the set of lagged predic- tors as the input vector to an ordinary feedforward neural network (10.1), and hence add more flexibility. This achieved a test R2 = 0.42, slightly better than the linear AR, and the same as the RNN.
- 438. 10.5 Recurrent Neural Networks 431 All the models can be improved by including the variable day of week corresponding to the day t of the target vt (which can be learned from the calendar dates supplied with the data); trading volume is often higher on Mondays and Fridays. Since there are five trading days, this one-hot en- codes to five binary variables. The performance of the AR model improved to R2 = 0.46 as did the RNN, and the nonlinear AR model improved to R2 = 0.47. We used the most simple version of the RNN in our examples here. Additional experiments with the LSTM extension of the RNN yielded small improvements, typically of up to 1% in R2 in these examples. We give details of how we fit all three models in Section 10.9.6. 10.5.3 Summary of RNNs We have illustrated RNNs through two simple use cases, and have only scratched the surface. There are many variations and enhancements of the simple RNN we used for sequence modeling. One approach we did not discuss uses a one- dimensional convolutional neural network, treating the sequence of vectors (say words, as represented in the embedding space) as an image. The con- volution filter slides along the sequence in a one-dimensional fashion, with the potential to learn particular phrases or short subsequences relevant to the learning task. One can also have additional hidden layers in an RNN. For example, with two hidden layers, the sequence Aℓ is treated as an input sequence to the next hidden layer in an obvious fashion. The RNN we used scanned the document from beginning to end; alter- native bidirectional RNNs scan the sequences in both directions. bidirectional In language translation the target is also a sequence of words, in a language different from that of the input sequence. Both the input se- quence and the target sequence are represented by a structure similar to Figure 10.12, and they share the hidden units. In this so-called Seq2Seq Seq2Seq learning, the hidden units are thought to capture the semantic meaning of the sentences. Some of the big breakthroughs in language modeling and translation resulted from the relatively recent improvements in such RNNs. Algorithms used to fit RNNs can be complex and computationally costly. Fortunately, good software protects users somewhat from these complexi- ties, and makes specifying and fitting these models relatively painless. Many of the models that we enjoy in daily life (like Google Translate) use state- of-the-art architectures developed by teams of highly skilled engineers, and have been trained using massive computational and data resources.
- 439. 432 10. Deep Learning 10.6 When to Use Deep Learning The performance of deep learning in this chapter has been rather impres- sive. It nailed the digit classification problem, and deep CNNs have really revolutionized image classification. We see daily reports of new success sto- ries for deep learning. Many of these are related to image classification tasks, such as machine diagnosis of mammograms or digital X-ray images, ophthalmology eye scans, annotations of MRI scans, and so on. Likewise there are numerous successes of RNNs in speech and language translation, forecasting, and document modeling. The question that then begs an an- swer is: should we discard all our older tools, and use deep learning on every problem with data? To address this question, we revisit our Hitters dataset from Chapter 6. This is a regression problem, where the goal is to predict the Salary of a baseball player in 1987 using his performance statistics from 1986. After removing players with missing responses, we are left with 263 players and 19 variables. We randomly split the data into a training set of 176 players (two thirds), and a test set of 87 players (one third). We used three methods for fitting a regression model to these data. • A linear model was used to fit the training data, and make predictions on the test data. The model has 20 parameters. • The same linear model was fit with lasso regularization. The tuning parameter was selected by 10-fold cross-validation on the training data. It selected a model with 12 variables having nonzero coefficients. • A neural network with one hidden layer consisting of 64 ReLU units was fit to the data. This model has 1,409 parameters.21 Table 10.2 compares the results. We see similar performance for all three models. We report the mean absolute error on the test data, as well as the test R2 for each method, which are all respectable (see Exercise 5). We spent a fair bit of time fiddling with the configuration parameters of the neural network to achieve these results. It is possible that if we were to spend more time, and got the form and amount of regularization just right, that we might be able to match or even outperform linear regression and the lasso. But with great ease we obtained linear models that work well. Linear models are much easier to present and understand than the neural network, which is essentially a black box. The lasso selected 12 of the 19 variables in making its prediction. So in cases like this we are much better off following the Occam’s razor principle: when faced with several methods Occam’s razor 21The model was fit by stochastic gradient descent with a batch size of 32 for 1,000 epochs, and 10% dropout regularization. The test error performance flattened out and started to slowly increase after 1,000 epochs. These fitting details are discussed in Sec- tion 10.7.
- 440. 10.6 When to Use Deep Learning 433 Model # Parameters Mean Abs. Error Test Set R2 Linear Regression 20 254.7 0.56 Lasso 12 252.3 0.51 Neural Network 1409 257.4 0.54 TABLE 10.2. Prediction results on the Hitters test data for linear models fit by ordinary least squares and lasso, compared to a neural network fit by stochastic gradient descent with dropout regularization. Coefficient Std. error t-statistic p-value Intercept -226.67 86.26 -2.63 0.0103 Hits 3.06 1.02 3.00 0.0036 Walks 0.181 2.04 0.09 0.9294 CRuns 0.859 0.12 7.09 < 0.0001 PutOuts 0.465 0.13 3.60 0.0005 TABLE 10.3. Least squares coefficient estimates associated with the regres- sion of Salary on four variables chosen by lasso on the Hitters data set. This model achieved the best performance on the test data, with a mean absolute error of 224.8. The results reported here were obtained from a regression on the test data, which was not used in fitting the lasso model. that give roughly equivalent performance, pick the simplest. After a bit more exploration with the lasso model, we identified an even simpler model with four variables. We then refit the linear model with these four variables to the training data (the so-called relaxed lasso), and achieved a test mean absolute error of 224.8, the overall winner! It is tempting to present the summary table from this fit, so we can see coefficients and p- values; however, since the model was selected on the training data, there would be selection bias. Instead, we refit the model on the test data, which was not used in the selection. Table 10.3 shows the results. We have a number of very powerful tools at our disposal, including neu- ral networks, random forests and boosting, support vector machines and generalized additive models, to name a few. And then we have linear mod- els, and simple variants of these. When faced with new data modeling and prediction problems, its tempting to always go for the trendy new methods. Often they give extremely impressive results, especially when the datasets are very large and can support the fitting of high-dimensional nonlinear models. However, if we can produce models with the simpler tools that perform as well, they are likely to be easier to fit and understand, and po- tentially less fragile than the more complex approaches. Wherever possible, it makes sense to try the simpler models as well, and then make a choice based on the performance/complexity tradeoff. Typically we expect deep learning to be an attractive choice when the sample size of the training set is extremely large, and when interpretability of the model is not a high priority.
- 441. 434 10. Deep Learning 10.7 Fitting a Neural Network Fitting neural networks is somewhat complex, and we give a briefoverview here. The ideas generalize to much more complex networks. Readers who find this material challenging can safely skip it. Fortunately, as we see in the lab at the end of the chapter, good software is available to fit neural network models in a relatively automated way, without worrying about the technical details of the model-fitting procedure. We start with the simple network depicted in Figure 10.1 in Section 10.1. In model (10.1) the parameters are β = (β0, β1, . . . , βK), as well as each of the wk = (wk0, wk1, . . . , wkp), k = 1, . . . , K. Given observations (xi, yi), i = 1, . . . , n, we could fit the model by solving a nonlinear least squares problem minimize {wk}K 1 , β 1 2 n 0 i=1 (yi − f(xi))2 , (10.23) where f(xi) = β0 + K 0 k=1 βkg 1 wk0 + p 0 j=1 wkjxij 2 . (10.24) The objective in (10.23) looks simple enough, but because of the nested arrangement of the parameters and the symmetry of the hidden units, it is not straightforward to minimize. The problem is nonconvex in the param- eters, and hence there are multiple solutions. As an example, Figure 10.17 shows a simple nonconvex function of a single variable θ; there are two solutions: one is a local minimum and the other is a global minimum. Fur- local minimum global minimum thermore, (10.1) is the very simplest of neural networks; in this chapter we have presented much more complex ones where these problems are com- pounded. To overcome some of these issues and to protect from overfitting, two general strategies are employed when fitting neural networks. • Slow Learning: the model is fit in a somewhat slow iterative fash- ion, using gradient descent. The fitting process is then stopped when gradient descent overfitting is detected. • Regularization: penalties are imposed on the parameters, usually lasso or ridge as discussed in Section 6.2. Suppose we represent all the parameters in one long vector θ. Then we can rewrite the objective in (10.23) as R(θ) = 1 2 n 0 i=1 (yi − fθ(xi))2 , (10.25) where we make explicit the dependence of f on the parameters. The idea of gradient descent is very simple.
- 442. 10.7 Fitting a Neural Network 435 −1.0 −0.5 0.0 0.5 1.0 0 1 2 3 4 5 6 θ R(θ) θ0 θ1 θ2 θ7 ● ● ● ● R(θ0 ) R(θ1 ) R(θ2 ) R(θ7 ) FIGURE 10.17. Illustration of gradient descent for one-dimensional θ. The objective function R(θ) is not convex, and has two minima, one at θ = −0.46 (local), the other at θ = 1.02 (global). Starting at some value θ0 (typically ran- domly chosen), each step in θ moves downhill — against the gradient — until it cannot go down any further. Here gradient descent reached the global minimum in 7 steps. 1. Start with a guess θ0 for all the parameters in θ, and set t = 0. 2. Iterate until the objective (10.25) fails to decrease: (a) Find a vector δ that reflects a small change in θ, such that θt+1 = θt + δ reduces the objective; i.e. such that R(θt+1 ) < R(θt ). (b) Set t ← t + 1. One can visualize (Figure 10.17) standing in a mountainous terrain, and the goal is to get to the bottom through a series of steps. As long as each step goes downhill, we must eventually get to the bottom. In this case we were lucky, because with our starting guess θ0 we end up at the global minimum. In general we can hope to end up at a (good) local minimum. 10.7.1 Backpropagation How do we find the directions to move θ so as to decrease the objective R(θ) in (10.25)? The gradient of R(θ), evaluated at some current value gradient θ = θm , is the vector of partial derivatives at that point: ∇R(θm ) = ∂R(θ) ∂θ V V V θ=θm . (10.26) The subscript θ = θm means that after computing the vector of derivatives, we evaluate it at the current guess, θm . This gives the direction in θ-space in which R(θ) increases most rapidly. The idea of gradient descent is to move θ a little in the opposite direction (since we wish to go downhill): θm+1 ← θm − ρ∇R(θm ). (10.27)
- 443. 436 10. Deep Learning For a small enough value of the learning rate ρ, this step will decrease the learning rate objective R(θ); i.e. R(θm+1 ) ≤ R(θm ). If the gradient vector is zero, then we may have arrived at a minimum of the objective. How complicated is the calculation (10.26)? It turns out that it is quite simple here, and remains simple even for much more complex networks, because of the chain rule of differentiation. chain rule Since R(θ) = )n i=1 Ri(θ) = 1 2 )n i=1(yi − fθ(xi))2 is a sum, its gradient is also a sum over the n observations, so we will just examine one of these terms, Ri(θ) = 1 2 1 yi − β0 − K 0 k=1 βkg ' wk0 + p 0 j=1 wkjxij (22 . (10.28) To simplify the expressions to follow, we write zik = wk0 + )p j=1 wkjxij. First we take the derivative with respect to βk: ∂Ri(θ) ∂βk = ∂Ri(θ) ∂fθ(xi) · ∂fθ(xi) ∂βk = −(yi − fθ(xi)) · g(zik). (10.29) And now we take the derivative with respect to wkj: ∂Ri(θ) ∂wkj = ∂Ri(θ) ∂fθ(xi) · ∂fθ(xi) ∂g(zik) · ∂g(zik) ∂zik · ∂zik ∂wkj = −(yi − fθ(xi)) · βk · g′ (zik) · xij. (10.30) Notice that both these expressions contain the residual yi − fθ(xi). In (10.29) we see that a fraction of that residual gets attributed to each of the hidden units according to the value of g(zik). Then in (10.30) we see a similar attribution to input j via hidden unit k. So the act of differen- tiation assigns a fraction of the residual to each of the parameters via the chain rule — a process known as backpropagation in the neural network backprop- agation literature. Although these calculations are straightforward, it takes careful bookkeeping to keep track of all the pieces. 10.7.2 Regularization and Stochastic Gradient Descent Gradient descent usually takes many steps to reach a local minimum. In practice, there are a number of approaches for accelerating the process. Also, when n is large, instead of summing (10.29)–(10.30) over all n ob- servations, we can sample a small fraction or minibatch of them each time minibatch we compute a gradient step. This process is known as stochastic gradient descent (SGD) and is the state of the art for learning deep neural networks. stochastic gradient descent Fortunately, there is very good software for setting up deep learning mod- els, and for fitting them to data, so most of the technicalities are hidden from the user.
- 444. 10.7 Fitting a Neural Network 437 0 5 10 15 20 25 30 0.1 0.2 0.3 0.4 Epochs Value of Objective Function Training Set Validation Set 0 5 10 15 20 25 30 0.00 0.02 0.04 0.06 0.08 0.10 0.12 Epochs Classification Error FIGURE 10.18. Evolution of training and validation errors for the MNIST neural network depicted in Figure 10.4, as a function of training epochs. The objective refers to the log-likelihood (10.14). We now turn to the multilayer network (Figure 10.4) used in the digit recognition problem. The network has over 235,000 weights, which is around four times the number of training examples. Regularization is essential here to avoid overfitting. The first row in Table 10.1 uses ridge regularization on the weights. This is achieved by augmenting the objective function (10.14) with a penalty term: R(θ; λ) = − n 0 i=1 9 0 m=0 yim log(fm(xi)) + λ 0 j θ2 j . (10.31) The parameter λ is often preset at a small value, or else it is found using the validation-set approach of Section 5.3.1. We can also use different values of λ for the groups of weights from different layers; in this case W1 and W2 were penalized, while the relatively few weights B of the output layer were not penalized at all. Lasso regularization is also popular as an additional form or regularization, or as an alternative to ridge. Figure 10.18 shows some metrics that evolve during the training of the network on the MNIST data. It turns out that SGD naturally enforces its own form of approximately quadratic regularization.22 Here the minibatch size was 128 observations per gradient update. The term epochs labeling the epochs horizontal axis in Figure 10.18 counts the number of times an equivalent of the full training set has been processed. For this network, 20% of the 60,000 training observations were used as a validation set in order to determine when training should stop. So in fact 48,000 observations were used for 22This and other properties of SGD for deep learning are the subject of much research in the machine learning literature at the time of writing.
- 445. 438 10. Deep Learning FIGURE 10.19. Dropout Learning. Left: a fully connected network. Right: net- work with dropout in the input and hidden layer. The nodes in grey are selected at random, and ignored in an instance of training. training, and hence there are 48,000/128 ≈ 375 minibatch gradient updates per epoch. We see that the value of the validation objective actually starts to increase by 30 epochs, so early stopping can also be used as an additional early stopping form of regularization. 10.7.3 Dropout Learning The second row in Table 10.1 is labeled dropout. This is a relatively new dropout and efficient form of regularization, similar in some respects to ridge reg- ularization. Inspired by random forests (Section 8.2), the idea is to ran- domly remove a fraction φ of the units in a layer when fitting the model. Figure 10.19 illustrates this. This is done separately each time a training observation is processed. The surviving units stand in for those missing, and their weights are scaled up by a factor of 1/(1 − φ) to compensate. This prevents nodes from becoming over-specialized, and can be seen as a form of regularization. In practice dropout is achieved by randomly set- ting the activations for the “dropped out” units to zero, while keeping the architecture intact. 10.7.4 Network Tuning The network in Figure 10.4 is considered to be relatively straightforward; it nevertheless requires a number of choices that all have an effect on the performance: • The number of hidden layers, and the number of units per layer. Mod- ern thinking is that the number of units per hidden layer can be large, and overfitting can be controlled via the various forms of regulariza- tion.
- 446. 10.8 Interpolation and Double Descent 439 • Regularization tuning parameters. These include the dropout rate φ and the strength λ of lasso and ridge regularization, and are typically set separately at each layer. • Details of stochastic gradient descent. These includes the batch size, the number of epochs, and if used, details of data augmentation (Sec- tion 10.3.4.) Choices such as these can make a difference. In preparing this MNIST exam- ple, we achieved a respectable 1.8% misclassification error after some trial and error. Finer tuning and training of a similar network can get under 1% error on these data, but the tinkering process can be tedious, and can result in overfitting if done carelessly. 10.8 Interpolation and Double Descent Throughout this book, we have repeatedly discussed the bias-variance trade- off, first presented in Section 2.2.2. This trade-off indicates that statistical learning methods tend to perform the best, in terms of test-set error, for an intermediate level of model complexity. In particular, if we plot “flexibility” on the x-axis and error on the y-axis, then we generally expect to see that test error has a U-shape, whereas training error decreases monotonically. Two “typical” examples of this behavior can be seen in the right-hand panel of Figure 2.9 on page 31, and in Figure 2.17 on page 42. One implica- tion of the bias-variance trade-off is that it is generally not a good idea to interpolate the training data — that is, to get zero training error — since interpolate that will often result in very high test error. However, it turns out that in certain specific settings it can be possible for a statistical learning method that interpolates the training data to perform well — or at least, better than a slightly less complex model that does not quite interpolate the data. This phenomenon is known as double descent, and is displayed in Figure 10.20. “Double descent” gets its name from the fact that the test error has a U-shape before the interpolation threshold is reached, and then it descends again (for a while, at least) as an increasingly flexible model is fit. We now describe the set-up that resulted in Figure 10.20. We simulated n = 20 observations from the model Y = sin(X) + ϵ, where X ∼ U[−5, 5] (uniform distribution), and ϵ ∼ N(0, σ2 ) with σ = 0.3. We then fit a natural spline to the data, as described in Section 7.4, with d degrees of freedom.23 Recall from Section 7.4 that fitting a natural spline 23This implies the choice of d knots, here chosen at d equi-probability quantiles of the training data. When d > n, the quantiles are found by interpolation.
- 447. 440 10. Deep Learning 2 5 10 20 50 0.0 0.5 1.0 1.5 2.0 Degrees of Freedom Error Training Error Test Error FIGURE 10.20. Double descent phenomenon, illustrated using error plots for a one-dimensional natural spline example. The horizontal axis refers to the num- ber of spline basis functions on the log scale. The training error hits zero when the degrees of freedom coincides with the sample size n = 20, the “interpolation threshold”, and remains zero thereafter. The test error increases dramatically at this threshold, but then descends again to a reasonable value before finally increas- ing again. with d degrees of freedom amounts to fitting a least-squares regression of the response onto a set of d basis functions. The upper-left panel of Figure 10.21 shows the data, the true function f(X), and ˆ f8(X), the fitted natural spline with d = 8 degrees of freedom. Next, we fit a natural spline with d = 20 degrees of freedom. Since n = 20, this means that n = d, and we have zero training error; in other words, we have interpolated the training data! We can see from the top-right panel of Figure 10.21 that ˆ f20(X) makes wild excursions, and hence the test error will be large. We now continue to fit natural splines to the data, with increasing values of d. For d > 20, the least squares regression of Y onto d basis functions is not unique: there are an infinite number of least squares coefficient es- timates that achieve zero error. To select among them, we choose the one with the smallest sum of squared coefficients, )d j=1 β̂2 j . This is known as the minimum-norm solution. The two lower panels of Figure 10.21 show the minimum-norm natural spline fits with d = 42 and d = 80 degrees of freedom. Incredibly, ˆ f42(X) is quite a bit less less wild than ˆ f20(X), even though it makes use of more degrees of freedom. And ˆ f80(X) is not much different. How can this be? Essentially, ˆ f20(X) is very wild because there is just a single way to inter- polate n = 20 observations using d = 20 basis functions, and that single way results in a somewhat extreme fitted function. By contrast, there are an
- 448. 10.8 Interpolation and Double Descent 441 −4 −2 0 2 4 −3 −2 −1 0 1 2 3 8 Degrees of Freedom seq(−5, 5, len = 1000) −4 −2 0 2 4 −3 −2 −1 0 1 2 3 20 Degrees of Freedom seq(−5, 5, len = 1000) f(seq(−5, 5, len = 1000)) −4 −2 0 2 4 −3 −2 −1 0 1 2 3 42 Degrees of Freedom −4 −2 0 2 4 −3 −2 −1 0 1 2 3 80 Degrees of Freedom f(seq(−5, 5, len = 1000)) FIGURE 10.21. Fitted functions ˆ fd(X) (orange), true function f(X) (black) and the observed 20 training data points. A different value of d (degrees of free- dom) is used in each panel. For d ≥ 20 the orange curves all interpolate the training points, and hence the training error is zero. infinite number of ways to interpolate n = 20 observations using d = 42 or d = 80 basis functions, and the smoothest of them — that is, the minimum norm solution — is much less wild than ˆ f20(X)! In Figure 10.20, we display the training error and test error associated with ˆ fd(X), for a range of values of the degrees of freedom d. We see that the training error drops to zero once d = 20 and beyond; i.e. once the interpolation threshold is reached. By contrast, the test error shows a U- shape for d ≤ 20, grows extremely large around d = 20, and then shows a second region of descent for d > 20. For this example the signal-to-noise ratio — Var(f(X))/σ2 — is 5.9, which is quite high (the data points are close to the true curve). So an estimate that interpolates the data and does not wander too far inbetween the observed data points will likely do well. In Figures 10.20 and 10.21, we have illustrated the double descent phe- nomenon in a simple one-dimensional setting using natural splines. How- ever, it turns out that the same phenomenon can arise for deep learning. Basically, when we fit neural networks with a huge number of parameters, we are sometimes able to get good results with zero training error. This is particularly true in problems with high signal-to-noise ratio, such as natural image recognition and language translation, for example. This is because the techniques used to fit neural networks, including stochastic gradient descent, naturally lend themselves to selecting a “smooth” interpolating model that has good test-set performance on these kinds of problems.
- 449. 442 10. Deep Learning Some points are worth emphasizing: • The double-descent phenomenon does not contradict the bias-variance trade-off, as presented in Section 2.2.2. Rather, the double-descent curve seen in the right-hand side of Figure 10.20 is a consequence of the fact that the x-axis displays the number of spline basis functions used, which does not properly capture the true “flexibility” of models that interpolate the training data. Stated another way, in this exam- ple, the minimum-norm natural spline with d = 42 has lower variance than the natural spline with d = 20. • Most of the statistical learning methods seen in this book do not exhibit double descent. For instance, regularization approaches typically do not interpolate the training data, and thus double descent does not occur. This is not a drawback of regularized methods: they can give great results without interpolating the data! In particular, in the examples here, if we had fit the natural splines using ridge regression with an appropriately-chosen penalty rather than least squares, then we would not have seen double descent, and in fact would have obtained better test error results. • In Chapter 9, we saw that maximal margin classifiers and SVMs that have zero training error nonetheless often achieve very good test error. This is in part because those methods seek smooth minimum norm solutions. This is similar to the fact that the minimum-norm natural spline can give good results with zero training error. • The double-descent phenomenon has been used by the machine learn- ing community to explain the successful practice of using an over- parametrized neural network (many layers, and many hidden units), and then fitting all the way to zero training err