Chapter12-Regression-PolynomialRegression.pdf

Regression Analysis | Chapter 12 | Polynomial Regression Models | Shalabh, IIT Kanpur
1
1
1
Chapter 12
Polynomial Regression Models
A model is said to be linear when it is linear in parameters. So the model
2
0 1 2
y x x
   
   
and
2 2
0 1 1 2 2 11 1 22 2 12 1 2
y x x x x x x
      
      
are also the linear model. In fact, they are the second-order polynomials in one and two variables,
respectively.
The polynomial models can be used in those situations where the relationship between study and explanatory
variables is curvilinear. Sometimes a nonlinear relationship in a small range of explanatory variable can also
be modelled by polynomials.
Polynomial models in one variable
The th
k order polynomial model in one variable is given by
2
0 1 2 ... .
k
k
y x x x
    
     
If , 1,2,..., ,
j
j
x x j k
  then the model is multiple linear regressions model in k explanatory variables
1 2
, ,..., .
k
x x x So the linear regression model y X  
  includes the polynomial regression model. Thus
the techniques for fitting linear regression model can be used for fitting the polynomial regression model.
For example:
2
0 1 2
y x x
   
   
or
2
0 1 2
( )
E y x x
  
  
is a polynomial regression model in one variable and is called a second-order model or quadratic
model. The coefficients 1
 and 2
 are called the linear effect parameter and quadratic effect
parameter, respectively.

2
2
2
The interpretation of parameter 0
 is 0 ( )
E y
  when 0
x  and it can be included in the model provided
the range of data includes 0.
x  If 0
x  is not included, then 0
 has no interpretation. An example of
the quadratic model is like as follows:
The polynomial models can be used to approximate a complex nonlinear relationship. The polynomial
models is just the Taylor series expansion of the unknown nonlinear function in such a case.
Considerations in fitting polynomial in one variable
Some of the considerations in the fitting polynomial model are as follows:
1. Order of the model
The order of the polynomial model is kept as low as possible. Some transformations can be used to keep
the model to be of the first order. If this is not satisfactory, then the second-order polynomial is tried.
Arbitrary fitting of higher-order polynomials can be a serious abuse of regression analysis. A model
which is consistent with the knowledge of data and its environment should be taken into account. It is
always possible for a polynomial of order ( 1)
n  to pass through n points so that a polynomial of
sufficiently high degree can always be found that provides a “good” fit to the data. Such models neither
enhance the understanding of the unknown function nor be a good predictor.

3
3
3
2. Model building strategy:
A good strategy should be used to choose the order of an approximate polynomial.
One possible approach is to successively fit the models in increasing order and test the significance of
regression coefficients at each step of model fitting. Keep the order increasing until t -test for the highest
order term is nonsignificant. This is called a forward selection procedure.
Another approach is to fit the appropriate highest order model and then delete terms one at a time starting
with the highest order. This is continued until the highest order remaining term has a significant t -
statistic. This is called a backward elimination procedure.
The forward selection and backward elimination procedures do not necessarily lead to the same model.
The first and second-order polynomials are mostly used in practice.
3. Extrapolation:
One has to be very cautioned in extrapolation with polynomial models. The curvatures in the region of
data and the region of extrapolation can be different. For example, in the following figure, the trend of
data in the region of original data is increasing, but it is decreasing in the region of extrapolation. So
predicted response would not be based on the true behaviour of the data.
In general, polynomial models may have unanticipated turns in inappropriate directions. This may
provide incorrect inferences in interpolation as well as extrapolation.

4
4
4
4. Ill-conditioning:
A basic assumption in linear regression analysis is that rank of X-matrix is full column rank. In
polynomial regression models, as the order increases, the '
X X matrix becomes ill-conditioned. As a
result, the 1
( ' )
X X 
may not be accurate, and parameters will be estimated with a considerable error.
If values of x lie in a narrow range, then the degree of ill-conditioning increases and multicollinearity in
the columns of X matrix enters. For example, if x varies between 2 and 3, then 2
x varies between 4
and 9. This introduces strong multicollinearity between x and 2
x .
5. Hierarchy:
A model is said to be hierarchical if it contains the terms 2 3
, ,
x x x , etc. in a hierarchy. For example, the
model
2 3 4
0 1 2 3 4
y x x x x
     
     
is hierarchical as it contains all the terms up to order four. The model
2 4
0 1 2 4
y x x x
    
    
is not hierarchical as it does not contain the term of 3
x .
It is expected that all polynomial models should have this property because only hierarchical models are
invariant under linear transformation. This requirement is more attractive from the mathematics point of
view. In many situations, the need for the model may be different. For example, the model
0 1 1 12 1 2
y x x x
   
   
needs a two-factor interaction which is provided by the cross-product term. A hierarchical model would
need inclusion of 2
x which is not needed from the point of view of statistical significance perspective.

5
5
5
Orthogonal polynomials:
While fitting a linear regression model to a given set of data, we begin with a simple linear regression
model. Suppose later we decide to change it to a quadratic or wish to increase the order from quadratic to
a cubic model etc. In each case, we have to begin the modeling from scratch, i.e., from the simple linear
regression model. It would be preferable to have a situation in which adding an extra term merely refine
the model in the sense that by increasing the order, we do not need to do all the calculations from scratch.
This aspect was of more importance in the pre-computer era when all the calculations were done
manually. This cannot be achieved by using the powers 0 2 3
1, , , ...
x x x x
 in succession. But it can be
achieved by a system of orthogonal polynomials. The th
k orthogonal polynomial has a degree k . Such
polynomials may be constructed by using Gram-Schmidt orthogonalization.
Another issue in fitting the polynomials in one variable is ill-conditioning. An assumption in usual
multiple linear regression analysis is that all the independent variables are independent. In the polynomial
regression model, this assumption is not satisfied. Even if the ill-conditioning is removed by centering,
there may exist still high levels of multicollinearity. Such difficulty is overcome by orthogonal
polynomials.
The classical cases of orthogonal polynomials of special kinds are due to Legendre, Hermite and
Tehebycheff polynomials. These are continuous orthogonal polynomials (where the orthogonality
relation involve integrating) whereas in our case, we have discrete orthogonal polynomials (where the
orthogonality relation involves summation).
Analysis:
Consider the polynomial model of order k is one variable as
2
0 1 2 ... , 1,2,..., .
k
i i i k i i
y x x x i n
    
      
When writing this model as
,
y X  
 
the columns of X will not be orthogonal. If we add another term 1
1 ,
k
k i
x
 
 then the matrix 1
( ' )
X X 
has
to be recomputed and consequently, the lower order parameters 0 1
ˆ ˆ ˆ
, ,..., k
   will also change.

6
6
6
Consider the fitting of the following model:
0 0 1 1 2 2
( ) ( ) ( ) ... ( ) , 1,2,...,
i i i i k k i i
y P x P x P x P x i n
    
      
where ( )
u i
P x is the th
u order orthogonal polynomial defined as
1
0
( ) ( ) 0, , , 0,1,2,...,
( ) 1.
n
r i s i
i
i
P x P x r s r s k
P x

  


In the context of ,
y X  
  the X  matrix, in this case, is given by
0 1 1 1 1
0 2 1 2 2
0 1
( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( )
k
k
n n k n
P x P x P x
P x P x P x
X
P x P x P x
 
 
 

 
 
 


   

.
Since this X -matrix has orthogonal columns, so '
X X matrix becomes
2
0
1
2
1
1
2
1
( ) 0 0
0 ( ) 0
'
0 0 ( )
n
i
i
n
i
i
n
k i
i
P x
P x
X X
P x



 
 
 
 
 

 
 
 
 
 
 





   

.
The ordinary least squares estimator is 1
ˆ ( ' ) '
X X X y
 
 which for j
 is
1
2
1
( )
ˆ , 0,1,2,...,
( )
n
j i i
i
j n
j i
i
P x y
j k
P x
 

 


and its variance is obtained from 2 1
ˆ
( ) ( ' )
V X X
  
 as
2
2
1
ˆ
( )
( )
j n
j i
i
Var
P x




 
 

.
When 2
 is unknown, it can be estimated from the analysis of variance table.

7
7
7
Since 0 ( )
i
P x is a polynomial of order zero, set it as 0 ( ) 1
i
P x  and consequently
0
ˆ ŷ y
   .
The residual sum of squares is
res
1 1
ˆ
( ) ( ) .
k n
T j j i i
j i
SS k SS P x y

 
 
   
 
 
The regression sum of squares is
1
2
1
2
1
ˆ ˆ
( ) ( )
( )
.
( )
n
reg j j j i i
i
n
j i i
i
n
j i
i
SS P x y
P x y
P x
 




 
 
 




This regression sum of squares does not depend on other parameters in the model.
The analysis of variance table, in this case, is given as follows
Source of variation Degrees of freedom Sum of squares Mean squares
0
̂ 1 0
ˆ
( )
SS  -
1
̂ 1 1
ˆ
( )
SS  1
ˆ
( )
SS 
2
̂ 1 2
ˆ
( )
SS  2
ˆ
( )
SS 
   
ˆk
 1 ˆ
( )
k
SS  ˆ
( )
k
SS 
Residual 1
n k
  res ( ) (by subtraction)
SS k res
SS
______________________________________________________________________________
Total n T
SS
_____________________________________________________________________________
If we add another term 1 1
( )
k i k
P x 
  in the model, then the model is
0 0 1 1 1 1
( ) ( ) ... ( ) ; 1,2,...,
i i i k k i i
y P x P x P x i n
   
 
     
then we just need 1
ˆk
  and that can be obtained as
 
1
1
1
2
1
1
( )
ˆ .
( )
n
k i i
i
k n
k i
i
P x y
P x










8
8
8
Notice that:
 We need not to bother for other terms in the model.
 Simply concentrate on the newly added term only.
 No re-computation of 1
( ' )
X X 
or any other ˆ ( 1)
j j k
   is necessary due to orthogonality of
polynomials.
 Thus higher-order polynomials can be fitted with ease.
 Terminate the process when a suitably fitted model is obtained.
Test of significance:
To test the significance of the highest order term, we test the null hypothesis
0 : 0.
k
H  
This hypothesis is equivalent to 0 : 0
k
H   in polynomial regression model.
We would use
0
1
0
( )
( )/( 1)
ˆ ( )
( )/( 1)
~ (1, 1) under .
reg k
res
n
k k i i
i
res
SS
F
SS k n k
P x y
SS k n k
F n k H




 

 
 

If the order of the model is changed to ( )
k r
 , we need to compute only r new coefficients. The
remaining coefficients 0 1
ˆ ˆ ˆ
, ,..., k
   do not change due to the orthogonality property of polynomials. Thus
the sequential fitting of the model is computationally easy.
When i
X are equally spaced, the tables of orthogonal polynomials are available, and the orthogonal
polynomials can be easily constructed.
First 7 orthogonal polynomials are as follows:
Let d be the spacing between levels of x and  
j
 be the constants chosen so that polynomials will
have integer values. The tables are available.

9
9
9
0
1 1
2 2
2 2
3 2
3 3
4 2 2 2 2
4 4
( ) 1
( )
( ) 1
( )
12
( ) 3 7
( )
20
( ) 3 13 3( 1)( 9)
( )
14 560
i
i
i
i
i
i i
i
i i
i
P x
x x
P x
d
x x n
P x
d
x x x x n
P x
d d
x x x x n n n
P x
d d






 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
   
 
 
 
   
   
 
 
 
 
 
    
   
  
  
   
   
  

   
   
5 3
2 4 2
5 5
6 4 2
2 4 2
6 6
2 2
( ) 5 1
( ) 7 15 230 407)
18 1008
( ) 5 1
( ) 3 31 5 110 329)
44 176
5
( 1)(
14784
i i i
i
i i i
i
x x x x x x
P x n n n
d d d
x x x x x x
P x n n n
d d d
n n





 
  
     
     
 
     
     
 
 
   
     
     
     
     


   2
9)( 25)
n

 

An example of the table for 5
n  is as follows:
________________________________________________________
i
x 1
P 2
P 3
P 4
P
________________________________________________________
1 -2 2 -1 1
2 -1 -1 - 2 -4
    
5
 
2
1
( )
n
j i
i
P x

 10 14 10 70
 1 1
5
6
35
12
______________________________________________________
The orthogonal polynomials can also be constructed when x’s are not equally spaced.

10
10
10
Piecewise polynomial (Splines):
Sometimes it is exhibited in the data that a lower order polynomial does not provide a good fit. A possible
solution in such a situation is to increase the order of the polynomial, but it may always not work. The
higher-order polynomial may not improve the fit significantly. Such situations can be analyzed through
residuals, e.g., the residual sum of squares may not stabilize, or the residual plots fail to explain the
unexplained structure. One possible reason for such happening is that the response function has different
behaviour in different ranges of independent variables. This type of problems can be overcome by fitting
an appropriate function in different ranges of the explanatory variable. So polynomial will be fitted into
pieces. The spline function can be used for such fitting of the polynomial in pieces.
Splines and knots:
The piecewise polynomials are called splines. The joint points of such pieces are called knots. If the
polynomial is of order k , then the spline is a continuous function with ( 1)
k  continuous derivatives.
For this, the function values and first ( 1)
k  derivatives agree at the knots.
Cubic spline:
For example, consider a cubic spline with h knots. Suppose the knots are 1 2 ... h
t t t
   and cubic spline
has continuous first and second derivatives at these knots. This can be expressed as
3
3
0 1
( ) ( ) ( )
h
j
oj i i
j i
E y S x x x t
  
 
   
 
where
if 0
( )
0 if 0.
i i
i
i
x t x t
x t
x t

  

  
 

It is assumed that the position of knots are known. Under this assumption, this model can be fitted using
the usual fitting methods of regression analysis like least-squares principle.
In case, the knot positions are unknown; then they can be considered as unknown parameters which can
be estimated. But in such situation, the model becomes non-linear, and methods of non-linear regression
can be used.

11
11
11
Issue of number and position of knots:
It is not so simple to know the number and position of knots in a given set of data. It is tried to keep the
number of knots as minimum as possible, and each segment should have minimum four or five data
points. There should not be more than one extreme point and one point of inflexion in each segment. If
such points are to be accommodated, then it is suggested to keep the extreme point in the center of the
segment and point of inflexion near the knots.
It is also possible to fit the polynomials of different orders in each segment and to impose different
continuity restrictions at the knots. Suppose it is to be accomplished in a cubic spline model. If all
( 1)
h  pieces of the polynomial are cubic, then a cubic spline model without continuity restrictions is
3 3
0 1 0
( ) ( ) ( )
h
j j
oj i i
j i j
E y S x x x t
  
  
   
 
where
0 1 if 0
( )
0 if 0.
i
x
x t
x



  


If the term ( )j
ij i
x t
 
 is in the model, then th
j derivative of ( )
S x at i
t is discontinuous.
If the term ( )j
ij i
x t
 
 is not in the model, then th
j derivative of ( )
S x is continuous at i
t .
So the model is fitted better when required continuity restrictions are fewer because then more parameters
will be included in the model.
If more continuity restrictions are needed, then it indicates that the model is not well fitted, but the finally
fitted curve will be smoother. The test of hypothesis in multiple regression model can be used to
determine the order of polynomial segments and continuity restrictions.
Example:
Suppose there is only one knot at t in a cubic spline without continuity restrictions given by
2 3 0 1 2 3
00 01 02 03 10 11 12 13
( ) ( ) ( ) ( ) ( ) ( ) .
E y S x x x x x t x t x t x t
       
   
            
The term involving 10 11
,
  and 12
 are present in the model, so ( )
S x , its first derivative '( )
S x and
second derivative "( )
S x are not necessarily continuous at t . Next question arises now is to judge that the
quality of fit will reduce the quality of fit. This can be done by the test of hypothesis as follows:

12
12
12
0 10
: 0
H   tests the continuity of ( )
S x
0 10 11
: 0
H  
  tests the continuity of ( )
S x and '( )
S x
0 10 11 12
: 0
H   
   tests the continuity of ( ), '( )
S x S x and "( )
S x .
The test 0 10 11 12 13
: 0
H    
    indicates that cubic spline fits data better than a single cubic
polynomial over the range of the explanatory variable x .
This approach is not satisfactory if the knots are large in number as this makes '
X X ill-conditioned. This
problem is solved by using cubic B spline which are defined as
3
4
4
4
1
( )
( ) , 1,2,..., 4
( )
( ) ( ) ( )
m j
i
j
i i
j i
j m
m i
h
i i
i
x t
B x i h
t t
E y S x B x



 
 


 
 

 
  
 

 
 
 
 



where ' ( 1,2,..., 4)
i s i h
   are parameters to be estimated. There are eight more knots- 3 2 1 0
t t t t
  
  
and 1 2 3 4.
h h h h
t t t t
   
   Choose 0 min 1 max
, h
t x t x

  and other knots arbitrarily.
Polynomial models in two or more variables:
The techniques of fitting of the polynomial model in one variable can be extended to the fitting of
polynomial models in two or more variables.
A second-order polynomial is more used in practice, and its model is specified by
2 2
0 1 1 2 2 11 1 22 2 12 1 2
y x x x x x x
      
       .
This is also termed as response surface. The methodology of response surface methodology is used to fit
such models and helps in designing an experiment. This type is generally covered in the topics in the
design of experiment.

Chapter12-Regression-PolynomialRegression.pdf

More Related Content

Similar to Chapter12-Regression-PolynomialRegression.pdf (20)

Recently uploaded (20)

Chapter12-Regression-PolynomialRegression.pdf