Stepwise Logistic Regression - Lecture for Students /Faculty of Mathematics and Informatics/

© Experian Limited 2007. All rights reserved. Experian and the marks used herein are service marks or registered trademarks of Experian Limited.
Other product and company names mentioned herein may be the trademarks of their respective owners. No part of this copyrighted work
may be reproduced, modified, or distributed in any form or manner without the prior written permission of Experian Limited.
Confidential and proprietary.
Stepwise Logistic Regression
Lecture for FMI Students 27.05.2010
Alexander Efremov

© Experian Limited 2007. All rights reserved.
Confidential and proprietary. 2
Agenda
Introduction
Applications of the Logistic Regression
System Identification & Stepwise Regression
Part I. Logistic Regression Model Development
Logistic Model
Maximum Likelihood Estimator
Potential Problems
Model Analysis and Validation
Part II. Stepwise Logistic Regression (SWR)
Basic Idea
SWR Algorithm
Potential Problems
Summary

Agenda
Introduction
Logistic Model
Potential Problems
Basic Idea
SWR Algorithm
Potential Problems
Summary

Introduction
Medicine – diagnostics, modeling of disease growth, treatment effect
Psychology – learn process modeling, psychological tests evaluation
Economics – risk analysis, countries debt investigation, occupational choices
Marketing – products consumption, retailers actions effect
Criminology – risk factors for performing of criminal act
Sociology – employment, graduation, vote analysis
Ecology – modeling population growth
linguistics – language changes
Chemistry – reaction models
Media – news effects, copycat reaction
Finance – credit scoring, fraud detection
Physics, Biology, etc.
The Logistic Model

Introduction
System Under Investigation
Individuals /rough data/ => System => Model
=>
=>

Introduction
System Identification Stages

Agenda
Introduction
Logistic Model
Potential Problems
Basic Idea
SWR Algorithm
Potential Problems
Summary

Logistic Model
Linear relation Logistic relation

k
kyˆ
ky
N
– index of current individual – intercept
– number of observations – the i+1-th model parameter
– dependent variable – the i-th independent variable
/prob. of good/
– model output – i-th independent variable
/predicted prob. of good/
Logistic Model
Logistic Relation – General Form “Linear” Log. Regression Model
k
k
M
M
k
e
e
y
+
=
1
ˆ
kMk
e
y −
+
=
1
1
ˆ
knnkk xxM ,,110 ... θθθ +++=
)...( ,,110
1
1
ˆ
knnk xxk
e
y θθθ +++−
+
=
knnky
y
xx
k
k
,,110ˆ1
ˆ
...ln θθθ +++=−
0θ
iθ
kix ,
ni ,1=
Nk ,1=

Logistic Model
Notation
Parameters vector
Regression vector
Logistic model
1+
∈ n
Rθ
1+
∈ n
k Rϕ
T
n ]...[ 10 θθθθ =
T
knkk xx ]...1[ ,,1=ϕ
θϕθθθ T
kknnk
ee
y xxk
−+++−
+
=
+
=
1
1
1
1
ˆ )...( ,,110

Residual
The Residual
kkkk eye
e
y T
k
+=+
+
=
−
ˆ
1
1
θϕ



=−
=−
=−=
0,ˆ
1,ˆ1
ˆ
for
for
kk
kk
kkk
yy
yy
yye
Sources of Uncertainty
Unavailable significant factors
Simplified relations
Time-varying performance
Database errors
Fraud

Agenda
Introduction
Logistic Model
Potential Problems
Basic Idea
SWR Algorithm
Potential Problems
Summary

Cost Function
Model output
Likelihood contribution
Likelihood function
Log-likelihood function
Maximum Likelihood Criterion
kk y
k
y
kk yyl −
−= 1
, )ˆ1(ˆθ
θ
θ
θ
θ
LL ln2minlnmax −⇔
∏
=
=
N
k
klL
1
,θθ
∑
=
−−+=
N
k
kkkk yyyyL
1
))ˆ1ln()1(ˆln(ln θ
)|1(ˆ kkk yPy ϕ==

Cost Function /-2 Log L/ for a Real Life Case

Tailor Series Expansion
Cost Function Models
Linear model
Quadratic model
)()()1( ˆˆ iii
θθθ ∆+=+
)()()(
ˆ
)(
)( iTiii
gfM θ
θ
∆+=
)()()(
2
1)()()(
ˆ
)(
)()( iiTiiTiii
HgfM θθθ
θ
∆∆+∆+=
3
)()()(
2
1)()()(
ˆ
)(
ˆ )()( OHgff iiTiiTiii
+∆∆+∆+=
∆+
θθθ
θθθ
)(
ˆ
)( iTi
fg
θ
∇=
)(
ˆ
2)( ii
fH
θ
∇=
Cost function
Gradient
Hessian
)(
ˆ
)(
ˆ ln ii
Lf
θθ
−=
?)(
=∆ i
θ
Estimates Update

Gradient Hessian
I-st Order Methods II-nd Order Method
/e.g. Steepest Descent/ /e.g. Newton-Raphson/
gαθ −=∆ gH 1−
−=∆ αθ
[ ] 1
10
+
∂
∂
∂
∂
∂
∂
∈= nTfff
Rg
nθθθ
L
11
2
2
1
2
0
2
1
2
2
1
2
01
2
0
2
10
2
2
0
2
+×+
∂
∂
∂∂
∂
∂∂
∂
∂∂
∂
∂
∂
∂∂
∂
∂∂
∂
∂∂
∂
∂
∂
∈


















= nn
fff
fff
fff
RH
nnn
n
n
θθθθθ
θθθθθ
θθθθθ
L
MOMM
L
L
θ
(0)
1
2
θ*θopt
1
2
θ
(0)
θ*
θopt

Steepest Newton-
Descent Raphson
(NR)
NR with NR with
Line Search Quadratic
Interpolation
1
2
θ
(0)
θ*
θopt
θ
(0)
1
2
θ*θopt
gαθ −=∆
gH 1−
−=∆ αθ
gH 1* −
−=∆ αθ
gH 1* −
−=∆ αθ
θ
(0)
1
2
θ*θopt
θ
(0)
1
2
θ*θopt

Agenda
Introduction
Logistic Model
Potential Problems
Basic Idea
SWR Algorithm
Potential Problems
Summary

Numerical Problems
Matrix inversion, hence SVD, EVD, QR, etc.
Local Minima
Potential problems
Model Overfitting
αθθ −=+ )()1( ˆˆ ii 1−
H g
-2lnL
k
y2,k
yk
1,ky

Agenda
Introduction
Logistic Model
Potential Problems
Basic Idea
SWR Algorithm
Potential Problems
Summary

Frequently Used Statistics for Model Analysis
Individual Estimate Measures
Standard error
Wald statistic
p-value
Overall Model Measures
Coefficient of determination (R2)
generalized R2
gen. max. resc. R2
Cost function
2
1
ˆ)ˆ(
~2
ˆ
2
2
ˆ
2
χ
θθ
σ
θ
σ
θθ
i
i
i
ii
iW ==
−
N
LL
eR
θθ ˆln0
ˆln
2
12
−
−=
1
0
ˆln2
1 −−= N
L
esR
θ
Rs
R
mR
22
=
)(
ˆ
)(
ˆ ln2 ii
Lf
θθ
−=
iH
i
)][diag( 1
ˆ
−
=θ
σ
2
1Pr χ>
χ
p-value
WWi

Frequently Used Statistics for Model Analysis
Modified criteria
Akaike Information Criterion (AIC)
Schwarz Criterion (SC)
Minimum Description Length (MDL), Final Prediction Error (FPE), etc.
Model Validation
Data split into development and validation samples
nLAIC 2ln2 ˆˆ +−= θθ
)1ln(ln2 ˆˆ −+−= NnLSC θθ
AIC
-2lnL

Agenda
Introduction
Logistic Model
Potential Problems
Basic Idea
SWR Algorithm
Potential Problems
Summary

Part II. Stepwise Logistic Regression
Stepwise Logistic Regression – Basic Idea
xo, xe – sets of all variables, out/entered in the model
xoi, xei – the most/less significant variable
SLE – Significance Level to Enter
SLS – Significance Level to Stay
SWR

Available information

1
Initialization

Forward Selection
1
2

1
2 3
Forward Selection

2 3
Backward Elimination

Agenda
Introduction
Logistic Model
Potential Problems
Basic Idea
SWR Algorithm
Potential Problems
Summary

Step 0. Initialization
Logistic model
1. Intercept Model
2. Full model
3. One Factor Model
Check for Enter
Score Chi-Sq for all potential models
Maximum Score Chi-Square
p-value & threshold
Model Determination (Optimization)
θϕT
ke
yk
−
+
=
1
1
ˆ
ii
T
ii gHgS 1−
=
R∈θ 1=kϕ
1+
∈ n
Rθ T
knkk xx ]1[ ,,1 K=ϕ
i
i
Smaxarg1 =l
SLEvalue-p 1
<l
T
kk x ]1[ ,1l=ϕ2
R∈θ

Step 1. Forward Selection
1. Check for Enter
Score Chi-Square of all potential models
Maximum Score Chi-Square
p-value & threshold
2. Model Determination (Optimization)
3. Statistics for Model Analysis
standard error
Wald statistic & p-value
ii
T
ii gHgS 1−
=
i
i
i Smaxarg=l
SLEvalue-p <il
T
kkk i
xx ]1[ ,,1 ll K=ϕ1+
∈ i
Rθ

Step 1. Forward Selection
3. Statistics for Model Analysis (part 2)
Coefficients of determination
Cost function
Modified criteria

SWR

Step 2. Backward Elimination
1. Check for Leave
Wald statistic & p-value of all potential models
p-value & threshold
2. Model Determination (Optimization)
3. Statistics for Model Analysis
standard error
Wald statistic & p-value
T
kkkkk ijj
xxxx ]1[ ,,,, 111 llll KK +−
=ϕi
R∈θ
SLLvalue-pmax >il

3. Statistics for Model Analysis (part 2)
Coefficients of determination
Cost function
Modified criteria
Step 2. Backward Elimination

Agenda
Introduction
Logistic Model
Potential Problems
Basic Idea
SWR Algorithm
Potential Problems
Summary

Potential problems in the Stepwise Regression
Local Minima & Initial Conditions
Numerical Problems /SVD, EVD, QR, etc./
Model Overfitting

Summary
Introduction
Logistic Model
Potential Problems
Basic Idea
SWR Algorithm
Potential Problems
Summary

© Experian Limited 2007. All rights reserved. Experian and the marks used herein are service marks or registered trademarks of Experian Limited.
Other product and company names mentioned herein may be the trademarks of their respective owners. No part of this copyrighted work
may be reproduced, modified, or distributed in any form or manner without the prior written permission of Experian Limited.
Confidential and proprietary.
Lecture for FMI Students 27.05.2010
Alexander Efremov
Thank You!
http://anp.tu-sofia.bg/aefremov/index.htm

Stepwise Logistic Regression - Lecture for Students /Faculty of Mathematics and Informatics/

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Stepwise Logistic Regression - Lecture for Students /Faculty of Mathematics and Informatics/ (20)

Recently uploaded (20)

Stepwise Logistic Regression - Lecture for Students /Faculty of Mathematics and Informatics/