Use of Linear Regression in Machine Learning for Ranking

IJSRD - International Journal for Scientific Research & Development| Vol. 1, Issue 5, 2013 | ISSN (online): 2321-0613
All rights reserved by www.ijsrd.com 1135
Abstract-- Machine Learning is a growing field today in AI.
We discuss use of a Supervised Learning algorithm called as
Regression Learning in this paper for ranking. Regression
Learning is used as Prediction Model. The values of
dependent variable are predicted by Regression Model based
on values of Independent Variables. By Regression
Learning if after Experience E, program improves its
performance P, then program is said to be doing Regression
Learning. We chose to use Linear Regression for Ranking
and discuss approaches for Rank Regression Model
Building by selecting best Ranking parameters from
Knowledge and confirming their selection further by
performing Regression Analysis during Model building.
Example is explained. Analysis of Results and we discuss
the Combined Regression and Ranking approach how it is
better for enhancing use of Linear Regression for Ranking
purpose. We conclude and suggesting future work Ranking
and Regression.
Keywords: Regression Learning, Ranking, Models
I. INTRODUCTION
THIS paper presents a use of Regression Learning in
Machine Learning for Ranking. The machine learning is a
growing field in AI provides Regression Learning as a
Supervised Learning.[1] The regression model can be with
Single and Multiple Variables.[2][3][4]
Single Variable Regression
Y=a+b*X (1)
Y=Dependent Variable X = Independent Variable
Multiple Variables Regression
Y=a+b1*X1+b2*X2+…bn*Xn (2)
Using there Models for Ranking we discuss various
approaches for Rank model building. Lets start by knowing
how the regression model is built with best parameters and
coefficients.
How to find coefficients of regression?
1. Normal equation when features are less than 10000.
W= (XT* X)-1*XT* Y
2. Gradient Descent for all cases.
Gradient descent algorithm
Repeat until convergence [5]
{ For j = 1 to n
{ Temp j: = M j+ α * 1/N ∑Ni=1(Y(i)-
H( M,X,i))*Xj}
Correct Simultaneous Update
For j= 1 to n
{
M j= Temp j
} } [5]
II. LITERATURE SURVEY
A. All Possible Subset Regression
Method of selecting the variables for inclusion in the
regression model that considers all possible combinations of
independent variables. Eg. 4 variables. The technique would
estimate all possible regression models with 1,2,3,4
variables. The technique would then identify the model(s)
with best prediction accuracy.
B. Backward Elimination
Method of selecting variables for inclusion in the model that
starts by including all variables in model and then
eliminating those variables not making significant
contribution to prediction.
C. Prediction without an Independent Variable
Mean of dependent variable values gives value of the
prediction without Independent variable.
D. Over fitting because of more features
Fig. 1: Over fitting because of growing features
E. Regularization to avoid overfitting
[11] L2 Regularization:
E(w)=1/2*∑n-0
m-1
(Yn-H(Xn,w))2
+ lambda/2*||w||2
L1 Regularization:
E(w)=1/2*∑n-0
m-1
(Yn-H(Xn,w))2
+ lambda*|w|1
L0 Regularization:
E(w)=1/2*∑n-0
m-1
(Yn-H(Xn,w))2
+ lambda∑n-0
m-
1
del(w≠0)
Use of Linear Regression in Machine Learning for Ranking
Yashavant S. Ingle1
Prof. Anil Mokhade2
S.P.B.Patel Engineering College, Mehsana, Gujarat

(IJSRD/Vol. 1/Issue 5/2013/025)
L0-regularization is feature selection for linear
models.
L1- and L2-regularizations approximate feature
selection and regularize the function.
L2 Regularized Normal Equation:
W= (XT
*X+lambda*I)-1
XT
* Y –[11]
F. Role of Correlation Coefficient
1) Coefficient of determination R2
Measure of the proportion of the variance of the
dependent variable about its mean that is explained
by indicator or predictor variables, The coefficient
can vary between 0 and 1.Researcher says if
regression model is properly estimated and applied
higher the value of R2,
the greater the explanatory
power of the regression equation, and therefore
better prediction of the dependent variable.
R2
=Sum of squares regression /Total sum of
squares
2) Adjusted coefficient of determination (adjusted R2
)
Modified measure of the coefficient of
determination that takes into account the number of
independent variables included in the regression
equation and the sample size. Although the
addition of independent variables will always cause
the coefficient of determination to rise, the adjusted
coefficient of determination may fall if the added
independent variables have little explanatory power
or if the degrees of freedom become too small. This
statistic is quite useful for comparison between
equations with different numbers of independent
variables, differing sample sizes.
G. Degrees of freedom (df)
Value calculated from the total number of observation
minus number of estimated parameters. These parameters
explain the restriction on the data because, once made, they
determine population from which data is assumed to have
been drawn.
If degrees of freedom is small , the resulting prediction may
be less generalized because all but a few observations were
incorporated in the prediction. Conversely, a large degrees
of freedom value indicates the prediction is fairly robust
with regard to being representative of the overall sample of
respondents.
H. P-value
Using the P- value Alone
1) Oftentimes researchers determine significance without
the usage of a critical value. The chart below is an
example of the criterion used to determine the statistical
significance of a given P- value.
Less than
0.01
Strong evidence against Null Hypothesis,
Very statistically significant
0.01 to 0.05 Some evidence against Null Hypothesis
Greater than
0.05
Insufficient evidence against Null
Hypothesis
Table 1: statistical significance of a given P- value
Statisticians who favor this approach argue that because no
significance level has been specified, the burden of
determining significance is left to the researcher.
I. General Regression Model Building Procedure
For building any Regression Model we follow a Procedure
for selecting best Parameters for Model.
To build a regression model we need to select regression
parameters for model .we need to form a model by selecting
the dependent and independent variables.
This process is as follows.[7]
1. We select output parameters that are suitable for our
purpose by knowledge
2. We will select input parameters by knowledge. We
select the input parameters by knowing the relationship
between output parameter and input parameters.
3. Regression analysis to decide the best parameters for
model
1. We build Single Variable Regression Model from
each selected parameters. Do the training of Model
using x, y data. We note their p-values and R-
squares, adjusted R squares.
Here, we select the Variables whose p-values are
<Significance otherwise we reject the Variables.
2. We can directly go for backward elimination or all
possible subset regression to get best model or We build
Multiple Variable regression model from the selected
variables in step given in 3A.We again do the training
and check for the p-values of all variables in model. We
also note R-square, Adjusted R-square i.e. correlation
coefficients.
Here we Select the variables from the model whose p-values
are < significance otherwise remove variables for model
whose p-values are >significance.
If some variables are dropped then we do the training of our
model and see if R-Square is falling too much or little. If R-
square is not falling much our Final model is declared.
Otherwise we may think about few other parameters and
think of including them after their evaluation discussed in 1,
2, and 3.
If we are having very good variables in our model with
sufficient R-Square required for our model. Then We
declare it as our final regression model.
Note:
1. If P- value<=Significance - the variable will
produce significant regression estimates otherwise
estimates will be erroneous.
2. If the R-Square is suggesting the % change in
dependent variable by change in independent
variable present in model.
. Correlation
Coefficient
R2
Suggested % Change in dependent
variable by change in independent
variable/s present in Model by R2
0.0 to 0.2 Very weak
0.2 to 0.4 Weak
0.4 to 0.7 Medium
0.7 to 0.9 Strong
0.9 to 1.0 Very strong
Table 2: Correlation Coefficient R2
Meaning
J. Regression Raking Model Building
The Linear Regression learning when used for ranking we
need to think on what parameters ranking is dependent. The
ranking may depend upon Single or Multiple Parameters
called as Ranking Parameters. So, initially we have two

Regression Models as below
Rank depends upon Single Variable
Rank=a + b*RP Model –I
RP = Ranking Parameter
Rank depends upon Multiple Variables
Rank=a+b1*RP1+b2*RP2+…..bn*RPn Model-II
RP1, RP2, RPn are Ranking Parameters for model
Both the models will be regression models to be trained by
data as-
Rank depends upon Single Variable
trainingdata<Rank,RP> for Model –I
Rank depends upon Multiple Variables
trainingdata<Rank, RP1, RP2,…..,RPn> for Model-II
Gradient Descent is used to minimize error while training to
get best regression coefficients. If data is moderate we use
batch gradient descent otherwise if data is very large we use
stochastic gradient descent.
The Model for ranking is built by selection of the best
Ranking Parameters for Regression by intuition or
knowledge further the model can be reevaluated using
regression analysis depending on training and final best
regression parameters can be selected for Regression Model.
This training will help to get the best regression coefficients
for Model by minimizing the mean square error using
gradient descent algorithm. Thus our model is complete for
Ranking with the training.
The regression model can be perfect regression model if the
mean squared error is zero. So, the prediction for rank will
be perfect with perfect regression model. But, this is not
always true because often the mean square error will not be
zero. The model can be nearly perfect model or erroneous.
This is the drawback of linear regression when used for
Ranking. The error in the regression will produce error in
the Rank. Though the error is minimized such models will
not always predict the correct ranks.
Now there is one more thing i.e. Think of when we need to
predict rank we have model but no direct value for Ranking
Parameter is available to us. In this Case we need to
construct another regression model for prediction of the
Ranking Parameters selecting the best independent variables
on which the specific Ranking Parameter depends. Thus we
can get the predicted value of Ranking parameter from this
model and submit the result in the our Ranking Model I or
II. Thus the Rank will be predicted by Model I or II.
III. RELATED WORK
A. Discussion on Regression Parameter Selection
Let’s discuss the best regression parameter selection process
for any regression model.
The parameter selection for building any regression model is
intuitive, knowledgeable task initially, followed by training
and regression analysis to decide the best parameters for
regression.
Thus the parameter selection is performed initially
intuitively and using knowledge about dependency of
dependent variable on independent variables. And the
variables are selected for regression that we think will truly
contribute to decide value of dependent variable.
Step I- We build single regression models with the selected
variables. We do the training of these models and observe
the P-values and Adjusted R-square. If our chosen
confidence is say 90% our significance will be 100-90
=10%. So, we select the variables whose P-values are less
than 10% because our estimate will be satisfying the
confidence criteria of 90%. We will be 90% sure about the
result if the particular variable is selected for regression.
Otherwise if P-value for the variable is greater than 10% the
estimates will be erroneous, so we reject the variable.
Step II-Further we build the multiple regression model if
selected variables from Step I are more than one. We
observe the P-values and Adjusted R-square values for
variables. First we reject the variables whose P-values are
far greater than the required Significance. We redo the
regression training and observe the change in the P-values
and Adjusted R-Square. We reject the variables whose P-
values are greater than the required Significance. We redo
the regression and we observe P-values and adjusted R-
square. Now we need to see Adjusted R-Square very
critically because if Adjusted R-square is fallen too much
after removing the variables whose P-values are slightly
greater than required Significance we can think of adding
them for regression to improve the Adjusted R-Square
provided the Standard Error don’t rise too much and P-value
of added variable don’t go far than required Significance.
Thus we complete our selection of best variables for
Regression.
B. Discussion on Ranking Parameter Selection
We need to select the best Ranking Parameters by intuition
and knowledge and perform the Regression Training and
Regression Result analysis explained above to keep best
Ranking Parameters for our Rank by Regression Model.
Further we can build the Regression Model to predict each
of the Ranking Parameters by regression. We need to select
the best independent variables by intuition and knowledge
on which the Ranking Parameter depends. We do the
regression training and Result Analysis explained above and
selects best regression parameters on which particular
Ranking Parameter depends. We do build regression model
for each Ranking Parameter.
We will now are equipped to use of Linear Regression for
anything and so for Ranking. We should keep in mind that
we need to use the methodologies explained to any Ranking
in real life example. We will now discuss the example of
Ranking by Regression and do the result analysis.
IV. USE OF REGRESSION FOR RANKING
A. Ranking depends on Single Variable
Rank =a + b*RP Model –I
Example:
Rank =a + b*CGPA (1)
Here CGPA is only Ranking Parameter i.e. RP
Now we need to also build a prediction model for CGPA
by selecting best Regression Parameters to predict CGPA.
Intuitively or with knowledge we can think of CGPA
depends upon these parameters
CGPA=a+b1*GateScore+b2*Health+b3*Attendance+b4*
AvgU1+b5*AvgU2+b6*AvgENDSEM (2)
Gate Score, Health, Attendance, AvgU1, AvgU2,
AvgENDSEM
And so on. We built the single Regression Models with each
Individual Parameter selected and select those parameters

whose P-values are less than 0.05 i.e. confidence>=95%
about correctness of estimate and keep note of R2
for all
parameters.[10]
We then build multiple regression model of all selected
variables which yield significant estimates and follow
backward elimination method. We can also directly use
backward elimination without building single variable
model for each parameter but it’s advisable if parameters are
less to build single variable models and know contribution
of each parameter R2
and P-values. This will help to decide
up on inclusion of variables in multivariate regression
model.
We can also use all Possible Subset Method to get best
model but when features are more the backward elimination
is preferred. Following any one of backward elimination or
all subset method we build model to predict CGPA.
Applying the model building procedure we get following
model…
CGPA=a+ b1*AvgU2+b2*AvgENDSEM (3)
B. Ranking depends on Multiple Variables
Rank =a + b1*RP1+b2*RP2+…..+bn*RPn------Model –II
Example: Selection of candidates in department by
ranking them.
Rank depends up on many ranking parameters such as
Education School Level
1-IIT 2-NIT 3-State GOV 4-PrivateA 5-PrivateB
Degree%
>=90 – 1 ,>=80-2, >=75-3,>=60-4 ,<60- 5
Extracurricular
International-1 National-2 State-3 District-4 Interschool -
5
Interschool IIT/NIT-2
Gate Score- out of 100
Gate Percentile- out of 100
Final model of selected ranking parameters,
Rank=alpha+b1*EduSchoolLv+b2*Degree%+b3*sports+
b4*GateScore (1)
Fig. 1: Weighted Ranking Models
Ranking depends on Multiple Variables
Rank =a + b1*w1*RP1+b2*w2*RP2+…..
…… +bn*wn*RPn Model –III
Example: Selection of candidates in department by
ranking them by assigning weights to Ranking
Parameters.
Rank depends up on many ranking parameters such as
Education School Level
1-IIT 2-NIT 3-State GOV 4-PrivateA 5-PrivateB
Degree%
>=90 – 1 ,>=80-2, >=75-3,>=60-4 ,<60- 5
Extracurricular
International-1 National-2 State-3 District-4 Interschool -
5
Interschool IIT/NIT-2
Gate Score- out of 100
Gate Percentile- out of 100
Final model of selected ranking parameters,
Rank=alpha+b1*w1*EduSchoolLv+b2*w2*Degree%+b3
*w3*sports+ b4*w4*GateScore (1)
Fig. 2: Weighted Ranking Final Models
Ranking depends on Multiple Variables
C. Error in regression affects ranking
Error in Regression affects ranking [6] so Combined Rank
and Regression approach to find rank yields better result.
D. Supervised Regression:
This aggregate loss L (w,D) is given by:
L(w,D) =1/|D| * ∑ ((x,y,q)∈ D) l(y, f(w, x))
Here, l(y, y′) is a loss function on a single example, defined
on the true target value y and the predicted value y′, and f(w,
x) returns the predicted value y′ using the model represented
by w.
The general formulation for L2 regularized empirical risk
minimization is:
Min w∈Rm L (w,D) + lambda/2 *||w||2
2 (1)
That is, we seek a linear model represented by weight vector
w that both minimizes the loss of w on the training data D
and also has low model complexity, represented by the
squared norm of the weight vector. The parameter lambda
controls the amount of regularization performed; tuning this
parameter trades off the (possibly conflicting) goals of
finding a model that is simple and finding a model that fits
the data with little loss.
E. Supervised Ranking:
The goal of a supervised ranking method is to learn a model
w that incurs little loss over a set of previously unseen data,
using a prediction function f(w, x) for each previously
unseen feature vector in the set, with respect to a rank-based
loss function.

A simple and successful approach to learning to rank is the
pairwise approach, used by RankSVM. In this pairwise
approach, the original distribution of training examples D is
expanded into a set P of candidate pairs, and learning
proceeds over a set of pairwise example vectors.
Formally, the set of candidate pairs P implied by a fixed
data set D is the set of example pairs (a, ya, qa), (b, yb, qb)
drawn from all examples in D where ya 6= yb and qa = qb.
When ya > yb, then a is preferred over b (or, equivalently,
ranked better than b). In general for fixed D, |P| is O(|D|2
),
but sharding by query identifier can result in |P| € |D|2
.
With P defined, we find w by optimizing a pairwise
objective function:
Min w∈Rm L(w,P) + lambda/2 *||w||2
2 (2)
Here, the loss function L(w, P) is defined over pairwise
difference vectors from
P: L(w, P) =1/|P| ∑ ((a,ya,qa),(b,yb,qb))∈ P) l(t(ya−yb), f(w, a−b))
The transformation function t(y) transforms the difference of
the labels, and is instantiated differently for different loss
Functions. Standard loss functions l(·, ·) are applicable on
these pairwise difference vectors, given an appropriate
transform t(·).
Squared Loss:
The squared loss for a single predicted value y′ compared
with true label y is given by l(y, y′) = (y − y′)2
. This loss
function is convex. The associated transform function is the
identity function t(y) = y.
Logistic Loss:
The logistic loss function for y € [0, 1] and y′ € [0, 1] is
l(y, y′) = y log y′ + (1 − y) log(1 − y′).
This loss function is convex. The associated prediction
function is
f(w, x) = 1/(1+e−<w,x> ).
F. Combined Rank and Regression CRR Model
To create an optimization problem with terms for regression
loss L(w,D) and pairwise ranking loss
L(w, P). The combined CRR optimization problem is:
Min w€ Rm α * L(w,D) + (1 − α)L(w, P) + lambda/2*
||w||2
2 (3)
Here, the parameter α € [0, 1] trades off between optimizing
regression loss and optimizing pairwise loss. Note that
setting α = 1 recovers the standard regression problem and
setting α = 0 recovers the pairwise ranking problem. Setting
α to an intermediate value forces the optimization to
consider both regression and ranking loss terms.. We have
found that CRR is not overly sensitive to specific values of
α.
Algorithm 1 Combined Regression and Ranking Given:
trade-off parameter α, regularization parameter lambda,
training data D, iterations t
1: w0 ← any initial values
2: for i = 1 to t do
3: pick z uniformly at random from [0, 1]
4: if z < α then
5: (x, y, q) ← RandomExample(D)
6: else
7: ((a, ya, q), (b, yb, q)) ← RandomCandidatePair(P)
8: x ← (a − b)
9: y ← t(ya − yb)
10: end if
11: ni ← 1/lambda
12: wi ← StochasticGradientStep(wi−1, x, y, lambda,ni)
13: end for
14: return wt
V. CONCLUSION
Based on the algorithm of Linear Regression Learning the
Gradient Descent is implemented and best coefficients for
regression are found initially. The pvalue,R2
,Std Dev, Mean
Error, Standard Error, t-value, Confidence Interval,
Confidence, Significance are studied ,computed and verified
correctly.
The models of Linear Regression use are built for real life
examples. The Linear Regression Model for Ranking are
built and tested. We find that if it is perfect regression i.e.
error is zero, ranking by such model is perfect i.e. correct.
But if there is error in regression the error affects Ranking
performance of Model.[8]
Further, the weighted Linear Regression Ranking Model is
built and tested. The results are correct. Performance for
Ranking is ok. But it is concluded that there is error in
regression and when rank is critical error will affect rank.[8]
So, Combined Rank Regression Model Approach is studied
and Results discussed. The CRR has good performance over
Rank by Regression model. Implementation of CRR is also
attempted.
VI. FUTURE WORK
A simple and successful approach to learning to rank is the
pairwise approach, used by RankSVM [9] and several
related methods. [10] Exploring the use of other learning to
rank methods in a combined ranking and regression
framework is left to future work.[8] In this pair wise
approach, the original distribution of training examples D is
expanded into a set P of candidate pairs, and learning
proceeds over a set of pairwise example vectors.
Better approaches discovery to decrease the error further is
left as future work.
ACKNOWLEDGMENT
I am thankful to the Google, YouTube, and Lectures on
Machine Learning for helping me to understand the
approaches for discovering them that excited me think up on
my own and I hope I got them right.
I would like to thank my Project Guide Professor Anil
Mokhade Sir for consistent encouragement for doing
something innovative and guidance.
REFERENCES
[1] ML 1.2 “What is Supervised Learning ?
”,mathematicalmonk,
www.youtube.com/watch?v=Wpxk__Sk2aO
[2] Andrew Ng, ”Linear Regression with one Variable”,
Stanford University, USA
[3] Hair,Black,Babin,Anderson, Tathan, “Multivariate
Data Analysis”, Sixth Edition,Pearson.
[4] ”What is Linear Regression?|how to do it in
matlab|How to use functions in matlab” ,satendra
kumar, www.youtube.com/watch?v=nkhadlenijo
[5] Tom Arnold, Jonathan M. Godbey, “Introducing
Linear Regression: An Example Using Basketball

Statistics”, University of Richmond-E, Claiborne
Robins School of Business, Georgia State University-
Department of Finance
[6] D. Sculley , “Combined regression and Ranking”,
Google, Inc., Pittsburg, PA USA
[7] T. Joachims. Optimizing search engines using
clickthrough data. In KDD ’02: Proceedings of the
eighth ACM SIGKDD international conference on
Knowledge discovery and data mining, 2002.
[8] J. L. Elsas, V. R. Carvalho, and J. G. Carbonell. Fast
learning of document ranking functions with the
committee perceptron. In WSDM ’08: Proceedings of
the international conference on Web search and web
data mining, 2008.
[9] Andrew Ng, Machine Learning Course at
www.Coursera.org Normal Equation Lecture,
https://class.coursera.org/ml-003/lecture/24
[10] P-value,
http://mips.stanford.edu/courses/stats_data_analsys/le
sson_10/Palone.html
[11] Regularization
http://eniac.cs.qc.cuny.edu/andrew/gcml/lecture5.pdf

Use of Linear Regression in Machine Learning for Ranking

More Related Content

What's hot (20)

Viewers also liked (8)

Similar to Use of Linear Regression in Machine Learning for Ranking (20)

More from ijsrd.com (20)

Recently uploaded (20)

Use of Linear Regression in Machine Learning for Ranking