Supervised Learning Ensemble Techniques Machine Learning

Sanjivani Rural Education Society’s
Sanjivani College of Engineering, Kopargaon-423 603
(An Autonomous Institute, Affiliated to Savitribai Phule Pune University, Pune)
NACC ‘A’ Grade Accredited, ISO 9001:2015 Certified
Department of Computer Engineering
(NBA Accredited)
Prof. S. A. Shivarkar
Assistant Professor
Contact No.8275032712
Email- shivarkarsandipcomp@sanjivani.org.in
Subject- Supervised Modeling and AI Technologies (CO9401)
Unit –III: Supervised Learning- Ensemble Techniques

Content
 Ensemble primer- Bias Vs variance trade off
 Generative models vs Non generative models
 Bagging-Random Forest trees
 Voting types
 Boosting -Ada boost, gradient boosting, XGBoost, and stacking, selection
criterion, constraints of ensemble techniques

Ensemble Learning
 Ensemble Learning Techniques in models suffer from bias and/or variance. Bias
is the difference between the predicted value and actual value by the model.
Bias is introduced when the model doesn’t consider the variation of data and
creates a simple model. The simple model doesn’t follow the patterns of data,
and hence the model gives errors in predicting training as well as testing data
i.e. the model with high bias and high variance. You Can also read Covariance
and Correlation In Machine Learning
 When the model follows even random quirks of data, as pattern of data, then
the model might do very well on training dataset i.e. it gives low bias, but it
fails on test data and gives high variance.

Ensemble Learning
 Therefore, to improve the accuracy (estimate) of the model, ensemble learning
methods are developed. Ensemble is a machine learning concept, in which several
models are trained using machine learning algorithms. It combines low performing
classifiers (also called as weak learners or base learner) and combine individual
model prediction for the final prediction.
 Ensemble learning is a machine learning technique involves training
multiple models and combining their predictions.
 The purpose of using ensemble learning is to reduce overfitting and
improve generalization.

Why do Ensemble Methods Work?
Ensemble learning algorithms can improve model performance by
reducing variance

Ensemble Learning Algorithms
 Ensemble learning algorithms are useful when the dataset is large and high-
dimensional.
 Ensemble learning algorithms can be computationally expensive when the
dataset is large.

Bagging
 Bagging, also known as bootstrap aggregation, is the ensemble learning
method that is commonly used to reduce variance within a noisy data set.
 In bagging, a random sample of data in a training set is selected with
replacement—meaning that the individual data points can be chosen more
than once. After generating several data samples, these weak models are
then trained independently. Depending on the type of task—regression or
classification, for example—the average or majority of those predictions yield
a more accurate estimate.
 Bagging is an ensemble technique that trains multiple models on different subsets
of the data.
 Bagging algorithms are effective in handling imbalanced datasets.
 Bagging ensemble learning algorithm can handle missing values in the dataset.
 Bagging ensemble learning algorithm can handle both numerical and categorical
features effectively.

Bagging
 Consider a scenario where you are looking at the users’ ratings
for a product. Instead of approving one user’s good/bad rating,
we consider average rating given to the product.
 With average rating, we can be considerably sure of quality of
the product. Bagging makes use of this principle.
 Instead of depending on one model, it runs the data through
multiple models in parallel, and average them out as model’s final
output.

What is Bagging and How it Works?
 Bagging is an
acronym for
Bootstrapped
Aggregation.
 Bootstrapping means
random selection of
records with
replacement from the
training dataset.
‘Random selection
with replacement.

What is Bagging and How it Works?
1. Consider that there are 8 samples in the training dataset. Out
of these 8 samples, every weak learner gets 5 samples as
training data for the model. These 5 samples need not be
unique, or non-repetitive.
2. The model (weak learner) is allowed to get a sample multiple
times. For example, as shown in the figure, Rec5 is selected 2
times by the model. Therefore, weak learner1 gets Rec2,
Rec5, Rec8, Rec5, Rec4 as training data.
3. All the samples are available for selection to next weak
learners. Thus all 8 samples will be available for next weak
learner and any sample can be selected multiple times by
next weak learners.

Bagging is a Parallel Method
 Bagging is a parallel method, which means several weak learners
learn the data pattern independently and simultaneously. This can
be best shown in the below diagram:

Bagging is a Parallel Method
 The output of each weak learner is averaged to generate final output
of the model.
 Since the weak learner’s outputs are averaged, this mechanism helps
to reduce variance or variability in the predictions. However, it does
not help to reduce bias of the model.
 Since final prediction is an average of output of each weak learner, it
means that each weak learner has equal say or weight in the final
output.

Bagging Summary
 Bagging is Bootstrapped Aggregation
 It is Parallel method
 Final output is calculated by averaging the outputs produced by
individual weak learner
 Each weak learner has equal say
 Bagging reduces variance

Boosting
 We saw that in bagging every model is given equal preference, but if
one model predicts data more correctly than the other, then higher
weightage should be given to this model over the other.
 Also, the model should attempt to reduce bias.
 These concepts are applied in the second ensemble method that we
are going to learn, that is Boosting.
 Boosting is an ensemble technique that constructs an ensemble by
iteratively updating weights.
 Boosting ensemble learning algorithm is prone to overfitting if the base
models are too complex.

What is Boosting
1. To start with, boosting assigns equal weights to all data points as all
points are equally important in the beginning. For example, if a
training dataset has N samples, it assigns weight = 1/N to each
sample.
2. The weak learner classifies the data. The weak classifier classifies
some samples correctly, while making mistake in classifying others.
3. After classification, sample weights are changed. Weight of correctly
classified sample is reduced, and weight of incorrectly classified
sample is increased. Then the next weak classifier is run.
4. This process continues until model as a whole gives strong predictions.
 Note: Adaboost is the ensemble learning method used in binary
classification only.

What is Boosting
1. To start with, boosting assigns equal weights to all data points as all points are
equally important in the beginning. For example, if a training dataset has N
samples, it assigns weight = 1/N to each sample.
2. The weak learner classifies the data. The weak classifier classifies some
samples correctly, while making mistake in classifying others.
3. After classification, sample weights are changed. Weight of correctly classified
sample is reduced, and weight of incorrectly classified sample is increased.
Then the next weak classifier is run.
4. This process continues until model as a whole gives strong predictions.
 Note: Adaboost is the ensemble learning method used in binary classification
only.
 AdaBoost is an example of Boosting algorithm.
 AdaBoost ensemble learning algorithm assigns weights to base models based
on their performance?

Adaboost -Example
Blocked Artery Chest Pain Weight Heart Disease
Y Y 200 Y
Y N 185 Y
N Y 200 Y
Y Y 160 Y
Y N 140 N
Y N 130 N
N Y 170 N
Y Y 170 N

Adaboost -Example
1. Initialize Weights To
All Training Points
First step is to assign
equal weights to all
samples as all samples
are equally important.
Always, sum of weights
of all samples equals 1.
There are 8 samples, so
each sample will get
weight = 1/8 = 0.125
Blocked
Artery
Chest Pain Weight Heart
Disease
Weights
Assigned
Y Y 200 Y 1/8=0.125
Y N 185 Y 1/8=0.125
N Y 200 Y 1/8=0.125
Y Y 160 Y 1/8=0.125
Y N 140 N 1/8=0.125
Y N 130 N 1/8=0.125
N Y 170 N 1/8=0.125
Y Y 170 N 1/8=0.125

Adaboost -Example
2. Create Stump
 After assigning weight, next step is to create stumps.
 Stump is a decision tree with one node and two leaves.
 Adaboost creates forest of decision stumps.
 To create stump, only one attribute should be chosen.
 But, it is not randomly selected.
 The attribute that does the best job of classifying the sample is
selected first.

Adaboost -Example
2. Create Stump

Adaboost -Example
3. Calculate Total Error And Voting Power (Amount Of Say) Of The Stump
Where ln is Natural Log
Total Error = Number of misclassified samples * Weight of sample.
Since the stump misclassified only 1 sample,
Total Error made by stump = weight of 1 sample = 1/8 = 0.125
Substituting Total Error in the equation,
Amount of say = ½ * ln((1-0.125)/0.125)
= 0.9729

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 26
Reference
 Han, Jiawei Kamber, Micheline Pei and Jian, “Data Mining: Concepts and
Techniques”,Elsevier Publishers, ISBN:9780123814791, 9780123814807.
 Python Machine Learning, 2nd Ed.,

Supervised Learning Ensemble Techniques Machine Learning

More Related Content

Similar to Supervised Learning Ensemble Techniques Machine Learning (20)

More from ShivarkarSandip (20)

Recently uploaded (20)

Supervised Learning Ensemble Techniques Machine Learning