Machine Learning Unit 4 Semester 3 MSc IT Part 2 Mumbai University

TOPICS TO
BE
COVERED…
• DISTANCE BASED MODELS: NEIGHBOURS AND
EXAMPLES, NEAREST NEIGHBOURS
CLASSIFICATION, DISTANCE BASED CLUSTERING
K-MEANS ALGORITHM, HIERARCHICAL
CLUSTERING,
• RULE BASED MODELS: ASSOCIATION RULE
MINING.
• TREE BASED MODELS: DECISION TREES,
REGRESSION TREES, CLUSTERING TREES.
• TRENDS IN MACHINE LEARNING: MODEL AND
SYMBOLS- BAGGING AND BOOSTING, ENSEMBLE
LEARNING, ONLINE LEARNING AND SEQUENCE
PREDICTION, DEEP LEARNING, REINFORCEMENT
LEARNING
PPT: MADHAV MISHRA 2

NEIGHBOURS AND EXAMPLES
• THE ‘NEAREST NEIGHBOURS’ ALGORITHM IS ONE OF THE EASIEST MACHINE
LEARNING ALGORITHMS TO UNDERSTAND MATHEMATICALLY. DESPITE ITS
SIMPLICITY, THE ALGORITHM IS FAIRLY ACCURATE AND MODELS BASED ON IT
USUALLY GENERATE OPTIMAL RESULTS.
• THE ALGORITHM IS BEST UNDERSTOOD THROUGH ITS USE TO PERFORM DATA
CLASSIFICATION. IN A SCENARIO WHERE WE ARE GIVEN A DATA-POINT FOR
WHICH WE NEED TO DETERMINE A PREDICTED LABEL (CLASS).
• WE WOULD FIND THE CLOSEST DATA-POINT TO THE ONE IN QUESTION AND
ASSUME THAT THE LABEL OF OUR DATA POINT MATCHES THAT OF THE ONE
THAT IT IS CLOSEST TO.

• CONSIDER THE EXAMPLE IN THE GRAPH
ALONGSIDE.
• WE ARE GIVEN A DATASET OF POINTS THAT ARE
EITHER RED OR BLUE.
• THESE HAVE BEEN PLOTTED ON A GRAPH BASED
ON SOME SET OF PARAMETERS.
• NOW, WE ARE GIVEN A NEW POINT (PLOTTED IN
GREEN) AND ASKED TO PREDICT WHETHER IT
SHOULD BE RED OR BLUE.
• IN THE NEAREST NEIGHBOURS (NN) ALGORITHM,
WE FIRST PLOT THE GREEN POINT ON THE
GRAPH BASED ON THE SAME PARAMETERS WE
USED TO PLOT THE EARLIER POINTS AND THEN
WE SEARCH FOR THE CLOSEST POINT TO THE
POINT IN QUESTION.
• IN THIS CASE, THE RED POINT WITH
COORDINATES (2,2) IS THE CLOSEST TO THE
GREEN POINT. SINCE THE CLOSEST POINT IS RED,
WE PREDICT THAT THIS NEW POINT IS ALSO

• ON A SIMPLE GRAPH LIKE THIS ONE, THE DISTANCE FORMULA CAN BE USED TO
CALCULATE THE DISTANCES TO FIND THE CLOSEST POINT, I.E. THE ‘NEAREST
NEIGHBOUR’.

NEAREST NEIGHBOURS CLASSIFICATION (KNN)
• K-NEAREST NEIGHBORS (KNN) ALGORITHM IS A TYPE OF SUPERVISED ML
ALGORITHM WHICH CAN BE USED FOR BOTH CLASSIFICATION AS WELL AS
REGRESSION PREDICTIVE PROBLEMS. HOWEVER, IT IS MAINLY USED FOR
CLASSIFICATION PREDICTIVE PROBLEMS IN INDUSTRY. THE FOLLOWING TWO
PROPERTIES WOULD DEFINE KNN WELL −
• LAZY LEARNING ALGORITHM − KNN IS A LAZY LEARNING ALGORITHM BECAUSE
IT DOES NOT HAVE A SPECIALIZED TRAINING PHASE AND USES ALL THE DATA
FOR TRAINING WHILE CLASSIFICATION.
• NON-PARAMETRIC LEARNING ALGORITHM − KNN IS ALSO A NON-PARAMETRIC
LEARNING ALGORITHM BECAUSE IT DOESN’T ASSUME ANYTHING ABOUT THE
UNDERLYING DATA.

• WORKING OF KNN ALGORITHM:
• K-NEAREST NEIGHBOURS (KNN) ALGORITHM USES ‘FEATURE SIMILARITY’ TO PREDICT THE VALUES OF NEW
DATAPOINTS WHICH FURTHER MEANS THAT THE NEW DATA POINT WILL BE ASSIGNED A VALUE BASED ON
HOW CLOSELY IT MATCHES THE POINTS IN THE TRAINING SET. WE CAN UNDERSTAND ITS WORKING WITH
THE HELP OF FOLLOWING STEPS-
• STEP 1 − FOR IMPLEMENTING ANY ALGORITHM, WE NEED DATASET. SO DURING THE FIRST STEP OF KNN,
WE MUST LOAD THE TRAINING AS WELL AS TEST DATA.
• STEP 2 − NEXT, WE NEED TO CHOOSE THE VALUE OF K (I.E. THE NEAREST DATA POINTS). K CAN BE ANY
INTEGER.
• STEP 3 − FOR EACH POINT IN THE TEST DATA DO THE FOLLOWING −
3.1 − CALCULATE THE DISTANCE BETWEEN TEST DATA AND EACH ROW OF TRAINING DATA WITH
THE HELP OF ANY OF THE METHOD NAMELY: EUCLIDEAN, MANHATTAN OR HAMMING DISTANCE. THE
MOST COMMONLY USED METHOD TO CALCULATE DISTANCE IS EUCLIDEAN.
3.2 − NOW, BASED ON THE DISTANCE VALUE, SORT THEM IN ASCENDING ORDER.
3.3 − NEXT, IT WILL CHOOSE THE TOP K ROWS FROM THE SORTED ARRAY.
3.4 − NOW, IT WILL ASSIGN A CLASS TO THE TEST POINT BASED ON MOST FREQUENT CLASS OF
THESE ROWS.
• STEP 4 − END
PPT: MADHAV
MISHRA
7

• EXAMPLE
• THE FOLLOWING IS AN EXAMPLE TO UNDERSTAND THE CONCEPT OF K AND
WORKING OF KNN ALGORITHM −
• SUPPOSE WE HAVE A DATASET WHICH CAN BE PLOTTED AS FOLLOWS −
• NOW, WE NEED TO CLASSIFY NEW DATA POINT WITH BLACK DOT (AT POINT 60,60)
INTO BLUE OR RED CLASS. WE ARE ASSUMING K = 3 I.E. IT WOULD FIND THREE
NEAREST DATA POINTS. IT IS SHOWN IN THE NEXT DIAGRAMS
• WE CAN SEE IN THE ABOVE DIAGRAM THE THREE NEAREST NEIGHBOURS OF THE
DATA POINT WITH BLACK DOT. AMONG THOSE THREE, TWO OF THEM LIES IN RED
CLASS HENCE THE BLACK DOT WILL ALSO BE ASSIGNED IN RED CLASS.

• PROS AND CONS OF KNN
• PROS
• IT IS VERY SIMPLE ALGORITHM TO UNDERSTAND AND INTERPRET.
• IT IS VERY USEFUL FOR NONLINEAR DATA BECAUSE THERE IS NO ASSUMPTION ABOUT
DATA IN THIS ALGORITHM.
• IT IS A VERSATILE ALGORITHM AS WE CAN USE IT FOR CLASSIFICATION AS WELL AS
REGRESSION.
• IT HAS RELATIVELY HIGH ACCURACY BUT THERE ARE MUCH BETTER SUPERVISED
LEARNING MODELS THAN KNN.
• CONS
• IT IS COMPUTATIONALLY A BIT EXPENSIVE ALGORITHM BECAUSE IT STORES ALL THE
TRAINING DATA.
• HIGH MEMORY STORAGE REQUIRED AS COMPARED TO OTHER SUPERVISED LEARNING
ALGORITHMS.
• PREDICTION IS SLOW IN CASE OF BIG N.
• IT IS VERY SENSITIVE TO THE SCALE OF DATA AS WELL AS IRRELEVANT FEATURES.
PPT: MADHAV
MISHRA
9

• APPLICATIONS OF KNN:
THE FOLLOWING ARE SOME OF THE AREAS IN WHICH KNN CAN BE APPLIED
SUCCESSFULLY −
• BANKING SYSTEM
• KNN CAN BE USED IN BANKING SYSTEM TO PREDICT WEATHER AN INDIVIDUAL IS FIT
FOR LOAN APPROVAL? DOES THAT INDIVIDUAL HAVE THE CHARACTERISTICS
SIMILAR TO THE DEFAULTERS ONE?
• CALCULATING CREDIT RATINGS
• KNN ALGORITHMS CAN BE USED TO FIND AN INDIVIDUAL’S CREDIT RATING BY
COMPARING WITH THE PERSONS HAVING SIMILAR TRAITS.
• POLITICS
• WITH THE HELP OF KNN ALGORITHMS, WE CAN CLASSIFY A POTENTIAL VOTER INTO
VARIOUS CLASSES LIKE “WILL VOTE”, “WILL NOT VOTE”, “WILL VOTE TO PARTY
‘CONGRESS’, “WILL VOTE TO PARTY ‘BJP’.
• OTHER AREAS IN WHICH KNN ALGORITHM CAN BE USED ARE SPEECH RECOGNITION,
HANDWRITING DETECTION, IMAGE RECOGNITION AND VIDEO RECOGNITION.

DISTANCE BASED CLUSTERING K-MEANS
ALGORITHM
• K-MEANS CLUSTERING ALGORITHM COMPUTES THE CENTROIDS AND ITERATES
UNTIL WE IT FINDS OPTIMAL CENTROID. IT ASSUMES THAT THE NUMBER OF
CLUSTERS ARE ALREADY KNOWN.
• IT IS ALSO CALLED FLAT CLUSTERING ALGORITHM.
• THE NUMBER OF CLUSTERS IDENTIFIED FROM DATA BY ALGORITHM IS
REPRESENTED BY ‘K’ IN K-MEANS.
• IN THIS ALGORITHM, THE DATA POINTS ARE ASSIGNED TO A CLUSTER IN SUCH A
MANNER THAT THE SUM OF THE SQUARED DISTANCE BETWEEN THE DATA
POINTS AND CENTROID WOULD BE MINIMUM.
• IT IS TO BE UNDERSTOOD THAT LESS VARIATION WITHIN THE CLUSTERS WILL
LEAD TO MORE SIMILAR DATA POINTS WITHIN SAME CLUSTER.

• HOW THE K- MEANS CLUSTERING ALGORITHM WORKS?
K- MEANS CLUSTERING ALGORITHM NEEDS THE FOLLOWING INPUTS:
K = NUMBER OF SUBGROUPS OR CLUSTERS
SAMPLE OR TRAINING SET = {X1, X2, X3,………XN}
• NOW LET US ASSUME WE HAVE A DATA SET THAT IS UNLABELED AND WE NEED
TO DIVIDE IT INTO CLUSTERS.

• NOW WE NEED TO FIND THE NUMBER OF CLUSTERS. THIS CAN BE
DONE BY TWO METHODS:
 ELBOW METHOD.
 PURPOSE METHOD.
• ELBOW METHOD
IN THIS METHOD, A CURVE IS DRAWN BETWEEN “WITHIN THE SUM
OF SQUARES” (WSS) AND THE NUMBER OF CLUSTERS. THE CURVE
PLOTTED RESEMBLES A HUMAN ARM.
IT IS CALLED THE ELBOW METHOD BECAUSE THE POINT OF ELBOW IN
THE CURVE GIVES US THE OPTIMUM NUMBER OF CLUSTERS. IN THE
GRAPH OR CURVE, AFTER THE ELBOW POINT, THE VALUE OF WSS
CHANGES VERY SLOWLY SO ELBOW POINT MUST BE CONSIDERED TO
GIVE THE FINAL VALUE OF THE NUMBER OF CLUSTERS.
• PURPOSE-BASED
IN THIS METHOD, THE DATA IS DIVIDED BASED ON DIFFERENT
METRICS AND AFTER THEN IT IS JUDGED HOW WELL IT PERFORMED
FOR THAT CASE. FOR EXAMPLE, THE ARRANGEMENT OF THE SHIRTS
IN THE MEN’S CLOTHING DEPARTMENT IN A MALL IS DONE ON THE
CRITERIA OF THE SIZES. IT CAN BE DONE ON THE BASIS OF PRICE
AND THE BRANDS ALSO. THE BEST SUITABLE WOULD BE CHOSEN TO
GIVE THE OPTIMAL NUMBER OF CLUSTERS I.E. THE VALUE OF K.

WORKING OF K-MEANS ALGORITHM
• WE CAN UNDERSTAND THE WORKING OF K-MEANS CLUSTERING ALGORITHM WITH THE HELP
OF FOLLOWING STEPS −
STEP 1 − FIRST, WE NEED TO SPECIFY THE NUMBER OF CLUSTERS, K, NEED TO BE GENERATED BY
THIS ALGORITHM.
STEP 2 − NEXT, RANDOMLY SELECT K DATA POINTS AND ASSIGN EACH DATA POINT TO A
CLUSTER. IN SIMPLE WORDS, CLASSIFY THE DATA BASED ON THE NUMBER OF DATA POINTS.
STEP 3 − NOW IT WILL COMPUTE THE CLUSTER CENTROIDS.
STEP 4 − NEXT, KEEP ITERATING THE FOLLOWING UNTIL WE FIND OPTIMAL CENTROID WHICH IS
THE ASSIGNMENT OF DATA POINTS TO THE CLUSTERS THAT ARE NOT CHANGING ANY MORE
4.1 − FIRST, THE SUM OF SQUARED DISTANCE BETWEEN DATA POINTS AND CENTROIDS
WOULD BE COMPUTED.
4.2 − NOW, WE HAVE TO ASSIGN EACH DATA POINT TO THE CLUSTER THAT IS CLOSER
THAN OTHER CLUSTER (CENTROID).
4.3 − AT LAST COMPUTE THE CENTROIDS FOR THE CLUSTERS BY TAKING THE AVERAGE
OF ALL DATA POINTS OF THAT CLUSTER.

ADVANTAGES OF K- MEANS CLUSTERING
ALGORITHM
• IT IS FAST, ROBUST, COMPARATIVELY EFFICIENT
• IF DATA SETS ARE DISTINCT THEN GIVES THE BEST RESULTS
• PRODUCE TIGHTER CLUSTERS, FLEXIBLE, EASY TO INTERPRET, BETTER
COMPUTATIONAL COST
• ENHANCES ACCURACY, WORKS BETTER WITH SPHERICAL CLUSTERS
PPT: MADHAV MISHRA
15
DISADVANTAGES OF K- MEANS CLUSTERING
ALGORITHM
• NEEDS PRIOR SPECIFICATION FOR THE NUMBER OF CLUSTER CENTERS
• IF THERE ARE TWO HIGHLY OVERLAPPING DATA THEN IT CANNOT BE DISTINGUISHED
AND CANNOT TELL THAT THERE ARE TWO CLUSTERS
• WITH THE DIFFERENT REPRESENTATION OF THE DATA, THE RESULTS ACHIEVED ARE
ALSO DIFFERENT
• CANNOT HANDLE OUTLIERS AND NOISY DATA
• DO NOT WORK FOR THE NON-LINEAR DATA SET
• LACKS CONSISTENCY

APPLICATIONS OF K- MEANS CLUSTERING
ALGORITHM
• MARKET SEGMENTATION
• DOCUMENT CLUSTERING
• IMAGE SEGMENTATION
• IMAGE COMPRESSION
• CLUSTER ANALYSIS
• INSURANCE FRAUD DETECTION
• PUBLIC TRANSPORT DATA ANALYSIS

HIERARCHICAL CLUSTERING
• ALSO KNOWN AS HIERARCHICAL CLUSTER ANALYSIS OR HCA
• IT IS AN UNSUPERVISED CLUSTERING ALGORITHM WHICH INVOLVES CREATING
CLUSTERS THAT HAVE PREDOMINANT ORDERING FROM TOP TO BOTTOM.
• FOR E.G: ALL FILES AND FOLDERS ON OUR HARD DISK ARE ORGANIZED IN A
HIERARCHY.
• THE ALGORITHM GROUPS SIMILAR OBJECTS INTO GROUPS CALLED CLUSTERS.
THE ENDPOINT IS A SET OF CLUSTERS OR GROUPS, WHERE EACH CLUSTER IS
DISTINCT FROM EACH OTHER CLUSTER, AND THE OBJECTS WITHIN EACH
CLUSTER ARE BROADLY SIMILAR TO EACH OTHER.
• THIS CLUSTERING TECHNIQUE IS DIVIDED INTO TWO TYPES:
1. AGGLOMERATIVE HIERARCHICAL CLUSTERING
2. DIVISIVE HIERARCHICAL CLUSTERING

• AGGLOMERATIVE HIERARCHICAL CLUSTERING
THE AGGLOMERATIVE HIERARCHICAL CLUSTERING IS THE MOST COMMON TYPE
OF HIERARCHICAL CLUSTERING USED TO GROUP OBJECTS IN CLUSTERS BASED ON
THEIR SIMILARITY. IT’S ALSO KNOWN AS AGNES (AGGLOMERATIVE NESTING). IT'S A
“BOTTOM-UP” APPROACH: EACH OBSERVATION STARTS IN ITS OWN CLUSTER,
AND PAIRS OF CLUSTERS ARE MERGED AS ONE MOVES UP THE HIERARCHY.
• HOW DOES IT WORK?
1. MAKE EACH DATA POINT A SINGLE-POINT CLUSTER → FORMS N CLUSTERS
2. TAKE THE TWO CLOSEST DATA POINTS AND MAKE THEM ONE CLUSTER →
FORMS N-1 CLUSTERS
3. TAKE THE TWO CLOSEST CLUSTERS AND MAKE THEM ONE CLUSTER → FORMS
N-2 CLUSTERS.
4. REPEAT STEP-3 UNTIL YOU ARE LEFT WITH ONLY ONE CLUSTER.

• HAVE A LOOK AT THE VISUAL REPRESENTATION OF AGGLOMERATIVE
HIERARCHICAL CLUSTERING FOR BETTER UNDERSTANDING:

• THERE ARE SEVERAL WAYS TO MEASURE THE DISTANCE BETWEEN CLUSTERS IN
ORDER TO DECIDE THE RULES FOR CLUSTERING, AND THEY ARE OFTEN CALLED
LINKAGE METHODS. SOME OF THE COMMON LINKAGE METHODS ARE:
• COMPLETE-LINKAGE: THE DISTANCE BETWEEN TWO CLUSTERS IS DEFINED AS
THE LONGEST DISTANCE BETWEEN TWO POINTS IN EACH CLUSTER.
• SINGLE-LINKAGE: THE DISTANCE BETWEEN TWO CLUSTERS IS DEFINED AS
THE SHORTEST DISTANCE BETWEEN TWO POINTS IN EACH CLUSTER. THIS
LINKAGE MAY BE USED TO DETECT HIGH VALUES IN YOUR DATASET WHICH MAY
BE OUTLIERS AS THEY WILL BE MERGED AT THE END.
• AVERAGE-LINKAGE: THE DISTANCE BETWEEN TWO CLUSTERS IS DEFINED AS THE
AVERAGE DISTANCE BETWEEN EACH POINT IN ONE CLUSTER TO EVERY POINT IN
THE OTHER CLUSTER.
• CENTROID-LINKAGE: FINDS THE CENTROID OF CLUSTER 1 AND CENTROID OF
CLUSTER 2, AND THEN CALCULATES THE DISTANCE BETWEEN THE TWO BEFORE
MERGING.

• THE CHOICE OF LINKAGE METHOD ENTIRELY DEPENDS ON
YOU AND THERE IS NO HARD AND FAST METHOD THAT
WILL ALWAYS GIVE YOU GOOD RESULTS. DIFFERENT
LINKAGE METHODS LEAD TO DIFFERENT CLUSTERS.
• THE POINT OF DOING ALL THIS IS TO DEMONSTRATE THE
WAY HIERARCHICAL CLUSTERING WORKS, IT MAINTAINS A
MEMORY OF HOW WE WENT THROUGH THIS PROCESS AND
THAT MEMORY IS STORED IN DENDROGRAM.
• WHAT IS A DENDROGRAM?
• A DENDROGRAM IS A TYPE OF TREE DIAGRAM SHOWING
HIERARCHICAL RELATIONSHIPS BETWEEN DIFFERENT SETS
OF DATA.
• AS ALREADY SAID A DENDROGRAM CONTAINS THE
MEMORY OF HIERARCHICAL CLUSTERING ALGORITHM, SO
JUST BY LOOKING AT THE DENDROGRAM YOU CAN TELL
HOW THE CLUSTER IS FORMED.
• NOTE:-
• DISTANCE BETWEEN DATA POINTS REPRESENTS
DISSIMILARITIES.
• HEIGHT OF THE BLOCKS REPRESENTS THE DISTANCE
PPT: MADHAV MISHRA
21

PARTS OF A DENDROGRAM
• A DENDROGRAM CAN BE A COLUMN GRAPH (AS IN THE GIVEN IMAGE ) OR A
ROW GRAPH.
• SOME DENDROGRAMS ARE CIRCULAR OR HAVE A FLUID-SHAPE, BUT THE
SOFTWARE WILL USUALLY PRODUCE A ROW OR COLUMN GRAPH. NO MATTER
WHAT THE SHAPE, THE BASIC GRAPH COMPRISES THE SAME PARTS:
• THE CLADES ARE THE BRANCH AND ARE ARRANGED ACCORDING TO HOW
SIMILAR (OR DISSIMILAR) THEY ARE.
• CLADES THAT ARE CLOSE TO THE SAME HEIGHT ARE SIMILAR TO EACH
OTHER;
• CLADES WITH DIFFERENT HEIGHTS ARE DISSIMILAR — THE GREATER THE
DIFFERENCE IN HEIGHT, THE MORE DISSIMILARITY.
• EACH CLADE HAS ONE OR MORE LEAVES.
• LEAVES A, B, AND C ARE MORE SIMILAR TO EACH OTHER THAN THEY ARE TO
LEAVES D, E, OR F.
• LEAVES D AND E ARE MORE SIMILAR TO EACH OTHER THAN THEY ARE TO
LEAVES A, B, C, OR F.
• LEAF F IS SUBSTANTIALLY DIFFERENT FROM ALL OF THE OTHER LEAVES.
PPT: MADHAV MISHRA
22

DIVISIVE HIERARCHICAL CLUSTERING
• IN DIVISIVE OR DIANA(DIVISIVE ANALYSIS CLUSTERING) IS A TOP-DOWN
CLUSTERING METHOD
• WHERE WE ASSIGN ALL OF THE OBSERVATIONS TO A SINGLE CLUSTER AND THEN
PARTITION THE CLUSTER TO TWO LEAST SIMILAR CLUSTERS.
• FINALLY, WE PROCEED RECURSIVELY ON EACH CLUSTER UNTIL THERE IS ONE
CLUSTER FOR EACH OBSERVATION.
• SO THIS CLUSTERING APPROACH IS EXACTLY OPPOSITE TO AGGLOMERATIVE
CLUSTERING.

RULE BASED MODELS: ASSOCIATION RULE
MINING
• HAS IT EVER HAPPENED THAT YOU’RE OUT TO BUY SOMETHING, AND YOU END
UP BUYING A LOT MORE THAN YOU PLANNED?
• IT’S A PHENOMENON KNOWN AS IMPULSIVE BUYING AND BIG RETAILERS TAKE
ADVANTAGE OF MACHINE LEARNING AND APRIORI ALGORITHM AND MAKE SURE
THAT WE TEND TO BUY MORE.
• SO LET’S UNDERSTAND HOW THE APRIORI ALGORITHM WORKS IN THE
FOLLOWING ORDER:
• MARKET BASKET ANALYSIS
• ASSOCIATION RULE MINING
• APRIORI ALGORITHM

• MARKET BASKET ANALYSIS
• IN TODAY’S WORLD, THE GOAL OF ANY ORGANIZATION IS TO INCREASE
REVENUE. CAN THIS BE DONE BY PITCHING JUST ONE PRODUCT AT A TIME TO
THE CUSTOMER? THE ANSWER IS A CLEAR NO.
• HENCE, ORGANIZATIONS BEGAN MINING DATA RELATED TO FREQUENTLY
BOUGHT ITEMS.
• MARKET BASKET ANALYSIS IS ONE OF THE KEY TECHNIQUES USED BY LARGE
RETAILERS TO UNCOVER ASSOCIATIONS BETWEEN ITEMS.
• THEY TRY TO FIND OUT ASSOCIATIONS BETWEEN DIFFERENT ITEMS AND
PRODUCTS THAT CAN BE SOLD TOGETHER, WHICH GIVES ASSISTING IN RIGHT
PRODUCT PLACEMENT.
• TYPICALLY, IT FIGURES OUT WHAT PRODUCTS ARE BEING BOUGHT TOGETHER
AND ORGANIZATIONS CAN PLACE PRODUCTS IN A SIMILAR MANNER.
• LET’S UNDERSTAND THIS BETTER WITH AN EXAMPLE:

26
PPT: MADHAV MISHRA
• PEOPLE WHO BUY BREAD USUALLY BUY BUTTER
TOO.
• THE MARKETING TEAMS AT RETAIL STORES
SHOULD TARGET CUSTOMERS WHO BUY BREAD
AND BUTTER AND PROVIDE AN OFFER TO THEM
SO THAT THEY BUY THE THIRD ITEM, LIKE EGGS.
• SO IF CUSTOMERS BUY BREAD AND BUTTER AND
SEE A DISCOUNT OR AN OFFER ON EGGS, THEY
WILL BE ENCOURAGED TO SPEND MORE AND BUY
THE EGGS.
• THIS IS WHAT MARKET BASKET ANALYSIS IS ALL
ABOUT.
• THIS IS JUST A SMALL EXAMPLE. SO, IF YOU TAKE
10000 ITEMS DATA OF YOUR SUPERMART TO A
DATA SCIENTIST, JUST IMAGINE THE NUMBER OF
INSIGHTS YOU CAN GET.

• ASSOCIATION RULE MINING
• ASSOCIATION RULES CAN BE THOUGHT OF AS AN IF-THEN RELATIONSHIP.
• SUPPOSE ITEM A IS BEING BOUGHT BY THE CUSTOMER, THEN THE CHANCES OF
ITEM B BEING PICKED BY THE CUSTOMER TOO UNDER THE SAME TRANSACTION ID IS
FOUND OUT.
• THERE ARE TWO ELEMENTS OF THESE RULES:
• ANTECEDENT (IF): THIS IS AN ITEM/GROUP OF ITEMS THAT ARE TYPICALLY FOUND IN
THE ITEM SETS OR DATASETS.
• CONSEQUENT (THEN): THIS COMES ALONG AS AN ITEM WITH AN
ANTECEDENT/GROUP OF ANTECEDENTS.
• BUT HERE COMES A CONSTRAINT. SUPPOSE YOU MADE A RULE ABOUT AN ITEM, YOU
STILL HAVE AROUND 9999 ITEMS TO CONSIDER FOR RULE-MAKING.
• THIS IS WHERE THE APRIORI ALGORITHM COMES INTO PLAY.
• SO BEFORE WE UNDERSTAND THE APRIORI ALGORITHM, LET’S UNDERSTAND THE
MATH BEHIND IT.
PPT: MADHAV MISHRA
27

• THERE ARE 3 WAYS TO MEASURE ASSOCIATION:
• SUPPORT
• CONFIDENCE
• LIFT
• SUPPORT: IT GIVES THE FRACTION OF TRANSACTIONS WHICH CONTAINS ITEM A
AND B. BASICALLY SUPPORT TELLS US ABOUT THE FREQUENTLY BOUGHT ITEMS
OR THE COMBINATION OF ITEMS BOUGHT FREQUENTLY.
• SO WITH THIS, WE CAN FILTER OUT THE ITEMS THAT HAVE A LOW FREQUENCY.

• CONFIDENCE: IT TELLS US HOW OFTEN THE ITEMS A AND B OCCUR TOGETHER,
GIVEN THE NUMBER TIMES A OCCURS.
• TYPICALLY, WHEN YOU WORK WITH THE APRIORI ALGORITHM, YOU DEFINE THESE
TERMS ACCORDINGLY.
• BUT HOW DO YOU DECIDE THE VALUE?
• HONESTLY, THERE ISN’T A WAY TO DEFINE THESE TERMS. SUPPOSE YOU’VE
ASSIGNED THE SUPPORT VALUE AS 2.
• WHAT THIS MEANS IS, UNTIL AND UNLESS THE ITEM/S FREQUENCY IS NOT 2%, YOU
WILL NOT CONSIDER THAT ITEM/S FOR THE APRIORI ALGORITHM.
• THIS MAKES SENSE AS CONSIDERING ITEMS THAT ARE BOUGHT LESS FREQUENTLY IS
A WASTE OF TIME.
• NOW SUPPOSE, AFTER FILTERING YOU STILL HAVE AROUND 5000 ITEMS LEFT.
• CREATING ASSOCIATION RULES FOR THEM IS A PRACTICALLY IMPOSSIBLE TASK FOR
ANYONE.
• THIS IS WHERE THE CONCEPT OF LIFT COMES INTO PLAY.
PPT: MADHAV MISHRA
29

• LIFT: LIFT INDICATES THE STRENGTH OF A RULE OVER THE RANDOM OCCURRENCE
OF A AND B. IT BASICALLY TELLS US THE STRENGTH OF ANY RULE.
• FOCUS ON THE DENOMINATOR, IT IS THE PROBABILITY OF THE INDIVIDUAL SUPPORT
VALUES OF A AND B AND NOT TOGETHER. LIFT EXPLAINS THE STRENGTH OF A RULE.
• MORE THE LIFT MORE IS THE STRENGTH. LET’S SAY FOR A -> B, THE LIFT VALUE IS
4. IT MEANS THAT IF YOU BUY A THE CHANCES OF BUYING B IS 4 TIMES.
• NOTE:
APRIORI ALGORITHM:
APRIORI ALGORITHM USES FREQUENT ITEMSETS TO GENERATE ASSOCIATION RULES.
IT IS BASED ON THE CONCEPT THAT A SUBSET OF A FREQUENT ITEMSET MUST ALSO
BE A FREQUENT ITEMSET. FREQUENT ITEMSET IS AN ITEMSET WHOSE SUPPORT VALUE
IS GREATER THAN A THRESHOLD VALUE(SUPPORT).
PPT: MADHAV MISHRA
30

TREE BASED MODELS: DECISION TREES, REGRESSION TREES, CLUSTERING
TREES
• CLASSIFICATION IS A TWO-STEP PROCESS, LEARNING STEP AND PREDICTION STEP, IN MACHINE
LEARNING.
• IN THE LEARNING STEP, THE MODEL IS DEVELOPED BASED ON GIVEN TRAINING DATA.
• IN THE PREDICTION STEP, THE MODEL IS USED TO PREDICT THE RESPONSE FOR GIVEN DATA.
• DECISION TREE IS ONE OF THE EASIEST AND POPULAR CLASSIFICATION ALGORITHMS TO
UNDERSTAND AND INTERPRET.
DECISION TREE ALGORITHM
• DECISION TREE ALGORITHM BELONGS TO THE FAMILY OF SUPERVISED LEARNING ALGORITHMS.
UNLIKE OTHER SUPERVISED LEARNING ALGORITHMS, THE DECISION TREE ALGORITHM CAN BE USED
FOR SOLVING REGRESSION AND CLASSIFICATION PROBLEMS TOO.
• THE GOAL OF USING A DECISION TREE IS TO CREATE A TRAINING MODEL THAT CAN USE TO
PREDICT THE CLASS OR VALUE OF THE TARGET VARIABLE BY LEARNING SIMPLE DECISION
RULES INFERRED FROM PRIOR DATA(TRAINING DATA).
• IN DECISION TREES, FOR PREDICTING A CLASS LABEL FOR A RECORD WE START FROM THE ROOT OF
THE TREE. WE COMPARE THE VALUES OF THE ROOT ATTRIBUTE WITH THE RECORD’S ATTRIBUTE. ON
THE BASIS OF COMPARISON, WE FOLLOW THE BRANCH CORRESPONDING TO THAT VALUE AND JUMP
TO THE NEXT NODE.
PPT: MADHAV MISHRA
3
1

• TYPES OF DECISION TREES
TYPES OF DECISION TREES ARE BASED ON THE TYPE OF TARGET VARIABLE WE
HAVE. IT CAN BE OF TWO TYPES:
• CATEGORICAL VARIABLE DECISION TREE: DECISION TREE WHICH HAS A
CATEGORICAL TARGET VARIABLE THEN IT CALLED A CATEGORICAL VARIABLE
DECISION TREE.
• CONTINUOUS VARIABLE DECISION TREE: DECISION TREE HAS A CONTINUOUS
TARGET VARIABLE THEN IT IS CALLED CONTINUOUS VARIABLE DECISION TREE.
• EXAMPLE:- LET’S SAY WE HAVE A PROBLEM TO PREDICT WHETHER A CUSTOMER
WILL PAY HIS RENEWAL PREMIUM WITH AN INSURANCE COMPANY (YES/ NO).
HERE WE KNOW THAT THE INCOME OF CUSTOMERS IS A SIGNIFICANT VARIABLE
BUT THE INSURANCE COMPANY DOES NOT HAVE INCOME DETAILS FOR ALL
CUSTOMERS. NOW, AS WE KNOW THIS IS AN IMPORTANT VARIABLE, THEN WE
CAN BUILD A DECISION TREE TO PREDICT CUSTOMER INCOME BASED ON
OCCUPATION, PRODUCT, AND VARIOUS OTHER VARIABLES. IN THIS CASE, WE
ARE PREDICTING VALUES FOR THE CONTINUOUS VARIABLES.
PPT: MADHAV MISHRA
32

33
PPT: MADHAV MISHRA
• IMPORTANT TERMINOLOGY RELATED TO
DECISION TREES
• ROOT NODE: IT REPRESENTS THE ENTIRE
POPULATION OR SAMPLE AND THIS FURTHER
GETS DIVIDED INTO TWO OR MORE
HOMOGENEOUS SETS.
• SPLITTING: IT IS A PROCESS OF DIVIDING A NODE
INTO TWO OR MORE SUB-NODES.
• DECISION NODE: WHEN A SUB-NODE SPLITS INTO
FURTHER SUB-NODES, THEN IT IS CALLED THE
DECISION NODE.
• LEAF / TERMINAL NODE: NODES DO NOT SPLIT IS
CALLED LEAF OR TERMINAL NODE.
• PRUNING: WHEN WE REMOVE SUB-NODES OF A
DECISION NODE, THIS PROCESS IS CALLED
PRUNING. YOU CAN SAY THE OPPOSITE PROCESS
OF SPLITTING.
• BRANCH / SUB-TREE: A SUBSECTION OF THE
ENTIRE TREE IS CALLED BRANCH OR SUB-TREE.
• PARENT AND CHILD NODE: A NODE, WHICH IS
DIVIDED INTO SUB-NODES IS CALLED A PARENT

• DECISION TREES CLASSIFY THE EXAMPLES BY SORTING THEM DOWN THE TREE
FROM THE ROOT TO SOME LEAF/TERMINAL NODE, WITH THE LEAF/TERMINAL
NODE PROVIDING THE CLASSIFICATION OF THE EXAMPLE.
• EACH NODE IN THE TREE ACTS AS A TEST CASE FOR SOME ATTRIBUTE, AND
EACH EDGE DESCENDING FROM THE NODE CORRESPONDS TO THE POSSIBLE
ANSWERS TO THE TEST CASE.
• THIS PROCESS IS RECURSIVE IN NATURE AND IS REPEATED FOR EVERY SUBTREE
ROOTED AT THE NEW NODE.

HOW DO DECISION TREES WORK?
• THE DECISION OF MAKING STRATEGIC SPLITS HEAVILY AFFECTS A TREE’S
ACCURACY. THE DECISION CRITERIA ARE DIFFERENT FOR CLASSIFICATION AND
REGRESSION TREES.
• DECISION TREES USE MULTIPLE ALGORITHMS TO DECIDE TO SPLIT A NODE INTO
TWO OR MORE SUB-NODES.
• THE CREATION OF SUB-NODES INCREASES THE HOMOGENEITY OF RESULTANT
SUB-NODES. IN OTHER WORDS, WE CAN SAY THAT THE PURITY OF THE NODE
INCREASES WITH RESPECT TO THE TARGET VARIABLE.
• THE DECISION TREE SPLITS THE NODES ON ALL AVAILABLE VARIABLES AND THEN
SELECTS THE SPLIT WHICH RESULTS IN MOST HOMOGENEOUS SUB-NODES.

• THE ALGORITHM SELECTION IS ALSO BASED ON THE TYPE OF TARGET VARIABLES. LET US LOOK AT
SOME ALGORITHMS USED IN DECISION TREES:
• ID3 → (EXTENSION OF D3)
C4.5 → (SUCCESSOR OF ID3)
CART → (CLASSIFICATION AND REGRESSION TREE)
CHAID → (CHI-SQUARE AUTOMATIC INTERACTION DETECTION PERFORMS MULTI-LEVEL SPLITS
WHEN COMPUTING CLASSIFICATION TREES)
• THE ID3 ALGORITHM BUILDS DECISION TREES USING A TOP-DOWN GREEDY SEARCH APPROACH
THROUGH THE SPACE OF POSSIBLE BRANCHES WITH NO BACKTRACKING. A GREEDY ALGORITHM,
AS THE NAME SUGGESTS, ALWAYS MAKES THE CHOICE THAT SEEMS TO BE THE BEST AT THAT
MOMENT.
• STEPS IN ID3 ALGORITHM:
1. IT BEGINS WITH THE ORIGINAL SET S AS THE ROOT NODE.
2. ON EACH ITERATION OF THE ALGORITHM, IT ITERATES THROUGH THE VERY UNUSED ATTRIBUTE
OF THE SET S AND CALCULATES ENTROPY(H) AND INFORMATION GAIN(IG) OF THIS ATTRIBUTE.
3. IT THEN SELECTS THE ATTRIBUTE WHICH HAS THE SMALLEST ENTROPY OR LARGEST
INFORMATION GAIN.
4. THE SET S IS THEN SPLIT BY THE SELECTED ATTRIBUTE TO PRODUCE A SUBSET OF THE DATA.
5. THE ALGORITHM CONTINUES TO RECUR ON EACH SUBSET, CONSIDERING ONLY ATTRIBUTES
NEVER SELECTED BEFORE.
PPT: MADHAV
MISHRA
36

• ATTRIBUTE SELECTION MEASURES
• IF THE DATASET CONSISTS OF N ATTRIBUTES THEN DECIDING WHICH ATTRIBUTE TO PLACE AT
THE ROOT OR AT DIFFERENT LEVELS OF THE TREE AS INTERNAL NODES IS A COMPLICATED
STEP.
• BY JUST RANDOMLY SELECTING ANY NODE TO BE THE ROOT CAN’T SOLVE THE ISSUE. IF WE
FOLLOW A RANDOM APPROACH, IT MAY GIVE US BAD RESULTS WITH LOW ACCURACY.
• FOR SOLVING THIS ATTRIBUTE SELECTION PROBLEM, RESEARCHERS WORKED AND DEVISED
SOME SOLUTIONS. THEY SUGGESTED USING SOME CRITERIA LIKE :
• ENTROPY,
INFORMATION GAIN,
GINI INDEX,
GAIN RATIO,
REDUCTION IN VARIANCE
CHI-SQUARE
• THESE CRITERIA WILL CALCULATE VALUES FOR EVERY ATTRIBUTE. THE VALUES ARE SORTED,
AND ATTRIBUTES ARE PLACED IN THE TREE BY FOLLOWING THE ORDER I.E, THE ATTRIBUTE
WITH A HIGH VALUE(IN CASE OF INFORMATION GAIN) IS PLACED AT THE ROOT.
WHILE USING INFORMATION GAIN AS A CRITERION, WE ASSUME ATTRIBUTES TO BE
CATEGORICAL, AND FOR THE GINI INDEX, ATTRIBUTES ARE ASSUMED TO BE CONTINUOUS.
PPT: MADHAV
MISHRA
3
7

3
8
PPT: MADHAV MISHRA
• ENTROPY
ENTROPY IS A MEASURE OF THE RANDOMNESS IN
THE INFORMATION BEING PROCESSED. THE HIGHER
THE ENTROPY, THE HARDER IT IS TO DRAW ANY
CONCLUSIONS FROM THAT INFORMATION.
FLIPPING A COIN IS AN EXAMPLE OF AN ACTION
THAT PROVIDES INFORMATION THAT IS RANDOM.
• FROM THE GRAPH, IT IS QUITE EVIDENT THAT
THE ENTROPY H(X) IS ZERO WHEN THE
PROBABILITY IS EITHER 0 OR 1. THE ENTROPY IS
MAXIMUM WHEN THE PROBABILITY IS 0.5
BECAUSE IT PROJECTS PERFECT RANDOMNESS IN
THE DATA AND THERE IS NO CHANCE IF
PERFECTLY DETERMINING THE OUTCOME.
• MATHEMATICALLY ENTROPY FOR 1 ATTRIBUTE
IS REPRESENTED AS:
• WHERE S → CURRENT STATE, AND PI →
PROBABILITY OF AN EVENT I OF STATE S OR
PERCENTAGE OF CLASS I IN A NODE OF STATE S.

• MATHEMATICALLY ENTROPY FOR MULTIPLE ATTRIBUTES IS REPRESENTED AS:
• WHERE T→ CURRENT STATE AND X → SELECTED ATTRIBUTE
PPT: MADHAV MISHRA
39

INFORMATION GAIN
• INFORMATION GAIN OR IG IS A STATISTICAL PROPERTY THAT MEASURES HOW
WELL A GIVEN ATTRIBUTE SEPARATES THE TRAINING EXAMPLES ACCORDING TO
THEIR TARGET CLASSIFICATION. CONSTRUCTING A DECISION TREE IS ALL ABOUT
FINDING AN ATTRIBUTE THAT RETURNS THE HIGHEST INFORMATION GAIN AND
THE SMALLEST ENTROPY.

• INFORMATION GAIN IS A DECREASE IN ENTROPY. IT COMPUTES THE DIFFERENCE
BETWEEN ENTROPY BEFORE SPLIT AND AVERAGE ENTROPY AFTER SPLIT OF THE
DATASET BASED ON GIVEN ATTRIBUTE VALUES. ID3 (ITERATIVE DICHOTOMISER)
DECISION TREE ALGORITHM USES INFORMATION GAIN.
• MATHEMATICALLY, IG IS REPRESENTED AS:
• IN A MUCH SIMPLER WAY, WE CAN CONCLUDE THAT:
WHERE “BEFORE” IS THE DATASET BEFORE THE SPLIT, K IS THE NUMBER OF SUBSETS
GENERATED BY THE SPLIT, AND (J, AFTER) IS SUBSET J AFTER THE SPLIT.
PPT: MADHAV
MISHRA
41

GINI INDEX
• YOU CAN UNDERSTAND THE GINI INDEX AS A COST FUNCTION USED TO EVALUATE SPLITS IN
THE DATASET. IT IS CALCULATED BY SUBTRACTING THE SUM OF THE SQUARED PROBABILITIES
OF EACH CLASS FROM ONE. IT FAVORS LARGER PARTITIONS AND EASY TO IMPLEMENT
WHEREAS INFORMATION GAIN FAVORS SMALLER PARTITIONS WITH DISTINCT VALUES.
• GINI INDEX WORKS WITH THE CATEGORICAL TARGET VARIABLE “SUCCESS” OR “FAILURE”. IT
PERFORMS ONLY BINARY SPLITS.
• STEPS TO CALCULATE GINI INDEX FOR A SPLIT
• CALCULATE GINI FOR SUB-NODES, USING THE ABOVE FORMULA FOR SUCCESS(P) AND
FAILURE(Q) (P²+Q²).
• CALCULATE THE GINI INDEX FOR SPLIT USING THE WEIGHTED GINI SCORE OF EACH NODE OF
THAT SPLIT.
• CART (CLASSIFICATION AND REGRESSION TREE) USES THE GINI INDEX METHOD TO CREATE
SPLIT POINTS.
PPT: MADHAV MISHRA
42

GAIN RATIO
• INFORMATION GAIN IS BIASED TOWARDS CHOOSING ATTRIBUTES WITH A LARGE
NUMBER OF VALUES AS ROOT NODES. IT MEANS IT PREFERS THE ATTRIBUTE
WITH A LARGE NUMBER OF DISTINCT VALUES.
• C4.5, AN IMPROVEMENT OF ID3, USES GAIN RATIO WHICH IS A MODIFICATION
OF INFORMATION GAIN THAT REDUCES ITS BIAS AND IS USUALLY THE BEST
OPTION. GAIN RATIO OVERCOMES THE PROBLEM WITH INFORMATION GAIN BY
TAKING INTO ACCOUNT THE NUMBER OF BRANCHES THAT WOULD RESULT
BEFORE MAKING THE SPLIT. IT CORRECTS INFORMATION GAIN BY TAKING THE
INTRINSIC INFORMATION OF A SPLIT INTO ACCOUNT.
• WHERE “BEFORE” IS THE DATASET BEFORE THE SPLIT, K IS THE NUMBER OF
SUBSETS GENERATED BY THE SPLIT, AND (J, AFTER) IS SUBSET J AFTER THE SPLIT.

• REDUCTION IN VARIANCE
• REDUCTION IN VARIANCE IS AN ALGORITHM USED FOR CONTINUOUS TARGET
VARIABLES (REGRESSION PROBLEMS). THIS ALGORITHM USES THE STANDARD
FORMULA OF VARIANCE TO CHOOSE THE BEST SPLIT. THE SPLIT WITH LOWER
VARIANCE IS SELECTED AS THE CRITERIA TO SPLIT THE POPULATION:
• ABOVE X-BAR IS THE MEAN OF THE VALUES, X IS ACTUAL AND N IS THE NUMBER
OF VALUES.
• STEPS TO CALCULATE VARIANCE:
• CALCULATE VARIANCE FOR EACH NODE.
• CALCULATE VARIANCE FOR EACH SPLIT AS THE WEIGHTED AVERAGE OF EACH
NODE VARIANCE.
PPT: MADHAV MISHRA
44

• CHI-SQUARE
• THE ACRONYM CHAID STANDS FOR CHI-SQUARED AUTOMATIC INTERACTION
DETECTOR. IT IS ONE OF THE OLDEST TREE CLASSIFICATION METHODS. IT FINDS
OUT THE STATISTICAL SIGNIFICANCE BETWEEN THE DIFFERENCES BETWEEN SUB-
NODES AND PARENT NODE. WE MEASURE IT BY THE SUM OF SQUARES OF
STANDARDIZED DIFFERENCES BETWEEN OBSERVED AND EXPECTED FREQUENCIES
OF THE TARGET VARIABLE.
• IT WORKS WITH THE CATEGORICAL TARGET VARIABLE “SUCCESS” OR “FAILURE”.
IT CAN PERFORM TWO OR MORE SPLITS. HIGHER THE VALUE OF CHI-SQUARE
HIGHER THE STATISTICAL SIGNIFICANCE OF DIFFERENCES BETWEEN SUB-NODE
AND PARENT NODE.
• IT GENERATES A TREE CALLED CHAID (CHI-SQUARE AUTOMATIC INTERACTION
DETECTOR).
• MATHEMATICALLY, CHI-SQUARED IS REPRESENTED AS:
PPT: MADHAV MISHRA
45

• STEPS TO CALCULATE CHI-SQUARE FOR A SPLIT:
• CALCULATE CHI-SQUARE FOR AN INDIVIDUAL NODE BY CALCULATING THE
DEVIATION FOR SUCCESS AND FAILURE BOTH
• CALCULATED CHI-SQUARE OF SPLIT USING SUM OF ALL CHI-SQUARE OF
SUCCESS AND FAILURE OF EACH NODE OF THE SPLIT

• HOW TO AVOID/COUNTER OVERFITTING IN DECISION TREES?
• THE COMMON PROBLEM WITH DECISION TREES, ESPECIALLY HAVING A TABLE
FULL OF COLUMNS, THEY FIT A LOT. SOMETIMES IT LOOKS LIKE THE TREE
MEMORIZED THE TRAINING DATA SET. IF THERE IS NO LIMIT SET ON A DECISION
TREE, IT WILL GIVE YOU 100% ACCURACY ON THE TRAINING DATA SET BECAUSE
IN THE WORSE CASE IT WILL END UP MAKING 1 LEAF FOR EACH OBSERVATION.
THUS THIS AFFECTS THE ACCURACY WHEN PREDICTING SAMPLES THAT ARE NOT
PART OF THE TRAINING SET.
• HERE ARE TWO WAYS TO REMOVE OVERFITTING:
• PRUNING DECISION TREES.
• RANDOM FOREST

• PRUNING DECISION TREES
• THE SPLITTING PROCESS RESULTS IN FULLY GROWN TREES UNTIL THE STOPPING
CRITERIA ARE REACHED. BUT, THE FULLY GROWN TREE IS LIKELY TO OVERFIT
THE DATA, LEADING TO POOR ACCURACY ON UNSEEN DATA.
PPT: MADHAV MISHRA
48

• IN PRUNING, YOU TRIM OFF THE BRANCHES OF THE TREE, I.E., REMOVE THE
DECISION NODES STARTING FROM THE LEAF NODE SUCH THAT THE OVERALL
ACCURACY IS NOT DISTURBED.
• THIS IS DONE BY SEGREGATING THE ACTUAL TRAINING SET INTO TWO SETS:
TRAINING DATA SET, D AND VALIDATION DATA SET, V.
• PREPARE THE DECISION TREE USING THE SEGREGATED TRAINING DATA SET, D. THEN
CONTINUE TRIMMING THE TREE ACCORDINGLY TO OPTIMIZE THE ACCURACY OF THE
VALIDATION DATA SET, V.
• IN THE ABOVE DIAGRAM, THE ‘AGE’ ATTRIBUTE IN THE LEFT-HAND SIDE OF THE
TREE HAS BEEN PRUNED AS IT HAS MORE IMPORTANCE ON THE RIGHT-HAND SIDE
PPT: MADHAV
MISHRA
49

50
PPT: MADHAV MISHRA
• RANDOM FOREST
• RANDOM FOREST IS AN EXAMPLE OF ENSEMBLE LEARNING, IN
WHICH WE COMBINE MULTIPLE MACHINE LEARNING
ALGORITHMS TO OBTAIN BETTER PREDICTIVE PERFORMANCE.
• WHY THE NAME “RANDOM”?
• TWO KEY CONCEPTS THAT GIVE IT THE NAME RANDOM:
• A RANDOM SAMPLING OF TRAINING DATA SET WHEN
BUILDING TREES.
• RANDOM SUBSETS OF FEATURES CONSIDERED WHEN
SPLITTING NODES.
• A TECHNIQUE KNOWN AS BAGGING IS USED TO CREATE AN
ENSEMBLE OF TREES WHERE MULTIPLE TRAINING SETS ARE
GENERATED WITH REPLACEMENT.
• IN THE BAGGING TECHNIQUE, A DATA SET IS DIVIDED
INTO N SAMPLES USING RANDOMIZED SAMPLING. THEN,
USING A SINGLE LEARNING ALGORITHM A MODEL IS BUILT ON
ALL SAMPLES. LATER, THE RESULTANT PREDICTIONS ARE
COMBINED USING VOTING OR AVERAGING IN PARALLEL.

• WHICH IS BETTER LINEAR OR TREE-BASED MODELS?
• WELL, IT DEPENDS ON THE KIND OF PROBLEM YOU ARE SOLVING.
• IF THE RELATIONSHIP BETWEEN DEPENDENT & INDEPENDENT VARIABLES IS WELL
APPROXIMATED BY A LINEAR MODEL, LINEAR REGRESSION WILL OUTPERFORM
THE TREE-BASED MODEL.
• IF THERE IS A HIGH NON-LINEARITY & COMPLEX RELATIONSHIP BETWEEN
DEPENDENT & INDEPENDENT VARIABLES, A TREE MODEL WILL OUTPERFORM A
CLASSICAL REGRESSION METHOD.
• IF YOU NEED TO BUILD A MODEL THAT IS EASY TO EXPLAIN TO PEOPLE, A
DECISION TREE MODEL WILL ALWAYS DO BETTER THAN A LINEAR MODEL.
DECISION TREE MODELS ARE EVEN SIMPLER TO INTERPRET THAN LINEAR
REGRESSION!

MODEL AND SYMBOLS

BAGGING AND BOOSTING
• BAGGING AND BOOSTING ARE BOTH ENSEMBLE LEARNING METHODS IN MACHINE LEARNING.
• BAGGING AND BOOSTING ARE SIMILAR IN THAT THEY ARE BOTH ENSEMBLE TECHNIQUES, WHERE A SET OF WEAK
LEARNERS ARE COMBINED TO CREATE A STRONG LEARNER THAT OBTAINS BETTER PERFORMANCE THAN A SINGLE
ONE.
• ENSEMBLE LEARNING HELPS TO IMPROVE MACHINE LEARNING MODEL PERFORMANCE BY COMBINING SEVERAL
MODELS. THIS APPROACH ALLOWS THE PRODUCTION OF BETTER PREDICTIVE PERFORMANCE COMPARED TO A
SINGLE MODEL.
• THE BASIC IDEA BEHIND ENSEMBLE LEARNING IS TO LEARN A SET OF CLASSIFIERS (EXPERTS) AND TO ALLOW THEM
TO VOTE.
• THIS DIVERSIFICATION IN MACHINE LEARNING IS ACHIEVED BY A TECHNIQUE CALLED ENSEMBLE LEARNING.
• THE IDEA HERE IS TO TRAIN MULTIPLE MODELS, EACH WITH THE OBJECTIVE TO PREDICT OR CLASSIFY A SET OF
RESULTS.
• BAGGING AND BOOSTING ARE TWO TYPES OF ENSEMBLE LEARNING TECHNIQUES. THESE TWO DECREASE THE
VARIANCE OF SINGLE ESTIMATE AS THEY COMBINE SEVERAL ESTIMATES FROM DIFFERENT MODELS.
• SO THE RESULT MAY BE A MODEL WITH HIGHER STABILITY.
• THE MAIN CAUSES OF ERROR IN LEARNING ARE DUE TO NOISE, BIAS AND VARIANCE.
• ENSEMBLE HELPS TO MINIMIZE THESE FACTORS. BY USING ENSEMBLE METHODS, WE’RE ABLE TO INCREASE THE
STABILITY OF THE FINAL MODEL AND REDUCE THE ERRORS MENTIONED PREVIOUSLY.

• BAGGING HELPS TO DECREASE THE MODEL’S VARIANCE.
• BOOSTING HELPS TO DECREASE THE MODEL’S BIAS.
• THESE METHODS ARE DESIGNED TO IMPROVE THE STABILITY AND THE ACCURACY OF
MACHINE LEARNING ALGORITHMS.
• COMBINATIONS OF MULTIPLE CLASSIFIERS DECREASE VARIANCE, ESPECIALLY IN THE CASE
OF UNSTABLE CLASSIFIERS, AND MAY PRODUCE A MORE RELIABLE CLASSIFICATION THAN
A SINGLE CLASSIFIER.
• TO USE BAGGING OR BOOSTING YOU MUST SELECT A BASE LEARNER ALGORITHM.
• FOR EXAMPLE, IF WE CHOOSE A CLASSIFICATION TREE, BAGGING AND BOOSTING WOULD
CONSIST OF A POOL OF TREES AS BIG AS WE WANT AS SHOWN IN THE FOLLOWING
DIAGRAM:

BAGGING
• BAGGING ( OR BOOTSTRAP AGGREGATION), IS A SIMPLE AND VERY POWERFUL ENSEMBLE
METHOD. BAGGING IS THE APPLICATION OF THE BOOTSTRAP PROCEDURE TO A HIGH-
VARIANCE MACHINE LEARNING ALGORITHM, TYPICALLY DECISION TREES.
• THE IDEA BEHIND BAGGING IS COMBINING THE RESULTS OF MULTIPLE MODELS (FOR
INSTANCE, ALL DECISION TREES) TO GET A GENERALIZED RESULT. NOW, BOOTSTRAPPING
COMES INTO PICTURE.
• BAGGING (OR BOOTSTRAP AGGREGATING) TECHNIQUE USES THESE SUBSETS (BAGS) TO
GET A FAIR IDEA OF THE DISTRIBUTION (COMPLETE SET). THE SIZE OF SUBSETS CREATED
FOR BAGGING MAY BE LESS THAN THE ORIGINAL SET.
• IT CAN BE REPRESENTED AS FOLLOWS:

• BAGGING WORKS AS FOLLOWS:-
• MULTIPLE SUBSETS ARE CREATED FROM THE ORIGINAL DATASET, SELECTING
OBSERVATIONS WITH REPLACEMENT.
• A BASE MODEL (WEAK MODEL) IS CREATED ON EACH OF THESE SUBSETS.
• THE MODELS RUN IN PARALLEL AND ARE INDEPENDENT OF EACH OTHER.
• THE FINAL PREDICTIONS ARE DETERMINED BY COMBINING THE PREDICTIONS FROM ALL
THE MODELS.
• NOW, BAGGING CAN BE REPRESENTED DIAGRAMMATICALLY AS FOLLOWS:

BOOSTING
• BOOSTING IS A SEQUENTIAL PROCESS, WHERE EACH SUBSEQUENT MODEL
ATTEMPTS TO CORRECT THE ERRORS OF THE PREVIOUS MODEL. THE
SUCCEEDING MODELS ARE DEPENDENT ON THE PREVIOUS MODEL.
• IN THIS TECHNIQUE, LEARNERS ARE LEARNED SEQUENTIALLY WITH EARLY
LEARNERS FITTING SIMPLE MODELS TO THE DATA AND THEN ANALYZING DATA
FOR ERRORS. IN OTHER WORDS, WE FIT CONSECUTIVE TREES (RANDOM SAMPLE)
AND AT EVERY STEP, THE GOAL IS TO SOLVE FOR NET ERROR FROM THE PRIOR
TREE.
• WHEN AN INPUT IS MISCLASSIFIED BY A HYPOTHESIS, ITS WEIGHT IS INCREASED
SO THAT NEXT HYPOTHESIS IS MORE LIKELY TO CLASSIFY IT CORRECTLY. BY
COMBINING THE WHOLE SET AT THE END CONVERTS WEAK LEARNERS INTO
BETTER PERFORMING MODEL.

• LET’S UNDERSTAND THE WAY BOOSTING WORKS IN THE BELOW STEPS.
• A SUBSET IS CREATED FROM THE ORIGINAL DATASET.
• INITIALLY, ALL DATA POINTS ARE GIVEN EQUAL WEIGHTS.
• A BASE MODEL IS CREATED ON THIS SUBSET.
• THIS MODEL IS USED TO MAKE PREDICTIONS ON THE WHOLE DATASET.
• ERRORS ARE CALCULATED USING THE ACTUAL VALUES AND PREDICTED VALUES.
• THE OBSERVATIONS WHICH ARE INCORRECTLY PREDICTED, ARE GIVEN HIGHER
WEIGHTS. (HERE, THE THREE MISCLASSIFIED BLUE-PLUS POINTS WILL BE GIVEN
HIGHER WEIGHTS)
• ANOTHER MODEL IS CREATED AND PREDICTIONS ARE MADE ON THE DATASET.
(THIS MODEL TRIES TO CORRECT THE ERRORS FROM THE PREVIOUS MODEL)

• SIMILARLY, MULTIPLE MODELS ARE CREATED,
EACH CORRECTING THE ERRORS OF THE
PREVIOUS MODEL.
• THE FINAL MODEL (STRONG LEARNER) IS THE
WEIGHTED MEAN OF ALL THE MODELS (WEAK
LEARNERS).
• THUS, THE BOOSTING ALGORITHM COMBINES A
NUMBER OF WEAK LEARNERS TO FORM A STRONG
LEARNER.
• THE INDIVIDUAL MODELS WOULD NOT PERFORM
WELL ON THE ENTIRE DATASET, BUT THEY WORK
WELL FOR SOME PART OF THE DATASET.
• THUS, EACH MODEL ACTUALLY BOOSTS THE
PERFORMANCE OF THE ENSEMBLE.

ENSEMBLE LEARNING
• LET’S UNDERSTAND THE CONCEPT OF ENSEMBLE LEARNING WITH AN EXAMPLE.
• SUPPOSE YOU ARE A MOVIE DIRECTOR AND YOU HAVE CREATED A SHORT MOVIE ON A VERY
IMPORTANT AND INTERESTING TOPIC.
• NOW, YOU WANT TO TAKE PRELIMINARY FEEDBACK (RATINGS) ON THE MOVIE BEFORE MAKING
IT PUBLIC.
• WHAT ARE THE POSSIBLE WAYS BY WHICH YOU CAN DO THAT?
 A: YOU MAY ASK ONE OF YOUR FRIENDS TO RATE THE MOVIE FOR YOU.
NOW IT’S ENTIRELY POSSIBLE THAT THE PERSON YOU HAVE CHOSEN LOVES YOU VERY MUCH
AND DOESN’T WANT TO BREAK YOUR HEART BY PROVIDING A 1-STAR RATING TO THE
HORRIBLE WORK YOU HAVE CREATED.
 B: ANOTHER WAY COULD BE BY ASKING 5 COLLEAGUES OF YOURS TO RATE THE MOVIE.
THIS SHOULD PROVIDE A BETTER IDEA OF THE MOVIE. THIS METHOD MAY PROVIDE HONEST
RATINGS FOR YOUR MOVIE. BUT A PROBLEM STILL EXISTS. THESE 5 PEOPLE MAY NOT BE
“SUBJECT MATTER EXPERTS” ON THE TOPIC OF YOUR MOVIE. SURE, THEY MIGHT UNDERSTAND
THE CINEMATOGRAPHY, THE SHOTS, OR THE AUDIO, BUT AT THE SAME TIME MAY NOT BE THE
BEST JUDGES OF DARK HUMOUR.
PPT: MADHAV
MISHRA
60

C: HOW ABOUT ASKING 50 PEOPLE TO RATE THE MOVIE?
SOME OF WHICH CAN BE YOUR FRIENDS, SOME OF THEM CAN BE YOUR
COLLEAGUES AND SOME MAY EVEN BE TOTAL STRANGERS.
• THE RESPONSES, IN THIS CASE, WOULD BE MORE GENERALIZED AND DIVERSIFIED
SINCE NOW YOU HAVE PEOPLE WITH DIFFERENT SETS OF SKILLS.
• WITH THESE EXAMPLES, YOU CAN INFER THAT A DIVERSE GROUP OF PEOPLE ARE
LIKELY TO MAKE BETTER DECISIONS AS COMPARED TO INDIVIDUALS.
• SIMILAR IS TRUE FOR A DIVERSE SET OF MODELS IN COMPARISON TO SINGLE
MODELS.
• THIS DIVERSIFICATION IN MACHINE LEARNING IS ACHIEVED BY A TECHNIQUE
CALLED ENSEMBLE LEARNING.

SIMPLE ENSEMBLE TECHNIQUES:
• IN THIS SECTION, WE WILL LOOK AT A FEW SIMPLE BUT POWERFUL TECHNIQUES,
NAMELY:
 MAX VOTING.
 AVERAGING.
 WEIGHTED AVERAGING.
• MAX VOTING:
 THE MAX VOTING METHOD IS GENERALLY USED FOR CLASSIFICATION PROBLEMS.
 IN THIS TECHNIQUE, MULTIPLE MODELS ARE USED TO MAKE PREDICTIONS FOR EACH
DATA POINT.
 THE PREDICTIONS BY EACH MODEL ARE CONSIDERED AS A ‘VOTE’.
 THE PREDICTIONS WHICH WE GET FROM THE MAJORITY OF THE MODELS ARE USED
AS THE FINAL PREDICTION.

• FOR EXAMPLE, WHEN YOU ASKED 5 OF YOUR COLLEAGUES TO RATE YOUR MOVIE (OUT OF 5);
WE’LL ASSUME THREE OF THEM RATED IT AS 4 WHILE TWO OF THEM GAVE IT A 5. SINCE THE
MAJORITY GAVE A RATING OF 4, THE FINAL RATING WILL BE TAKEN AS 4. YOU CAN CONSIDER
THIS AS TAKING THE MODE OF ALL THE PREDICTIONS.
• THE RESULT OF MAX VOTING WOULD BE SOMETHING LIKE THIS:
• AVERAGING
• SIMILAR TO THE MAX VOTING TECHNIQUE, MULTIPLE PREDICTIONS ARE MADE FOR EACH DATA
POINT IN AVERAGING. IN THIS METHOD, WE TAKE AN AVERAGE OF PREDICTIONS FROM ALL
THE MODELS AND USE IT TO MAKE THE FINAL PREDICTION. AVERAGING CAN BE USED FOR
MAKING PREDICTIONS IN REGRESSION PROBLEMS OR WHILE CALCULATING PROBABILITIES FOR
CLASSIFICATION PROBLEMS.
• FOR EXAMPLE, IN THE BELOW CASE, THE AVERAGING METHOD WOULD TAKE THE AVERAGE OF
ALL THE VALUES.
• I.E. (5+4+5+4+4)/5 = 4.4

• WEIGHTED AVERAGE:
• THIS IS AN EXTENSION OF THE AVERAGING METHOD. ALL MODELS ARE
ASSIGNED DIFFERENT WEIGHTS DEFINING THE IMPORTANCE OF EACH MODEL
FOR PREDICTION. FOR INSTANCE, IF TWO OF YOUR COLLEAGUES ARE CRITICS,
WHILE OTHERS HAVE NO PRIOR EXPERIENCE IN THIS FIELD, THEN THE ANSWERS
BY THESE TWO FRIENDS ARE GIVEN MORE IMPORTANCE AS COMPARED TO THE
OTHER PEOPLE.
• THE RESULT IS CALCULATED AS :
[(5*0.23) + (4*0.23) + (5*0.18) + (4*0.18) + (4*0.18)] = 4.41.
PPT: MADHAV
MISHRA
64

• ADVANCED ENSEMBLE TECHNIQUES
• NOW THAT WE HAVE COVERED THE BASIC ENSEMBLE TECHNIQUES, LET’S MOVE
ON TO UNDERSTANDING THE ADVANCED TECHNIQUES.
STACKING.
BLENDING.
BAGGING.
BOOSTING.

• STACKING
• STACKING IS AN ENSEMBLE LEARNING TECHNIQUE THAT USES PREDICTIONS
FROM MULTIPLE MODELS (FOR EXAMPLE DECISION TREE, KNN OR SVM) TO BUILD
A NEW MODEL.
• THIS MODEL IS USED FOR MAKING PREDICTIONS ON THE TEST SET. BELOW IS A
STEP-WISE EXPLANATION FOR A SIMPLE STACKED ENSEMBLE:
1 THE TRAIN SET IS SPLIT INTO 10 PARTS:
2 A BASE MODEL (SUPPOSE A DECISION TREE) IS FITTED ON 9 PARTS AND
PREDICTIONS ARE MADE FOR THE 10TH PART. THIS IS DONE FOR EACH PART OF
THE TRAIN SET.
PPT: MADHAV MISHRA
66

3. THE BASE MODEL (IN THIS CASE, DECISION TREE) IS THEN FITTED ON THE
WHOLE TRAIN DATASET.
4. USING THIS MODEL, PREDICTIONS ARE MADE ON THE TEST SET
5. STEPS 2 TO 4 ARE REPEATED FOR ANOTHER BASE MODEL (SAY KNN) RESULTING
IN ANOTHER SET OF PREDICTIONS FOR THE TRAIN SET AND TEST SET.
PPT: MADHAV MISHRA
67

6. THE PREDICTIONS FROM THE TRAIN SET ARE USED AS FEATURES TO BUILD A
NEW MODEL.
7. THIS MODEL IS USED TO MAKE FINAL PREDICTIONS ON THE TEST PREDICTION
SET.
PPT: MADHAV MISHRA
68

• BLENDING:
• BLENDING FOLLOWS THE SAME APPROACH AS STACKING BUT USES ONLY A
HOLDOUT (VALIDATION) SET FROM THE TRAIN SET TO MAKE PREDICTIONS.
• IN OTHER WORDS, UNLIKE STACKING, THE PREDICTIONS ARE MADE ON THE
HOLDOUT SET ONLY. THE HOLDOUT SET AND THE PREDICTIONS ARE USED TO
BUILD A MODEL WHICH IS RUN ON THE TEST SET. HERE IS A DETAILED
EXPLANATION OF THE BLENDING PROCESS:
1. THE TRAIN SET IS SPLIT INTO TRAINING AND VALIDATION SETS.
PPT: MADHAV MISHRA
69

2. MODEL(S) ARE FITTED ON THE TRAINING SET.
3. THE PREDICTIONS ARE MADE ON THE VALIDATION SET AND THE TEST SET.
4. THE VALIDATION SET AND ITS PREDICTIONS ARE USED AS FEATURES TO BUILD
A NEW MODEL.
5. THIS MODEL IS USED TO MAKE FINAL PREDICTIONS ON THE TEST AND META-
FEATURES.
PPT: MADHAV MISHRA
70

• BAGGING & BOOSTING: COVERED AHEAD IN THE SLIDES
• ALGORITHMS BASED ON BAGGING AND BOOSTING
• BAGGING AND BOOSTING ARE TWO OF THE MOST COMMONLY USED TECHNIQUES IN
MACHINE LEARNING. IN THIS SECTION, WE WILL LOOK AT THEM IN DETAIL.
FOLLOWING ARE THE ALGORITHMS WE WILL BE FOCUSING ON:
BAGGING ALGORITHMS:
• BAGGING META-ESTIMATOR
• RANDOM FOREST
BOOSTING ALGORITHMS:
• ADABOOST
• GBM
• XGBM
• LIGHT GBM
• CATBOOST
PPT: MADHAV MISHRA
71

ONLINE LEARNING AND SEQUENCE
PREDICTION
• ONLINE MACHINE LEARNING IS A METHOD OF MACHINE LEARNING IN WHICH
DATA BECOMES AVAILABLE IN A SEQUENTIAL ORDER AND IS USED TO UPDATE
THE BEST PREDICTOR FOR FUTURE DATA AT EACH STEP, AS OPPOSED TO BATCH
LEARNING TECHNIQUES WHICH GENERATE THE BEST PREDICTOR BY LEARNING
ON THE ENTIRE TRAINING DATA SET AT ONCE.
• ONLINE LEARNING IS A COMMON TECHNIQUE USED IN AREAS OF MACHINE
LEARNING WHERE IT IS COMPUTATIONALLY INFEASIBLE TO TRAIN OVER THE
ENTIRE DATASET, REQUIRING THE NEED OF OUT-OF-CORE ALGORITHMS.
• IT IS ALSO USED IN SITUATIONS WHERE IT IS NECESSARY FOR THE ALGORITHM
TO DYNAMICALLY ADAPT TO NEW PATTERNS IN THE DATA, OR WHEN THE DATA
ITSELF IS GENERATED AS A FUNCTION OF TIME, E.G., STOCK PRICE PREDICTION

• SEQUENCE PREDICTION IS A POPULAR MACHINE LEARNING TASK, WHICH CONSISTS OF
PREDICTING THE NEXT SYMBOL(S) BASED ON THE PREVIOUSLY OBSERVED SEQUENCE OF
SYMBOLS. THESE SYMBOLS COULD BE A NUMBER, AN ALPHABET, A WORD, AN EVENT,
OR AN OBJECT LIKE A WEBPAGE OR PRODUCT. FOR EXAMPLE:
 A SEQUENCE OF WORDS OR CHARACTERS IN A TEXT.
 A SEQUENCE OF PRODUCTS BOUGHT BY A CUSTOMER.
 A SEQUENCE OF EVENTS OBSERVED ON LOGS.
• SEQUENCE PREDICTION IS DIFFERENT FROM OTHER TYPES OF SUPERVISED LEARNING
PROBLEMS, AS IT IMPOSES THAT THE ORDER IN THE DATA MUST BE PRESERVED WHEN
TRAINING MODELS AND MAKING PREDICTIONS.
• SEQUENCE PREDICTION IS A COMMON PROBLEM WHICH FINDS REAL-LIFE APPLICATIONS
IN VARIOUS INDUSTRIES. WE WILL INTRODUCE YOU THREE TYPES OF SEQUENCE
PREDICTION PROBLEMS:
 PREDICTING THE NEXT VALUE.
 PREDICTING A CLASS LABEL.
 PREDICTING A SEQUENCE.
PPT: MADHAV
MISHRA
73

• PREDICTING THE NEXT VALUE
• BEING ABLE TO GUESS THE NEXT ELEMENT OF A SEQUENCE IS AN IMPORTANT
QUESTION IN MANY APPLICATIONS.
• A SEQUENCE PREDICTION MODEL LEARNS TO IDENTIFY THE PATTERN IN THE
SEQUENTIAL INPUT DATA AND PREDICT THE NEXT VALUE.

DEEP LEARNING
• DEEP LEARNING IS A MACHINE LEARNING TECHNIQUE THAT TEACHES COMPUTERS
TO DO WHAT COMES NATURALLY TO HUMANS: LEARN BY EXAMPLE. DEEP LEARNING
IS A KEY TECHNOLOGY BEHIND DRIVERLESS CARS, ENABLING THEM TO RECOGNIZE A
STOP SIGN.
• IT IS THE KEY TO VOICE CONTROL IN CONSUMER DEVICES LIKE PHONES, TABLETS,
TVS, AND HANDS-FREE SPEAKERS.
• DEEP LEARNING IS GETTING LOTS OF ATTENTION LATELY AND FOR GOOD REASON.
• IT’S ACHIEVING RESULTS THAT WERE NOT POSSIBLE BEFORE.
• IN DEEP LEARNING, A COMPUTER MODEL LEARNS TO PERFORM CLASSIFICATION
TASKS DIRECTLY FROM IMAGES, TEXT, OR SOUND.
• DEEP LEARNING MODELS CAN ACHIEVE STATE-OF-THE-ART ACCURACY, SOMETIMES
EXCEEDING HUMAN-LEVEL PERFORMANCE.
• MODELS ARE TRAINED BY USING A LARGE SET OF LABELED DATA AND NEURAL
NETWORK ARCHITECTURES THAT CONTAIN MANY LAYERS.

• MOST DEEP LEARNING METHODS USE NEURAL NETWORK ARCHITECTURES, WHICH IS
WHY DEEP LEARNING MODELS ARE OFTEN REFERRED TO AS DEEP NEURAL
NETWORKS.
• THE TERM “DEEP” USUALLY REFERS TO THE NUMBER OF HIDDEN LAYERS IN THE
NEURAL NETWORK. TRADITIONAL NEURAL NETWORKS ONLY CONTAIN 2-3 HIDDEN
LAYERS, WHILE DEEP NETWORKS CAN HAVE AS MANY AS 150.
• DEEP LEARNING MODELS ARE TRAINED BY USING LARGE SETS OF LABELED DATA AND
NEURAL NETWORK ARCHITECTURES THAT LEARN FEATURES DIRECTLY FROM THE
DATA WITHOUT THE NEED FOR MANUAL FEATURE EXTRACTION.

EXAMPLES OF DEEP LEARNING
• AS WE KNOW DEEP LEARNING AND MACHINE LEARNING ARE SUBSETS OF ARTIFICIAL
INTELLIGENCE BUT DEEP LEARNING TECHNOLOGY REPRESENTS THE NEXT
EVOLUTION OF MACHINE LEARNING.
• AS MACHINE LEARNING WILL WORK BASED ON ALGORITHMS AND PROGRAMS
DEVELOPED BY HUMANS WHEREAS DEEP LEARNING LEARNS THROUGH A NEURAL
NETWORK MODEL WHICH ACTS LIKE SIMILAR TO HUMANS AND ALLOWS MACHINE
OR COMPUTER TO ANALYZE THE DATA IN A SIMILAR WAY AS HUMANS DO. THIS
BECOMES POSSIBLE AS WE TRAIN THE NEURAL NETWORK MODELS WITH A HUGE
AMOUNT OF DATA AS DATA IS THE FUEL OR FOOD FOR NEURAL NETWORK
MODELS.
• BELOW ARE SOME OF THE EXAMPLES IN THE REAL WORLD:
COMPUTER VISION: COMPUTER VISION DEALS WITH ALGORITHMS FOR COMPUTERS
TO UNDERSTAND THE WORLD USING AN IMAGE AND VIDEO DATA AND TASKS SUCH
AS IMAGE RECOGNITION, IMAGE CLASSIFICATION, OBJECT DETECTION, IMAGE
SEGMENTATION, IMAGE RESTORATION ETC.
PPT: MADHAV
MISHRA
79

SPEECH AND NATURAL LANGUAGE PROCESSING: NATURAL LANGUAGE PROCESSING
DEALS WITH ALGORITHMS FOR COMPUTERS TO UNDERSTAND, INTERPRET, AND
MANIPULATE IN HUMAN LANGUAGE. NLP ALGORITHMS WORK WITH TEXT AND AUDIO
DATA AND TRANSFORM THEM INTO AUDIO OR TEXT OUTPUT. USING NLP WE CAN DO
TASKS SUCH AS SENTIMENT ANALYSIS, SPEECH RECOGNITION, LANGUAGE
TRANSITION, AND NATURAL LANGUAGE GENERATION ETC.
AUTONOMOUS VEHICLES: DEEP LEARNING MODELS ARE TRAINED WITH A HUGE
AMOUNT OF DATA FOR IDENTIFYING STREET SIGNS; SOME MODELS SPECIALIZE IN
IDENTIFYING PEDESTRIANS, IDENTIFYING HUMANS ETC. FOR DRIVERLESS CARS WHILE
DRIVING.
IMAGE FILTERING: BY USING DEEP LEARNING MODELS SUCH AS ADDING COLOR TO
BLACK-AND-WHITE IMAGES CAN BE DONE BY DEEP LEARNING MODELS WHICH WILL
TAKE MORE TIME IF WE DO MANUALLY.
PPT: MADHAV
MISHRA
80

APPLICATION OF DEEP LEARNING
• APPLICATIONS OF DEEP LEARNING ARE VAST, BUT WE WOULD TRY TO COVER THE MOST
USED APPLICATION OF DEEP LEARNING TECHNIQUES. HERE ARE SOME OF THE DEEP
LEARNING APPLICATIONS, WHICH ARE NOW CHANGING THE WORLD AROUND US VERY
RAPIDLY.
TOXICITY DETECTION FOR DIFFERENT CHEMICAL STRUCTURES
HERE DEEP LEARNING METHOD IS VERY EFFICIENT, WHERE EXPERTS USED TO TAKE
DECADES OF TIME TO DETERMINE THE TOXICITY OF A SPECIFIC STRUCTURE, BUT WITH
DEEP LEARNING MODEL IT IS POSSIBLE TO DETERMINE TOXICITY IN VERY LESS AMOUNT
OF TIME (DEPENDS ON COMPLEXITY COULD BE HOURS OR DAYS).
MITOSIS DETECTION/ RADIOLOGY
DETERMINING CANCER DETECTION DEEP LEARNING MODEL HAS 6000 FACTORS WHICH
COULD HELP IN PREDICTING THE SURVIVAL OF A PATIENT. FOR BREAST CANCER
DIAGNOSIS DEEP LEARNING MODEL HAS BEEN PROVEN EFFICIENT AND EFFECTIVE. CNN
MODEL OF DEEP LEARNING IS NOW ABLE TO DETECT AS WELL AS CLASSIFY MITOSIS
INPATIENT. DEEP NEURAL NETWORKS HELP IN THE INVESTIGATION OF THE CELL LIFE
CYCLE
PPT: MADHAV
MISHRA
81

TEXT EXTRACTION AND TEXT RECOGNITION
TEXT EXTRACTION ITSELF HAS A LOT OF APPLICATIONS IN THE REAL WORLD. FOR
EXAMPLE, AUTOMATIC TRANSLATION FROM ONE LANGUAGE TO OTHER,
SENTIMENTAL ANALYSIS OF DIFFERENT REVIEWS. THIS WIDELY IS KNOWN AS
NATURAL LANGUAGE PROCESSING. WHEN WRITING AN EMAIL WE SEE AUTO-
SUGGESTION TO COMPLETE THE SENTENCE IS ALSO THE APPLICATION OF DEEP
LEARNING.
MARKET PREDICTION
DEEP LEARNING MODELS CAN PREDICT BUY AND SELL CALLS FOR TRADERS,
DEPENDING ON THE DATASET HOW THE MODEL HAS BEEN TRAINED, IT IS USEFUL
FOR BOTH SHORT TERM TRADING GAME AS WELL AS LONG TERM INVESTMENT BASED
ON THE AVAILABLE FEATURES.
FRAUD DETECTION
A DEEP LEARNING MODEL USES MULTIPLE DATA SOURCES TO FLAG A DECISION AS A
FRAUD IN REAL-TIME. WITH DEEP LEARNING MODELS, IT IS ALSO POSSIBLE TO FIND
OUT WHICH PRODUCT AND WHICH MARKETS ARE MOST SUSCEPTIBLE TO FRAUD AND
PROVIDE OR EXTRA CARE IN SUCH CASES.
PPT: MADHAV
MISHRA
82

REINFORCEMENT LEARNING
• REINFORCEMENT IS THE FIELD OF MACHINE LEARNING THAT INVOLVES
LEARNING WITHOUT THE INVOLVEMENT OF ANY HUMAN INTERACTION AS IT
HAS AN AGENT THAT LEARNS HOW TO BEHAVE IN AN ENVIRONMENT BY
PERFORMING ACTIONS AND THEN LEARN BASED UPON THE OUTCOME OF THESE
ACTIONS TO OBTAIN THE REQUIRED GOAL THAT IS SET BY THE SYSTEM TWO
ACCOMPLISH.
• BASED UPON THE TYPE OF GOALS IT IS CLASSIFIED AS POSITIVE AND NEGATIVE
LEARNING METHODS WITH THERE APPLICATION IN THE FIELD OF HEALTHCARE,
EDUCATION, COMPUTER VISION, GAMES, NLP, TRANSPORTATION, ETC.

UNDERSTAND REINFORCEMENT LEARNING
• LET US TRY TO UNDER THE WORKING OF
REINFORCEMENT LEARNING WITH THE
HELP OF 2 SIMPLE USE CASES:
• CASE #1
• THERE IS A BABY IN THE FAMILY AND SHE
HAS JUST STARTED WALKING AND
EVERYONE IS QUITE HAPPY ABOUT IT. ONE
DAY, THE PARENTS TRY TO SET A GOAL,
LET US BABY REACH THE COUCH, AND SEE
IF THE BABY IS ABLE TO DO SO.
• RESULT OF CASE 1: THE BABY
SUCCESSFULLY REACHES THE SETTEE AND
THUS EVERYONE IN THE FAMILY IS VERY
HAPPY TO SEE THIS. THE CHOSEN PATH
NOW COMES WITH A POSITIVE REWARD.
• POINTS: REWARD + (+N) → POSITIVE
REWARD.

• CASE #2
• THE BABY WAS NOT ABLE TO REACH THE
COUCH AND THE BABY HAS FALLEN.
• IT HURTS! WHAT POSSIBLY COULD BE THE
REASON?
• THERE MIGHT BE SOME OBSTACLES IN THE
PATH TO THE COUCH AND THE BABY HAD
FALLEN TO OBSTACLES.
• RESULT OF CASE 2: THE BABY FALLS TO
SOME OBSTACLES AND SHE CRIES! OH,
THAT WAS BAD, SHE LEARNED, NOT TO
FALL IN THE TRAP OF OBSTACLE THE NEXT
TIME. THE CHOSEN PATH NOW COMES
WITH A NEGATIVE REWARD.
• POINTS: REWARDS + (-N) →NEGATIVE
REWARD.

TYPES OF REINFORCEMENT LEARNING
• BELOW ARE THE TWO TYPES OF REINFORCEMENT LEARNING WITH THEIR
ADVANTAGES AND DISADVANTAGES:
1. POSITIVE
• WHEN THE STRENGTH AND FREQUENCY OF THE BEHAVIOR ARE INCREASED DUE TO
THE OCCURRENCE OF SOME PARTICULAR BEHAVIOR, IT IS KNOWN AS POSITIVE
REINFORCEMENT LEARNING.
ADVANTAGES: THE PERFORMANCE IS MAXIMIZED AND THE CHANGE REMAINS FOR A
LONGER TIME.
DISADVANTAGES: RESULTS CAN BE DIMINISHED IF WE HAVE TOO MUCH
REINFORCEMENT.
2. NEGATIVE
• IT IS THE STRENGTHENING OF BEHAVIOR, MOSTLY BECAUSE OF THE NEGATIVE TERM
VANISHES.
ADVANTAGES: BEHAVIOR IS INCREASED.
DISADVANTAGES: ONLY THE MINIMUM BEHAVIOR OF THE MODEL CAN BE REACHED
WITH THE HELP OF NEGATIVE REINFORCEMENT LEARNING.
PPT: MADHAV
MISHRA
86

THANK YOU…!!

Machine Learning Unit 4 Semester 3 MSc IT Part 2 Mumbai University

More Related Content

What's hot (20)

Similar to Machine Learning Unit 4 Semester 3 MSc IT Part 2 Mumbai University (20)

Recently uploaded (20)

Machine Learning Unit 4 Semester 3 MSc IT Part 2 Mumbai University