Understanding Bagging and Boosting
4 likes4,150 views
The document discusses supervised learning techniques, focusing on ensemble methods like bagging and boosting that combine weak learners to improve performance. Bagging reduces variance by generating multiple datasets and creating diverse models, while boosting sequentially enhances accuracy by reducing bias through a cost-based model combination. Additionally, it analyzes feature importance and accuracy evaluation for algorithms like XGBoost and Random Forest, highlighting differences in handling correlated features and model performance metrics.
Understanding Bagging and Boosting
- 2. Understanding Bagging and Boosting Both are ensemble techniques, where a set of weak learners are combined to create a strong learner that obtains better performance than a single one. Error = Bias + Variance + Noise
- 3. Bagging short for Bootstrap Aggregating It’s a way to increase accuracy by Decreasing Variance Done by Generating additional dataset using combinations with repetitions to produce multisets of same cardinality/size as original dataset. Example: Random Forest Develops fully grown decision trees (low bias high variance) which are uncorrelated to maximize the decrease in variance. Since cannot reduce bias therefore req. large unpruned trees.
- 4. Boosting It’s a way to increase accuracy by Reducing Bias 2- step Process Done by Develop averagely performing models over subsets of the original data. Boost these model performance by combining them using a cost function (eg.majority vote). Note: every subsets contains elements that were misclassified or were close by the previous model. Example: Gradient Boosted Tree Develops shallow decision trees (high bias low variance) aka weak larner. Reduce error mainly by reducing bias developing new learner taking into account the previous learner (Sequential).
- 10. Comparison Both are ensemble methods to get N learners from 1 learner… … but, while they are built independently for Bagging, Boosting tries to add new models that do well where previous models fail. Both generate several training data sets by random sampling… … but only Boosting determines weights for the data to tip the scales in favor of the most difficult cases. Both make the final decision by averaging the N learners (or taking the majority of them)… … but it is an equally weighted average for Bagging and a weighted average for Boosting, more weight to those with better performance on training data. Both are good at reducing variance and provide higher stability… … but only Boosting tries to reduce bias. On the other hand, Bagging may solve the overfitting problem, while Boosting can increase it. Similarities Differences
- 11. Exploring the Scope of Supervised Learning in Current Setup Areas where Supervised Learning can be useful Feature Selection for Clustering Evaluating Features Increasing the Aggressiveness of the Current setup Bringing New Rules Idea
- 12. Feature Selection/ Feature Importance & Model Accuracy and Threshold Evaluation Algorithm Used Feature Importance Metric XGBoost F Score Random Forest Gini Index, Entropy
- 14. Feature Selection/ Importance RF - Gini Index
- 16. Feature Selection/ Importance Comparison b/w Important Feature by Random Forest & XGBoost feature_21w feature_sut feature_du1 feature_sc3 feature_drh feature_1a2 feature_sc18 feature_drl feature_snc feature_sc1 feature_2c3 feature_npb feature_3e1 feature_bst feature_nub RF - Entropy feature_sut feature_sc3 feature_21w feature_sc18 feature_du1 feature_sc1 feature_drh feature_drl feature_1a2 feature_snc feature_npb feature_3e1 feature_tbu feature_nub feature_bst RF - GiniXGBoost - F Score feature_1a2 feature_2c3 feature_hhs feature_nrp feature_urh feature_nub feature_nup feature_psc feature_sncp feature_3e1 feature_tpa feature_snc feature_bst feature_tbu feature_nub Analysis of Top 15 important variable
- 17. Feature Selection/ Importance Comparison b/w Important Feature by Random Forest & XGBoost Reason for difference in Feature Importance b/w XGB & RF Basically, when there are several correlated features, boosting will tend to choose one and use it in several trees (if necessary). Other correlated features won t be used a lot (or not at all). It makes sense as other correlated features can't help in the split process anymore -> they don't bring new information regarding the already used feature. And the learning is done in a serial way. Each tree of a Random forest is not built from the same features (there is a random selection of features to use for each tree). Each correlated feature may have the chance to be selected in one of the tree. Therefore, when you look at the whole model it has used all features. The learning is done in parallel so each tree is not aware of what have been used for other trees. Tree Growth XGB When you grow too many trees, trees are starting to be look very similar (when there is no loss remaining to learn). Therefore the dominant feature will be an even more important. Having shallow trees reinforce this trend because there are few possible important features at the root of a tree (shared features between trees are most of the time the one at the root of it). So your results are not surprising. In this case, you may have interesting results with random selection of columns (rate around 0.8). Decreasing ETA may also help (keep more loss to explain after each iteration).
- 18. Model Accuracy and Threshold Evaluation XGBoost
- 19. Model Accuracy and Threshold Evaluation XGBoost A A A A BB B B
- 20. Model Accuracy and Threshold Evaluation XGBoost A A A A BB B B
- 21. Model Accuracy and Threshold Evaluation XGBoost A A A A BB B B
- 22. Model Accuracy and Threshold Evaluation XGBoost A A A A BB B B
- 23. Model Accuracy and Threshold Evaluation XGBoost A A A A BB B B
- 24. Model Accuracy and Threshold Evaluation XGBoost Threshold Accuracy TN FP FN TP 0 0.059% 0 46990 0 2936 0.1 87.353% 42229 4761 1553 1383 0.2 93.881% 46075 915 2140 796 0.3 94.722% 46691 299 2336 600 0.4 94.894% 46866 124 2425 511 0.5 94.902% 46923 67 2478 458 0.6 94.866% 46956 34 2529 407 0.7 94.856% 46973 17 2551 385 0.8 94.824% 46977 13 2571 365 0.9 94.776% 46982 8 2600 336 1 94.119% 46990 0 2936 0 A A B B
- 25. Model Accuracy and Threshold Evaluation Random Forest Criteria - Gini Index Random Forest Criteria - Entropy Criteria Accuracy TN FP FN TP Gini 94.800% 46968 22 2574 362 Entropy 94.788% 46967 23 2579 357 A A A A BB B B
- 26. Model Accuracy and Threshold Evaluation Comparison b/w Random Forest & XGBoost Criteria Accuracy TN FP FN TP Gini 94.800% 46968 22 2574 362 Entropy 94.788% 46967 23 2579 357 Threshold Accuracy TN FP FN TP 0 0.059% 0 46990 0 2936 0.1 87.353% 42229 4761 1553 1383 0.2 93.881% 46075 915 2140 796 0.3 94.722% 46691 299 2336 600 0.4 94.894% 46866 124 2425 511 0.5 94.902% 46923 67 2478 458 0.6 94.866% 46956 34 2529 407 0.7 94.856% 46973 17 2551 385 0.8 94.824% 46977 13 2571 365 0.9 94.776% 46982 8 2600 336 1 94.119% 46990 0 2936 0
- 27. Bringing New Rules Idea Comparison b/w Random Forest & XGBoost
- 28. Bringing New Rules Idea Comparison b/w Random Forest & XGBoost