TreeNet Tree Ensembles and CART Decision Trees: A Winning Combination
Download as PPTX, PDF1 like1,615 views
This document discusses how combining CART decision trees with TreeNet tree ensembles can overcome some of the shortcomings of CART alone. It outlines the advantages of CART, including its ability to handle different variable types and its interpretability. However, it also describes CART's disadvantages like high variance and data fragmentation. TreeNet is introduced as a method that can dramatically increase accuracy by growing many weak decision trees on residuals of previous trees. Using both CART and TreeNet harnesses the interpretability of CART models with TreeNet's ability to produce more accurate and stable predictions.
TreeNet Tree Ensembles and CART Decision Trees: A Winning Combination
- 1. TreeNet Tree Ensembles and CART Decision Trees: A Winning Combination October 2012 Mikhail Golovnya Salford Systems CART® software is a trademark of California Statistical Software, Inc. and is licensed exclusively to Salford Systems. TreeNet® software is a trademark of Salford Systems
- 2. Course Outline • CART decision tree pros/cons • TreeNet stochastic gradient boosting: a promising way to overcome the shortcomings of a single tree • Introducing TreeNet, a powerful modern ensemble of boosted trees o Methodology o Reporting o Interpretability o Post-processing o Interaction detection • Advantages of using both CART and TreeNet o Contribution from CART o Contribution from TreeNet © Salford Systems 2012
- 3. Demonstration Dataset 108,376 bank customers (commercial and individual) with 6,564 in bad standing over the past two years Goal: identify customers in bad standing using the following predictors Revolving utilization of credit Age of the primary account holder Debt ratio of the primary account holder Monthly income Number of open credit lines Number of mortgages Number of dependents © Salford Systems 2012
- 4. CART Advantages 1. Relatively fast 2. All types of variables 1. Numeric, binary, categorical, missing values 3. Invariant under monotone transformations 1. Variable scales are irrelevant 2. Immunity to outliers 3. Most variables can be used “as is” 4. Resistance to many irrelevant variables 5. Few tunable parameters “off-the-shelf” procedure 6. Interpretable model representation © Salford Systems 2012
- 5. CART Disadvantages 1. Trade-off: accuracy vs. interpretability 2. Piecewise-constant model 1. Big errors near region boundaries 2. Impossible to detect fine differences within the segment 3. Instability => high variance 1. Small data change => big model change (especially for large trees) 4. Data fragmentation – splitting 5. High interaction order model, unreasonably complicated way to represent simple additive dependencies © Salford Systems 2012
- 6. TreeNet Tree Ensembles • Complements CART advantages, while dramatically increasing accuracy Tree 1 Tree 2 Tree 3 + + First tree grown 2nd tree grown on 3rd tree grown on on original residuals from residuals from target. first. Predictions model consisting Intentionally made to improve of first two trees “weak” model first tree © Salford Systems 2012
- 7. TreeNet Overcomes CART’s Shortcomings Piecewise-Constant CART TreeNet Model Big errors near region Fine predictions, nearly boundaries, coarse emulating smooth predictions continuous response surface Instability and Variance CART TreeNet Small data changes Stable models due to induce big model changes averaging of individual (especially for large trees) tree responses Data Fragmentation CART TreeNet Relatively few predictors Each tree works with the make it into the model entire data – many opportunities for variables to enter High Interaction Order CART TreeNet Model Always enforced Allows precise control © Salford Systems 2012 over the interactions
- 8. TreeNet and CART A Winning Combination © Salford Systems 2012