Leakage in Meta Modeling And Its Connection to HCC Target-Encoding - Mathias Müller, Data Scientist, H2O.ai
Download as PPTX, PDF3 likes1,668 views
Mathias Müller discusses data leakage in meta modeling, particularly with HCC target encoding, and its effects on machine learning predictions. He explains how improper handling of data splits and target information can bias model performance and provides methods to mitigate these risks, including nested cross-validation. The presentation highlights the importance of avoiding target leakage to enhance model accuracy and reliability.
Leakage in Meta Modeling And Its Connection to HCC Target-Encoding - Mathias Müller, Data Scientist, H2O.ai
- 2. Mathias Müller faron@h2o.ai kaggle.com/mmueller - github.com/Far0n Leakage in Meta Modeling & Its Connection to HCC Target-Encoding
- 3. Background • Born & raised in Berlin • Diplom in Computer Science from Humboldt University of Berlin • Joined H2O two month ago • Data Scientist • Development of Driverless AI
- 5. Leakage “Data Leakage is the creation of unexpected additional information in the training data, allowing a model or machine learning algorithm to make unrealistically good predictions.” kaggle.com/wiki/leakage • Many different sources • ID-Leaks • Leaking future information into past • Validating models on already seen data • Leaking target information into Feature Matrices • Feedback loops / adaptive data analysis • … • Caused damage varies from case to case
- 6. 6 Meta Modeling / Stacking
- 7. Leakage in Meta Modeling • Suppose a 3-fold split of our training data into (A,B,C) • Creating of out-of-fold predictions (A2,B2,C2) • train((A,B)) followed by predict(C) to get C2 • train((A,C)) followed by predict(B) to get B2 • train((B,C)) followed by predict(A) to get A2 AB -> C2 A2B2 -> C3 (BC)(AC) -> C3 AC -> B2 A2C2 -> B3 (BC)(AB) -> B3 BC -> A2 B2C2 -> A3 (AC)(AB) -> A3 Base Level 1st Meta Level Leaked Target Information
- 8. HCC Target Encoding • In general, tree based models like XGBoost, LightGBM, RF, etc. struggle with (non-ordinal) High Cardinal Categoricals (HCC) features • Order of mapped HCC values determines the required amount of splits to get “useful” data partitions • Idea: Replace HCC values by their likelihoods to get a “good order”
- 9. K-Fold Target Encoding - Example X y Fold AB -> C AC -> B BC -> A X_lhood_cv blue 1 p(y | X = blue) = 0 0 red 1 p(y | X = red) = 1.0 1 blue 0 0.5 blue 0 0.5 blue 0 p(y | X = blue) = 0.333 0.333 red 1 p(y | X = red) = 1.0 1 A B C p(y | X = blue) = 0.5 • We want to replace the categorical values blue and red by their likelihoods in a k-fold cross-validated fashion:
- 10. Recap: Leakage in Meta Modeling AB -> C2 A2B2 -> C3 (BC)(AC) -> C3 AC -> B2 A2C2 -> B3 (BC)(AB) -> B3 BC -> A2 B2C2 -> A3 (AC)(AB) -> A3 Base Level 1st Meta Level Leaked Target Information
- 11. K-Fold Target Encoding - Example X y Fold AB -> C AC -> B BC -> A X_lhood_cv blue 1 p(y | X = blue) = 0 0 red 1 p(y | X = red) = 1.0 1 blue 0 0.5 blue 0 0.5 blue 0 p(y | X = blue) = 0.333 0.333 red 1 p(y | X = red) = 1.0 1 A B C p(y | X = blue) = 0.5 • X_lhood_cv values are basically “out-of-fold” predictions of a maximum likelihood estimator • Using X_lhood_cv as feature is pretty much the same procedure as stacking • Same leakage issue .. but fails more often than strong model stacking, because of no regularization
- 12. Counter-Measures • Using a fixed holdout set to calculate likelihoods / to generate out-of-fold predictions • Loss of training data at later stages • Using a 2-fold scheme with fixed seed • Not ideal regarding bias-variance-tradeoff • Adding Noise to likelihoods / out-of-fold predictions • Hard to get the noise level right (heavily dataset dependent) • Avoiding target leakage by nested cross validation • Order of magnitude higher complexitity: O(k) => O(kouter * kinner)
- 13. Nested Cross Validation AB -> C2 A2B2 -> C3 (B)(A) -> C3 A -> B2 B -> A2 AC -> B2 A2C2 -> B3 (C)(A) -> B3 A -> C2 C -> A2 BC -> A2 B2C2 -> A3 (C)(B) -> A3 B -> C2 C -> B2 Base Level 1st Meta Level No Leaked Target Information
- 14. Thank you for your attention! Any Questions?