XGBoost: the algorithm that wins every competition
- 1. XGBoost: the algorithm that wins every competition Poznań Univeristy of Technology; April 28th, 2016 meet.ml #1 - Applied Big Data and Machine Learning By Jarosław Szymczak
- 3. Some recent competitions from Kaggle (Binary classification problem) Product Classification Challenge (Multi-label classification problem) Store Sales (Regression problem)
- 4. Some recent competitions from Kaggle Features: 93
- 5. Some recent competitions from Kaggle CTR – click through rate (~ equal to probability of click) Features:
- 6. Some recent competitions from Kaggle Sales prediction for store at certain date Features: • store id • date • school and state holidays • store type • assortment • promotions • competition
- 7. Solutions evaluation 0 2 4 6 8 10 12 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 logarithmic loss for positive example • Root Mean Square Percentage Error (RMSPE) • Binary logarithmic loss (loss is capped at ~35 for a single observation) • Multi-class logarithmic loss
- 8. What algorithms we tackle the problems with?
- 9. Bias related errors Bagging Random forest Adaptive boosting Gradient boosting Variance related errors
- 10. Shall we play tenis?
- 11. Decision tree standard example – shall we play tenis?
- 12. Bias related errors Bagging Random forest Adaptive boosting Gradient boosting Variance related errors
- 13. But one tree for complicated problems is simply not enough, and we will need to reduce erros coming from: • variance • bias
- 14. Bias related errors Bagging Random forest Adaptive boosting Gradient boosting Variance related errors
- 15. 44 2 Bagging – bootstrap aggregation 2 5 5 6 2 3 3 4 5 5 1 1 3 4 4 6 1 2 2 + 5 1 2 3 4 5 6Our initial dataset: First draw: Second draw: Third draw: Fourth draw: 3 Average: Model 1 Model 2 Model 3 Model 4 Final model
- 16. Bias related errors Bagging Random forest Adaptive boosting Gradient boosting Variance related errors
- 17. Random forest
- 18. Bias related errors Bagging Random forest Adaptive boosting Gradient boosting Variance related errors
- 19. AdaBoost – adaptive boosting + + + - + + - - - -
- 20. AdaBoost – adaptive boosting + + + - + + - - - - + + + - + + - - - - + + + - + + - - - - + + + - + + - - - - ℎ1 ε1 = 0,30 α1 = 0,42 ℎ2 ε2 = 0,21 α2 = 0,65 ℎ3 ε3 = 0,14 α3 = 0,92 𝐻 = 0,42ℎ1 +0,65ℎ2 +0,92ℎ3
- 21. Bias related errors Bagging Random forest Adaptive boosting Gradient boosting Variance related errors
- 22. Another boosting approach What if we, instead of reweighting examples, made some corrections to prediction errors directly? Note: Residual is a gradient of single observation error contribution in one of the most common evaluation measure for regression: root mean squared error (RMSE) We add new model, to the one we already have, so: Ypred = X1(Y) + NEW(Y) NEW(Y) = X1(Y) - Ypred our residual
- 23. Gradient descent
- 24. Gradient boosting It can be: • same algorithm as for further steps • or something very simple (like uniform probabilities or average target in regression) For any function that: • agrregates the error from examples (e.g. log-loss, RMSE, but not AUC) • you can calculate gradient on example level (it is called pseudo-residual) Fit model to initial data Fit pseudo-residuals Finalization Sum up all the models
- 25. Bias related errors Bagging Random forest Adaptive boosting Gradient boosting Variance related errors
- 26. XGBoost by Tianqi Chen eXtreme Gradient Boosting Used for: • classification • regression • ranking with custom loss functions Interfaces for Python and R, can be executed on YARN custom tree building algorithm
- 27. XGBoost – handling the features Numeric values • for each numeric value, XGBoost finds the best available split (it is always a binary split) • algorithm is designed to work with numeric values only Nominal values • need to be converted to numeric ones • classic way is to perform one-hot-encoding / get dummies (for all values) • for variables with large cardinality, some other, more sophisticated methods may need to be used Missing values • XGBoost handles missing values separately • missing value is always added to a branch in which it would minimize the loss (so it is either treated as very large / very small value) • it is a great advantage over, e.g. scikit-learn gradient boosting implementation
- 28. XGBoost – loss functions Used as optimization objective • logloss (optimized in complicated manner) for objectives binary:logistic and reg:logistic (they only differ by evaluation measure, all the rest is the same) • softmax (which is logloss transofrmed to handle multiple classes) for objective multi:softmax • RMSE (root mean squared error) for objective reg:linear Only measured • AUC – area under ROC curve • MAE – mean absolute error • error / merror – error rate for binary / multi-class classification Custom • only measured • simply implement the evaluation function • used as optimization objective • provided that you have a measure that is aggregation of element-wise losses and is differentiable • you create a function for element-wise gradient (derivative of prediction function) • then hessian (second derivative of prediction function) • that’s it, algorithm will approximate your loss function with Taylor series (pre-defined loss functions are implemented in the same manner)
- 29. XGBoost – setup for Python environment Installing XGBoost in your environment Check out this wonderful tutorial to install Docker with kaggle Python stack: http://goo.gl/wSCkmC http://xgboost.readthedocs.org/en/latest/build.html Link as above, expect potential troubles Clone: https://github.com/dmlc/xgboost At revision: 219e58d453daf26d6717e2266b83fca0071f3129 And follow the instructions in it (it is of course older version)
- 30. XGBoost code example on Otto data Train dataset Test dataset Sample submission
- 31. XGBoost – simple use example (direct) Link to script: https://goo.gl/vwQeXs for direct use we need to specify number of classes scikit-learn conventional names to account for examples importance we can assign weights to them in DMatrix (not done here)
- 32. XGBoost – feature importance Link to script: http://pastebin.com/uPK5aNkf
- 33. XGBoost – simple use example (in scikit-learn style) Link to script: https://goo.gl/IPxKh5
- 34. XGBoost – most common parameters for tree booster subsample • ratio of instances to take • example values: 0.6, 0.9 colsample_by_tree • ratio of columns used for whole tree creation • example values: 0.1, …, 0.9 colsample_by_level • ratio of columns sampled for each split • example values: 0.1, …, 0.9 eta • how fast algorithm will learn (shrinkage) • example values: 0.01, 0.05, 0.1 max_depth • maximum numer of consecutive splits • example values: 1, …, 15 (for dozen or so or more, needs to be set with regularization parametrs) min_child_weight • minimum weight of children in leaf, needs to be adopted for each measure • example values: 1 (for linear regression it would be one example, for classification it is gradient of pseudo-residual)
- 35. XGBoost – regularization parameters for tree booster alpha • L1 norm (simple average) of weights for whole objective function lambda • L2 norm (root from average of squares) of weights, added as penalty to objective function gamma • L0 norm, multiplied by numer of leafs in a tree is used to decide whether to make a split lambda is in denominator because of various transformation, in original objective it is „classic” L2 penalty
- 36. XGBoost – some cool things not necesarilly present elswhere Watchlist You can create some small validation set to prevent overfitting without need to guess how many iterations would be enough Warm start Thanks to the nature of gradient boosting, you can take some already woking solution and start the „improvement” from it Easy cut- off at any round when classyfing Simple but powerful while tuning the parameters, allows you to recover from overfitted model in easy way
- 37. Last, but not least – who won?
- 38. Winning solutions The winning solution consists of over 20 xgboost models that each need about two hours to train when running three models in parallel on my laptop. So I think it could be done within 24 hours. Most of the models individually achieve a very competitive (top 3 leaderboard) score. by Gert Jacobusse I did quite a bit of manual feature engineering and my models are entirely based on xgboost. Feature engineering is a combination of “brutal force” (trying different transformations, etc that I know) and “heuristic” (trying to think about drivers of the target in real world settings). One thing I learned recently was entropy based features, which were useful in my model. by Owen Zhang
- 39. Winning solutions by Gilberto Titericz Jr and Stanislav Semenov
- 40. Even deeper dive • XGBoost: • https://github.com/dmlc/xgboost/blob/master/demo/README.md • More info about machine learning: • Uczenie maszynowe i sieci neuronowe. Krawiec K., Stefanowski J., Wydawnictwo PP, Poznań, 2003. (2 wydanie 2004) • The Elements of Statistical Learning: Data Mining, Inference, and Prediction by Trevor Hastie, Robert Tibshirani, Jerome Friedman (available on-line for free) • Sci-kit learn and pandas official pages: • http://scikit-learn.org/ • http://pandas.pydata.org/ • Kaggle blog and forum: • http://blog.kaggle.com/ (no free hunch) • https://www.datacamp.com/courses/kaggle-python-tutorial-on-machine-learning • https://www.datacamp.com/courses/intro-to-python-for-data-science • Python basics: • https://www.coursera.org/learn/interactive-python-1 • https://www.coursera.org/learn/interactive-python-2