Bayesian model averaging
5 likes5,169 views
The document discusses Bayesian model averaging (BMA) as a Bayesian approach to combining multiple models, explaining how to implement BMA using R packages, highlighting that BMA works well for linear models but its application to more complex models is still limited, and concludes by noting that BMA provides useful tools for model interpretation and combination beyond just prediction.
Bayesian model averaging
- 1. Volodymyrk Bayesian Model Averaging Bayesian Mixer, 27.09.2016 London, UK
- 2. Volodymyrk Bayesian Model Averaging (BMA) - 1 minute version New Project - how much does it worth? CFO VP of Growth Net Present Value: $50m $100m Model M1 Model M2 30%CEO belief: after evaluating both models and market data 70% $15m + $70m = $85m K = 2
- 3. Volodymyrk Bayesian Model Averaging (BMA) - 3 minute version VP of Growth CLV assumptions $10 $12 $15 CAC $4 72 129 149 $6 62 112 133 $8 51 92 101 Average= $100.11m Sensitivity Analysis for M2 DATA
- 4. Volodymyrk Bayesian Model Averaging (BMA) - 5 minute version Bayesian Model Averaging: A Tutorial Jennifer A. Hoeting, David Madigan, Adrian E. Raftery and Chris T. Volinsky How much do you trust your VP and CFO, before you look at models? Scary normalising term that you can ignore Prior probability for model parameter
- 5. Volodymyrk Bayesian answer to overfitting Frequentist: - model selection - regularisation Bayesian: - BMA - marginalisation
- 6. Volodymyrk Case Study You just get the best job in the galaxy
- 7. Volodymyrk Your new Boss Business domain Modelling case Always test your models on synthetic data that you understand and control
- 8. Volodymyrk Use Cases: - Fraud Detection - Inventory Sourcing Data
- 9. Volodymyrk Modelling goals - Prediction range is needed, so that you can identify fraudulent transactions (sand people under-reporting real transaction size and pocketing profit) - Sale price should be easily explainable, as a function of various Droid Features so that Jabba can invest in appropriate scavenging/sourcing projects - You want lowest prediction error possible so that you are not feeded to Sarlacc
- 12. Volodymyrk Model Selection - classical method credits ~ height + weight + power + dents + rad + wheels + legs + red + blue + black + temperature + lat + long + ir_emit + dents_log + height_log + weight_log + power_log + rad_log Adj. R2: 0.884974385182
- 13. Volodymyrk Model Selection - backward elimination
- 14. Volodymyrk Final Model credits ~ weight + power + dents + rad + wheels + blue + black + temperature + lat + dents_log + height_log + weight_log + power_log Adj. R2: 0.903544333611
- 16. Volodymyrk Ridge regression (L2 regularisation)
- 17. Volodymyrk Bayesian Model Averaging for Linear Models - a special case Inclusion probability for (regression coefficients) are weighted across all possible models Number of models = combinations of all K features (include/exclude) = 2K
- 18. Volodymyrk How to actually do BMA? (in R) cran.r-project.org/web/packages/BMA cran.r-project.org/web/packages/BAScran.r-project.org/web/packages/BMS Mature. A.k.a. “the original” Developed by PhD duringresearch. Not maintained Newest. Maintained by Chair of the Department ofStatistical Science at Duke
- 19. Volodymyrk BMA using BMS (R) package Model Selection L2 Regularisation BMA MSE 9736.49 7782.21 7329.44 It worked! But you can find inputs into data generator script that will not work as well!
- 20. Volodymyrk Nice things you get from BMA Posterior Inclusion Probability! How cool is that!
- 21. Volodymyrk Model ranking! MCMC can beused, if number of features is large Best model, according toBMA
- 22. Volodymyrk Can we use it for more complex models? normalising term that you can ignore http://www.ssc.wisc.edu/~bhansen/718/NonParametrics15.pdf http://www.ejwagenmakers.com/2004/aic.pdf Warning:Very questionable math. Does not work
- 23. Volodymyrk Can we use BMA to combine complex (incl. hierarchical) models? 1 3 2 Model order is somewhat similar. Relative probabilities are not. We need working Reverse-Jump MCMC or something more sophisticated. Not available in common bayesian MCMC packages yet.
- 24. Volodymyrk In Summary - BMA is a Bayesian version of ML Model Ensembles - Math behind is quite beautiful - Model Averaging is useful for interpretation, not only prediction - Invest in synthetic data generation, - before applying new modelling techniques to real-world data - Even if you are not using BMA, fit different models - And combine them, if your goal is prediction - BMA works very well for common GLMs, but does not work yet for arbitrary models - Do try it next time you need to fit OLS, though!
- 25. Volodymyrk Of course we are hiring! ● (Snr, Mid) Data Scientists ● Solutions Architect ● Ruby Developer ● Data Engineer ● Senior Artist ● Technical Artist ● Unity Developers ● Senior Product Manager ● Product Director http://jobs.productmadness.com/