Statistical Modeling: The Two Cultures
24 likes10,530 views
This document discusses two cultures of statistical modeling: data modeling, focused on understanding the underlying data-generating mechanisms, and algorithmic modeling, which emphasizes optimization and predictive accuracy. It highlights the limitations of traditional data modeling and advocates for the increased use of algorithmic methods in statistics, particularly in machine learning. The conclusion suggests that data analysts should prioritize predictive accuracy and incorporate machine learning techniques into their practices.
![Data Modeling
Algorithmic Modeling
Principles
Experiences
Conclusions
OUTLINE
1. Statistics: Data Modeling Culture
2. Machine Learning: Algorithmic Modeling Culture
3. Statistical Learning Principles
4. Personal Experience
5. Summary
Content heavily based on: ``Statistical Modeling: The two cultures''
from Leo Breiman [1]
1 / 19](https://image.slidesharecdn.com/presentation-140120075829-phpapp02/85/Statistical-Modeling-The-Two-Cultures-3-320.jpg)
![2014-01-20
.
.
Statistical Modeling: The Two Cultures
OUTLINE
1. Statistics: Data Modeling Culture
2. Machine Learning: Algorithmic Modeling Culture
3. Statistical Learning Principles
4. Personal Experience
Outline
5. Summary
.
Content heavily based on: ``Statistical Modeling: The two cultures''
from Leo Breiman [1]
.
This presentation is based on ``Statistical Modeling: The two
cultures'' from Leo Breiman [1].
The first segment introduces the data modeling culture and
analogously the second segment explains the algorithmic
modeling culture together with the presentation of three
algorithms. The part about statistical learning principals
presents aspects which help to compare both of cultures.
Personal experiences in both cultures are addressed. The
conclusion summarizes the message of the paper [1].](https://image.slidesharecdn.com/presentation-140120075829-phpapp02/85/Statistical-Modeling-The-Two-Cultures-4-320.jpg)
![Data Modeling
Algorithmic Modeling
Principles
Experiences
Conclusions
ALGORITHM IN MACHINE LEARNING
→ Boosting
→ Support Vector Machines
→ Artificial neural networks
→ Random Forests
→ Hidden Markov
→ Bayes-Nets
→ …1
1
Details and more algorithms in ``Elements of statistical learning''[2]
8 / 19](https://image.slidesharecdn.com/presentation-140120075829-phpapp02/85/Statistical-Modeling-The-Two-Cultures-17-320.jpg)
![2014-01-20
.
.
Statistical Modeling: The Two Cultures
ALGORITHM IN MACHINE LEARNING
→ Boosting
Algorithmic Modeling
→ Support Vector Machines
→ Artificial neural networks
→ Random Forests
→ Hidden Markov
→ Bayes-Nets
Algorithm in Machine Learning
→ …1
.
1
Details and more algorithms in ``Elements of statistical learning''[2]
.
The algorithms used in machine learning are motivated
differently. Three algorithms are presented in short.](https://image.slidesharecdn.com/presentation-140120075829-phpapp02/85/Statistical-Modeling-The-Two-Cultures-18-320.jpg)
Statistical Modeling: The Two Cultures
- 1. STATISTICAL MODELING: THE TWO CULTURES Based on Leo Breiman's paper Christoph Molnar Department of Statistics, LMU Munich
- 2. 2014-01-20 . . Statistical Modeling: The Two Cultures STATISTICAL MODELING: THE TWO CULTURES Based on Leo Breiman's paper . Christoph Molnar Department of Statistics, LMU Munich . Abstract: This presentation compares two cultures of statistical modeling: the data modeling culture, which assumes a stochastic process that produced the data. This culture is associated with traditional statistics. The other culture is called algorithmic modeling culture, which can be reduced to optimization of a loss function with an algorithm. This culture is associated with Machine Learning. It is argued to use algorithmic modeling more often in statistics.
- 3. Data Modeling Algorithmic Modeling Principles Experiences Conclusions OUTLINE 1. Statistics: Data Modeling Culture 2. Machine Learning: Algorithmic Modeling Culture 3. Statistical Learning Principles 4. Personal Experience 5. Summary Content heavily based on: ``Statistical Modeling: The two cultures'' from Leo Breiman [1] 1 / 19
- 4. 2014-01-20 . . Statistical Modeling: The Two Cultures OUTLINE 1. Statistics: Data Modeling Culture 2. Machine Learning: Algorithmic Modeling Culture 3. Statistical Learning Principles 4. Personal Experience Outline 5. Summary . Content heavily based on: ``Statistical Modeling: The two cultures'' from Leo Breiman [1] . This presentation is based on ``Statistical Modeling: The two cultures'' from Leo Breiman [1]. The first segment introduces the data modeling culture and analogously the second segment explains the algorithmic modeling culture together with the presentation of three algorithms. The part about statistical learning principals presents aspects which help to compare both of cultures. Personal experiences in both cultures are addressed. The conclusion summarizes the message of the paper [1].
- 6. Data Modeling Algorithmic Modeling Principles Experiences Conclusions WORK OF A DATA ANALYST → Predict → Reveal associations → Munge data, design experiments, visualize data, … 2 / 19
- 7. 2014-01-20 . . Statistical Modeling: The Two Cultures WORK OF A DATA ANALYST Data Modeling → Predict → Reveal associations → Munge data, design experiments, visualize data, … Work of a Data Analyst . . The work of a data analyst is very diverse. This presentation focuses on the modeling of data, which can be reduced to two targets: Learn a model to predict the outcome for new covariates and get a better understanding about the relationship between covariates and outcome.
- 8. Data Modeling Algorithmic Modeling Principles Experiences Conclusions SIMPLIFIED WORLDVIEW y . nature x 3 / 19
- 9. 2014-01-20 . . Statistical Modeling: The Two Cultures SIMPLIFIED WORLDVIEW Data Modeling y . nature Simplified worldview . . In a strongly simplified world an arbitrary outcome y is produced by the nature given the covariates x. The knowledge about the natures true mechanisms range between entirely unknown and established (scientific) explanations of the mechanism. One example: Outcome y is the rent for apartments and covariates x are size, number of bathrooms and location. x
- 10. Data Modeling Algorithmic Modeling Principles Experiences Conclusions DATA MODELING CULTURE y Logistic Regression, . Cox Model, GEE, … x Find a stochastic model of the data-generating process: y = f(x, parameters, random error) 4 / 19
- 11. 2014-01-20 . . Statistical Modeling: The Two Cultures DATA MODELING CULTURE Data Modeling y Data Modeling Culture Logistic Regression, . Cox Model, GEE, … x Find a stochastic model of the data-generating process: y = f(x, parameters, random error) . . The direct modeling of the mechanism in the ``box'' is labeled »Data Modeling Culture« by Leo Breiman. In this culture a stochastic model for the data- generating process is assumed. A common formulation of these model is: y is a function of x with corresponding weights and a random error. For example: Given the covariates size, number of bathrooms and location, the rent of apartments is normal distributed.
- 12. Data Modeling Algorithmic Modeling Principles Experiences Conclusions TYPICAL ASSUMPTIONS AND RESTRICTIONS → Specific stochastic model that generated the data → Distribution of residuals → Linearity (e.g. linear predictor) → Manual specification of interactions 5 / 19
- 13. Data Modeling Algorithmic Modeling Principles Experiences Conclusions PROBLEMS → Conclusions about model, not about nature → Assumptions often violated → Often no model evaluation → ⇒ can lead to irrelevant theory and questionable statistical conclusions → Focus not on prediction → Data models fail in areas like image and speech recognition 6 / 19
- 15. Data Modeling Algorithmic Modeling Principles Experiences Conclusions MACHINE LEARNING y . unknown x algorithm Find a function f (X) that minimizes the loss: L(Y, f (X)) 7 / 19
- 16. 2014-01-20 . . Statistical Modeling: The Two Cultures MACHINE LEARNING y Algorithmic Modeling Machine Learning . unknown x algorithm Find a function f (X) that minimizes the loss: L(Y, f (X)) . . In the »Algorithmic Modeling Culture«, the true mechanism is treated as unknown. It is not the target to find the true data-generating mechanism but to use an algorithm that imitates the mechanism as good as possible. Modeling is reduced to a mere problem of function optimization: Given the covariates x, outcome y and a loss function find a function f(x) which minimizes the loss for the prediction of the outcome. This culture is lived in the machine learning area. Summary: The data modeling culture tries to find the true data-generating mechanism, the algorithmic modeling culture tries to imitate the true mechanism as good as possible.
- 17. Data Modeling Algorithmic Modeling Principles Experiences Conclusions ALGORITHM IN MACHINE LEARNING → Boosting → Support Vector Machines → Artificial neural networks → Random Forests → Hidden Markov → Bayes-Nets → …1 1 Details and more algorithms in ``Elements of statistical learning''[2] 8 / 19
- 18. 2014-01-20 . . Statistical Modeling: The Two Cultures ALGORITHM IN MACHINE LEARNING → Boosting Algorithmic Modeling → Support Vector Machines → Artificial neural networks → Random Forests → Hidden Markov → Bayes-Nets Algorithm in Machine Learning → …1 . 1 Details and more algorithms in ``Elements of statistical learning''[2] . The algorithms used in machine learning are motivated differently. Three algorithms are presented in short.
- 19. Data Modeling Algorithmic Modeling Principles Experiences Conclusions ARTIFICIAL NEURAL NETWORKS 2 2 http://commons.wikimedia.org/wiki/File: Mouse_cingulate_cortex_neurons.jpg http://commons.wikimedia.org/wiki/File:Neural_network.svg 9 / 19
- 20. 2014-01-20 . . Statistical Modeling: The Two Cultures ARTIFICIAL NEURAL NETWORKS Algorithmic Modeling Artificial neural networks 2 . 2 http://commons.wikimedia.org/wiki/File: Mouse_cingulate_cortex_neurons.jpg http://commons.wikimedia.org/wiki/File:Neural_network.svg . Artificial neural networks are used in classification and regression. They are inspired by the brain, which consists of a network of brain cells (neurons). Mathematically artificial neural networks are a concatenation of weighted functions. An exemplary application is image processing.
- 21. Data Modeling Algorithmic Modeling Principles Experiences Conclusions SUPPORT VECTOR MACHINES 3 3 http://commons.wikimedia.org/wiki/File:Svm_10_perceptron.JPG 10 / 19
- 22. 2014-01-20 . . Statistical Modeling: The Two Cultures SUPPORT VECTOR MACHINES Algorithmic Modeling Support Vector Machines 3 . 3 http://commons.wikimedia.org/wiki/File:Svm_10_perceptron.JPG . Support Vector Machines (SVM) were originally a classification method (regression is also possible). SVMs try to draw a border between two classes in the covariate space. The distance between the border and the class points is maximized. They use a mathematical trick to implicitly project the covariates in a space with higher dimensions (yes, it sounds a bit crazy) in order to achieve class separation. Text classification is an exemplary usage.
- 23. Data Modeling Algorithmic Modeling Principles Experiences Conclusions RANDOM FORESTS 4 11 / 19
- 24. 2014-01-20 . . Statistical Modeling: The Two Cultures RANDOM FORESTS Algorithmic Modeling Random Forests . . 4 4 http://openclipart.org/detail/175304/forest-by-z-175304 Random Forests™(invented by Leo Breiman) are used for regression and classification. A Random Forest consists of many decision trees. Two random mechanisms are used to train different trees on the data. Results from all trees are averaged for the prediction.
- 26. Data Modeling Algorithmic Modeling Principles Experiences Conclusions RASHOMON EFFECT (Often) Many different models describe a situation equally accurate. 12 / 19
- 27. 2014-01-20 . . Statistical Modeling: The Two Cultures RASHOMON EFFECT Principles (Often) Many different models describe a situation equally accurate. Rashomon Effect . . Rashomon is a Japanese movie in which four witnesses tell different versions of a crime. All versions account for the facts but they contradict each other. In terms of statistical learning this means, that often different models (e.g. same y but different covariates) can be equally accurate. Each model has a different interpretation which makes it difficult to find the true mechanism in the data modeling cultures. In the algorithmic modeling culture the Rashomon effect is exploited by aggregating over many models. Random forests use this effect by aggregating over many trees. It is also common to average the predictions of different algorithms.
- 28. Data Modeling Algorithmic Modeling Principles Experiences Conclusions DIMENSIONALITY OF THE DATA → The higher the dimensionality of the data (# covariates) the more difficult is the separation of signal and noise → Common practice in data modeling: variable selection (by expert selection or data driven) and reduction of dimensionality (e.g. PCA) → Common practice in algorithmic modeling: Engineering of new features (covariates) to increase predictive accuracy; algorithms robust for many covariates 13 / 19
- 29. Data Modeling Algorithmic Modeling Principles Experiences Conclusions PREDICTION VS. INTERPRETATION Predictive accuracy Interpretability . Tree Logist. Regression … Random Forest Neuronal networks … 14 / 19
- 30. 2014-01-20 . . Statistical Modeling: The Two Cultures PREDICTION VS. INTERPRETATION Principles Predictive accuracy Interpretability . Tree Logist. Regression Prediction vs. Interpretation … . . There is a trade-off between interpretability and predictive accuracy: the models that are good in prediction are often complex and models that are easy to interpret are often bad predictors. See for example trees and Random Forests: A single decision tree is very intuitive and easy to read for non-professionals, but they are unstable and give weak predictions. A complex aggregation of decision trees (Random Forest) has an excellent prediction accuracy, but it is impossible to interpret the model structure. Random Forest Neuronal networks …
- 31. Data Modeling Algorithmic Modeling Principles Experiences Conclusions GOODNESS OF MODEL → Data modeling culture: Goodness of fit often based on model assumptions (e.g. AIC) and calculated on training data. → Algorithmic modeling culture: Evaluation of predictive accuracy with an extra test set or cross validation. How good is a statistical model if the predictive accuracy is weak? Is it legit to interpret parameters and p-values? 15 / 19
- 33. Data Modeling Algorithmic Modeling Principles Experiences Conclusions STATISTICAL CONSULTING Stereotypical user ... → are e.g. veterinarians, linguists, biologists, … → crave p-values → want interpretability → ignore model diagnosis 16 / 19
- 34. 2014-01-20 . . Statistical Modeling: The Two Cultures STATISTICAL CONSULTING Experiences Stereotypical user ... → are e.g. veterinarians, linguists, biologists, … → crave p-values → want interpretability → ignore model diagnosis Statistical Consulting . . From my experience in the statistical consulting unit of our university, most user want to test their scientific hypothesis with models. They want models which are easy to interpret regarding their questions. Thus it is more important to have a model that gives parameters associated with covariates and p-values than to have a model that predicts the data well.
- 35. Data Modeling Algorithmic Modeling Principles Experiences Conclusions KAGGLE Algorithms of winners on kaggle, a platform for prediction challenges: → Job Salary Prediction: »I used deep neural networks« → Observing Dark Worlds: »Bayesian analysis provided the winning recipe for solving this problem« → Give Me Some Credit: »In the end we only used five supervised learning methods: a random forest of classification trees, a random forest of regression trees, a classification tree boosting algorithm, a regression tree boosting algorithm, and a neural network.« 17 / 19
- 36. 2014-01-20 . . Statistical Modeling: The Two Cultures KAGGLE Algorithms of winners on kaggle, a platform for prediction challenges: Experiences → Job Salary Prediction: »I used deep neural networks« → Observing Dark Worlds: »Bayesian analysis provided the winning recipe for solving this problem« → Give Me Some Credit: »In the end we only used five supervised learning methods: a random forest of classification trees, a random forest of regression trees, a classification tree boosting algorithm, a regression tree boosting algorithm, and a neural network.« kaggle . . Kaggle (http://www.kaggle.com) is a platform for prediction challenges. Data are provided and the participant with the best prediction on a test set wins the challenge. To generate insights about the mechanisms in the data is secondary because the prediction is all that counts to win. That's why the algorithmic modeling culture is superior in this field.
- 37. Data Modeling Algorithmic Modeling Principles Experiences Conclusions CONCLUSION Data analysts should: → Use predictive accuracy to evaluate models → Seek the best model → Add Machine Learning to their toolbox 18 / 19
- 38. Data Modeling Algorithmic Modeling Principles Experiences Conclusions FURTHER LITERATURE L. Breiman Statistical modeling: The two cultures (with comments and a rejoinder by the author) Institute of Mathematical Statistics, 2001. T. Hastie, R. Tibshirani and J. Friedman The elements of statistical learning Springer New York, 2001 19 / 19
- 39. »All models are wrong, but some are useful« (G. Box)