Statistical Modeling in 3D: Describing, Explaining and Predicting
Download as PPTX, PDF2 likes331 views
This document discusses statistical modeling approaches for explaining, predicting, and describing. It notes that explanatory modeling focuses on testing causal hypotheses, predictive modeling focuses on predicting new observations, and descriptive modeling approximates distributions or relationships. The document argues that these goals are different and the best model for one purpose is not necessarily best for another. It cautions against conflating explanation and prediction, and notes that explanatory power does not necessarily indicate predictive power or vice versa. The document examines differences in how data is approached and models are designed and evaluated for these different purposes.
![Predict ≠ Explain
+ ?
“we tried to benefit from an
extensive set of attributes
describing each of the movies in
the dataset. Those attributes
certainly carry a significant signal
and can explain some of the user
behavior. However… they could
not help at all for improving the
[predictive] accuracy.”
Bell et al., 2008](https://image.slidesharecdn.com/explainpredictdescribeupadovajune2018-180615125259/85/Statistical-Modeling-in-3D-Describing-Explaining-and-Predicting-27-320.jpg)
![Predict ≠ Describe
Election Polls
“There is a subtle, but important, difference between
reflecting current public sentiment and predicting the
results of an election. Surveys have focused largely on
the former… [as opposed to] survey based prediction
models [that are] focused entirely on analysis and
projection”
Kenett, Pfefferman & Steinberg (2017) “Election Polls – A Survey, A Critique,
and Proposals”, Annual Rev of Stat & its Applications](https://image.slidesharecdn.com/explainpredictdescribeupadovajune2018-180615125259/85/Statistical-Modeling-in-3D-Describing-Explaining-and-Predicting-28-320.jpg)
Statistical Modeling in 3D: Describing, Explaining and Predicting
- 1. Statistical Modeling in 3D: Describing Explaining Predicting University of Padova, Jun 15, 2018 Galit Shmueli 徐茉 莉 Institute of Service Science
- 2. 1997-2000 (PhD, Statistics) Israel Institute of Technology Faculty of IE & M 2000-2002 Carnegie Mellon Univ. Department of Statistics 2002-2012 Univ. of Maryland Smith School of Business 2011-2014 Indian School of Business Hyderabad, India 2014-… National Tsing Hua Univ. Institute of Service Science My Academic Path My Research ‘Entrepreneurial’ statistical & data mining modeling Interdisciplinary Statistical Strategy • To Explain or To Predict? • Information Quality • Data Mining for Causality • Predicting with Causal Models
- 3. Road Map 1. Definitions 2. Monopolies & confusion in academia & industry 3. Explanatory, predictive, descriptive modeling & evaluation are different Why? Different modeling paths Explanatory power vs. predictive power 4. Where next?
- 4. Definitions: Explain Explanatory modeling theory-based, statistical testing of causal hypotheses Explanatory power strength of relationship in statistical model
- 5. Definitions: Predict Predictive modeling empirical method for predicting new observations Predictive power ability to accurately predict new observations
- 6. Definitions: Describe Descriptive modeling statistical model for approximating a distribution or relationship Descriptive power goodness of fit, generalizable to population
- 7. Explai n Predict Describ e Social Sciences Machine Learning Statistics Monopolies in Different Fields
- 9. Social sciences & management research Domination of ”Explain” Purpose: test causal theory (“explain”) Association-based statistical models Prediction & description nearly absent
- 10. Start with a causal theory Generate causal hypotheses on constructs Operationalize constructs → measurable variables Fit statistical model Classic journal paper Statistical inference → causal conclusions
- 11. Misconception #1: The same model is best for explaining, describing, predicting Social Sci & Mgmt: Build explanatory model and use it to ”predict” “A good explanatory model will also predict well” “You must understand the underlying causes in order to predict” “To examine the predictive power of the proposed model, we compare it to four models in terms of R2 adjusted”
- 12. Misconception #1: The same model is best for explaining, describing, predicting CS/eng/stat: Build a predictive model and use it to ”explain” in cs / stat / engineering / industry 2014 6th International Conference on Mobile Computing, Applications and Services (Agent-based modeling using census data) “our model is able to provide both predictions of how the population may vote and why they are voting this way”… 2009 IEEE International Conference on Systems, Man and Cybernetics
- 13. Misconception #2: explain > predict or predict > explain Emanuel Parzen, Comment on “Statistical Modeling: The Two Cultures” Statistical Science 2001 “Correlation supersedes causation, and science can advance even without coherent models, unified theories, or really any mechanistic explanation at all” *Chris Anderson is the editor in chief of Wired
- 15. Philosophy of Science “Explanation and prediction have the same logical structure” Hempel & Oppenheim, 1948 “It becomes pertinent to investigate the possibilities of predictive procedures autonomous of those used for explanation” Helmer & Rescher, 1959 “Theories of social and human behavior address themselves to two distinct goals of science: (1) prediction and (2) understanding” Dubin, Theory Building, 1969
- 16. Why statistical explanatory modeling predictive modeling descriptive modeling are different
- 17. Explanatory Model: test/quantify causal effect between constructs for “average” unit in population Descriptive Model: test/quantify distribution or correlation structure for measured “average” unit in population Predictive Model: predict values for new/future individual units Different Scientific Goals Different generalization
- 18. Theory vs. its manifestation ?
- 19. Notation Theoretical constructs: X, Y Causal theoretical model: Y=F(X) Measurable variables: X, Y Statistical model: E(y)=f(X) Breiman, “Stat Modeling: The Two Cultures”, Stat Science, 2001
- 20. Five aspects to consider Theory – Causation – Retrospective – Bias – Average unit – Data Association Prospective Variance Individual unit
- 21. “The goal of finding models that are predictively accurate differs from the goal of finding models that are true.”
- 22. But there’s more than bias-variance
- 23. Example: Regression Model for Explanation yi|xi = b0 + b1xi +b2 xcontrols + ei parameter of interest (inference) Chosen to avoid Omitted Var Bias (better to over-specify) Measures of X, Y constructs Underlying model: X Y Danger: endogeneity
- 24. yi|xi = b0 + b1 x1i +…+bp xpi + ei parameters of interest (inference) Chosen b/c related to Y Danger: multicollinearity All variables treated/interpreted as observable Remain in model only if statistically significant Residual analysis for GoF & test assumptions Example: Regression Model for Description
- 25. yi|xi = b0 + b1 x1i +…+bp xpi + ei Quantity of interest for new i’s (prediction) Chosen b/c possibly correlated with Y Danger: over-fitting All variables treated as observable, available at time of prediction Retain only if improve out- of-sample prediction Evaluate overfitting (train vs holdout) Example: Regression Model for Prediction
- 27. Predict ≠ Explain + ? “we tried to benefit from an extensive set of attributes describing each of the movies in the dataset. Those attributes certainly carry a significant signal and can explain some of the user behavior. However… they could not help at all for improving the [predictive] accuracy.” Bell et al., 2008
- 28. Predict ≠ Describe Election Polls “There is a subtle, but important, difference between reflecting current public sentiment and predicting the results of an election. Surveys have focused largely on the former… [as opposed to] survey based prediction models [that are] focused entirely on analysis and projection” Kenett, Pfefferman & Steinberg (2017) “Election Polls – A Survey, A Critique, and Proposals”, Annual Rev of Stat & its Applications
- 29. Goal Definition Design & Collection Data Preparation EDA Variables? Methods? Evaluation, Validation & Model Selection Model Use & Reporting
- 30. Observational or experiment? Primary or secondary data? Instrument (reliability+validity vs. measurement accuracy) How much data? How to sample? Study design & data collection predict: increase group size explain/describe: increase #groups Multilevel (nested) data School Class Student
- 31. Data preprocessing Reduced-Feature Models Saar-Tsechansky & Provost, JMLR 2007
- 32. Data exploration, viz, reduction PCA Factor Analysis (interpretable) Dimension Reduction (fast, small)
- 33. Which variables? multicollinearity causation associations endogeneity ex-post availability identifiability A, B, A*B leading, coincident, lagging indicators
- 34. ensembles long/short regression omitted variables bias shrinkage models variance bias Methods / Models blackbox / interpretable mapping to theory
- 35. Evaluation, Validation & Model Selection training datastatistical model holdout data Predictive power Over-fitting analysis theoretical model statistical model Data Validation Model fit ≠ Explanatory power
- 36. Point #2 Cannot infer one from the others explanatory power predictive power descriptive power
- 37. out-of-sample Performance Metrics type I,II errors goodness-of-fit p-values overall, specific over-fitting costs prediction accuracy interpretation training vs holdout R2
- 39. Convinced ?
- 41. Currently in Academia (social sciences, management) • Theory-based explanatory modeling • Prediction underappreciated • Distinction blurred • Unfamiliar with predictive modeling – getting better How/why use prediction (predictive models + evaluation) for scientific research beyond project-specific solution/utility/profit?
- 42. The predictive power of an explanatory/descriptive model has important scientific value relevance, reality check, predictability
- 43. Generate new theory Develop measures Compare theories Improve theory Assess relevance Evaluate predictability Prediction for Scientific Research Shmueli & Koppius, “Predictive Analytics in Information Systems Research” MIS Quarterly, 2011
- 44. Currently in Industry (and machine learning) • Data-driven predictive modeling • Prediction over-appreciated • Distinction blurred • A-B testing • Unfamiliar with theory-based explanatory modeling Will the customer pay? What causes non-payment?
- 45. Implications: Short-term solutions Shallow/no understanding Ethical, social, human pitfalls Shmueli (2017) “Research Dilemmas With Behavioral Big Data”, Big Data, vol 5(2), pp. 98-119 How to do theory-based explanatory modeling with Behavioral Big Data?
- 46. Explain + Predict + Describe