Predictive analytics in Information Systems Research (TSWIM 2015 keynote)
4 likes1,118 views
The document discusses the distinction between explanatory and predictive modeling in information systems research, emphasizing their different purposes and methodologies. It highlights that while explanatory models test causal theories, predictive models aim to forecast new observations with empirical methods. The author argues for the importance of understanding these differences to enhance scientific research and its applications in business analytics.
![Predict ≠ Explain
+ ?
“we tried to benefit from an extensive
set of attributes describing each of the
movies in the dataset. Those attributes
certainly carry a significant signal and
can explain some of the user behavior.
However… they could not help at all
for improving the [predictive]
accuracy.”
Bell et al., 2008](https://image.slidesharecdn.com/predictiveanalyticsinistswim2015-150713105306-lva1-app6891/85/Predictive-analytics-in-Information-Systems-Research-TSWIM-2015-keynote-26-320.jpg)
![Current State in Social Sciences
(and MIS)
“While the value of scientific prediction… is beyond
question… the inexact sciences [do not] have…the
use of predictive expertise well in hand.”
Helmer & Rescher, 1959
Distinction blurred
Unfamiliarity with predictive
modeling/assessment
Prediction underappreciated](https://image.slidesharecdn.com/predictiveanalyticsinistswim2015-150713105306-lva1-app6891/85/Predictive-analytics-in-Information-Systems-Research-TSWIM-2015-keynote-42-320.jpg)
Predictive analytics in Information Systems Research (TSWIM 2015 keynote)
- 1. To Explain or To Predict? Predictive Analytics in IS Research 3rd Taiwan Summer Workshop on Information Management July 2015 Galit Shmuéli
- 2. Galit Shmueli (徐茉莉) www.galitshmueli.com ❷ 2000-2002 Carnegie Mellon Univ. Visiting Assistant Prof. Dept. of Statistics ❸ 2002-2012 Univ. of Maryland College Park Assistant then Associate Prof. of Statistics & Management Science R H Smith School of Business 2008-2014 Rigsum Institute (Bhutan) Co-Director, Rigsum Research Lab ❹ 2011-2014 Indian School of Business SRITNE Chaired Prof. of Data Analytics, Associate Prof. of Statistics & Info Systems ❶ 1994-2000 Israel Institute of Technology MSc + PhD, Statistics 2014-… NTHU Institute of Service Science Director, Center for Service Innovation & Analytics
- 3. Research in Data Analytics www.galitshmueli.com • Statistical strategy • ‘Entrepreneurial’ statistical & data mining modeling (new conditions & environments) • Business analytics In progress…
- 5. Road Map Definitions Explanatory-dominated MIS Explanatory modeling ≠ predictive modeling Why? Different modeling paths Explanatory power vs. predictive power How do I use this?
- 6. Definitions Explanatory modeling: Theory-based, statistical testing of causal hypotheses Explanatory power: Strength of relationship in statistical model
- 7. Definitions Predictive modeling: Empirical method for predicting new observations Predictive power: Ability to accurately predict new observations
- 8. Explain PredictDescribe Matching Game Social Sciences (MIS included) Machine learning Statistics
- 9. Statistical modeling in MIS research Purpose: test causal theory (“explain”) Association-based statistical models Prediction nearly absent
- 10. Start with a causal theory Generate causal hypotheses on constructs Operationalize constructs → Measurable variables Fit statistical model Statistical inference → Causal conclusions Explanatory modeling à-la MIS
- 11. In MIS, data analysis is mainly used for testing causal theory. “If it explains, it predicts”
- 12. “Empirical prediction alone is un-scientific” Some statisticians share this view: The two goals in analyzing data... I prefer to describe as “management” and “science”. Management seeks profit... Science seeks truth. - Parzen, Statistical Science 2001
- 13. Prediction in top research journals in Information Systems Predictive goal? Predictive modeling? Predictive assessment? 1990-2006
- 14. 52 “predictive” articles among 1,072 in Information Systems top journals
- 15. generate new theory develop measures compare theories improve theory assess relevance evaluate predictability Why Predict? for Scientific Research Shmueli & Koppius, “Predictive Analytics in IS Research” MIS Quarterly, 2011
- 16. “A good explanatory model will also predict well” “You must understand the underlying causes in order to predict”
- 17. Philosophy of Science “Explanation and prediction have the same logical structure” Hempel & Oppenheim, 1948 “It becomes pertinent to investigate the possibilities of predictive procedures autonomous of those used for explanation” Helmer & Rescher, 1959 “Theories of social and human behavior address themselves to two distinct goals of science: (1) prediction and (2) understanding” Dubin, Theory Building, 1969
- 18. Why statistical explanatory modeling differs from predictive modeling
- 19. Explanatory Model: Test/quantify causal effect for “average” record in population Predictive Model: Predict new individual observations Different Scientific Goals Different generalization
- 20. Theory vs. its manifestation ?
- 21. Notation Theoretical constructs: X, Y Causal theoretical model: Y=F(X) Measurable variables: X, Y Statistical model: E(y)=f(X)
- 22. Four aspects 1. Theory – Data 2. Causation – Association 3. Retrospective – Prospective 4. Bias - Variance Y=F(X) E(Y)=f(X)
- 23. “The goal of finding models that are predictively accurate differs from the goal of finding models that are true.”
- 24. Best explanatory model Best predictive model ≠ Point #1
- 25. Four aspects 1. Theory - Data 2. Causation – Association 3. Retrospective – Prospective 4. Bias - Variance Y=F(X) Y=f(X)
- 26. Predict ≠ Explain + ? “we tried to benefit from an extensive set of attributes describing each of the movies in the dataset. Those attributes certainly carry a significant signal and can explain some of the user behavior. However… they could not help at all for improving the [predictive] accuracy.” Bell et al., 2008
- 28. Explain ≠ Predict The FDA considers two products bioequivalent if the 90% CI of the relative mean of the generic to brand formulation is within 80%-125% “We are planning to… develop predictive models for bioavailability and bioequivalence” Lester M. Crawford, 2005 Acting Commissioner of Food & Drugs
- 29. “For a long time, we thought that Tamoxifen was roughly 80% effective for breast cancer patients. But now we know much more: we know that it’s 100% effective in 70%-80% of the patients, and ineffective in the rest.”
- 30. Goal Definition Design & Collection Data Preparation EDA Variables? Methods? Evaluation, Validation & Model Selection Model Use & Reporting
- 31. Study design Hierarchical data Observational or experiment? Primary or secondary data? Instrument (reliability+validity vs. measur accuracy) How much data? How to sample? & data collection
- 33. PCA SVD Interactive visualization Data exploration & reduction
- 34. Which Variables? Multicollinearity? causation associations endogeneity ex-post availability A, B, A*B?
- 35. ensembles Shrinkage models variance bias Methods / Models Blackbox / interpretable Mapping to theory
- 36. Evaluation, Validation & Model Selection Training dataEmpirical model Holdout data Predictive power Over-fitting analysis Theoretical model Empirical model Data Validation Model fit ≠ Explanatory power
- 37. Inference Model Use test causal theory generate new theory develop measures compare theories improve theory assess relevance Evaluate predictability Predictive performance Over-fitting analysis Null hypothesis Naïve/baseline
- 38. Point #2 Explanatory Power Predictive Power≠ Cannot infer one from the other
- 39. out-of-sample Performance Metrics type I,II errors goodness-of-fit p-values over-fitting costs prediction accuracy interpretation Training vs. holdout R2
- 41. The predictive power of an explanatory model has important scientific value Relevance, reality check, predictability
- 42. Current State in Social Sciences (and MIS) “While the value of scientific prediction… is beyond question… the inexact sciences [do not] have…the use of predictive expertise well in hand.” Helmer & Rescher, 1959 Distinction blurred Unfamiliarity with predictive modeling/assessment Prediction underappreciated
- 43. How does this impact Scientific Research?
- 44. State-of-the-art in Industry Distinction blurred Prediction over-appreciated “Big Data” synonymous with prediction
- 45. How does this impact an organization’s actions? …and our lives?
- 46. What can be done? Acknowledge difference Learn/teach prediction Leverage prediction in research BUT focus on its scientific uses:
- 47. generate new theory develop measures compare theories improve theory assess relevance evaluate predictability Why Predict? for Scientific Research Shmueli & Koppius, “Predictive Analytics in IS Research” MIS Quarterly, 2011
- 48. Shmueli (2010) “To Explain or To Predict?”, Statistical Science Shmueli & Koppius (2011) “Predictive Analytics in IS Research”, MISQ