![The elephant in the room of:
1) metascience
●
e.g. reproducibility
2) statistics
●
e.g. model “complexity”
●
[see SW seminar 2021, Fanelli 2019, 2022]](https://image.slidesharecdn.com/fanelliswtalkdraft2red-221212221058-7b824147/85/The-neglected-importance-of-complexity-in-statistics-and-Metascience-3-320.jpg)
![(Bonifay and Cai 2017)
Bifactor
confirmatory
(theoretical)
20 parameters
EIFA:
exploratory
(a-theoretical)
20 parameters
K(Bifactor) ≥ K(EIFA)
“The [bifactor-encoded] theory
is specifically supported
by the data”
(Fanelli & Bonifay, in progress)
P<0.05
when is a theory actually supported?
example of application](https://image.slidesharecdn.com/fanelliswtalkdraft2red-221212221058-7b824147/85/The-neglected-importance-of-complexity-in-statistics-and-Metascience-44-320.jpg)
The neglected importance of complexity in statistics and Metascience
- 1. The neglected importance of complexity in statistics and Metascience Daniele Fanelli
- 2. In this talk: 1) what’s missing in the current paradigm 2) what a new paradigm might look like 3) evidence in support of this proposal
- 3. The elephant in the room of: 1) metascience ● e.g. reproducibility 2) statistics ● e.g. model “complexity” ● [see SW seminar 2021, Fanelli 2019, 2022]
- 4. What is “complex”? Level of complexity Many, diverse, interacting parts. Long to describe, difficult to predict.
- 5. example 1) complexity deflates the “reproducibility crisis”. year project discipline N result 2014 Many labs 1 psychology, misc. 13, 36 labs 77% 2016 COS social+cognitive psychology 100 36-68% 2016 Camerer et al. experimental economics 18 61-78% 2018 Many labs 2 social+cognitive psychology 28, 62 samples, 36 countries 54% 2018 Camerer et al. social studies in Nature, Science 21 57-67% 2021 RPCB cancer biology 188, 50 exper., 23 papers 3-82% Lower reproducib: 1) complex phenomena 2) complex methods ● not just random noise ● structured, systematic diff.
- 6. example 2) complexity confuses statistical results AIC:−2log( L)+2k
- 7. models vary by “fitting propensity” (“complexity” beyond n. of parameters) (Bonifay and Cai 2017) universe of possible data When is a theory actually supported?
- 8. How might the elephant appear? 1) integrate complexity of phenomena & methods in measuring, forecasting, correcting reproducibility 2) penalize statistical models for complexity beyond number of parameters integrating “complexity” ~ severity of testing
- 9. https://www.wallpaperflare.com/elephant-during-daytime-grayscale-photo-of-elephant-portrait-wallpaper-zhdat Part 2: A candidate alternative
- 10. K theory in a nutshell (Fanelli 2019, Fanelli 2022) K = H(Y) - H(Y|X, τ)) H(Y) H(X) D(τ)) + nX nY +
- 11. K = consilience K = H(Y) - H(Y|X, τ)) H(Y) H(X) D(τ)) + nX nY explain/predict/control more/diverse phenomena with fewer/simpler theories/methods +
- 12. what the variables represent = H(Y) - H(Y|X, τ)) H(Y) H(X) D(τ)) + nX nY K explain/predict/control more/diverse phenomena with fewer/simpler theories/methods +
- 13. more information Y makes K GROW (Fanelli 2019, Fanelli 2022) K= H(Y) - H(Y|X, τ)) H(Y) H(X) D(τ)) + nX nY +
- 14. (Fanelli 2019, Fanelli 2022) = - H(Y|X, τ)) H(X) D(τ)) nX H(Y1 ) + H(Y2 ) + H(Y3 ) nY H(Y1 ) + H(Y2 ) + H(Y3 ) + K more information Y makes K GROW +
- 15. K theory in a nutshell (Fanelli 2019, Fanelli 2022) = H(Y) - H(Y|X, τ)) H(Y) H(X) D(τ)) + nX nY K +
- 16. more info. Y|X, X, τ makes K small (Fanelli 2019, Fanelli 2022) K = H(Y) - H(Y|X, τ)) H(Y) H(X) D(τ)) + nX nY +
- 17. (Fanelli 2019, Fanelli 2022) K = H(Y) - H(Y) H(X) + nX nY H(Y1 |X, τ))+H(Y2 |X, τ))+H(Y3 |X, τ)) D(τ)1 )+D(τ)2 )+D(τ)3 ) + more info. Y|X, X, τ makes K small
- 18. K theory in a nutshell (Fanelli 2019, Fanelli 2022) = H(Y) - H(Y|X, τ)) H(Y) H(X) D(τ)) + nX nY K + = ≈1, full consilience =0, no knowledge <0, wrong
- 19. K theory in a nutshell (Fanelli 2019, Fanelli 2022) = H(Y) - H(Y|X, τ)) H(Y) H(X) D(τ)) + nX nY K +
- 20. K of a regression model (Fanelli 2019, Fanelli 2022) Y = α + β X + error = H(Y) - H(Y|X, τ)) H(Y) H(X) D(τ)) + nX nY K +
- 21. (Fanelli 2019, Fanelli 2022) Y = α + β X + error this has been said before = H(Y) - H(Y|X, τ)) H(Y) H(X) D(τ)) + nX nY K +
- 22. (Fanelli 2019, Fanelli 2022) key theoretical innovations Y = α + β X + error = H(Y) - H(Y|X, τ)) H(Y) H(X) D(τ)) + nX nY K +
- 23. (Fanelli 2019, Fanelli 2022) key methodological innovations Y = α + β X + error = H(Y) - H(Y|X, τ)) H(Y) H(X) D(τ)) + nX nY K +
- 24. (Fanelli 2019, Fanelli 2022) graphs are everywhere in science methodologies theories (Fanelli 2019, Fanelli 2022) www.protocols.io/view/an-optimized-protocol- for-in-vivo-analysis-of-tumo-3byl471m2lo5/v1 = H(Y) - H(Y|X, τ)) H(Y) H(X) D(τ)) + nX nY K + (Mueller 2015, ICSS)
- 26. 1) K predicts perceived and actual reproducibility
- 27. “tau” of biological experiments (Fanelli, Tan, Amaral & Neves, 2022, MetaArxiv)
- 28. K vs. perceived reproducibility (Fanelli, Tan, Amaral & Neves, 2022, MetaArxiv)
- 29. K vs. actual reproducibility Kr=K o 2−λ⋅d kr hr=ko ho 2−λ⋅d log kr ko =log ho hr −λ⋅d R≡log H (Y )−H (Y∣X , τr) H (Y )−H (Y∣X , τo) =α+βlog 1 D (τr)/ N
- 30. (Fanelli, Tan, Amaral & Neves, in prep) 1) K predicts actual reproducibility (part of collaboration with Brazilian Reproducibility Initiative)
- 31. (Fanelli, Tan, Amaral & Neves, in prep) independent new predictor multiple regression, Y=reproducibility
- 32. (Fanelli, Tan, Amaral & Neves, in prep) multiple regression, Y=reproducibility (here D(τ) based on sentences in replication protocol!) very easy to measure, automatize
- 33. better than just P-values and N (Fanelli, Tan, Amaral & Neves, in prep) multiple regression, Y=reproducibility R
- 34. leads to progress in metascience (Fanelli, Tan, Amaral & Neves, in prep) multiple regression, Y=reproducibility
- 35. 2) D(τ) might predict fitting propensity
- 36. preregistered test (Fanelli & Bonifay, in prep)
- 37. 1) generated N=20 models, all with 36 parameters 2) derived D(τ)), predicted their fitting propensity 3) tested them on 20,000 random covariance matrices (Fanelli & Bonifay, in prep) preregistered test
- 38. D(τ) uniquely reflects model complexity (pre-registered test) (Fanelli & Bonifay, in progress)
- 39. (Fanelli & Bonifay, in progress) NO alternative theory explains this! (pre-registered test) multiple regression: Y=fitting propensity
- 41. Spearman’s ρ = 0.64, P<0.002 Spearman’s ρ = 0.73, P<0.001
- 42. (Bonifay and Cai 2017) Bifactor confirmatory (theoretical) 20 parameters EIFA: exploratory (a-theoretical) 20 parameters (Fanelli & Bonifay, in progress) how K improves theory testing example of application
- 43. theories invoking general factor (e.g. IQ, stress, psychosis) Bifactor widely used to “test” theories
- 44. (Bonifay and Cai 2017) Bifactor confirmatory (theoretical) 20 parameters EIFA: exploratory (a-theoretical) 20 parameters K(Bifactor) ≥ K(EIFA) “The [bifactor-encoded] theory is specifically supported by the data” (Fanelli & Bonifay, in progress) P<0.05 when is a theory actually supported? example of application
- 45. Summary of this talk: 1) what’s missing in the current paradigm? ● we pretend complexity is irrelevant 2) what might a new paradigm look like? ● measuring D(τ), integrating/penalizing with K 3) evidence in support of this proposal? ● increasingly promising ● any alternative, better suggestions?