Program theory evaluation
0 likes1,311 views
The document discusses challenges in evaluating the mechanisms and theories behind how interventions and programs work. It notes that while an intervention may predict an outcome like increased physical activity, understanding how and why it worked requires testing the underlying program theory against alternatives. Evaluating mechanisms of change is difficult due to sample size limitations and the complexity of interventions. Overfitting theories to limited data is also a problem. The document raises open questions about whether program theories can be treated like other scientific theories and subjected to replication, falsification, and cross-validation, and how best to study mechanisms of action in complex, real-world interventions.
Program theory evaluation
- 1. www.mattiheino.com How or why did my intervention work? Challenges in evaluating mechanisms and program theories @heinonmatti
- 2. www.mattiheino.com What do you mean “it worked”? • A predicted change in a pre-specified outcome: Physical activity
- 3. www.mattiheino.com What do you mean “this is how it worked”? (Note: simplified logic model)
- 4. www.mattiheino.com What do you mean “this is how it worked”? (Note: simplified logic model) Program theory draws from… • COM-B system • Control Theory • Theory of Planned Behaviour • Self-Determination Theory • (Goal Setting Theory • Self-Efficacy Theory • Self-Regulation Theory • Social Cognitive Theory…)
- 5. www.mattiheino.com Observations Theory testing in science(s) Statistical hypothesis Substantive theory and assumptions Substantive hypothesis
- 6. www.mattiheino.com Observations Theory testing in science(s) Statistical hypothesis Substantive theory and assumptions Substantive hypothesis The causal stream: each line means “From this, it follows that…” – deductive!
- 7. www.mattiheino.com Observations Theory testing in science(s) Statistical hypothesis Substantive theory and assumptions Substantive hypothesis The causal stream: each line means “From this, it follows that…” – deductive!
- 8. www.mattiheino.com Observations Theory testing in science(s) Statistical hypothesis Substantive theory and assumptions Substantive hypothesis The causal stream: each line means “From this, it follows that…” – deductive! The inferential stream: each line means “From this, what can we conclude about…?” – inductive!
- 9. www.mattiheino.com Theory testing in science(s) Theory Auxiliary theories All else equal (ceteris paribus) Assumptions of instruments Experimental conditions
- 10. www.mattiheino.com Theory Auxiliary theories All else equal (ceteris paribus) Assumptions of instruments Experimental conditions Program theory holds
- 11. www.mattiheino.com Theory Auxiliary theories All else equal (ceteris paribus) Assumptions of instruments Experimental conditions Program theory holds Other theories work in this context
- 12. www.mattiheino.com Theory Auxiliary theories All else equal (ceteris paribus) Assumptions of instruments Experimental conditions Program theory holds Other theories work in this context No systematic, unmeasured confounders
- 13. www.mattiheino.com Theory Auxiliary theories All else equal (ceteris paribus) Assumptions of instruments Experimental conditions Program theory holds Other theories work in this context No systematic, unmeasured confounders Accelerometers, bioimpedance devices etc. measure what they’re supposed to
- 14. www.mattiheino.com Theory Auxiliary theories All else equal (ceteris paribus) Assumptions of instruments Experimental conditions Program theory holds Other theories work in this context No systematic, unmeasured confounders Accelerometers, bioimpedance devices etc. measure what they’re supposed to Intervention delivered as planned
- 15. www.mattiheino.com Intervention PA link tells us something’s going on Theory Auxiliary theories All else equal (ceteris paribus) Assumptions of instruments Experimental conditions Intervention Physical activity
- 16. www.mattiheino.com Intervention PA link tells us something’s going on Theory Auxiliary theories All else equal (ceteris paribus) Assumptions of instruments Experimental conditions Intervention Physical activity … But only to the extent, that increased PA (in comparison to controls) was a “damn strange coincidence” absent intervention. (see Meehl 1990)
- 17. www.mattiheino.com Intervention PA link tells us something’s going on Theory Auxiliary theories All else equal (ceteris paribus) Assumptions of instruments Experimental conditions Intervention Physical activity “the only admissible positive evidence for a theory are the corpses of its rivals.” ‒ Lakatos (1978, on Popper’s solution)
- 18. www.mattiheino.com Absence of link only tells us something was wrong! Theory Auxiliary theories All else equal (ceteris paribus) Assumptions of instruments Experimental conditions Intervention Physical activity
- 19. www.mattiheino.com Theory testing in science(s) 1. Replicate experiment to see that finding holds • “we may say that a phenomenon is experimentally demonstrable when we know how to conduct an experiment which will rarely fail to give us a statistically significant result.” ‒ Fisher (1947); see Mayo 2016 • Also, see Heino, Fried & LeBel (2017)
- 20. www.mattiheino.com Theory testing in science(s) 1. Replicate experiment to see that finding holds 2. Investigate boundary conditions and mechanisms of action
- 22. www.mattiheino.com About those mechanisms… Intervention Physical activity Autonomous motivation (See here for an additional sample size problem)
- 23. www.mattiheino.com About those mechanisms… Intervention Physical activity Autonomous motivation - Spousal support? - Media exposure? - Habit? - Past experiences? - …
- 26. www.mattiheino.com Overfitting (2016) Link Seemingly supported theories fail to generalise to slightly different situations, or other samples from the same population!
- 27. www.mattiheino.com Link“Attempts to corroborate statistically significant subgroup differences are rare; when done, the initially observed subgroup differences are not reproduced.”
- 28. www.mattiheino.com Thanks! Summary and open questions: • Scientific knowledge grows by risky tests of theories; replication and falsification. Currently, we do neither. – Are program theories inherently different to other scientific theories?
- 29. www.mattiheino.com Thanks! Summary and open questions: • Scientific knowledge grows by risky tests of theories; replication and falsification. Currently, we do neither. – Are program theories inherently different to other scientific theories? • Mechanisms are extremely difficult to study. Currently, we claim mediation but remain ignorant about causation. – (How) should we study mechanisms of action in complex interventions for complex systems?
- 30. www.mattiheino.com Thanks! Summary and open questions: • Scientific knowledge grows by risky tests of theories; replication and falsification. Currently, we do neither. – Are program theories inherently different to other scientific theories? • Mechanisms are extremely difficult to study. Currently, we claim mediation but remain ignorant about causation. – (How) should we study mechanisms of action in complex interventions for complex systems? • Overfitting is a problem, though cross-validation can help. Currently, sample sizes remain inadequate. – How to make models simple enough, but not simpler?
- 31. www.mattiheino.com Bonus slide I’d be happy to have a conversation on how all this applies to real life decision making!
- 32. www.mattiheino.com Bonus slide 2 Link Link How do we figure out, which pieces of information truly count, so that we can ditch the misleading ones?