IDS Impact, Innovation and Learning Workshop March 2013: Day 2, Paper session 4 Barbara Befani
1 like1,040 views
This document discusses using set theory and Bayesian approaches to better understand impact evidence in complex contexts. Set theory can help define impact and success more flexibly as being "on the right track" rather than a specific goal. It also helps analyze causal contribution and deal with uncertainty. Bayesian approaches allow quantifying uncertainty with probabilities and measuring the strength of qualitative evidence based on sensitivity, specificity, and predictive value. When theories of change explaining impact are supported by evidence confirming multiple components, it represents strong evidence as the probability of observing all components is low but high if the theory of change holds true.
IDS Impact, Innovation and Learning Workshop March 2013: Day 2, Paper session 4 Barbara Befani
- 1. 27:03:2013 Set-theoretic, diagnostic and Bayesian approaches to impact evidence Barbara Befani Impact, Innovation and Learning: Towards a Research and Practice Agenda for the Future Brighton, 26-27 March 2013
- 2. Outline Set-theoretic Methods (e.g. QCA) and the new challenges – Uncertainty (equifinality) – Causal contribution (multiple-conjunctural causality) – Causal asymmetry (necessity and sufficiency) Diagnostic and Bayesian approaches – Uncertainty (can be quantified with probabilities) – The strength of qualitative evidence can be measured
- 3. Defining & explaining events with Set Theory In uncertain and emergent contexts, we cannot define “impact” (or success) precisely An Impact “space” of possible events, all desirable – All compatible with given values and goals Success is likely to look like ANY of a number of events = a LOGICAL UNION Success looks more like “being on the right track” than achieving a specific goal Being “on the right track” means avoiding a number of pitfalls / dead ends Sets can be defined as NEGATION of other sets The three main operators in set theory are – Negation, union, intersection
- 6. Causal Asymmetry and Contribution Analysis What is a sufficient causal package (a branch of blue nodes) Principal Contributory Cause = INUS, a necessary part of the (sufficient) combination (each blue node of a given branch) In Set Theory terminology, a causal package is an INTERSECTION of contributory causes A combination of necessary causes (necessary within that causal package) Set Theory provides the mathematical basis for 1. analyzing causal contribution 2. dealing with uncertainty (particularly Fuzzy Sets)
- 7. UNION of A and B A U B A OR B A + B INTERSECTION of A and B A ∩ B A AND B A * B A A B B NEGATION of A = ~A, NOT A
- 8. Conf ID Cold Chain Integrity (necessary) Vaccine (V) (intervention) Health System Quality (HSQ) Success (S) (reduction in specific morbidity) No. Cases A 1 1 1 1 3 B 1 1 0 1 1 C 1 0 1 1 2 D 1 0 0 0 2 E 0 1 1 1/0 0 F 0 1 0 1/0 0 G 0 0 1 1/0 0 H 0 0 0 1/0 0 Why are Causal Combinations important? Impact is contingent on the context Finding ONE counterfactual is not enough QCA helps finding many counterfactuals through systematic cross- case comparison
- 9. Probability and Diagnosis: what the evidence says Realm of “unknown knowns” General problem of the strength / quality of evidence: how to assess it? In clinical practice, physicians use tests – Specificity • Probability that absence of the disease will return negative evidence on that test – Sensitivity • Probability that presence of the disease will return positive evidence on that test – (Positive) Predictive Power • Probability that positive evidence signals presence of the disease
- 10. When is evidence strong? When it is sensitive and specific – Sensitive: P ( Evidence | Impact ) high – Specific: <= P ( Evidence ) low • false positives are low – Predictive: P ( Impact | Evidence ) high – The latter can be calculated with the Bayes formula P ( I | E ) = P ( I ) * P ( E | I ) / P ( E ) Two important principles of high-quality evidence – of all kinds, quali, quanti, etc. Evidence is strong when: – The prior probability of observing positive evidence P ( E ) is LOW (~specificity) – The probability of observing positive evidence IF the intervention was successful / had an impact P ( E | I ) is HIGH (sensitivity)
- 11. Impact ‘I’ Predictive Evidence Weak Evidence Strong Evidence Sensitive Evidence Specific Evidence Evidence ‘E’ (of ‘I’)
- 12. REALITY about success of intervention Intervention Successful (I) Intervention Unsuccessful (~I) EVIDENCE about success of intervention Evidence Positive (E) True Positive False Positive Evidence Negative (~E) False Negative True Negative Seeking evidence of impact: diagnostic tests
- 13. REALITY about success of intervention Intervention Successful (I) Intervention UNsuccessful (~I) EVIDENCE about success of intervention Evidence Positive (E) True Positive False Positive Positive predictive value = Σ True Positive Σ Evidence Posi tive Evidence Negative (~E) False Negative True Negative Negative predictive value = Σ True Negative Σ Evidence Neg ative Sensitivity = Σ True Positive Σ Intervention Successful Specificity = Σ True Negative Σ Intervention UNsuccessful
- 14. When articulated ToCs explaining impact are supported by evidence IT IS STRONG EVIDENCE OF IMPACT The prior probability of observing a sophisticated ToC with several components is LOW, because The probability of a combination is the product of the probability of components (a very SMALL #) P (a, b, c, ... N ) = P (a) * P (b) * P (c) * ... * P (n) When ToCs with many components are confirmed, it is strong evidence of impact, because: – the chances of all components being observed simultaneously were LOW • P ( E ) low, specificity high – If the ToC explaining impact holds true, the probability of observing evidence of all components is HIGH • P ( E | I ) = sensitivity high
- 15. Conclusion: let’s use other branches of mathematics In the form of SET THEORY or PROBABILITY THEORY (used differently than in frequentist statistics) They provide new ways of dealing with uncertainty SET THEORY helps with: – Defining success in a more flexible, open and inclusive way (being “on the right track”) – Explaining success by defining and identifying contributory causes rigorously through data analysis (eg. with QCA) PROBABILITY THEORY helps with: – Assessing the strength of evidence in terms of sensitivity, specificity and predictive value – Qualitative evidence CAN be strong if a number of conditions are met – Carefully weigh each piece of evidence as in a court of law, using conditional and subjective probabilities
- 16. References Befani, B. (2013) “Between Complexity and Rigour: addressing evaluation challenges with QCA” in Evaluation (forthcoming) Befani, B. (2013) “What were the chances? Diagnostic Tests and Bayesian Tools to Assess the Strength of Evidence in Impact Evaluation”, CDI Practice Paper (forthcoming) John Mayne (2013) “Making Causal Claims” (presentation to this event) Bruno Marchal (2013) “Conceptual distinctions: Complexity and Systems – Making sense of evaluation of complex programmes” (presentation to this event)