Evaluating Revalidation: Lessons from the Evaluation of Complex Interventions
Download as PPTX, PDF0 likes435 views
This document discusses key considerations for evaluating complex interventions like revalidation of physicians. It makes three main points: 1. Revalidation involves complex interactions that are nonlinear and context-dependent, so evaluations must account for these complexities. Randomized trials may not be possible. 2. Evaluations should have both formative and summative components to support learning and improvement, as well as assess merit and worth. Longitudinal, multilevel designs can help link processes to outcomes over time. 3. Evaluations must develop a clear theory of change and understand how different elements of revalidation interact and impact outcomes like quality of care. Stakeholder engagement and mixed methods can provide a more comprehensive understanding.
Evaluating Revalidation: Lessons from the Evaluation of Complex Interventions
- 1. Evaluating Revalidation: Lessons from the Evaluation of Complex Interventions IAMRA Revalidation Symposium Sanjeev Sridharan The Evaluation Centre for Complex Health Interventions University of Toronto & St. Michael’s Hospital 1
- 2. Standpoint Evaluation and complexity Evaluation Framework Application to GMC Domains Ten Questions
- 3. What is evaluation? A useful but perhaps incomplete definition • Evaluation is defined both as a means of assessing performance and to identify alternative ways to deliver • “evaluation is the systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance, performance and alternative ways to deliver them or to achieve the same results.” 3
- 4. Purpose of evaluation • Assessing merit and worth o Causal questions, RCT, observational studies • Programme and organizational improvement o Formative evaluation • Oversight and compliance • Knowledge development o Neglected purpose of many evaluations o Processes and active ingredients o Testing assumptions 4
- 5. An Example: Primary Prevention Have a Heart Paisley 5
- 6. Features of complex interventions The intervention is a theory or theories The intervention involves the actions of people. The intervention consists of a chain of steps These chains of steps or processes are often not linear, and involve negotiation and feedback at each stage. Interventions are embedded in social systems and how they work is shaped by this context. Interventions are prone to modification as they are implemented. Interventions are open systems and change through learning as stakeholders come to understand them. 6
- 7. System Dynamic Approaches (Sterman, 2006) • Constantly changing; • Governed by feedback; • Non-linear, History-dependent; • Adaptive and evolving; • Characterized by trade-offs; • Policy resistance: “The result is policy resistance, the tendency for interventions to be defeated by the system’s response to the intervention itself.” 7
- 8. “Solutions” Can Also Create New Problems Meadows DH, Richardson J, Bruckmann G. Groping in the dark: the first decade of global modelling. New York, NY: Wiley, 1982. Merton RK. The unanticipated consequences of purposive social action. American Sociological Review 1936;1936:894-904. Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68. Policy resistance is the tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention itself. -- Meadows, Richardson, Bruckman 8
- 9. System-as-Cause Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68. Meadows DH. Leverage points: places to intervene in a system. Sustainability Institute, 1999. Available at <http://www.sustainabilityinstitute.org/pubs/Leverage_Points.pdf>. Richardson GP. Feedback thought in social science and systems theory. Philadelphia, PA: University of Pennsylvania Press, 1991. Sterman JD. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000. 9
- 10. 10
- 11. Intervention System Components of the Intervention Stability of the components Impacts on evaluation design 1. Elements of the intervention: What are the elements of a revalidation system? 11
- 12. revalidation can enhance quality of care? What are the connections between the various domains (trust, knowledge, communication, quality)? • Context and Support Conditions • Mechanisms • Heterogeneity • Leverage 12
- 13. 3. Evidence Synthesis: What can we learn from the literature on revalidation? 13
- 14. 4. Anticipated Timeline of Impact: What is your timeline of impact? Would revalidation impact quality of care within the time frame of the evaluation? Outcome 1 Outcome 2 Outcome 3 Program Years 1 2 3 4 5 60 14
- 15. B. Structure of Evaluation Influence 5. Framework of Learning 6. Pathways of evaluation influence 15
- 16. 5. What do you wish to learn from the evaluation of revalidation? Improvement vs. Merit and Worth 16
- 17. Evaluation/Per formance Measurement Pathways of Evaluation Influence: Collective, Individual and Interpersonal 6. How will the evaluation of Revalidation impact practice? Impacts on Policy, Practice and the Political Agenda 17
- 18. Design, Methods and Data • 7. Design Issues • 8. Learning about Program Theory 18
- 19. Some ideas for design • Is a comparison group possible? • Longitudinal • Ecology of Evidence • Involving the stakeholders • Balance an improvement vs. Merit and worth lens • Multilevel • Attribution vs. Contribution 19
- 20. Program Evaluation Design/Metho d Program Impacts Initial Program Theory Initial Impacts Areas of Uncertainty Learning from Innovative Methods Emergent Program Theory (8) Learning from Methods: How do you learn along the way? 20
- 22. Program Larger System Spread Innovations Components of System Innovative Practice Contextual Learning (9) Framework of Spread 22
- 23. Framework of Learning Decision to Sustain Match between Anticipated Findings of Impact/ Performance Trajectory and Actual Performance Evaluation Results Other Factors (10) A Framework for Sustainability 23
- 24. How will this approach help with learning about key domains discussed by GMC? • 1. Knowledge, skills, and performance • 2. Safety and quality • 3. Communication, partnership, and teamwork • 4. Maintaining Trust. 24
- 25. Clarifying revalidation o Theory of Change; Realist Review • Connect the different domains • Connections between different mechanisms • Support conditions o Timeline of Impact o What are you uncertain of? What evidence do you need? Whom are you trying to persuade? • Ecology of Evidence o Processes Tough tests of processes Mechanisms and differential mechanisms Formative vs. Summative o Knowledge Translation o Learning, accountability and accountability towards learning 25
- 26. Linking processes to outcomes Variations in ‘revalidation’ implementation Longitudinal Multilevel Connect changes in practice to patient outcomes Patient involvement—qualitative longitudinal research 26