Developmental evaluation learning as you go
Download as PPT, PDF1 like1,479 views
Developmental evaluation (DE) is an approach that is useful when the connection between solutions and problems is not clear or when situations are complex and continuously evolving. In DE, the evaluator acts as a co-creator with the project partners to support learning and adaptation. The evaluator helps partners assess what factors they should consider and structures feedback to facilitate collective learning and effective collaboration. Typical results of DE include identifying principles and relationships to focus on, discussing how they could apply in different contexts, and establishing mechanisms for real-time feedback to promote social learning and transformation.
Developmental evaluation learning as you go
- 1. Developmental Evaluation Fostering collective learning & effective collaboration through developmental evaluation 07/13/12 Christopher Wilson & Associates 1
- 2. Traditional Evaluation • Traditional evaluation is typically designed to assess whether an intervention has reached its intended target either through a formative or summative evaluation. • Formative Evaluation - improves – provides interim feedback to various initiative stakeholders, such as funders, staff, or community residents during the implementation period to ensure that the implementation aligns with the initial plan. • Summative Evaluation – validates – provides evidence of progress towards program & policy goals over a period of time by looking at outcomes and impacts over the longer term. Obviously, the outcomes and impacts must be achievable within the time frame being evaluated. 07/13/12 Christopher Wilson & Associates 2
- 3. Traditional Assumptions • The connection between ends and means is well established • The situation being evaluated remains stable • Key variables are known and controllable • Outcomes are predictable and measurable 07/13/12 Christopher Wilson & Associates 3
- 4. Traditional Orientation • Solutions are centrally defined, then evaluated in field tests for their validity and then ‘best practices’ are implemented broadly • Evaluation is conducted in the middle (formative) or at the end (summative) to test programs and policies • Works well in linear, simple and complicated social situations where conditions are stable and standardized in many different locales • The evaluator is an independent, objective assessor 07/13/12 Christopher Wilson & Associates 4
- 5. Two Other Evaluation Needs • One focuses on supporting the group’s social learning in the face of complexity and uncertainty – This evaluation assesses many things: whether the initial plan continues to fit with the current assessment of needs and circumstances; whether there are unforeseen factors involved; how an innovation might be traditionally evaluated; to acquire more information from experimentation – This is particularly prevalent in areas of ‘wicked’ problems where there is a need to learn both ‘ends’ and ‘means’ at the same time. Feedback is obtained to ascertain whether the right thing is indeed being done. Partners can then collectively reflect on this feedback to determine the ongoing fitness between problem understanding and response. • The other need of evaluation promotes the ongoing contingent cooperation of partners – This assessment is conducted to validate the commitments, trust, and participation of an initiative’s partners to each other, especially when the evaluation tools are used as a means of supporting and empowering the ownership of stakeholders to be part of defining and delivering a solution. It helps satisfy the caveat “I will if you will.” • These will not be met by traditional evaluation approaches 07/13/12 Christopher Wilson & Associates 5
- 6. Developmental Evaluation - when to use it • Adaptation of an existing programs and ongoing development to an evolving context • Application of principles and best practice transferred from one context to a new one • Exploring impacts of small innovations and policies in preparation for larger interventions using formative or summative evaluation • Providing feedback on major systems change • Rapid response to crisis or sudden change Source: Michael Quinn-Patton, Developmental Evaluation, Guilford Press, New York, NY, 2011 07/13/12 Christopher Wilson & Associates 6
- 7. DE Assumptions • The connection between ends and means is not well understood. There is no known solution. – This is being learned via the intervention • The situation being evaluated will continue to change & evolve – The situation continues to respond to a number of factors not all of which are under the control of the evaluators • Key variables may not be known nor is their potential influence on outcomes – Intervention is a means of learning what key variables might be • Outcomes are not directly measurable, predictable – Interveners are likely to be surprised by their experience • The system under consideration may respond in unpredictable ways to small local changes • Evaluation is not a tool for validation but a tool of ‘learning while doing’ through experimentation, prototyping and serious play 07/13/12 Christopher Wilson & Associates 7
- 8. DE Approach • Map system dynamics, partner assumptions, and interdependencies • Develop mechanisms to provide quick feedback on group decisions and assumptions- in-use • DE must be flexible to changes in collective understanding and to changes in the nature of the intervention and be able to adjust the evaluation accordingly • Emphasis on evaluation to support the judgement of participants • Attention to unanticipated results 07/13/12 Christopher Wilson & Associates 8
- 9. DE Evaluator Role • Evaluator seen as co-creator with participants, providing them with the information they require to make sense of what’s going on • The evaluator is part of a team whose members collaborate to conceptualize, design and test new approaches in a long-term, on-going process of continuous improvement, adaptation & intentional change. • The developmental evaluator helps partners to ‘pay attention to the ‘right’ things’ • The evaluator's primary function is to foster various group conversations contributing evaluative questions, data and logic, and thereby facilitate social learning, contingent cooperation and data-based decision-making. • Evaluation is a stewardship function: encouraging reality testing; cross learning; joint ownership; and shared commitment 07/13/12 Christopher Wilson & Associates 9
- 10. Typical Results from DE • What are the principles, factors, relationships that people need to pay attention to? • How could these principles, factors, relationships be applied in a local context? • How can real time feedback be structured? • How does that feedback foster collective learning? • How does feedback contribute to effective, ongoing collaboration and teamwork? 07/13/12 Christopher Wilson & Associates 10
- 11. “Theory of Change” Orientation • Instead of linear “logic models”, DE makes use of “theory of change” models” – The presumed building blocks and pathways of change that are required to bring about a given long-term goal. • Evaluation conducted to support meaning making, reframing, belonging, trust, commitment and the conceptualization of a “theory of change” • Evaluation conducted to promote conversations that challenge the mental models that contribute to the status quo and to possible transformation • Theories of change are reality tested and evaluation is conducted to support shared learning • Evaluation conducted to support the process of collaboration 07/13/12 Christopher Wilson & Associates 11
- 12. For further information please contact: Chris Wilson Christopher Wilson & Associates Tel: (613) 355-6505 chris@christopherwilson.ca www.christopherwilson.ca 07/13/12 Christopher Wilson & Associates 12