Using theories of change to evaluate information literacy initiatives
0 likes358 views
The document discusses the application of the Theory of Change (ToC) methodology in evaluating information literacy initiatives at the University of Sheffield. It outlines the steps involved in constructing a ToC, the role of reflection in the evaluation process, and presents examples from various programs that have successfully utilized this approach. The document emphasizes the participatory nature of the ToC, involving multiple stakeholders to link activities with desired outcomes and impacts.
Using theories of change to evaluate information literacy initiatives
- 1. Using Theories of Change to evaluate Information Literacy initiatives Information School, University of Sheffield, UK ECIL 2021 Dr Pam McKinney Sheila Webber Photo: Sheila Webber
- 2. Presentation outline 1. Introductions and our experiences of using Theory of Change 2. Overview of Theory of Change 3. How a Theory of Change is constructed 4. The role of reflection in Theory of Change 5. Developing an evaluation plan 6. Examples of using Theory of Change
- 3. Introducing ourselves - Sheila ● Senior Lecturer in the Information School, University of Sheffield ● Teach on the Information School's research methods modules ● Was leader for one of the projects in Centre for Inquiry-based Learning in the Arts and Social Sciences (CILASS) ● Am currently using Theory of Change working with educators in virtual worlds Pam McKinney/Sheila Webber 2021
- 4. Introducing ourselves - Pam ● Background as an educational developer, working at CILASS: Centre for Inquiry-based Learning in the Arts and Social Sciences ● Theory of Change methodology applied at project and programme level to evaluate the impact of the Centre ● I led Theory of Change creation process with stakeholders, and supported the development of evaluation plans ● Supported reflective evaluation process, and jointly published with project leaders using evaluation data ● Now: lecturer at the University of Sheffield Information School Logo: Sabine Little Pam McKinney/Sheila Webber 2021
- 5. What is Theory of Change? ● A theory-based, participatory approach to project evaluation developed by the Aspen Institute (Connell & Kubisch 1998) ● Designed to create a link between funding and outcomes for stakeholders ● Can be used as an organisational level tool across multiple projects to standardise evaluation practice and identify and share good practice. ● Theory-based evaluation explains how and why results are achieved through understanding the links stakeholders construct between activities and outcomes ● Participatory evaluation avoids feelings that evaluation is “done to” projects, instead it is a flexible tool applied throughout the project that involves multiple stakeholders. ● Longitudinal evaluation focuses on intermediate and longer term impacts, and can be revisited long after a project has ceased. Hart, Diercks-O’Brian & Powell, 2009 Pam McKinney/Sheila Webber 2021
- 6. The role of the facilitator: creating a Theory of Change with stakeholders ● Literature is positive about the role of an external facilitator to collaboratively design an educational intervention, define a Theory of Change and evaluation plan, and carry out evaluation activities ● In the CILASS project an educational developer led the ToC creation process with project stakeholders (e.g. academic staff, students, professional services staff and librarians) ● An audio-recorded planning session, featuring interviews with key stakeholders, with a draft ToC created ● The educational developer then led the revision and finalisation of the ToC, and the resulting evaluation plan. Hart, Diercks-O’Brian & Powell, 2009 Pam McKinney/Sheila Webber 2021
- 7. Constructing a Theory of Change Drivers Resources/ enabling factors Activities Outcomes Anticipated impact What is the current situation that has led to the project? What support is needed to do the project activities? What activities need to take place to achieve the project outcomes? What are the desirable and feasible outcomes for the project? What is the longer term impact of the project? There should be a consistent and credible narrative developed across the 5 columns, where an outcome has specific activities and enabling factors associated with it. Pam McKinney/Sheila Webber 2021
- 8. Example Theory of Change from the CILASS programme Pam McKinney/Sheila Webber 2021
- 9. Role of reflection in Theory of Change ● Reflection is a key aspect of the Theory of Change design ● Based on “reflective practitioner” model (Kolb, 1984; Lewin, 1946; Schön, 1983). ● Recognition that change is a complex, socially-constructed process, and reflection can help articulate this. ● Critical reflection from project stakeholders helps identify relationships between actions and effects ● Reflection is a desirable aspect of the evaluation plan Hart, Diercks-O’Brian & Powell 2009 Photo Nigel Ford: Pam McKinney/Sheila Webber 2021
- 10. Next step - developing an evaluation plan ● Once a Theory of Change has been agreed by project stakeholders, an evaluation plan is created. ● Each Enabler, Process and Outcome indicator should have an evaluation process associated with it ● Multiple data collection methods can be used, qualitative and quantitative ● A significant role for reflection in evaluation, often taking the form of reflective interviews with project stakeholders ● An evaluation matrix is created which articulates the evaluation method to be used for each indicator Pam McKinney/Sheila Webber 2021
- 11. Evaluation matrix ToC indicator Student focus group Librarian questionnaire Project leader reflection Enabler: Effective support structures are developed for librarians to help them design effective online IL tutorials X Process: A suite of online IL tutorials is created based on ACRL IL framework X Outcome: students have confidence in referencing practices and can accurately cite sources in their work. X X Pam McKinney/Sheila Webber 2021
- 12. Examples of Theory of Change in practice USA's Public Library Association (2018) - developed a ToC for Professional Development ● Identified goals, outcomes, strategies & competencies required to achieve outcomes ● Identified 6 key roles for LIS professionals to achieve outcomes The Youth Cafe - a "pan-African youth empowerment organisation" - uses ToC and stresses participative element "We have incorporated the views of young people, youth-led and youth-serving organizations, and experts so that our map reflects young people’s view of how change occurs" (The Youth Cafe, 2021) Pam McKinney/Sheila Webber 2021
- 15. How could you use Theories of Change to evaluate Information Literacy initiatives? Photo: Sheila Webber
- 16. Sheila Webber Information School University of Sheffield s.webber@shef.ac.uk Twitter: @sheilayoshikawa http://information-literacy.blogspot.com/ http://www.slideshare.net/sheilawebber/ Dr Pamela McKinney Information School University of Sheffield p.mckinney@shef.ac.uk Twitter: @ischoolpam https://www.slideshare.net/PamelaMcKinney/