Evaluating without numbers: exploration of a story based evaluation method
- 1. Evaluating without numbers: Exploration of a story based evaluation method Ann Wilson, Learning and Teaching at Navitas
- 2. Welcome • Introductions – who is in the room?
- 3. Evaluating without numbers Evaluation is an important part of the improvement cycle and answers the question – how did we do, and how might we make it better? Quantitative evaluation methods are relatively easy to do, and meet the need for numbers to tell the story. In this presentation I will talk about my experience of a narrative or story based evaluation method, which has recently been published in the journal Studies in Educational Evaluation.
- 4. The background • 2007 Promoting Excellence Initiative funded by ALTC (Australian Learning and Teaching Council) and the networks began Networks were state based, representing Universities within each state • 2009 now OLT (Office of Learning and Teaching) funded the networks
- 5. OLT requirements The OLT was a government agency that “promotes and supports change in higher education institutions for the enhancement of learning and teaching” In 2011 the government determined to cease funding the agency, so in 2011 at a final conference the OLT offered to distribute $50 000 to each network. To do this the OLT required: • A budget • A plan of activities • Evaluation plan
- 6. Our proposal Budget – lots of things to spend money on airfares, catering, meetings, showcases Plan of activities – meetings, showcases, forums Evaluation –
- 7. Evaluation • The challenge was to demonstrate network value in a volatile funding environment • Would the numbers do it? We could count how many meetings, how many came, how many attended the showcases and forums. Does that show impact and value?
- 8. Evaluation of networks • Usual practice is an end of project approach focussing on quantitative methods we could still do that • While the literature suggests that learning and teaching networks are valuable – members struggle to express what the benefits are to them • Research led us to narrative based approaches to evaluation 8
- 9. What do you think of when you hear the phrase narrative based method?
- 10. This is what we did… Developed three evaluation questions to address the aims and purpose of the network – • What were the experiences of participants in NSW/ACT PEN (the network)? • How effective were network activities in relation to the achievement of the network aim? • What factors will help/hinder future network sustainability?
- 11. Process of story building – a three phase approach 1. Individual reflective stories 2. A collective story 3. A value creation story
- 12. Phase 1: Individual reflective stories Three times during the course of the project each network member was invite to respond to the following trigger questions: 1. What did the previous network mean to my institution? What could an on- going network mean to me as a network member 2. What does the network mean to my institution and to me? 3. What did the network mean to my institution and to me?
- 13. Analysis of the stories 1. Read all the stories 2. Extract significant statements, words of phrases 3. Group like themes together 4. Formulate meaning under each theme by returning to the stories 5. Integrate the themes into a description of the members experiences.
- 14. Phase 2: Constructing a collective story The purpose of this phase is to identify the meta-themes that emerge in the middle of the story. Two steps were undertaken to do this: 1. All themes were analysed from phase 1 to identify meta themes 2. Individual stories were included under the meta themes
- 15. Four meta themes emerged 1. Consolidation of existing connections and helping to foster new collegial and cross-institutional links 2. Travelling on a common journey creates a sense of community 3. Collaborative ways of working deepen learning and teaching conversations to facilitate personal and professional capacity building 4. Advancement of institutional practices and processes results when institutional and personal professional outcomes overlap
- 16. Phase 3: composing a value creation story This addresses the evaluator questions such as the network’s purpose, level of engagement. The value categories were: • Immediate value • Potential value • Applied value • Realised value • Reframed value
- 17. How did story building contribute to authentic evaluation? • Increasing visibility of the networks impacts by collecting evidence from multiple sources using a variety of methods • Exposing affective and relational dimensions of network life • Collating this evidence in a form that could easily be shared with others
- 18. Some conclusions • The data gathering and analysis was time consuming • The process captured the qualitative elements of the experience • The story can demonstrate current and future impacts of networks
- 19. Questions