Vicky Pelka's Training Session On Impact Evaluation
Download as PPTX, PDF3 likes1,033 views
The document outlines an impact evaluation session led by Vicky Pelka, focusing on defining impact evaluation, its importance, and methods of data collection. Participants will learn about measuring project impacts on quality of life and standard of living through various research methods such as questionnaires, interviews, and focus groups. The session aims to empower teams to effectively collect, assess, and communicate the outcomes of their projects to improve social, economic, and environmental factors for those in need.
Vicky Pelka's Training Session On Impact Evaluation
- 1. Impact Evaluation with Vicky Pelka
- 2. What awaits you… • Enactus UK first ever Impact Evaluation session!!! • 1 hour… • Reflect • Ask Questions • electronic Hand-outs available on request!
- 3. Who am I? • Vicky Pelka • Ex- Enactus Sussex President • Research and Evaluation Coordinator at a London-based charity • I evaluate the impact of our projects
- 4. Learning Outcome Statements 1. I can define impact evaluation. 2. I understand why impact evaluation is important and useful. 3. I know at what points in a project cycle impact evaluation should take place. 4. I know what kind of things can be measured in order to do impact evaluation. 5. I can name some research methods used in impact evaluation. 6. I can name some ways to communicate the impact of a project.
- 5. Definitions • Need • Empowerment • Quality of Life (QOL) • Standard of Living (SOL) • Impact • Evaluation
- 6. (PEOPLE in )Need Lack of something requisite, desirable, or useful. Needs range from basic survival needs, to cultural, intellectual, and social needs (Maslow’s pyramid of Needs). Empowerment The process of increasing the capacity of individuals (or groups) to apply new skills and gain self-confidence, so they achieve a stage of independence.
- 7. Quality of Life (QOL) Intangible components of someone’s life e.g. satisfaction with life, confidence, self-esteem, social status, emotional / psychological well-being, etc. Standard of Living (SOL) Tangible components of someone's’ life e.g. employment, income, housing, access to clean drinking water or sanitation, healthcare, food provision, etc.
- 8. Impact The positive and negative, short-term and long-term changes produced by a project. Impact can be direct or indirect, intended or unintended. Evaluation The objective assessment of an on-going or completed project, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, effectiveness, impact and sustainability.
- 9. WHY is it important? It’s in the Enactus Criterion Considering the relevant social, economic and environmental factors, which Enactus team most effectively empowered people in need by applying business and economic concepts and an entrepreneurial approach, to improve their quality of life and standard of living.
- 10. WHY is it important? It’s in the in the Enactus Criterion Considering the relevant social, economic and environmental factors, which Enactus team most effectively empowered people in need by applying business and economic concepts and an entrepreneurial approach, to improve their quality of life and standard of living.
- 11. Why is it important? • To know for yourself • To tell others (university, students, committed and potential funders) • To improve • To let beneficiaries have a say!
- 12. What Impact Evaluation are YOU already doing in your team and when? (Take 2 minutes to tell your neighbour)
- 13. Project plans you have seen so far Need • Find people in need Solution • Apply business and economic concepts • Take an entrepreneurial approach • Empower Impact • Improve quality of life and standard of living
- 14. Bringing in Impact Evaluation Need • Find people in need Solution • Apply business and economic concepts • Take an entrepreneurial approach • Empower Impact • Improve quality of life and standard of living Conduct Needs Assessment Collect Baseline Data Record Outputs Monitor Progress Adjust/ Improve if necessary Measure Outcomes: Measure QOL and SOL changes Review effectiveness and sustainability
- 15. What Impact Data is worth collecting? • Outputs – what you have done e.g. delivered 5 business workshops, carried out 30 hours of teaching • Outcomes - what this means… e.g. successfully established one business, doubled beneficiaries’ income, enabled a family to send 5 children to high school
- 16. What Impact Data is worth collecting? • Quantitative – relates to number, statistics, quantities, specifics e.g. “We increased the number of orphans attending school from 10 to 20” • Qualitative – relates to quality, but also subjective opinion, judgements or perceptions e.g. “The beneficiaries reported to be more confident in their writing skills“
- 17. What Impact Data is worth collecting? • Get into groups of 3 • Brainstorm data that you could collect to prove: Group1: A Need, e.g. unemployment Group 2: changes in SOL, e.g. number of households with access to clean water Group 3: changes in QOL, e.g. self-confidence in public speaking Group 4: Empowerment, e.g. ability to produce quality furniture independently
- 18. What Impact Data is worth collecting? Need Average income, unemployment rate, current state of business, child mortality rate, local poverty line, confidence, independence, SOL Income, literacy rate, number of books in a library, attendance at events, access to medicine, electricity, school attendance rate , employment, transport, QOL Self-confidence in own skills, appearance, mental well being, social inclusion, self-fulfilment, independence, Empower ment Occupational success, ability to provide for family, independence,
- 19. Collecting Impact Data Where do I find this data? • Government statistics, Census Data • International organisations e.g. UN / WB / OECD / Transparency International • Other charity publication Can’t find it anywhere? Collect it yourself using Research Methods…
- 20. What are Methods of Impact Evaluation? 1. Questionnaires 2. Interviews 3. Focus Groups 4. Conversations and Observations • Find a research method that suits you! • Tailor methods to your project and evaluation • All methods have benefits and weaknesses
- 21. Method 1: Questionnaire Method for qualitative and quantitative data collection Useful for e.g. knowledge testing, gathering baseline data for needs assessment, collecting testimonials
- 22. Method 1: Questionnaire Things to consider: • Language • Length of the questionnaire • Using open or closed questions • Using textboxes or tick boxes • An absolute no-go: leading questions! • Scales of rating (e.g. 1-5; strongly agree to disagree) to give you QUANT data • Logic of the question order • Online (surveymonkey) or paper?
- 23. Method 1: Questionnaire Things to do: • Trial the questionnaire • Explain purpose of the questionnaire • Ensure confidentiality Weaknesses: • Misunderstanding your question • Possible misreporting
- 24. Method 2: Interview Method for qualitative and quantitative data collection Useful for e.g. individual needs assessments, Project progress review, in-depth analysis of impact on an individual
- 25. Method 2: Interview Things to consider: • Language and use of translator (issues of culture, gender, etc.) • Choice between structures and semi-structured • Length of the interview • Location
- 26. Method 2: Interview Things to consider: • Incentive for interviewee to come • Open or closed questions • An absolute no-go: leading questions! • Logic of the question order • Video recording?
- 27. Method 2: Interview Things to do: • Practice the interview • Explain purpose of the interview • Ensure confidentiality Weaknesses: • Possible misreporting • Bad chemistry with interviewee • Only get perspective of one individual
- 28. Method 3: Focus Group Method for qualitative data collection mainly “A focus group is a planned discussion among a small group (4-12 persons) of stakeholders facilitated by a skilled moderator. It is designed to obtain information about people’s preferences, values, beliefs, perceptions pertaining to a defined topic and why these are held by observing the structured discussion of an interactive group.” Participatory Method Toolkit, Slocum, 2003
- 29. Method 3: Focus Group • Useful for e.g. needs assessment, project design, market research, end of project impact evaluation Things to consider: • Group size (4 -12 is recommended) • Makeup of the group (gender, caste, social class, etc.) • Location (indoor, outdoor, private, public?) • Providing incentives for people to attend (food, money?)
- 30. Method 3: Focus Group Things to do: • Use a local facilitator • Think of ice breakers and warm ups • Plan a series of questions • Appoint someone in charge of taking notes and observing. Record discussion points, arguments, behaviour.
- 31. Method 3: Focus Group Weaknesses: • Only reflects opinions of the people present • Domination of discussion by powerful group members • Unpredictable group dynamics • Facilitator is key!
- 32. Method 4: Conversation and Observation Method for qualitative data collection Conversational reporting such as: “I send my daughter to school now.” “I really like coming here.” “Used in marketing and the social sciences, observational research is a social research technique that involves the direct observation of phenomena in their natural setting.” Wikipedia
- 33. Method 4: Conversation and Observation Things to consider: • In what context did someone say something? • What do you want to observe – a process or result? Interactions between people; individual behaviour; physical facts? Things to do: • Write down quotes ASAP • Make an observation checklist • Separate observation and thought/interpretation!
- 34. Method 4: Conversation and Observation Weaknesses: • Possible misreporting • Observers’ bias • Presence of an observer can influence behaviour
- 35. How do you communicate your impact? • Consider the different ways of publicising / communicating your findings • Consider your audience! • Consider mix of Quant and Qual, Outputs and Outcomes
- 36. Figures and Stats Example: • 100 young people took part in a business skills course in 2009-2010. • 60% of homeless individuals now live in safe housing conditions.
- 38. The written word…. Describing what you do • Do not use jargon • Take the reader on a journey (Need, Solution, Impact, Meaning) • Explain Evaluation Process, methodology
- 39. The written word…. Back it up with a Testimonial "I feel I’m a changed person. I had some problems at school and I was badly bullied and threatened. On the Outward Bound course I was challenged to face my fears and get over them but when I finished it I felt really proud of myself.“ Ryan, 13 years
- 40. Pictures – Choose them well! From the Oxfam and ActionAid website
- 41. Video e.g. Windows Movie Maker • Of the community in which you work • Of project activities • Testimonials from beneficiaries • Animation to explain a complex project concept
- 42. Go home and put all information into one awesome Action Plan: 1. Know what you want to evaluate (what data you want to collect) 2. Think when you want to record this data (before, after, mid-project) 3. Decide what method of data collection you want to use? 4. Who will carry out the evaluation? 5. What resources do you need for it? 6. Do it! 7. Communicate it!
- 43. Learning Outcome Statements 1. I can define impact evaluation. 2. I understand why impact evaluation is important and useful. 3. I know at what points in a project cycle impact evaluation should take place. 4. I know what kind of things can be measured in order to do impact evaluation. 5. I can name some research methods used in impact evaluation. 6. I can name some ways to communicate the impact of a project.
- 44. Get in touch! • Vicky Pelka • Skype: vickyp101 • Email: pelka.v@gmail.com • London-based meetings • Emergency calls: 07 580 430 866 • Sign up/ email me to receive the hand-out
Editor's Notes
- #3: This is a current topic, real buzz and investment in the development and charity community about it! Electronic handouts available if you email me
- #5: This is what you will get out of this session
- #6: Some words I will use throughout the presentation and that come up in Enactus jargon. Definitions are not set in stone! There are varieties
- #10: Question: Where do you think Impact Evaluation is hinted at in the Criterion?
- #11: Highlighted areas are the focus of Impact Evaluation
- #13: Activity
- #18: Activity, 4 minutes
- #19: Answers on the slide were collected during the brainstorm at Grantham 2013
- #21: Think charities, ministries – they need to show accountability, therefore they are facing the sae challenges in evaluating their impact as you are!
- #23: Open Question: What did you think of the training provided? Closed question: Did you enjoy the training: Yes, No, Not sure Leading question: How would you say did you benefit from the project?
- #36: Tailor your communication to your audience: funders are interested in how you spent your money, students are more interested in the project process, the general public in your project impact, etc.
- #40: This example is taken from my own work with the Outward Bound Trust. A participant made this comment after taking part in an outdoor adventure course.
- #41: Choose pictures you use carefully. What message do you want to convey? Should pictures show a process, an activity, an output, beneficiaries? What mood are you trying to create?