Evaluating community projects
- 1. Evaluating community projects A practical guide Marilyn Taylor, Derrick Purdue, Mandy Wilson and Pete Wilde These guidelines were initially developed as part of the JRF Neighbourhood Programme. This programme is made up of 20 community or voluntary organisations all wanting to exercise a more strategic influence in their neighbourhood. The guidelines were originally written to help these organisations evaluate their work. They provide step-by-step advice on how to evaluate a community project which will be of interest to a wider audience. Contents All about evaluation 2 What is evaluation? 2 Why evaluate? 2 Key principles of evaluation How to evaluate: a step-by-step approach 3 Step 1: Review the situation 3 Step 2: Gather evidence for the evaluation 4 Step 3: Analyse the evidence 6 Step 4: Make use of what you have found out 6 Step 5: Share your findings with others 7 Annex A: Reviewing the situation (example) 8 Annex B: Reviewing the situation (record sheet) 10 Annex C: Evaluation planning table 11
- 2. 2 Evaluating community projects All about evaluation What is evaluation? Put simply, evaluation by members of a project or organisation will help people to learn from their day-to-day work. It can be used by a group of people, or by individuals working alone. It assesses the effectiveness of a piece of work, a project or a programme. It can also highlight whether your project is moving steadily and successfully towards achieving what it set out to do, or whether it is moving in a different direction. You can then celebrate and build on successes as well as learn from what has not worked so well. Why evaluate? Although evaluation may seem like an unnecessary additional task if you are already short of time and resources, it can save you both time and resources by keeping participants focused on, and working towards, the ultimate goal of the project. If necessary, it can refocus activity away from unproductive or unnecessary work. Evaluation can help you to: • learn from your experience; • record what you have learnt, and share it with other stakeholders; • check your progress; • check whether what you are doing is still what local people want or need; • identify strengths and weaknesses in your project; • create a basis for future planning; • demonstrate whether you have used your resources – time and money – effectively; • explain to funders, and others involved in your work, what you have achieved and how successful it is. Key principles of evaluation Evaluation is most effective when: • it is a continuous (not just one-off) process informing planning and delivery as the project develops; • it involves all those with an interest in the project in defining the questions they want answered; • it uses imaginative and creative approaches, which engage those involved; • it helps projects to be more accountable to the wider community; • it is used to challenge discriminatory and oppressive policies and practice, and to overcome inequality and disadvantage; • it highlights and celebrates successes and achievements; • it encourages an honest appraisal of progress, so that you can learn from what hasn’t worked as well as what has. 2
- 3. How to evaluate: a step- by-step approach Step 1: Review the situation Evaluating a project is like taking a journey. First you need to plan your journey – you need to decide where you want to get to; then work out how you are going to get there, and what you need to equip yourself with for the journey; you will also need to look out for the signposts and milestones that will tell you whether you are on the right road and whether you are likely to get there on time. When starting to evaluate a community project, it is helpful to begin by writing down how you are going to tackle each stage of this journey. To complete Annex B you will need to ask yourself the following questions: What problems are you trying to solve? Before you can evaluate your progress, you need to be clear about what you are trying to achieve. Community projects usually want to solve one or more local problem. What are these problems? What would make change happen? What needs to change if the problem is to be solved, and why? Think about what might have caused the problem, and what needs to happen to make a difference. How do you plan to make change happen? What can you do to solve the problem? Look at your answer to the last question and think about what your project can do. Try to be as precise as possible. Break your broad or long-term goals down into short- and medium-term goals. What actions are you going to take to achieve these goals? What resources will you need? What results do you want to see? How will you know when you have achieved your short-, medium- and long-term goals? What will be different about the original problem at the end of the project, if you are successful? Try to be as precise as possible about what you want to achieve. The achievements should all be related to the original problem you wanted to work on. How can you measure progress? In order to know whether you are on the right track to achieve your goals and how far you have got, you will need to decide on a few key questions, and collect evidence to answer them. What will tell you whether you are on the right track? What signs, milestones or ‘indicators’ should you be looking for? Annex A (pages 8-9) is a table reflecting the range of community empowerment issues being faced by projects in the JRF Neighbourhood Programme. It was produced to help projects think about what they were trying to achieve, Annex B (page 10) is a blank, expanded version of the same table. It can be used by any project, tackling any issue. It provides the framework for your own evaluation process. The best way of using it is to bring together a small group – say project staff, some board or committee members and any other key players – to discuss it and fill it in. It is a way of gathering the views of other people involved in the project, to see if their expectations are similar or different. 3Evaluating community projects
- 4. 4 Evaluating community projects Step 2: Gather evidence for the evaluation This is a key part of the evaluation process. You need to find evidence to show whether or not you have reached the goals you set yourself in Annex B (be they short-, medium- or long-term). Different types of evidence should be collected: • numbers (for example, the number of people you have reached, the number of people getting work, the number of good stories in the press, changes in crime levels); • people’s opinions, views and experiences (for example, people’s stories about their experience on the programme, photos of the area ‘before and after’, people’s views on whether they think they have more power); • who has benefited and who has not. Establish a baseline Be as clear as you can about where you are starting from. If you know where you are starting from, it will be easier to assess the distance you have travelled during the project. What information do you already have? Make sure you have collected together in a central file material that will be useful to you later. Relevant paperwork for creating a baseline might include: the programme proposal, past research reports and any statistical data you have on the area, press reports and, if relevant, minutes of key preliminary meetings. Set up systems to gather data on a regular basis Clarify what information you will require to help you to answer or explore the questions set out in Step 1. Think about the records you will be collecting anyway as part of your project management that will provide some of the information you need. Relevant paperwork that should be gathered on a regular basis might include: action plans and work programmes, minutes of meetings, reviews and reports, and records of attendance at groups and events. Decide what additional information you will gather specifically for the evaluation, and how you will gather it The following menu lists ways of gathering information. You should choose two or three methods in order to get a rounded picture. Also, different ways of gathering information will suit different projects. Which you choose to use, and how much time you spend gathering data, should by tailored to the capacity of your organisation. Annex C (page 11) will help you to plan what information you will need to gather in order to judge the project’s progress, and so support decisions about what needs to change. For each measure of progress (indicator), it will be important to find out what information you have already, what additional information you will need to collect, what methods you will use to gather it, who will take responsibility for collecting the information, and the timescale.
- 5. 5Evaluating community projects Ways of gathering additional information: a menu • A questionnaire survey A questionnaire survey can be used to find out more about the views and experiences of users, the wider community, agencies, etc. Use tick- boxes or questions that can be answered with a yes or no if you want to survey a lot of people, or ask a lot of questions. Questions that allow people to say more than just yes or no will give you more detailed information, but they take longer to fill in, a lot more time to analyse, and fewer people will fill them in. Responses to questionnaires are often low so think about offering a prize. • In-depth interviews It is usually best to limit the number of in-depth interviews to those people whose involvement with the project gives them particular insights or valuable experience – but try to talk to a range of people who are likely to have different perspectives and views on your project. • Feedback forms You can find out whether people have found your training and other events useful by asking them to fill in a short form. Ask them, for example, what they found most and least useful; what they might do differently as a result; what could be improved. • Focus groups and round tables A ‘focus group’ gathers together about half a dozen people who are broadly similar (for example, they are all single parents with young children) to discuss themes or questions you want to address in the evaluation. A ‘round table’ discussion is a similar idea, which brings together people with different perspectives (for example, teenage parents, teachers, health visitors). • Diaries Ask key people to keep diaries of their involvement with the project. • Press reports Gather and review press reports on the area (for example, you could see whether positive reports about the area are increasing). • Observation Take photographs of your area over time, to see if you can observe any changes. Observe who contributes to meetings or comes to your centre, and see whether this changes over time. This will give you an idea of which types of people you are reaching (men, women, younger, older) and which of these types of people are playing a more confident role in the project. • Case studies In order to make the evaluation manageable, you might want to pick a few pieces of work (case studies) to explore in detail, rather than trying to explore everything. Pick pieces of work that illustrate your main objectives. • Evaluation workshops and review meetings Hold special workshops/review meetings of people who are involved in your project and use pictures, photographs or models, as well as the spoken word, to get feedback from participants.
- 6. 6 Evaluating community projects Step 3: Analyse the evidence You will now have gathered together quite a lot of information. Working out what it is telling you is the next stage. Does the information you have gathered show that you have reached your goals? Does it highlight any achievements? Does it show up any problem issues that need to be tackled? Be alert to unexpected outcomes, both desirable and undesirable. For example, evidence of good community cohesion might be that events have been organised which draw together people from across the community or groups who were previously in conflict. Other evidence of community cohesion might be recording stories of how conflicts have been resolved, or the views of formerly distant groups now working as partners. The views of different people involved in, or affected by, the project (including those taking part and service users) can often be as important as numbers in providing evidence of change. Don’t collect too much information or you will be overwhelmed when you come to pull it together and write up your notes. Allow plenty of time to pull the information together. Even if only one or two people initially do the work, it is worth feeding the initial findings back to a wider group of stakeholders to add their insights. Step 4: Make use of what you have found out If there are goals or objectives you have not met, or if you haven’t got as far as you had hoped, you need to think about why, and what you can learn from that. There could be a variety of explanations. Here are a few ideas to consider: Problems with external circumstances • The environment in which you are working has changed or worked against you (for example, local or national policies have changed, or a major employer has closed down). Problems with carrying out your plans • You have departed from your original aims. • You didn’t allow enough time or resources. • Changes in your organisation have meant that you have not been able to do what you said you would do (for example, a key worker left). • The quality of performance of your organisation has been lower than expected (for example, workers or members have not completed tasks they said they would do). Problems with the ideas behind what you want to achieve • The plans you came up with initially to make things happen have not been successful. • Your original aims were inappropriate (perhaps they turned out to be not what people wanted). • Different people involved in the project were working against each other and towards different goals.
- 7. 7Evaluating community projects Step 5: Share your findings with others Sharing your findings with others is important because it can help other people in the project, or associated with it, to recognise any problems or issues that are preventing the project from making progress. It can help everyone to learn from any mistakes that have been made, or pick up on any successful ideas that have been put into action. Remember that sharing findings can provide an opportunity to celebrate success as much as to learn from difficulties. • Think about who you are sharing your findings with, and how best to communicate with them. • Is there going to be a written report and/or other ways of reporting the findings of the evaluation? Try to ensure that any report uses clear, plain language, and follows a logical order. • Will you need to report the findings to different audiences using different formats? It may be necessary to produce both a comprehensive report of the evaluation exercise and a much briefer report or summary of key findings. • How will you ensure that the findings inform practical changes in your project’s work plan? For example, you could hold a special meeting or workshop for those involved in the project at which you both feed back key findings and also collectively consider future action plans in the light of these findings. JRF Neighbourhood Programme The Joseph Rowntree Foundation Neighbourhood Programme offers a range of ‘light touch’ support to 20 groups and projects in England, Scotland and Wales. These resources include: access to facilitation support from a team of regionally based facilitators; access to small amounts of credit; national and regional networking opportunities; a website; brokerage; the opportunity to share effective practice and learning more widely. The programme includes cross-cutting studies on funding, community involvement, diversity and community cohesion, and changing relationships with power holders. Evaluation of the programme was built in from the start and has been carried out by Marilyn Taylor and Derrick Purdue (UWE) and Mandy Wilson and Pete Wilde (COGS).This document, and a summary based on the Interim Evaluation of the Neighbourhood Programme, 'Lending a hand', are both available from the Joseph Rowntree Foundation website (www.jrf.org.uk). About the authors The authors form the evaluation team for the JRF Neighbourhoods Programme. Marilyn Taylor and Derrick Purdue work in the Cities Research Centre, University of the West of England, Bristol. Marilyn Taylor, who leads the team, is Professor of Urban Governance and Regeneration; Derrick Purdue is a Research Fellow. Mandy Wilson and Pete Wilde are Directors of COGS Ltd (Communities and Organisations: Growth and Support), Sheffield.
- 8. 8 Evaluating community projects AnnexAReviewingthesituation(step1)–example Analysis NoonehasanalysedlocalproblemsandassetsConsultationisneededwithlocalcommunitiesProfilecommunityneedsandassetsAplanofactionthatlocal toidentifytheirneedsandplanactioncommunitiesfeelapartof Drawupanactionplanwithshort-and Lackofdirection–activitytendstobereactivemedium-termobjectivestoguidework ratherthanproactive Engagement Peoplearenotengaged,littleactivityisgoingCommunitydevelopmentisneededtoSetupcommunitydevelopmentandoutreachLocalcommunitiesareengagedinavarietyof onlocallytotackleproblemsencouragepeopletoengagewiththeprojectprojectsactivitiesandtacklinglocalproblems Developastrategytocommunicatewith peopleandtellpeoplewhattheprojectis allabout Capacity Lackofleadership;lackoforganisationalSustainableleadershipandorganisationalProvideopportunitiestodevelop:Effectiveandsustainableorganisationsand ability;lowlevelofskills;lowlevelofresourcesabilityareneededtoplanandco-ordinate•leadersandorganisationsaccountableleadership actionandraisefunds•managementskills •teambuildingandaccountabilitySustainablefundingandassets •fundraisingskills Cohesion ThecommunityisdividedandfragmentedCommonground,mutualrespectandDevelopnegotiating,mediationandconflictLocalcommunitiesactingeffectivelytogether understandingbetweencommunitiesandresolutionskills individuals Holdeventsandmeetingstobringdifferent groupstogether Whatproblemsareyoutryingtosolve?Whatwouldmakechangehappen?Howdoyouplantomakechangehappen?Whatresultsdoyouwanttosee? (Rationale)(AimsandObjectives)(Outcomes)
- 10. 10 Evaluating community projects AnnexBReviewingthesituation(Step1)–arecordsheetforprojects WhatproblemsareyoutryingWhatwouldmakechangeHowdoyouplantomakeWhatresultsdoyouwanttoHowcanyoumeasureprogress? tosolve?happen?(Rationale)changehappen?see?(Outcomes)(Indicators) (AimsandObjectives)
- 12. Ref: 0225 Published 2005 by the Joseph Rowntree Foundation, The Homestead, 40 Water End, York YO30 6WP. This project is part of the JRF’s research and development programme. The views expressed, however, are those of the authors and not necessarily those of the Foundation. Other formats available. Tel: 01904 615905, Email: info@jrf.org.uk