Humanitarian Indicator Tool (Dr Vivien Walden, Oxfam)
Download as PPT, PDF2 likes1,884 views
The Humanitarian Indicator Tool is a scoring system developed by Oxfam to improve the quality of humanitarian programmes. It assesses programmes against benchmarks in areas such as timeliness, coverage, technical standards, monitoring and evaluation, feedback mechanisms, partnerships, safety, gender equity, preparedness, and advocacy. An external consultant uses documentation and interviews to score each benchmark and provide an overall assessment of how well a programme meets recognized humanitarian standards. The results are reviewed by the country office and published online to increase accountability and drive quality improvements across Oxfam's humanitarian work.
Humanitarian Indicator Tool (Dr Vivien Walden, Oxfam)
- 1. Improving quality of humanitarian programmes through the use of a scoring system: the Humanitarian Indicator Tool Dr Vivien Margaret Walden and Nigel Timmins
- 2. Acknowledgements • We wish to acknowledge the consultants, Andy Featherstone, Peta Sandison, Marlise Turnbull and Sarah House who helped refine the tools and the 10 countries offices who have been through the process and have hopefully come out of it unscathed. • Photographs used in this presentation are by Caroline Gluck, Oxfam Page 2
- 3. Why a new tool? • In 1995, the ODI evaluation of the response to the Rwanda crisis raised concerns around the quality of service delivery • As a result of findings, the Sphere project was started • In 1997 the People in Aid Code of Best Practice was published • In 2000 the Humanitarian Ombudsman became HAP • In 2001 Griekspoor and Sondorp wrote a paper highlighting the various quality assurance measures and posed the question “Do all of the developments in the field of accountability and performance actually improve overall performance?” Page 3
- 4. Why a new tool? • Oxfam’s hypothesis – by improving the quality of a humanitarian programme, the likelihood of positive impact is increased • The tool does not try to prove the link • Oxfam has introduced the Global Performance Framework as part of reporting to DFID • The framework has global indictors of which the Humanitarian Indicator is one • Whereas the other indicators measure impact, the humanitarian indictor looks at quality of the response Page 4
- 5. Oxfam’s Global Performance Framework Global Output Indicators Global Outcome Indicators Humanitarian Assistance: Degree to which humanitarian responses meet Total number of people provided with appropriate recognised quality standards for humanitarian humanitarian assistance, disaggregated by sex programming (e.g. Sphere guidelines) Adaptation and Risk Reduction: % of supported households demonstrating greater # of people supported to understand current and likely ability to minimise risk from shocks and adapt to future hazards, reduce risk, and/or adapt to climatic changes and uncertainty, disaggregated by sex emerging trends & uncertainty Livelihood Enhancement Support: % of supported households demonstrating greater Improved quality # of women and men directly supported to increase income, as measured by daily consumption and of life for poor income via enhancing production and/or market access expenditure per capita women and men Women’s Empowerment: % of supported women demonstrating greater # of people reached to enable women to gain increased involvement in household decision-making and control over factors affecting their own priorities and interests influencing affairs at the community level Citizen Mobilisation: Degree to which selected interventions have # of a) citizens, CBO members and CSO staff supported contributed to affecting outcome change, as to engage with state institutions/other relevant actors; generated from findings of rigorous qualitative and b) duty bearers benefiting from capacity support evaluations Campaigning and Advocacy: Degree to which selected interventions have #of campaign actions directly undertaken or supported, contributed to affecting outcome change, as e.g. contacts made with policy targets, online and generated from findings of rigorous qualitative offline actions taken, media coverage, publications, etc. evaluations Degree to which selected interventions meet recognised standards for accountable programming Extent to which selected project delivery good value for money Page 5
- 6. Page 6
- 7. The indicator • Output indicator – the total number of people provided with appropriate humanitarian assistance, disaggregated by sex • Data collected annually through online system • Outcome indicator – the degree to which humanitarian responses meet recognised quality standards for humanitarian programming • Data collected through HIT Page 7
- 8. The tool – for rapid onset emergencies Global Humanitarian Indicator: Degree to which humanitarian responses meet recognised quality standards for humanitarian programming RAPID ONSET EMERGNECY – EARTHQUAKE, SUDDEN FLOODS, TSUNAMI, CYCLONES, TYPHOONS, HURRICANES, SUDDEN CONFLICT WITH DISPLACEMENT, AWD OUTBREAKS Number Quality standard Met Almost Partially Not met (score6) met (4) met (score (score 2) 0) 1 Timeliness - rapid appraisal/assessment enough to make decisions within 24 hours and initial implementation within three days 2 Coverage uses 25% of affected population as an planned figure (response should reflect the scale of the disaster) with clear justification for final count 3 Technical aspects of programme measured against Sphere standards Number Quality standard Met Almost Partially Not met (score3) met (score met (score 2) (score 1) 0) 4 MEAL strategy and plan in place and being implemented using appropriate indicators 5 Feedback/complaints system for affected population in place and functioning and documented evidence of information sharing, consultation and participation leading to a programme relevant to context and needs 6 Partner relationships defined, capacity assessed and partners fully engaged in all stages of programme cycle 7 Programme is considered a safe programme: action taken to avoid harm and programme considered conflict sensitive 8 Programme (including advocacy) addresses gender equity and specific concerns Page 8
- 9. The benchmarks • Timeliness - rapid appraisal/assessment enough to make decisions within 24 hours and initial implementation within three days • Coverage uses 25% of affected population as an planned figure (response should reflect the scale of the disaster) with clear justification for final count • Technical aspects of programme measured against Sphere standards • MEAL strategy and plan in place and being implemented using appropriate indicators • Feedback/complaints system for affected population in place and functioning and documented evidence of information sharing, consultation and participation leading to a programme relevant to context and needs Page 9
- 10. The benchmarks • Partner relationships defined, capacity assessed and partners fully engaged in all stages of programme cycle • Programme is considered a safe programme: action taken to avoid harm and programme considered conflict sensitive • Programme (including advocacy) addresses gender equity and specific concerns and needs of women, girls, men and boys and vulnerable groups • Evidence that preparedness measures were in place and effectively actioned • Programme has an advocacy/campaigns strategy and has incorporated advocacy into programme plans based on evidence from the field Page 10
- 11. The benchmarks • Country programme has an integrated approach including reducing and managing risk though existing longer-term development programmes and building resilience for the future • Evidence of appropriate staff capacity to ensure quality programming Page 11
- 12. The methodology • Done by an external consultant – although preferably one who has a knowledge of Oxfam • Done as a desk study using documentation and some telephone/Skype interviews • Follows a pre-determined scoring system and list of documents • Has to be commented on and accepted by the country • The country writes a management response • All reports and a summary for each are published on the Oxfam website – www. Oxfam.org Page 12
- 13. The scoring Quality standard Evidence needed Met (score 6) Almost met (4) Partially met (score 2) Not met (score 0) 3 Proposals Sphere standards NA Sphere standards Standards only MEAL strategy and proposed and put in proposed and adjusted mentioned in proposals plans place with adjusted to context but not replicated in PH and EFSL strategies standards for context Standards mentioned in plans Technical adviser visits Training in standards proposals and Or Training agendas and carried out for staff and LogFrames but not No mention of Sphere presentations partners monitored against in any document LogFrames and Indicators use Some evidence of monitoring frameworks standards and training but not donor reports monitoring against widespread (staff but RTE and other standards takes place not partners or only in evaluation reports regularly one area) learning event or Standards evaluated review reports Page 13
- 14. Instructions for use Benchmark Evidence Quality check Benchmark 3 Technical aspects of Proposals Check proposals and strategies to see if programme measured MEAL strategy and plans standards are mentioned not just as a possibility against Sphere standards PH and EFSL strategies but that they are considered in the context of Technical adviser visits the response – this might mean that Sphere has Training agendas and been adapted to suit the context presentations The indicators on the LogFrame for technical LogFrames and monitoring areas should reflect Sphere standards frameworks The MEAL strategy should have Sphere as donor reports indicators and for data collection methods RTE and other evaluation Check adviser reports for mention of standards reports and how these were implemented learning event or review Check the RTE report for mention of Sphere reports standards Check WASH and EFSL strategies and adviser reports to see if any training was carried out for staff and partners Check review and evaluation reports for mention of standards Page 14
- 15. Final score • First year total score could be 30 • Adjusted for non-applicable benchmarks • First year scoring • Somalia – 17/28 • Kenya – 24/30 • Ethiopia – 9/28 • Pakistan – 19/30 • Colombia – the test case for the HIT – 18/26 • Second year – only South Sudan is complete – 21/30 Page 15
- 16. The findings – Somalia • Benchmark 5 on accountability • Score Met – 2/2 • “Oxfam’s partners appeared to differ in the level of beneficiary participation in design and delivery. Some documented highly participatory process, with qualitative and quantitative data. As well as gathering information, rapid assessments were done to establish VRCs to improve participation (criteria, entitlements, payment points, registration, complaints and feedback). Mobile phone hotlines were set up where possible, with feedback protocol to guide staff on how to register and follow-up complaints” Page 16
- 17. Pakistan • Benchmark 7 – protection and gender • Partially met – ½ • Some protection concerns were identified relating to security for staff and women and girls using WASH facilities. Some actions were taken responding to dignity and protection including involving women in different activities. Post- distribution monitoring investigated some security concerns related to cashing cheques and distribution points, including analysis of responses from women. No protection problems observed were communicated to agencies or authorities responsible for, or specialising in, protection Page 17
- 18. Ethiopia • Benchmark 11 – advocacy • Not met – 0/2 • Advocacy activities were clearly part of the intended response and there is a regional advocacy action plan with Ethiopia objectives and a media, advocacy and campaign strategy which includes a number of plans for Ethiopia. However, no Ethiopia country advocacy strategy was provided for the evaluation. The sitreps do mention Oxfam’s participation in influential meetings, but are not tied to an explicit strategy. There is no record of the impact of Oxfam’s advocacy activities Page 18
- 19. Lessons learnt • There are limitations to doing a desk study – it relies heavily on documentation and scores do not reflect absence of documentation or actual absence of good programming • Several country teams objected to being judged solely on “little pieces of paper” • There is no opportunity to get the views of the affected population unless this is already documented • Telephone/Skype interviews can be biased – it is sometimes difficult to triangulate • The process needs the goodwill and buy-in from the country team Page 19
- 20. Advantages • The process is fairly inexpensive – under £6000 • The country does not have to host a consultant • The methodology can be used to track progress in subsequent responses in one country • The scores are comparable across programmes (although context should be considered) Page 20
- 21. She deserves the best we can give – thank you! Page 21
Editor's Notes
- #4: They then went on to discuss the fact that there were more agencies, more funding and more media attention. They even went as far as to say that 100,000 deaths could be attrivbuted to poor perfomraance by relief agencies
- #11: Elderly, disabled, HIV positive, single women, female-headed households are examples