Longing to do an Ex-post Sustainability Evaluation? How to support this work…

by | May 18, 2015 | Dissemination, Feedback loops, innovation, local capacity building, Local Participants, post-project evaluation, Sustainability, Sustainable development

Longing to do an Ex-post Sustainability Evaluation? How to support this work…

Just back from Niger where Catholic Relief Services, Rutere Kagendo and I are doing a fascinating post-project evaluation.  Fresh on my mind is the commitment we all had to make to get this quite ground-breaking research going.  Here is the full report, but there are three kinds of conditions we found were integral for success: client-ValuingVoices match; project and site selection; and resources.

Client – Valuing Voices match:

  1. The study needs to be appreciated as innovative, adding to the program quality and learning of the organization so funding is provided and there is in-house interest in the findings;
  2. The local office needs to allocate staff and technical time to support the study technically and logistically (see below);
  3. Shared clarity is needed among all involved that such a study looks for self-sustained activities and outcomes. While there are lessons that emerge about the quality of implementation, its focus is what participants and their country-partners could continue themselves after project close-out and withdrawal of resources. It also can include lessons about what the local non-profit and the national stakeholders are doing to support community success (or not) and unexpected outcomes. Our clients need an openness to honestly seeing what was not sustained and exploring why;
  4. While Valuing Voices provides expertise given review of the handful of post-project and exit evaluations that exist, client is interested in sharing findings and advocating to donors to fund more of these studies;
  5. Disseminating findings internally where a possibility of learning from this evaluation to support similar current implementation; research could help similar project learning, lessons for country nationals such as Ministries;
  6. Prioritizing local capacity – Valuing Voices believes in using regional M&E capacity to do the work; where possible we partner with regional evaluators while also building capacity within our client's staff to carry out such work;
  7. Sharing and discussing findings locally: Valuing Voices believes knowledge learned needs to be shared in immediate feedback loops. We present: a) to each village after each site’s research; b) to local key partners and representatives from each village at the end of the qualitative Rapid Rural Appraisal findings; c) to the non-profit in-country at the end of qualitative research and d) internationally to headquarters at the end of the combined analysis of the qualitative and quantitative research with findings in the final report.

Project and Site Selection:

  1. Non-profit programming projects has been closed out for at least two years and no more than seven years (for recall);
  2. No other NGO has done very similar work in the region in the intervening years;
  3. The region selected is representative of the project as a whole (e.g. agro-ecological zones, economic/ livelihood/ health, educational or other sectoral criteria);
  4. Research areas are secure and safe (e.g. from civil unrest, severe drought/ floods, epidemics, to the degree possible);
  5. Timing does not interfere with urgent priorities of those involved with the study (e.g. livelihoods are not jeapordized in communities, holidays are kept, other technical work is not disrupted).

Resources (Time, Material and Project Expertise):

  1. Time: the research is qualitative followed by quantitative, coming to 80-90 days of qualitative and quantitative research overall (roughly 5 weeks of fieldwork in teams of 4-10, analysis, report-writing and presentation;
  2. Project and evaluation documents are available to inform and contextualize approach including activities, outcomes and projected impacts;
  3. Data: key to the fieldwork are village and participant lists from pre-closeout days so participants can be interviewed both during a Rapid Rural Appraisal and a follow-on household survey;
  4. Internal/external sectoral staff and at least one past project staff are part of the team to inform and ‘ground truth’ research;
  5. Logistics support is provided by the client; from vehicle/driver and lodging support in the field to materials such as mobile phones, flipcharts, photocopying and advances;
  6. A consultant or staff prepares the sites before the research teams come, e.g. to confirm communities are willing to be visited (each visit will be 2-5 days) and to identify participants and partners still there;
  7. Partners familiar with the closed project can be identified so they can be interviewed by the research team;
  8. Local language expertise is needed, e.g. translator to local language, as well as data entry personnel afterwards.
  9. While Valuing Voices provides the technical lead experts and statistical back office analysis, including sampling and rigorous analysis, senior non-profit staff are needed in-country for contextualization and input on preliminary findings, as well as senior technical staff to review the final product;
  10. A home for findings and on the road for dissemination: good knowledge management is needed for data retention, for the findings to have a sustainable ‘home’– be that info-graphics and print copies distributed to villages and partners or online repositories created that are language-accessible both nationally and by foreign donors; webinars, conference presentations etc are needed to optimize the learning via sustainability dissemination campaigns.

Much of this is needed for the research to be the best quality and yield the highest results. It is exciting learning is to be had from not only what communities and supporters could sustain but what they exceeded or dropped!  Consider doing one to see how post-project sustainability research can improve your current implementation, future design, and long-term self-sustainability!