by Jindra Cekan | Jun 12, 2023 | Adaptation Fund, climate change, evaluability checklist, Learning, Sustainability, sustainability ratings, Sustainable development, UNDP, Valuing Voices, WFP, World Bank
Evaluating global aid projects ex-post closure: Evaluability, and how-to via Sustainability (and Resilience) Evaluation training materials:
How do we evaluate projects ex-post, and are all projects evaluable after donors leave? How do we learn from project documents to ascertain likely markers of sustainability (hint: see materials for Theory of Sustainability and Sustainability Ratings) that we can verify? How do we design projects to make sustainability more likely (hint: implement pre-exit the drivers in the second image, below)? How to consider evaluating resilience to climate change (hint: evaluate resilience characteristics)?
In 2017, Valuing Voices created an evaluability checklist for Michael Scriven’s foundation, as many projects are not implemented long enough, or too long ago, or have such weak monitoring and evaluation (M&E) data as to be very difficult to evaluate: https://valuingvoices.com/wp-content/uploads/2017/08/Valuing-Voices-Checklists.pdf. We have used it in multiple evaluations since. Now in 2023, we are sharing work we’ve done with the Adaptation Fund, where we have applied that evaluability process via a vetting evaluability process that eliminated over 90% of all projects done in their early years, which alone is sobering but typical in our field, and is leading to changes in how they monitor, retain information and learn. See page 24 – officially pg 15) of this Phase 1 report: https://www.adaptation-fund.org/wp-content/uploads/2021/10/2021-09-12-AF-TERG_Ex-Post-Phase-1-Final-Report_Final.pd
Ex-post evaluability at the Adaptation Fund
The Adaptation Fund has also committed to doing two ex-post sustainability and resilience evaluations per year. and learn from remote ex-posts as well. The first two ex-post evaluations are found here: Samoa/ UNDP on resilient infrastructure (https://www.adaptation-fund.org/wp-content/uploads/2023/01/Ex-Post-Evaluation-Samoa-final-ed-2.pdf) and Ecuador/ WFP on food security (https://www.adaptation-fund.org/document/ex-post-evaluation-summary-ecuador/). So many lessons from assumptions made to future design…
Not only are we evaluating projects ex-post, but we are also creating processes for others to follow, including this new Sustainability Framework. The left three columns project the likelihood of sustainability, which the right three columns verify. It includes the Valuing Voices ’emerging outcomes’ from local efforts to sustain results after donors leave. SO exciting:
Ex-post Sustainability Framework
Also, just today, we published our training materials which we are using in ex-posts 3 and 4 in Argentina (Ministry of the Environment and the World Bank); https://www.adaptation-fund.org/document/training-material-for-ex-post-pilots/. They include videos for those who like to listen and watch us, as well as Powerpoint slides/ PDFs and Excels of our training materials (including suggested methods) for those who like to read…
Take a look, use, and tell us what you think and what you are learning! Thanks, Jindra Cekan (Sustainability) and Meg Spearman (Resilience) and the Adaptation Fund team!
by Jindra Cekan | Aug 2, 2017 | evaluability checklist, ex-post evaluation, Faster Forward Fund (3F), measuring sustainability checklist, Participation, post-project evaluation, Project cycle, SEIE, Sustainability, Sustained and Emerging Impact Evaluation (SEIE), Sustained and Emerging Impacts Evaluation, Valuing Voices
Building the Evidence Base for Post Project Evaluation:
A report to the Faster Forward Fund
We are delighted to share Valuing Voices’ report on the value added of post-project evaluation, which compares findings from eight end-of-project and subsequent post project evaluations [1]. Many of you are aware of how rarely post project evaluations are undertaken. As a result, there is little real evidence about project impact on long-term sustainability. Valuing Voices received a grant from Michael Scriven’s Faster Forward Fund to begin to address this gap.
Our findings show that post project evaluations can contribute to better understanding of sustainability impacts, and reveal unexpected and emerging outcomes years after project close. They also indicate ways in which we can design and implement for sustainability.
Finding suitable projects for this review was difficult because so few post project evaluations are done, fewer are publically available, and fewer still had comparable final evaluations and included local voices. Agencies that fund post project evaluations offer a range of reasons for doing so: to learn, to promote a success, to inform replication or scale, to provide justification for future funding, to promote accountabilities. However, many funding agencies consider post project evaluation a luxury or not necessary. JICA and OECD are notable exceptions in this regard.
Highlights include:
- The review highlights the range of methods that have been used in post project evaluations, and point to the advantages of planning for sustainability measurement from the outset of the project.
- The cases reviewed in the study highlight the (sometime dramatic) difference between the anticipated trajectory of a project, what is happening as the project ends, and what actually continued, was adapted, ceased or changed course after close out.
- Taxonomies, knowledge management about evaluation, data retrieval/ retention, analysis, use and dissemination are elements of sustained impact evaluation that require attention.
- Little documentation is available about how post project evaluations have actually informed and influenced organizational learning, sectoral dialogue or future programming.
- Post project evaluations shed particularly interesting light on what emerged post-project that was entirely due to the efforts and resources of participants and partners after project investments stopped. More on these Sustained and Emerging Impacts Evaluations (SEIEs) at Better Evaluation.
As part of this report, Valuing Voices created an evaluability checklist for assessing whether a post project evaluation is viable, as well as a checklist for measuring sustainability starting at the beginning of the project cycle [2].
We welcome your comments on this report and checklists, and encourage you to share it in your networks and get us feedback on their use. Please use the report and findings to advocate for more post project sustainability impact evaluations which will contribute to greater evidence-based learning about project sustainability. Valuing Voices is among a handful of organizations who do post-project evaluations and we can either conduct one or refer you to another who does.
Thank you,
Laurie Zivetz, MPH, PhD and Jindra Cekan, PhD, with Kate Robins, MPH, PhD of Valuing Voices
The full report is available here:
https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf
Sources:
[1] Zivetz, L., Cekan, J., & Robbins, K. (2017, May). Building the Evidence Base for Post-Project Evaluation: Case Study Review and Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf
[2] Zivetz, L., & Cekan, J. (n.d.). Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2017/08/Valuing-Voices-Checklists.pdf
