Building the Evidence Base for Post Project Evaluation:
A report to the Faster Forward Fund
We are delighted to share Valuing Voices’ report on the value added of post-project evaluation, which compares findings from eight end-of-project and subsequent post project evaluations [1]. Many of you are aware of how rarely post project evaluations are undertaken. As a result, there is little real evidence about project impact on long-term sustainability. Valuing Voices received a grant from Michael Scriven’s Faster Forward Fund to begin to address this gap.
Our findings show that post project evaluations can contribute to better understanding of sustainability impacts, and reveal unexpected and emerging outcomes years after project close. They also indicate ways in which we can design and implement for sustainability.
Finding suitable projects for this review was difficult because so few post project evaluations are done, fewer are publically available, and fewer still had comparable final evaluations and included local voices. Agencies that fund post project evaluations offer a range of reasons for doing so: to learn, to promote a success, to inform replication or scale, to provide justification for future funding, to promote accountabilities. However, many funding agencies consider post project evaluation a luxury or not necessary. JICA and OECD are notable exceptions in this regard.
Highlights include:
- The review highlights the range of methods that have been used in post project evaluations, and point to the advantages of planning for sustainability measurement from the outset of the project.
- The cases reviewed in the study highlight the (sometime dramatic) difference between the anticipated trajectory of a project, what is happening as the project ends, and what actually continued, was adapted, ceased or changed course after close out.
- Taxonomies, knowledge management about evaluation, data retrieval/ retention, analysis, use and dissemination are elements of sustained impact evaluation that require attention.
- Little documentation is available about how post project evaluations have actually informed and influenced organizational learning, sectoral dialogue or future programming.
- Post project evaluations shed particularly interesting light on what emerged post-project that was entirely due to the efforts and resources of participants and partners after project investments stopped. More on these Sustained and Emerging Impacts Evaluations (SEIEs) at Better Evaluation.
As part of this report, Valuing Voices created an evaluability checklist for assessing whether a post project evaluation is viable, as well as a checklist for measuring sustainability starting at the beginning of the project cycle [2].
We welcome your comments on this report and checklists, and encourage you to share it in your networks and get us feedback on their use. Please use the report and findings to advocate for more post project sustainability impact evaluations which will contribute to greater evidence-based learning about project sustainability. Valuing Voices is among a handful of organizations who do post-project evaluations and we can either conduct one or refer you to another who does.
Thank you,
Laurie Zivetz, MPH, PhD and Jindra Cekan, PhD, with Kate Robins, MPH, PhD of Valuing Voices
The full report is available here:
https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf
Sources:
[1] Zivetz, L., Cekan, J., & Robbins, K. (2017, May). Building the Evidence Base for Post-Project Evaluation: Case Study Review and Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf
[2] Zivetz, L., & Cekan, J. (n.d.). Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2017/08/Valuing-Voices-Checklists.pdf
“I like the mention of a “Theory of Sustainability” for each project as a part of/complement to a Theory of Change…In other words, a sound TOC contains the assumptions around sustainability trajectories, contextual factors, and other assumptions that will ultimately dictate the sustainability of project outcomes/impacts.
In some project design processes I’ve wondered whether we should use a TOC goal that goes beyond the project lifespan. I.e. if your five-year project begins in 2015 and your Results Framework goal is “By 2020, participants will…” we could set the corresponding Theory of Change goal as “By 2025, participants will…” thereby forcing us to build in TOC assumptions about what we expect beyond the life of activity. You’d still only technically be liable to deliver what’s in the RF, but you’d have to show convincingly why you believe certain assumptions will hold true for the longer term goal, which would force you to start thinking about life after the project from Day One. We could even make this a standard procedure for CRS project TOCs” CRS l Field RegionaStaffer
Thank you