Building the Evidence Base for Post Project Evaluation:
A report to the Faster Forward Fund
We are delighted to share Valuing Voices’ report on the value added of post-project evaluation, which compares findings from eight end-of-project and subsequent post project evaluations . Many of you are aware of how rarely post project evaluations are undertaken. As a result, there is little real evidence about project impact on long-term sustainability. Valuing Voices received a grant from Michael Scriven’s Faster Forward Fund to begin to address this gap.
Our findings show that post project evaluations can contribute to better understanding of sustainability impacts, and reveal unexpected and emerging outcomes years after project close. They also indicate ways in which we can design and implement for sustainability.
Finding suitable projects for this review was difficult because so few post project evaluations are done, fewer are publically available, and fewer still had comparable final evaluations and included local voices. Agencies that fund post project evaluations offer a range of reasons for doing so: to learn, to promote a success, to inform replication or scale, to provide justification for future funding, to promote accountabilities. However, many funding agencies consider post project evaluation a luxury or not necessary. JICA and OECD are notable exceptions in this regard.
- The review highlights the range of methods that have been used in post project evaluations, and point to the advantages of planning for sustainability measurement from the outset of the project.
- The cases reviewed in the study highlight the (sometime dramatic) difference between the anticipated trajectory of a project, what is happening as the project ends, and what actually continued, was adapted, ceased or changed course after close out.
- Taxonomies, knowledge management about evaluation, data retrieval/ retention, analysis, use and dissemination are elements of sustained impact evaluation that require attention.
- Little documentation is available about how post project evaluations have actually informed and influenced organizational learning, sectoral dialogue or future programming.
- Post project evaluations shed particularly interesting light on what emerged post-project that was entirely due to the efforts and resources of participants and partners after project investments stopped. More on these Sustained and Emerging Impacts Evaluations (SEIEs) at Better Evaluation.
As part of this report, Valuing Voices created an evaluability checklist for assessing whether a post project evaluation is viable, as well as a checklist for measuring sustainability starting at the beginning of the project cycle .
We welcome your comments on this report and checklists, and encourage you to share it in your networks and get us feedback on their use. Please use the report and findings to advocate for more post project sustainability impact evaluations which will contribute to greater evidence-based learning about project sustainability. Valuing Voices is among a handful of organizations who do post-project evaluations and we can either conduct one or refer you to another who does.
Laurie Zivetz, MPH, PhD and Jindra Cekan, PhD, with Kate Robins, MPH, PhD of Valuing Voices
The full report is available here:
 Zivetz, L., Cekan, J., & Robbins, K. (2017, May). Building the Evidence Base for Post-Project Evaluation: Case Study Review and Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf
 Zivetz, L., & Cekan, J. (n.d.). Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2017/08/Valuing-Voices-Checklists.pdf
How ‘new’ are our projects… and who is aiming at the right outcomes?
Valuing Voices exciting news is we have received research grant funding from the esteemed evaluator, Michael Scriven’s Faster Forward Fund. We’re looking into the value-added of (ex-post) Sustained and Emerging Impacts Evaluations (SEIEs) and we are doing the research now. We will be documenting methods used and discuss how best to evaluate such sustained impacts after project close-out. Very exciting stuff in this staggeringly ‘new’ field of evaluation.
During this research, a senior international development evaluation expert told us that they can’t return to evaluate now-closed projects because they aren’t the same projects anymore (after closeout) and we are no longer responsible for the results. That took my breath away.
All new projects come from old projects… we recycle old project design most of the time, occasionally making substantive changes in targeting or design but much of how we design and implement remains the same. And while we thoroughly evaluate them during implementation, learning ex-post is a key missing link which all projects in the future can benefit from as we do similar interventions and track similar outcomes year after year but we rarely know which ones were sustained or emerged anew. There absolutely are aspects that get adapted but there are only so many ways to heal the sick, improve crop growth, save money, learn to read and so on, and there is a world we need to learn about what enabled some to be sustained and even morph into new results!
This excellent article, Do NGOs (non-governmental organization) help?, notes that “due to donor pressure [NGOs] are increasingly forced to respond with a discrete project with x number of deliverable outcomes” . It goes on to cite D. Sriskandarajah, the secretary-general of Civicus, a global network of civil society organizations and activists, wrote: ‘We have become a part of the problem rather than the solution… Since demonstrating bang for your buck has become all-important, we divide our work into neat projects, taking on only those endeavours that can produce easily quantifiable outcomes. Reliant on funding to service our own sizeable organizations, we avoid approaches or issues that might threaten our brand or upset our donors. We trade in incremental change’” .
We also settle for results while we control them, and don’t ask unpopular questions about who is to sustain these results, with what resources, and for goodness sake, why sustained impact was not funded, designed, implemented and monitored/ evaluated from the very onset in our rush to measurable results?
As this great NGO article by Dinyar Godrej goes on to say, “most media scrutiny of NGO accountability is of how they use funds, their accountability to donors. But what of their accountability towards the recipients of their interventions” ? They have no lobbyists to persuade our funders they would like this but nto that, and often such lobbying for their needs falls to the very NGOs that have won these large contracts and tasked with implementing a dizzying array of mandatory input, output, outcome and some impact indicators. We do care deeply about results! US State Department/ USAID has a “Standard Foreign Assistance Master Indicator List” of 2,300 lines in an excel spreadsheet . (There are more indicators still– custom and cross-cutting indicators, the mind boggles).
Wow. But are we asking the right questions? Are we asking what was sustained after all this hard work was done and ended? Rarely. Who should be?! “It is perhaps unrealistic to expect such large structural changes to be delivered by NGOs when governments don’t tackle them either.”
For the rub is this. When we take development over from national governments, largely do not involve country nationals in the funding, design and M&E of projects, then how sustained can these projects still be after we go? Millions are invested, then disappear… Last year, at local debrief at the end of one SEIE Valuing Voices did, the state of affairs became crystal clear when a government official asked us “Can you ever find some funds to fund us to do our own independent evaluations? Even if it is not the projects that they did themselves? We would be happy to get that support…”
When are we no longer responsible for doing great, sustained work? Valuing Voices will let you know what we found regarding the best ways to do SEIEs more. Stay tuned.
What do you think?
P.S. This blog topic prompted me to look for statistics on the number or percentage of funded projects that were renewed. Nothing. Does anyone know how many or what % of projects were extended/ funded again after showing good results? (Often this happens in the form that a successful project in one area of the country gets either funded again or repeated elsewhere in the country or in the world, as have two of our own SEIEs, Niger and Ethiopia). For that matter, what made them so excellent to be replicated? What can we learn?
 Godrej, D. (2014, December 1). NGOs – Do They Help? Retrieved from https://newint.org/features/2014/12/01/ngos-keynote/
 US Department of State. (n.d.). Standard Foreign Assistance Indicators. Retrieved 2017, from https://web.archive.org/web/20170404072145/https://www.state.gov/f/indicators/index.htm
Leading in Challenging Times:
Sustained and Emerging Impacts Evaluation (SEIEs)
Some American organizations are retrenching, focusing more attention on domestic rather than international programming. Some are pulling back from critique of international development to informing legislators of its benefits; the Center for Global Development’s changed ‘Rethinking US Development Policy’ blog to only “US Development Policy“. UN’s Refugee Agency questions whether to challenge Washington’s tough line on refugees from countries such as Syria, or should it stay quiet in the hopes of protecting its funding ?”
Reticence is understandable in this ‘climate’, so to speak, but fear does not change the world, leadership does. Envisioning and creating the world we want gets us there.
There may be no better time to build the evidence base on what works in sustainable development as these are low cost investments if we use national staff and focus research well. We have seen this in the fewer than 1% of all projects that have been evaluated post-closeout for sustainability . At the very least, we can learn what we should do differently in the next design, to fully foster sustainability, once more funding emerges. Many are interested in great results. Hundreds of ‘impact evaluations’ are happening on aid effectiveness; our industry wants to learn what works and what we could do better.
Our SEIE work goes beyond current understanding of ‘impact’ to see what projects our partners and participants can self-sustain ex-post for years to come which is an excellent investment in proving cost-effectiveness. While some governments’ investments can diminish in the short term, national governments, and other funders such as a range of international bilateral and multilateral donors, foundations corporate social responsibility and impact investors do want to invest in provably “sustainable” development .
Why should we invest in SEIEs?
- Hundreds of thousands of projects are still being implemented.
- Millions of participants are still hoping what we are doing together will be sustainable.
- Billions of dollars, euros, kwacha, pesos, rupees are being spent on new projects that need to be designed and implemented for future sustainability.
Implementing organizations could be fearful to see what remains once funding and technical assistance are withdrawn, but such a view not only robs our industry of exciting lessons on what did change and was so valued that it was sustained, but also what to not do again. Not returning post-project also short-changes our participants. In our SEIEs, we have found participants and partners creating new ways to carry on, innovating beyond what we could imagine during our assistance.
We also need to start now to design and implement for sustainability. doing SEIEs, we can start to understand the ‘drivers’ behind the Sustainable Development Goals (SDG) results with countries tracking some 120 indicators across 17 goals. Currently countries are tracking up to 230 indicators across the 17 Goals . But while such monitoring shows ‘GDP has increased or ‘under-nourishment has decreased’, there is little or no information on what has caused it. Yet doing and SEIE on a large donor-funded programme, we can explore what elements made projects sustainable and how to do more (or less) there and elsewhere. Such sentinel site support for learning about sustained and emerging impacts is key to understand some of the why, for example, did income or health improve.
Dare to lead, especially in these challenging times. We know of organizations that are doing these evaluations internally, others are publishing them on their sites. Leadership happens at all levels, from internal, technical to managerial and administrative work to external evaluators and consultants as well as public pressure.
How can you foster sustained impact?
- You can advocate for such evaluations
- You can share the SEIE guidance, below, and start to design and implement, monitor and evaluate sustainably in all projects/ proposals you are designing now.
- You can see if your organization has done any post-project sustainability evaluations and we can post them on Valuing Voices’ repository, celebrating your organization.
We can help you learn how to do these. Our partner, Better Evaluation, just published our Sustained and Emerging Impacts Evaluation as a ‘new’ evaluation ‘theme.’
Guidance there shows you :
1. What is SEIE?
2. Why do SEIE?
3. When to do SEIE?
4. Who should be engaged in the evaluation process?
5. What definitions and methods can be used to do an SEIE?
SEIEs will grow as will examples, discussions, and joy as embracing sustainability sprouts, and sends us progressing in yet-unforeseen ways! We are excited to be in the final stages of receiving a research grant to further guide SEIEs. We will share that news in our next blog.
We want to learn from you:
- What do you think needs to be in place for funders to move beyond the funding cycle and do an SEIE?
- What would help to make this type of evaluation more widely undertaken?
- If you have done a post-project evaluation, how did you do it? What were some of the barriers you faced and resources you were able to draw on to overcome them?
How can we lead together to Value the Voices of those we serve!?
(Reposted from https://medium.com/@WhatWeValue/leading-in-challenging-times-sustained-and-emerging-impacts-evaluation-seies-617b33bf4d27#.ec7fcg4ty)
 Foulkes, I. (2017, February 27). Is there a US diplomacy vacuum at the UN in Geneva? Retrieved from https://www.bbc.com/news/world-europe-39080204
 Cekan, J. (2015, March 13). When Funders Move On. Retrieved from https://ssir.org/articles/entry/when_funders_move_on
 UN DESA. (2011, March 2). Lasting impact of sustainable development. Retrieved from https://www.un.org/en/development/desa/news/sustainable/sustainable-development.html
 UN Statistics Division. SDG Indicators: Global indicator framework for the Sustainable Development Goals and targets of the 2030 Agenda for Sustainable Development. Retrieved March, 2017, from https://unstats.un.org/sdgs/indicators/indicators-list/
 Cekan, J., Zivetz, L., & P, R. (2016). Sustained and Emerging Impacts Evaluation (SEIE). Retrieved from https://www.betterevaluation.org/en/themes/SEIE
Making up your mind. Prioritizing and making it happen
* As Dr. Martin Luther King Jr. said, "every man must decide whether he will walk in the light of creative altruism or in the darkness of destructive selfishness".
* Our President, Barack Obama said in his farewell speech, "change only happens when ordinary people get involved and they get engaged, and they come together to demand it".
* OXFAM International demanded changing shocking inequity: "just eight billionaires own the same wealth as the 3.6 billion people who make up the poorest half of humanity".
* Caroline Heider of the World Bank's IEG asked we examine how we evaluate long-term impacts: "current considerations of efficiency, cost savings, or cost-benefit analyses are challenged to take long-term impacts into account".
What do you want to prioritize and demand of international development? In these times of shifting priorities in powerful nations, where politicians are questioning the needs of those whom many of us have been serving, what do you want to demand? What issue do you prioritize, and want to move forward?
I choose to prioritize sustained impact driven by country-nationals. Why? I grew up in large cities, and when I first worked in Africa’s Sahel desert 25 years ago, the herders and farmers making a living from the arid pastures and sandy soil, with wells 100 feet deep astonished me.
Without them, I’d last 3 days out there. They were the experts.
I always assumed we measured ‘sustainable’ development in work with such herders and farmers, but in 2013 I founded Valuing Voices after I began to see how rarely we return to evaluate what remained after our foreign aid projects stopped.
Reviewing thousands of “ex-post” or “post-project” documents in 30 organizations’ public databases, Valuing Voices has found the vast majority of documents only suggested a post-project be done, a small proportion were desk studies and fewer than 1% were original fieldwork post-project evaluations of sustainability. In these 370 post-project (ex-post) evaluations, development workers asked partners and participants what was still standing, showed what succeeded or failed and what unexpected successes participants created themselves from what we left behind.
Returning to learn, consulting our participant-experts seems so common sense as they are the ones that can tell us what we should replicate, adapt or abandon. In 2015 research we found only three World Bank IEG evaluations that asked participants their views in a methodologically clear way (out of 33 post-project PPAR evaluations), and only one was perceived as successful. On the other hand, in 2014, IRIN highlighted Rwanda’s very successful community based nutrition solutions, replete with participant voices. We have found 23 ‘catalytic’ (mostly NGO) organizations having done one or more (ex-) post-project evaluation that include participant input and each of them is filled with excellent lessons for doing ‘development’ well now and after closeout. Yet what are any of these organizations doing differently and why are so few doing more? Why do donors seem to care so little about sustained impact that such studies are so rarely funded by them, and NGOs use private funds? That is what drives me.
A seminal book, Time to Listen asked 6,000 such participant-experts in 20 countries what they wanted foreign aid to look like. “Very few people call for more aid; virtually everyone says they want “smarter” aid…. A majority criticize the “waste” of money and other resources through programs they perceive as misguided or through the failure of aid providers to be sufficiently engaged… [it is] a supply-driven approach that squeezes out the views of the recipients, and a focus on spending – both volume and speed, which undermines aid’s ability to listen, learn and adapt to local contexts.”
While Valuing Voices is not profitable (yet?) and growth is slow, I continue to evaluate and advocate, believing that designing, implementing, monitoring and evaluating for sustained impact by our true clients is key to successful work life well spent.
We need a sustained impact mindset.
We are getting there. Better Evaluation just featured our Sustained and Emerging Impact Evaluation (SEIE) approach as a new theme in evaluation. OXFAM and Save the Children recently wrote “The Power of Ownership: Transforming US Foreign Assistance” (2016). They ask: “country ownership is at the core of effective development… as the United States transitions to a new President and new leadership for development cooperation, how will the next administration build on current successes and chart a path forward?“ I fear the answer, as it takes trust and interest in countries’ capacity to chart their own way forward. USAID (and maybe other donors?) are ready to help. USAID alone has some done some exciting work recently through USAID Forward’s local partners (e.g. Afghanistan has done this in depth) and it has looked at Local Systems. Food For Peace’s strategy includes sustainability.
What will be a priority in 2017 onward? What each of us creates will remain.
The powerful Sidekick Manifesto beautifully proposes this new core belief which we can each espouse, that “Local leaders with local solutions to local problems” will end poverty. We will not.” We can, however, “always be listening, learning and seeking a deeper understanding…” I am delighted to be a sidekick in projects that prioritize participant and partner views, for that is how they end poverty.
What do you want to prioritize and create? What is so vital for you that you must work on it? What has been neglected? What difference do you want to make? GO!
Presenting Lessons on (post-project) Sustained and Emerging Impact Evaluations from the U.S. AEA Conference
Dear readers, attached please find the Barking up a Better Tree: Lessons about SEIE Sustained and Emerging Impact Evaluation presentation we did last week at the American Evaluation Association (AEA) conference in Atlanta GA . I had the pleasure of co-presenting with Beatrice Lorge Rogers PhD, Professor, Friedman Nutrition School, Tufts University (aka the famous Food for Peace/ Tufts Exit Strategy study ), Patricia Rogers PhD, Director, BetterEvaluation, Professor, Australia and New Zealand School of Government (where we recently published guidance on SEIE ) and Laurie Zivetz PhD, International Development Consultant and Valuing Voices evaluator.
We integrated our presentations from Africa, Asia and Latin America into this fascinating overview:
1.Sustained and Emerging Impact Evaluation: global context
2.SEIE: definitions and methods
3.Case studies: findings from post-project evaluations
4.Designing an SEIE: Considerations
5.Q&A — which fostered super comments, but since you couldn’t come, please tell us what you think and what questions you have…
There are amazing lessons to learn about design, implementation, M&E from doing post-project evaluation. We have also grown in appreciating that sustainability can be tracked throughout the project cycle, not just during post-project SEIE evaluation.
We’ll be building this into a white paper or a … (toolkit? webinar series? training? something else?). What’s your vote ___? (I know in this US election season, so… :)).
What would you like to get to support your learning about Sustained and Emerging Impact Evaluations? Look forward to hearing from you- Jindra@ValuingVoices.com
The full presentation is available here:
 Cekan, J., Rogers, B. L., Rogers, P., & Zivetz, L. (2016, October 26). Barking Up a Better Tree: Lessons about SEIE (Sustained and Emerging Impact Evaluation). Retrieved from https://valuingvoices.com/wp-content/uploads/2016/11/Barking-up-a-Better-Tree-AEA-Oct-26-FINAL.pdf
 Food and Nutrition Technical Assistance (FANTA). (n.d.). Effective Sustainability and Exit Strategies for USAID FFP Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
 Zivetz, L., Cekan, J., & Robbins, K. (2017, May). Building the Evidence Base for Post-Project Evaluation: Case Study Review and Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf
Maximizing what we've got… Time is now!
We had a stirring conversation here in D.C. with someone very knowledgeable about sustainability; this person is a strong proponent of local ownership of all development. They also said vehemently, why evaluate the sustainability of projects after closeout; we all know what that will show! What was implied is that our system of international development and aid is so flawed, so broken, that the inevitable result of not focusing on local ownership as the fundamental basis for our work means Nothing. Will. Be. Left…. All. Is. Lost.
We disagree. Our development industry does some good, some bad, and is ever-changing (albeit slowly). Billions of dollars each year are spent trying to improve people's lives and livelihoods around the world, and we've seen great good be done. While. We. Remain.
We know far, far less about what remains after our projects end because less than 1% of the time we return post-project (ex-post) to evaluate anything.
Our problem with time begins with fixed timelines within projects that say we have 1, 3, 5 years to get to success. They work with participants and partners who need to make substantial changes to how they use their resources and beliefs over a relatively short time of a few months to a few years. We expect immediate results from them, changing how they farm (use new seeds, new methods, new ways of interacting with markets) and save money (learn new concepts of profit and interest, repayment and re-lending), and improve their health and that of their families (get prenatal exams, vaccinate your children, exclusively breastfeed without adding water or tea). Everyone in our projects is 'on the clock' from the donor and implementer to partners and participants. This clock ticks down irrevocably as project closeout looms, promised-successes-to-donors at hand or a mirage in the distance. We assume sustained results.
How many of us have ever gone on a diet? How many have learned a new language? How many of us have transferred jobs and had to learn new skills on the job? How quickly have we managed to do all that successfully, all at once?! Probably many. How many of you have had to do this on a fixed timeline? Were you successful when there was a limited, fixed time and you did not set your own pace?
It takes time to implement projects well enough to ensure that most participants ‘got it’, not just the 'early adopters'. It takes time to hand over projects so well that our partners and participants are ready to take over at least some of what we worked so hard to transfer. It took leadership and staff two years in the very successful participatory USAID/ Food For Peace food security project by CRS Niger that was a continuation of similar programming for 15 years. It also takes time to pass for conditions to be ready for our return, to isolate what people could self-sustain from what the project supplied, to learn what was so well designed and implemented during projects that to 'took root' in people's lives, that they have made it their own. We estimate optimal evaluation time is 2-7 years after closeout. Valuing Voices also believes we should not just evaluate the sustainability of outputs and outcomes of what we put in place that we thought they would continue, and the sustained impact of those cumulative investments, but also the emerging, unintended new activities and impacts we never imagined people would innovate from our projects. We are doing just such evaluations in Zimbabwe and Uganda now and hope to do and catalyze much more fieldwork around the world.
And why does it matter? Why shouldn't we write off our time-limited donor-funded projects? Because:
1) It's all we've got. Our current development system is not going anywhere soon, and there is success to learn from.
2) We need to quickly learn from what worked sustainably best and stop wasting time and resources on what we refuse to admit fails because we are too scared to return to see. Go back with the intention to learn what does and focus on doing more of what works.
3) Such analysis – and design of new projects – must have country ownership as a centerpiece throughout the project cycle assumptions, but to throw out decades of good work simply because we are just learning the value of country ownership is foolish.
Finally, here's a lovely example from Brazil of how local, participation (and yes, as my colleague thought, local ownership) works best. And. It. Takes. Time.
"Our results also show that Participatory Budgeting’s influence strengthens over time… Participatory Budgeting’s increasing impact indicates that governments, citizens, and civil society organizations are building new institutions… cities incorporate citizens at multiple moments of the policy process, allowing community leaders and public officials to exchange better information." How often do we return to do what are called longitudinal reviews of our work abroad, using the same rigorous standards we evaluate our domestic projects? Not often. Shouldn't that change?
Only by working together, honoring the value of our participants, that they deserve the same chance at change that we take for granted will things change. We must value both the voices of our participants and our own expertise for development to improve for true aid effectiveness…. Let us begin anew!