Grow the .002% of all global development projects that are evaluated ex-post closure for sustainability

Grow the .002% of all global development projects that are evaluated ex-post closure for sustainability

It seems like ‘fake news’ that after decades of global development so few evaluations would have peered back in time to see what was sustained. While I was consulting to the Policy Planning and Learning Bureau at USAID, I asked the head of this M&E department who does ex-post sustainability evaluation as I knew USAID had done some in the 1980s, Cindy Clapp-Wincek answered ‘No one, there are no incentives to do it.’ (She later became our advisor.)

Disbelieving, I did a year of secondary keyword research before devoting my professional consulting life to advocating for and doing ex-post evaluations of sustained outcomes and impacts. I searched USAID, OECD, and other bilateral and later multilateral donors’ databases and found thousands of studies, most of which were inaccurately named ‘ex-post’ or ‘post-closure’ studies.  Some of the roughly 1,000 projects I looked at at USAID and OECD that came up under ‘ex-post’, ‘ex post’, ‘post closure’ were final evaluations that were slightly delayed, a few were evaluations that were at least one year after closure, but were desk studies without interviews. Surprisingly, the vast majority of final evaluations found were those that only recommended ex-post evaluation several years later to confirm projected sustainability.

 

 

 

 

 

 

 

 

In 2016 at the American Evaluation Association conference, a group of us did a presentation. In it, I cited these statistics from of 1st year of Valuing Voices’ research:

  • Of 900+ “ex-post” “ex post” “post closure” documents in USAID’s DEC database, there were only 12 actual post-project evaluations with fieldwork have been done in the last 20 years
  • Of 12,000 World Bank projects – only 33 post-project evaluations asked ‘stakeholders’ for input, and only 3 showed clearly they talked to participants
  • In 2010 Asian Development Bank conducted 491 desk reviews of completed projects, and returned to 18 actual field-based post-project evaluations that included participant voices; they have done only this 1 study.
  • We found no evaluations by recipient governments of aid projects’ sustainability

12 years of research, advocacy and fieldwork later, the ‘catalysts’ database on Valuing Voices now shows actual fieldwork-informed evaluations by 40 organizations that had actual ex-posts that returned to the field to ask participants and project partners what was sustained, highlighting 92 ex-posts.

How many ex-post project closure evaluations have been done? .002% of all projects. The 0.002% statistic looks at just public foreign development aid from 1960 (not even counting private funding such as foundations or gifts to organizations, which isn’t tracked in any publicly available database). Calculating aggregated OECD aid statistics (excluding private because it’s only recent data) over 62 years $5.6 trillion by 2022 (thanks to Rebecca Regan-Sachs for the updated #s).

I then estimated 3.000 actual ex-posts which comes from 2,500 JICA projects plus almost 500 other projects that I have either found looking through databases all across the spectrum from governments and multilaterals (almost 100 in our catalysts, and am assuming there must be 400 others done in the 1980s-2000 like USAID and the World Bank).

Without a huge research team it is improssible to aggregate data on the total number of projects by all donors. So I extrapolated from project activity disbursements of one year (2022) for Mali on the www.foreignassistance.gov page. In my 35 years of experience, Mali, where I did my doctoral research, typifies he average USAID aid recipient. They had 382 projects going in 2022. I rounded up to 400 projects x 70 years (since 1960 when OECD data began) x 100 countries by just one donor (of the 150 possible recipient countries, to be conservative). This comes to 2.8 million projects. So if we take 39 OECD countries as donors (given most have far less to give than US), in total 109 million publicly funded aid projects disbursed $5.6 trillion since 1960. While final evaluations are industry standard, only .002% is the estimated number of ex-post evaluations of projects the were evaluated with data from local participants and partners of the 109 million projects .

This became Valuing Voices focus, and we created an open-access database for learning, and conducted our own  My team and I identified 92 ex-posts that returned to ask locals what lasted, what didn’t, why, and what emerged from their own efforts. We also created evaluability checklists and created a new evaluation, Sustained and Emerging Impacts Evaluation that included examining not just what donors put in place to last, but also what emerged outcomes from local efforts to sustain results with more limited resources, partnerships, capacities and local ownership/motivation. These four drivers were found by Rogers and Coates for USAID’s food security exit study in 2015). We have done 15 ex-posts for 9 clients since 2006 and shared Adaptation Fund ex-post training materials in 2023.

 

Yet the public assumes we know our development is sustainable. 2015’s ‘Sustainable Development Goals‘ focused aid on 17 themes, which was to generate $12 trillion more in annual spending on SDG sectors than the;$21 trillion already being invested each year. Nonetheless, a recent UN report states that there is now a $4 trillion annual financing gap to achieve the SDGs. All this funding goes to projects that are currently implemented, not to evaluate what had been sustained from past projects that already closed. Such learning from what succeeded or failed, or what emerged from local efforts to keep activities and results going is pivotal to improving current and future programming is almost wholly missing from the dialogue; I know, I asked multiple SDG evaluation experts.

 

Why do we return to learn so rarely? There are many reasons, the most prosaic among them being administrative.

  • When aid funds are spent over 2-10 years, projects are closed, evaluated at the end, ‘handed over’ to national governments, and no additional funding exists to return ‘ex-post’ closure to learn.
  • Next is the push to continue to improve lives through implementation which means low rates of overhead allocated to M&E and learning during, much less after closure.
  • Another is the assumption that ‘old’ projects differ so much from new ones, but there are few differences. After all there are only so many ways to grow food, feed the malnourished, educate children; evaluating ‘old’ projects can teach ‘new’ projects.
  • A last major one, from Valuing Voices’ research of 12 years may be the largest: Fear of admitting failure. Please read Valuing Voices’ 2016 blog highlighted many Lessons about Funding, Assumptions and Fears (Part 3). One US aid lobbyist told me in 2017 that I must not share this lack of learning about sustained impacts because it could imperil US aid funding; I told her I had to tell people because lives were at stake.
  • Overall, there is much to learn; most ex-post evaluations show mixed results. None show 100% sustainability and while most show 30-60% sustainability, none are 0% sustained either. If we don’t learn to replicate what worked and cease what didn’t now, then future programming will be as flawed and successes, especially brilliant emerging locally designed ex-post outcomes such as Niger’s local funding of redesign of health incentives will remain hidden.

 

Occasionally donors invest in sets of ex-post learning evaluations such as USAID’s ‘global waters’ seven water/ sanitation evaluations linked to the E3 Bureau taking sustainability as a strategic goal. Yet the overall findings from USAID’s own staff of these ex-posts Drivers of WASH study were chilling. While 25 million gained access to drinking water and 18 million to basic sanitation, ‘they have largely not endured.’ But the good news in such research is that the donor learned that infrastructure fails when spare parts are not accessible and maintenance not funded or performed, which can be planned for and addressed during implementation by investing in resources and partnerships. They learned that relying on volunteers is unreliable and management needs to be bolstered, which can lead to some implementation funding to be focused on capacities and local ownership. We can plan better for sustainability by learning from ex-post and exit studies (see Valuing Voices’ checklists in this 2023 article on Fostering Values-Driven Sustainability).

 

And since 2019, three climate funds, the Adaptation Fund, the Global Environmental Facility, and the Climate Investment Funds have turned to ex-post evaluations to look at sustainability and longer-term resilience and even transformation, given environmental shocks may take years to affect the project sites. The Adaptation Fund has done four ex-posts, with more to come in 2024/25, and the CIF is beginning now. The GEF has done a Post-Completion Assessment Pilot for the Yellow Sea Region . Hopeful!

Making up your mind. Prioritizing and making it happen

Making up your mind. Prioritizing and making it happen

 

* As Dr. Martin Luther King Jr. said, "every man must decide whether he will walk in the light of creative altruism or in the darkness of destructive selfishness".

 

 

 

 

 

 

 

 

Source: https://shanemcdonnell45.wordpress.com/tag/darkness-into-light/

* Our President, Barack Obama said in his farewell speech, "change only happens when ordinary people get involved and they get engaged, and they come together to demand it".

* OXFAM International demanded changing shocking inequity: "just eight billionaires own the same wealth as the 3.6 billion people who make up the poorest half of humanity".

* Caroline Heider of the World Bank's IEG asked we examine how we evaluate long-term impacts: "current considerations of efficiency, cost savings, or cost-benefit analyses are challenged to take long-term impacts into account".

 

What do you want to prioritize and demand of international development? In these times of shifting priorities in powerful nations, where politicians are questioning the needs of those whom many of us have been serving, what do you want to demand? What issue do you prioritize, and want to move forward?

 

I choose to prioritize sustained impact driven by country-nationals. Why? I grew up in large cities, and when I first worked in Africa’s Sahel desert 25 years ago, the herders and farmers making a living from the arid pastures and sandy soil, with wells 100 feet deep astonished me.

 

Without them, I’d last 3 days out there. They were the experts.

 

I always assumed we measured ‘sustainable’ development in work with such herders and farmers, but in 2013 I founded Valuing Voices after I began to see how rarely we return to evaluate what remained after our foreign aid projects stopped.

 

Reviewing  thousands of “ex-post” or “post-project” documents in 30 organizations’ public databases, Valuing Voices has found the vast majority of documents only suggested a post-project be done, a small proportion were desk studies and fewer than 1% were original fieldwork post-project evaluations of sustainability. In these 370 post-project (ex-post) evaluations, development workers asked partners and participants what was still standing, showed what succeeded or failed and what unexpected successes participants created themselves from what we left behind.

 

Returning to learn, consulting our participant-experts seems so common sense as they are the ones that can tell us what we should replicate, adapt or abandon.  In 2015 research we found only three World Bank IEG evaluations that asked participants their views in a methodologically clear way (out of 33 post-project PPAR evaluations), and only one was perceived as successful.  On the other hand, in 2014, IRIN highlighted Rwanda’s very successful community based nutrition solutions, replete with participant voices.  We have found 23 ‘catalytic’ (mostly NGO) organizations having done one or more (ex-) post-project evaluation that include participant input and each of them is filled with excellent lessons for doing ‘development’ well now and after closeout. Yet what are any of these organizations doing differently and why are so few doing more? Why do donors seem to care so little about sustained impact that such studies are so rarely funded by them, and NGOs use private funds? That is what drives me.

 

A seminal book, Time to Listen asked 6,000 such participant-experts in 20 countries what they wanted foreign aid to look like. “Very few people call for more aid; virtually everyone says they want “smarter” aid…. A majority criticize the “waste” of money and other resources through programs they perceive as misguided or through the failure of aid providers to be sufficiently engaged… [it is] a supply-driven approach that squeezes out the views of the recipients, and a focus on spending – both volume and speed, which undermines aid’s ability to listen, learn and adapt to local contexts.”

Sobering stuff.

 

While Valuing Voices is not profitable (yet?) and growth is slow, I continue to evaluate and advocate, believing that designing, implementing, monitoring and evaluating for sustained impact by our true clients is key to successful work life well spent.

 

We need a sustained impact mindset.

 

We are getting there. Better Evaluation just featured our Sustained and Emerging Impact Evaluation (SEIE) approach as a new theme in evaluation. OXFAM and Save the Children recently wrote “The Power of Ownership: Transforming US Foreign Assistance” (2016). They ask: “country ownership is at the core of effective development… as the United States transitions to a new President and new leadership for development cooperation, how will the next administration build on current successes and chart a path forward?“  I fear the answer, as it takes trust and interest in countries’ capacity to chart their own way forward.  USAID (and maybe other donors?) are ready to help. USAID alone has some done some exciting work recently through USAID Forward’s local partners (e.g. Afghanistan has done this in depth) and it has looked at Local Systems. Food For Peace’s strategy includes sustainability.

 

What will be a priority in 2017 onward? What each of us creates will remain.

 

The powerful Sidekick Manifesto beautifully proposes this new core belief which we can each espouse, that “Local leaders with local solutions to local problems” will end poverty. We will not.” We can, however, “always be listening, learning and seeking a deeper understanding…” I am delighted to be a sidekick in projects that prioritize participant and partner views, for that is how they end poverty.

 

What do you want to prioritize and create? What is so vital for you that you must work on it?  What has been neglected?  What difference do you want to make?  GO!

What happens after the project ends?  Lessons about Funding, Assumptions and Fears (Part 3)

 

What happens after the project ends?
Lessons about Funding, Assumptions and Fears (Part 3)

 

In part 1 and part 2 of this blog, we showcased 11 of the 18 organizations that have done post-project evaluations.  While this was scratching the surface of all that is to be learned, we shared a few insights on How we do it Matters, Expect Unexpected Results and Country-national Ownership. We gained some champions in this process of sharing our findings, including Professor Zenda Ofir of South Africa, who said “we cannot claim to have had success in development interventions if the outcomes and/or impacts are not durable, or at least have a chance to sustain or endure.”

 

In this third blog of Lessons Learned from What Happens After the Project Ends, we turn to some of the curious factors that hold us back from undertaking more post project evaluations: Funding, Assumptions, and Fears.

 

Funding

  • Why haven’t we gone back? For the last 2+years Valuing Voices has been researching the issue, we have heard from colleagues: ‘we would love to evaluate post-project but we don’t have any money, ‘donors don’t fund this’, ‘it is too expensive’[*].  Funding currently from bilateral donors such as USAID is given in 1-5 year tranches with fixed terms for completion of results and learning from them and one-year close-out processes [1]. Much of the canon of evaluations conducted after close out that we amassed was from international NGOs that had used their private funds to evaluate large donor-funded projects for their own learning.  Many aimed also to show leadership in sustainability and admittedly dazzle their funders – join them!.
  • We fund capacity building during projects but if we do not return to evaluate how well we have supported our partners and communities to translate this to sustainability, then we fall short. Meetings convened by INTRAC on civil society sustainability are opening new doors for joint learning about factors such as “legitimacy… leadership, purpose, values, and structures” within organizations well beyond any project’s end [2]. The OECD’s DAC criteria for evaluating development assistance define sustainability as: “concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable“ [3]. We need to extend our view beyond typical criterion for sustainability being a focus primarily on continued funding.
  • We need funding to explore whether certain sectors lend themselves to sustainability. In addition to the cases in blog 1, a study by CARE/ OXFAM/ PACT on Cambodian Savings groups finds that we have some revisions to make on how we design and implement with communities to foster sustainability in this sector which typically promises greater sustainability because capital can be recycled [4]. Valuing Voices blogs show indications that once we amass a greater range of post-project evaluations (funders unite), the insights gleaned can illuminate cost efficient paths to more sustained programming, possibly leading to revisions in programming or interventions which have greater likelihood for country-ownership
  • Extend the program cycle to include post-project sustainability evaluation. Rare are donors such as the Australian government (forthcoming) and USAID’s Food For Peace that commission such studies. Rare is the initiative such as 3ie that has research funds allocated by major donors to explore an aspect of impact. We miss out on key opportunities to learn from the past for improved project design if we do not return to learn how sustained our outcomes and impacts have been. We miss learning how we could better implement so more national partners could take on the task of sustaining the changes we catalyzed.
  • We call on donors to fund a research initiative to comprehensively review sustainability evaluations.
  • We call on governments to ask for this in their discussion with donors. 
  • We call on implementers to invest in such learning to improve the quality of implementation today and sustained impact in the years to come.

 

 

Assumptions

Development assistance makes many assumptions about what happens after projects end in terms of people’s self-sufficiency,   partners’ capacity to continue to support activities, and projects’ financial independence and people’s ability to step into the shoes of donors and carry on.   Unless we take a hard look at our assumptions, we will not move from proving what we expect to learning what is actually there.

 

evaluation_-_Hledat_Googlem

 

Among them are these six assumptions:

  • All will be well once we exit; we have implemented so well that of course national participants and partners will be ready and able to carry on without us. We may assume the only important outcomes and impacts are within our Logical Frameworks and Theories of Change. Thus there is no need to return to explore unexpected negative ones, or ways in which the people we strengthened may have innovated in unexpectedly wonderful ways. Aysel Vazirova, a fellow international consultant wrote me: “Post-project evaluations provide data for a deeper analysis of sustainability and help to appreciate numerous avenues taken by the beneficiaries in incorporating development projects into their lives. The theory of change narratives presented by a majority of development programs and projects have a rather disturbing resemblance to the structure of magic tales: (from) Lack – (to) change – (to) happy ending. Post project evaluations have a power to change a rigid structure of this narrative.”
  • We assume evaluations are often used to inform new designs, yet dozens of colleagues have lamented that too often this does not happen in the race to new project design. But there is hope. World Wildlife Fund/UK M&E expert Clare Crawford says when following its new management standards, WWF “expects to see the recommendations of an evaluation before the next phase of design can happen (hence evaluations happen a little before the end of a strategic period).  WWF-UK, when reading new program plans is mandated to verify if – and how – the recommendations of the last evaluation(s) were made use of in the new design phase.  Equally we track management responses to evaluations to see how learning has been applied in current or in future work.” Such a link across the program cycle is not common in our experience and none of the post-project sustained impact evaluations we reviewed said how learning would be used.
  • We may assume data continues accessible from the projects we have evaluated, yet our team member Siobhan Green has found that until recently, with the move toward open data, often project data remains the province of the donors and implementers and to the best of our knowledge leaves the country when projects close. While some sectoral data such as health and education data remains local, we are finding in fieldwork that household level data has been rolled up or discarded once projects close, which makes interviews difficult.
  • We may assume that the participants and partners are not able to evaluate  projects, particularly after the fact.  Being vulnerable does not mean that people are not able to share insights or assess how projects helped or not. Methods such as empowerment evaluation and evaluative thinking are powerful supports [5] [6].
  • Some may assume that the situation has changed in the intervening years, that there is no benefit in returning to see what results remain. Change is inevitable and sometimes more rapid or dramatic than others.  But does that mean we shouldn’t want to understand what happened? This is the greatest disservice of all, for we are selling “sustainable development” so how well have we designed it to be so?
  • We assume that learning for our own benefit is enough. A potential client brought me in to discuss my working on a rare post-project evaluation last year. It was to cost hundreds of thousands of dollars and while would occur in several countries. What I discovered was that while the donor really wanted to learn what results remained more than a decade on, I asked ‘how would the countries themselves benefit from this research and findings?’  There was a long silence. Turns out, nothing from the research would benefit or even remain in country. No one had considered the learning needs of the countries themselves. This simply cannot continue if we are to be accountable to those we serve.

 

Fears

This may be the greatest barrier of all to returning to assess sustainability.

  • We assume our projects continue. We may be afraid to look for what will this tell us about the sustainability of our efforts to save lives and livelihoods so we only choose to publicly study what is successful. Valuing Voices has found that across most the post project studies there is some ‘selection bias’, as we repeatedly learned in our research from colleagues that organizations choose to evaluate projects that are most likely to be successfully sustained.  For instance, USAID Food For Peace’ study notes, “The countries included in this study—Bolivia, Honduras, India, and Kenya—were also chosen because of their attention to sustainability and exit.” Yet as an Appreciative Inquiry practitioner, I would argue that learning what worked best to know what to do more of may be the best way forward.
  • All too often the choice of evaluation design, and sensitivity to findings fly in the face of learning—particularly when findings are negative.  This raises fears around a discontinuation of funding (an implementer fear; a beneficiary fear; could also be a recipient government’s fear). Yet as Bill Gates says, “your most unhappy customers are your greatest source of learning.”>
  • Participants asked during the project cycle about interventions may be fearful of truth telling because of perceived vulnerabilities around promised future resources, local power imbalances in control over resources, or even political imperatives to adopt a particular position. Alternatively we may not believe them, thinking they may not tell us the truth were that to stop resources.

 

Those are ours.

  • Peter Kimeu, my wise advisor and 20-year friend and colleague from Kenya tells us some fears of the that we need to listen to – those that haunt our national partners and participants.

 

They are afraid we do not see their real desires:

  • “It is ‘not how many have you (the NGO) fed, but how many of us have the capability to feed ourselves and our community?’
  • ‘How can we (country national) support our fellow citizens to take our lives and livelihoods into our own hands and excel, sustainably?’
  • What is sustainability if it isn’t expanded opportunities, Isn’t the capability of one to make a choice of value/quality life out of the many choices that the opportunities present?”

 

Will you help us address these challenges? Will you join us in advocating filling the gap in the program cycle, and looking beyond it to how we design and implement with country nationals? Will you, in your own work foster their ownership throughout and beyond? We need to fund learning from sustained impact to transparently discuss assumptions and face our fears.  This is a sustained purpose we need to and can fill.

 

 

Sources:

[1] Capable Partners Program & FHI 36. (2010). Essential NGO Guide to Managing Your USAID Award: Chapter 6 – Close Out. Retrieved from https://www.ngoconnect.net/sites/default/files/resources/Essential%20NGO%20Guide%20-%20Chapter%206%20-%20Close%20Out.pdf

[2] Hayman, R. (2014, November 5). Civil society sustainability: Stepping up to the challenge. Retrieved from https://www.intrac.org/civil-society-sustainability-stepping-challenge/

[3] OECD. (n.d.). DAC Criteria for Evaluating Development Assistance. Retrieved 2015, from https://web.archive.org/web/20151206171605/http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm

[4] Emerging Markets Consulting. (2013). Sustainability of Savings Group Programs in Cambodia for CARE, Oxfam, and Pact. Retrieved from https://mangotree.org/Resource/Sustainability-of-Savings-Group-Programs-in-Cambodia-for-CARE-Oxfam-and-Pact

[5] Better Evaluation. (n.d.). Empowerment Evaluation. Retrieved from https://www.betterevaluation.org/plan/approach/empowerment_evaluation

[6] Griñó, L., Levine, C., Porter, S., & Roberts, G. (Eds.). (2016). Embracing Evaluative Thinking for Better Outcomes: Four NGO Case Studies. Retrieved from https://www.theclearinitiative.org/resources/embracing-evaluative-thinking-for-better-outcomes-four-ngo-case-studies

 


[*] It does not have to be. We have done these evaluations for under $170,000, all-inclusive.

 

Sustained Impact post-project (ex-post)? Little proof at 3ie

 

Sustained Impact post-project (ex-post)? Little proof at 3ie

 

Find_Evidence-3ie_International_Initiative_for_Impact_Evaluation___Evaluating_Impact__Informing_Policy__Improving_Lives

 

Surely an organization that has received tens of millions of dollars of funding must track ‘impact’ as the actual long-term impact of projects. That is what I thought when I first began exploring post-project sustainability. If you look at the International Initiative on Impact Evaluation (3ie), so respected in RCTs (randomized control trials), they do valuable work comparing international development interventions among participants versus control groups during implementation, showing which interventions work best. Valuable stuff. Yet in an earlier post on impact vs. sustainability, I looked at 3ie, and found that much of the impact seems to be projected, during implementation. Our projects’ ‘logical frameworks’ are designed to measure progress to get to outputs (e.g. farmers trained), outcomes (e.g. hectares planted with new seeds), impacts (e.g. higher harvests and lower malnutrition) and goals (e.g. hunger sustainably eliminated). So did we get to sustained impact and goals? Who is looking if we got there?

 

This kind of impact appears in OECD Post-2015: “how cost-effective is it? To what extent does aid help nurture solutions that are sustainable over time? In other words, is aid being delivered effectively and is it having an impact“ [1]? Post-project evaluations that would assess sustained impact don’t seem to widely be done or get much traction at the otherwise respected 3ie either. Returning a year later to 3ie since my first post, I found 18 documents under searches of ‘ex-post’ and ‘post project’.  Sadly, again and again, their definition of ex-post seems to be post the onset of implementation, ranging from 1-2 years after the baseline rather than the typical definition of 2-10 years after projects actually closed, where long-term impact can be found.

 

When I searched by the more technical term ‘ex-post’ on 3ie, I found 17 that were titled ex-post but in fact were not. This ex-post evaluation of a conditional transfer in Costa Rica in fact was not one as they were comparing the efficacy of the project on school attendees in 2002 while the project was still underway. The French study of Cambodian healthcare was done firmly within implementation yet was titled ex-post. Another, evaluating a women’s group project in Kenya says they have a post-project evaluation but how can it be post project if it was only one and a half years after he baseline while still during implementation? A third “ex-post” on Job Youth training in the Dominican Republic only looks at the efficacy one year after the training, hence again during implementation and firmly not ex-post.

 

The only true ex-post was a 15-yea.r-old study by the World Bank in Nicaragua, comparing the impact of an emergency social investment fund for primary education, rural health and water/sanitation projects that were done from January 1994 to June 1997.  It was evaluated two years after completion regarding targeting the poorest, community priorities and participation, projects’ utilization rates, and operational and physical sustainability and impact on beneficiaries’ health and education status.  The rest of the documents on 3ie under ex-post were secondary systemic reviews of other project evaluations and a hopeful post that a new UK collaboration will track and improve aid impact [2]. While DIFD is doing terrific work on value for money, it focuses on donor money well spent rather than country-national participants and partners’ views of how well they felt our project money served them. But I quibble.

 

So is it a matter only of definition of what impact we’re looking for and expectations that we can find it during implementation rather than the dusty ankles post-project/ ex-post research Valuing Voices promotes?  As a PhD I firmly expected to find many definitions and examples of real, sustained research on actual long-term impact and the 3ie site and on many donors’ sites.  I am flabbergasted today at not only the lack of post-project/ ex-post evaluations but also that these terms seem not to be defined. There are no definitions or examples on Betterevaluation, nowhere on the American Evaluation site or within USAID’s Evaluation Policy [3], none at all in UK’s DFID’s annals on overseas aid effectiveness. Other Valuing Voices blogs have outlined the dearth of post-project evaluations. Only today did I realize even their definitions are scarce. One definition I did find was from the UK’s Department of Education:

 

The purpose of this post-project evaluation (PPE) is to:

  • Evaluate the effectiveness of the project in realizing the proposed benefits [4]
  • Compare planned costs and benefits with actual costs and benefits to allow an assessment of the project’s overall value for money to be made [4]
  • Identify particular aspects of the project which have affected benefits either positively or negatively; [and make] recommendations for future projects [4]
  • Reveal opportunities for increasing the project’s yield of benefits, whether they were planned or became apparent during or after implementation, and to recommend the actions required to achieve their maximisation [4]

 

Sustainability is the hoped-for outcome of ex-post evaluation work; so I have long turned to the OECD/ DAC have five related criteria for evaluating development assistance [5]. They define sustainability as concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable. When evaluating the sustainability of a programme or a project, it is useful to consider the following questions:

  • To what extent did the benefits of a programme or project continue after donor funding ceased?
  • What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project?

 

While this definition is 15 years old, it works quite well. Are you equally surprised at how hard it is to find definitions and examples of post-project and ex-post evaluations? 3ie doesn’t seem to have it down; USAID has hundreds of documents listed as ex-post. Some are secondary analyses of previous evaluations and most simply say post-project evaluation should be done. At least USAID has now two new ex-post studies (one in 2014, one in 2015). They are the first studies done in 30 years by USAID.  The latter is a brave and fascinating one, where Tufts evaluates Food For Peace projects ex-post in four countries this month, with publicly mixed results.  Even J-Pal, the highly-respected Abdul Lateef Jameel Poverty Action Lab does rigorous impact evaluations, comparing the outcomes of project activities to non-participant control groups to ascertain the effectiveness of the interventions, but it too seems to have no documents on their impact evaluation site that show the sustainability of impacts.

Putting ourselves in the shoes of those we indend to serve… the sustained impact is what they actually want.

When will our organizations actually go back to the field and learn from those living with the projects years after donors and implementers leave? When will we walk our accountability talk? DO TELL!

 

 

Sources:

[1] OECD. (2013). Element 10, Paper 1: Effective development co-operation: An important enabler in a post-2015 global development framework. Retrieved from http://www.oecd.org/dac/FINAL%20POST-2015%20Effective%20Development%20Co-operation.pdf

[2] White, H. (2011, May 23). New UK watchdog to improve aid impact. Retrieved from https://www.3ieimpact.org/blogs/new-uk-watchdog-improve-aid-impact

[3] Levine, R., & DeRuiter, D. (Eds.). (2015, September 22). USAID Evaluation Policy. Retrieved from https://www.usaid.gov/documents/1865/usaid-evaluation-policy

[4] UK Department of Education. (n.d.). Post Project Evaluation. Retrieved from https://www.education-ni.gov.uk/articles/post-project-evaluation

[5] OECD. (n.d.). DAC Criteria for Evaluating Development Assistance. Retrieved 2015, from https://web.archive.org/web/20151206171605/http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm