Sustained Impact post-project (ex-post)? Little proof at 3ie
Surely an organization that has received tens of millions of dollars of funding must track 'impact' as the actual long-term impact of projects. That is what I thought when I first began exploring post-project sustainability. If you look at the International Initiative on Impact Evaluation (3ie), so respected in RCTs (randomized control trials), they do valuable work comparing international development interventions among participants versus control groups during implementation, showing which interventions work best. Valuable stuff. Yet in an earlier post on impact vs. sustainability, I looked at 3ie, and found that much of the impact seems to be projected, during implementation. Our projects' 'logical frameworks' are designed to measure progress to get to outputs (e.g. farmers trained), outcomes (e.g. hectares planted with new seeds), impacts (e.g. higher harvests and lower malnutrition) and goals (e.g. hunger sustainably eliminated). So did we get to sustained impact and goals? Who is looking if we got there?
This kind of impact appears in OECD Post-2015: “how cost-effective is it? To what extent does aid help nurture solutions that are sustainable over time? In other words, is aid being delivered effectively and is it having an impact?“ Post-project evaluations that would assess sustained impact don't seem to widely be done or get much traction at the otherwise respected 3ie either. Returning a year later to 3ie since my first post, I found 18 documents under searches of ‘ex-post’ and ‘post project’. Sadly, again and again, their definition of ex-post seems to be post the onset of implementation, ranging from 1-2 years after the baseline rather than the typical definition of 2-10 years after projects actually closed, where long-term impact can be found.
When I searched by the more technical term 'ex-post' on 3ie, I found 17 that were titled ex-post but in fact were not. This ex-post evaluation of a conditional transfer in Costa Rica in fact was not one as they were comparing the efficacy of the project on school attendees in 2002 while the project was still underway. The French study of Cambodian healthcare was done firmly within implementation yet was titled ex-post. Another, evaluating a women’s group project in Kenya says they have a post-project evaluation but how can it be post project if it was only one and a half years after he baseline while still during implementation? A third “ex-post” on Job Youth training in the Dominican Republic only looks at the efficacy one year after the training, hence again during implementation and firmly not ex-post.
The only true ex-post was a 15-yea.r-old study by the World Bank in Nicaragua, comparing the impact of an emergency social investment fund for primary education, rural health and water/sanitation projects that were done from January 1994 to June 1997. It was evaluated two years after completion regarding targeting the poorest, community priorities and participation, projects' utilization rates, and operational and physical sustainability and impact on beneficiaries' health and education status. The rest of the documents on 3ie under ex-post were secondary systemic reviews of other project evaluations and a hopeful post that a new UK collaboration will track and improve aid impact. While DIFD is doing terrific work on value for money, it focuses on donor money well spent rather than country-national participants and partners’ views of how well they felt our project money served them. But I quibble.
So is it a matter only of definition of what impact we’re looking for and expectations that we can find it during implementation rather than the dusty ankles post-project/ ex-post research Valuing Voices promotes? As a PhD I firmly expected to find many definitions and examples of real, sustained research on actual long-term impact and the 3ie site and on many donors’ sites. I am flabbergasted today at not only the lack of post-project/ ex-post evaluations but also that these terms seem not to be defined. There are no definitions or examples on Betterevaluation, nowhere on the American Evaluation site or within USAID's Evaluation Policy, none at all in UK's DFID's annals on overseas aid effectiveness. Other Valuing Voices blogs have outlined the dearth of post-project evaluations. Only today did I realize even their definitions are scarce. One definition I did find was from the UK's Department of Education:
The purpose of this post-project evaluation (PPE) is to:
- Evaluate the effectiveness of the project in realizing the proposed benefits
- Compare planned costs and benefits with actual costs and benefits to allow an assessment of the project's overall value for money to be made
- Identify particular aspects of the project which have affected benefits either positively or negatively; [and make] recommendations for future projects
- Reveal opportunities for increasing the project's yield of benefits, whether they were planned or became apparent during or after implementation, and to recommend the actions required to achieve their maximisation
Sustainability is the hoped-for outcome of ex-post evaluation work; so I have long turned to the OECD/ DAC have five related criteria for evaluating development assistance. They define sustainability as concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable. When evaluating the sustainability of a programme or a project, it is useful to consider the following questions:
- To what extent did the benefits of a programme or project continue after donor funding ceased?
- What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project?
While this definition is 15 years old, it works quite well. Are you equally surprised at how hard it is to find definitions and examples of post-project and ex-post evaluations? 3ie doesn’t seem to have it down; USAID has hundreds of documents listed as ex-post. Some are secondary analyses of previous evaluations and most simply say post-project evaluation should be done. At least USAID has now two new ex-post studies (one in 2014, one in 2015). They are the first studies done in 30 years by USAID. The latter is a brave and fascinating one, where Tufts evaluates Food For Peace projects ex-post in four countries this month, with publicly mixed results. Even J-Pal, the highly-respected Abdul Lateef Jameel Poverty Action Lab does rigorous impact evaluations, comparing the outcomes of project activities to non-participant control groups to ascertain the effectiveness of the interventions, but it too seems to have no documents on their impact evaluation site that show the sustainability of impacts.
Putting ourselves in the shoes of those we indend to serve… the sustained impact is what they actually want.
When will our organizations actually go back to the field and learn from those living with the projects years after donors and implementers leave? When will we walk our accountability talk? DO TELL!