Sustained Impact post-project (ex-post)? Little proof at 3ie

» Posted by on Dec 15, 2015 in 3ie, Aid effectiveness, DFID, Evaluation, ex-post evaluation, post-project evaluation, Sustainability, Sustainable development, USAID | 14 comments

Sustained Impact post-project (ex-post)? Little proof at 3ie

Find_Evidence-3ie_International_Initiative_for_Impact_Evaluation___Evaluating_Impact__Informing_Policy__Improving_Lives

Surely an organization that has received tens of millions of dollars of funding must track 'impact' as the actual long-term impact of projects. That is what I thought when I first began exploring post-project sustainability. If you look at the International Initiative on Impact Evaluation (3ie), so respected in RCTs (randomized control trials), they do valuable work comparing international development interventions among participants versus control groups during implementation, showing which interventions work best. Valuable stuff. Yet in an earlier post on impact vs. sustainability, I looked at 3ie, and found that much of the impact seems to be projected, during implementation. Our projects' 'logical frameworks' are designed to measure progress to get to outputs (e.g. farmers trained), outcomes (e.g. hectares planted with new seeds), impacts (e.g. higher harvests and lower malnutrition) and goals (e.g. hunger sustainably eliminated). So did we get to sustained impact and goals? Who is looking if we got there?

 

This kind of impact appears in OECD Post-2015: “how cost-effective is it? To what extent does aid help nurture solutions that are sustainable over time? In other words, is aid being delivered effectively and is it having an impact?“ Post-project evaluations that would assess sustained impact don't seem to widely be done or get much traction at the otherwise respected 3ie either. Returning a year later to 3ie since my first post, I found 18 documents under searches of ‘ex-post’ and ‘post project’.  Sadly, again and again, their definition of ex-post seems to be post the onset of implementation, ranging from 1-2 years after the baseline rather than the typical definition of 2-10 years after projects actually closed, where long-term impact can be found.

 

When I searched by the more technical term 'ex-post' on 3ie, I found 17 that were titled ex-post but in fact were not. This ex-post evaluation of a conditional transfer in Costa Rica in fact was not one as they were comparing the efficacy of the project on school attendees in 2002 while the project was still underway. The French study of Cambodian healthcare was done firmly within implementation yet was titled ex-post. Another, evaluating a women’s group project in Kenya says they have a post-project evaluation but how can it be post project if it was only one and a half years after he baseline while still during implementation? A third “ex-post” on Job Youth training in the Dominican Republic only looks at the efficacy one year after the training, hence again during implementation and firmly not ex-post. 

 

The only true ex-post was a 15-yea.r-old study by the World Bank in Nicaragua, comparing the impact of an emergency social investment fund for primary education, rural health and water/sanitation projects that were done from January 1994 to June 1997.  It was evaluated two years after completion regarding targeting the poorest, community priorities and participation, projects' utilization rates, and operational and physical sustainability and impact on beneficiaries' health and education status.  The rest of the documents on 3ie under ex-post were secondary systemic reviews of other project evaluations and a hopeful post that a new UK collaboration will track and improve aid impact. While DIFD is doing terrific work on value for money, it focuses on donor money well spent rather than country-national participants and partners’ views of how well they felt our project money served them. But I quibble.

 

So is it a matter only of definition of what impact we’re looking for and expectations that we can find it during implementation rather than the dusty ankles post-project/ ex-post research Valuing Voices promotes?  As a PhD I firmly expected to find many definitions and examples of real, sustained research on actual long-term impact and the 3ie site and on many donors’ sites.  I am flabbergasted today at not only the lack of post-project/ ex-post evaluations but also that these terms seem not to be defined. There are no definitions or examples on Betterevaluation, nowhere on the American Evaluation site or within USAID's Evaluation Policy, none at all in UK's DFID's annals on overseas aid effectiveness. Other Valuing Voices blogs have outlined the dearth of post-project evaluations. Only today did I realize even their definitions are scarce. One definition I did find was from the UK's Department of Education:

 

The purpose of this post-project evaluation (PPE) is to:

  • Evaluate the effectiveness of the project in realizing the proposed benefits
  • Compare planned costs and benefits with actual costs and benefits to allow an assessment of the project's overall value for money to be made
  • Identify particular aspects of the project which have affected benefits either positively or negatively; [and make] recommendations for future projects
  • Reveal opportunities for increasing the project's yield of benefits, whether they were planned or became apparent during or after implementation, and to recommend the actions required to achieve their maximisation

 

Sustainability is the hoped-for outcome of ex-post evaluation work; so I have long turned to the OECD/ DAC have five related criteria for evaluating development assistance. They define sustainability as concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable. When evaluating the sustainability of a programme or a project, it is useful to consider the following questions:

  • To what extent did the benefits of a programme or project continue after donor funding ceased?
  • What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project?

 

While this definition is 15 years old, it works quite well. Are you equally surprised at how hard it is to find definitions and examples of post-project and ex-post evaluations? 3ie doesn’t seem to have it down; USAID has hundreds of documents listed as ex-post. Some are secondary analyses of previous evaluations and most simply say post-project evaluation should be done. At least USAID has now two new ex-post studies (one in 2014, one in 2015). They are the first studies done in 30 years by USAID.  The latter is a brave and fascinating one, where Tufts evaluates Food For Peace projects ex-post in four countries this month, with publicly mixed results.  Even J-Pal, the highly-respected Abdul Lateef Jameel Poverty Action Lab does rigorous impact evaluations, comparing the outcomes of project activities to non-participant control groups to ascertain the effectiveness of the interventions, but it too seems to have no documents on their impact evaluation site that show the sustainability of impacts.

Putting ourselves in the shoes of those we indend to serve… the sustained impact is what they actually want.

When will our organizations actually go back to the field and learn from those living with the projects years after donors and implementers leave? When will we walk our accountability talk? DO TELL!

29 years of listening to participants in Africa, Latin America, the Balkans, Europe and the US. I Value their Voices. Let’s have sustained impact!

Twitter 

14 Comments

  1. Readers might be interested in ADB's 2010 report on the post completion  sustainability of ADB assisted projects. See:

    http://www.adb.org/documents/post-completion-sustainability-asian-development-bank-assisted-projects

    regards

    Scott Bayley

    (one of the authors)

    • Scott, thank you so much for sharing this resource. Is post-project evaluation widespread at ADB? It is so rare at USAID, MCC (US) and DIFD and NORAD which are the agencies we've looked at the most… Thank you!

    • A quick question – only read the summary that says “The objective of this special evaluation study is to assess ADB’s achievements in the sustainability of its projects and programs, and ADB’s approach to project sustainability. In a review of 491 ADB-assisted project completion reports between 2001 and 2009, 65% were rated most likely or likely to be sustainable; implying a “substantial task” after completion of enhancing the sustainability of the remaining 35% of the reviewed projects.” Amazing to review 491 projects — was this a secondary review of the evaluation documents or did the Asian Development Staff go out there and talk to participants to find out how it actually turned out in the 65%? 

  2. In my view there is a big definitional difference of the word impact as used by you and the organizations that conduct RCTs, so you are comparing very different things.  While the word impact is used by some to mean after a specific time period, in RCTs it carries a technical statistical definition and is based on a statistical calculation using very specific regression models with data collected  any point in time AFTER project implementation.  In simplistic terms this calculation is defined as the difference in outcomes between program and control/comparison group.  Usually theory drives when an RCT impact may be readily observed, post-intervention. This can be immediate, short-term, long-term and also often described in the project’s Theory of Change.  Statistical modelling can also be used to assess the likelihood of sustainability of the project, but what usually happens is if the project has an impact and is cost-effective then it is scaled up.  It is difficult to measure long-term RCT impacts if the control/comparison group begins receiving similar or the same services that was received by the intervention group.  There are RCT projects in Early Childhood Education, employment, etc. in the US and UK that have traced and measured impacts many, many years later (e.g., 25 years post-project implementation).  Since the use of RCTs is relatively new in a development context it is not surprising that this may not yet have taken place.  However, I would not be surprised to learn it is being done, but not yet published. 

    • Claudia – this is very helpful… It seems that much of what I looked at in 3ie seemed to be looking at projected impact during implementation not any point after… maybe there is much there that does as you say look at statistical impact after which an RCT expert would pick up! Could you share with me a couple of resources in our int’l dev field of “RCT projects in Early Childhood Education, employment, etc. in the US and UK that have traced and measured impacts many, many years later (e.g., 25 years post-project implementation).” Thanks again!

      • "Impact during implementation" does not make sense, since by definition RCT impact is post intervention. Many RCTs use process/implementation research to describe the project, establish fidelity to the model, and help explain the results, but impact can only be measured post-intervention.  The USA and UK projects I mentioned are not development projects. They are with vulnerable populations in these developed countries.  One example is the Perry PreSchool project in the USA.  The last published result I saw was 40 years post-intervention. The use of RCT in development is relatively recent.  There is a Mexican RCT cash-transfer project that has looked at long-term impacts.  If memory serves me right it was called PROGRESSO.

  3. Claudia, thank you again for your answer:

    I completely agree the term impact implies long-term. The term ex-post implies after the project ends. In an illustrative IADB evaluation listed on 3ie as an ex-post, the impact seemed to be projected 

    http://www.3ieimpact.org/en/evidence/impact-evaluations/details/393/

    "To rigorously assess the impact of the job training programme, participants were randomly assigned to job training or control groups. The individual organizations implementing the training collected baseline data, and an independent organization collected follow-up data 10–14 months after the completion of training on a subsample of 786 treatment and 563 control group members, using a stratified sampling method based on age, gender and educational level. Reported impacts are based on the difference in a given indicator between the treatment and control groups."

    J-PAL does the same here where the credit seems ongoing: https://www.povertyactionlab.org/evaluation/impact-microcredit-women-mexico

    "Baseline and endline data is from socioeconomic surveys administered to women 18-60 who had or were likely to start a business in both treatment and comparison areas. Researchers conducted follow-up surveys between 2011 and 2012, an average of 26 months after Compartamos entered the treatment areas."

    Claudia, could you mean this on Mexican RCT cash transfers? "https://www.rbfhealth.org/resource/mexico%E2%80%99s-model-conditional-cash-transfer-cct-program-fighting-poverty"

    Thanks again for discussing this with me, helping us all think more clearly….

  4. The link for the Mexico program did not work.  

     I quickly scanned the other two links and both are point-in-time impact estimates (i.e., statistical calculations) done after the "treatment"/ intervention was offered, using real data collected from program and control group members through surveys, and are not projections.  So for the 3ie project it was the impact (i.e., differences between the program and control groups) 10-14 months after the "treatment"/ intervention was offered. There can be very different results with data collected at 24 months or 60 months. Remember impact for RCTs refers to a statistical difference calculation, not the specific length of time since the program ended.

     

     

    • Claudia- so your ‘impact estimates’ is akin to what I’ve been calling ‘projected impact’ (snapshots taken during implementation). It is not going out after things closed out and folks were on their own sustaining outcomes/ impacts. Yes, the 25 to 60 months after closeout is more or what expected ‘impact’ to be!  Many thanks for the clarification on how they’re defining impact vs timeframe. FYI the ‘impact’ I am talking about is the Theory of Change ‘impact’. See how Impact is described here; it’s what we have on most Logical Frameworks: http://whitecenterpromise.org/about/theory-of-change/

      Pelican discussions are great…

      Here is the link again without the strange endquote – it was what I found

      https://www.rbfhealth.org/resource/mexico%E2%80%99s-model-conditional-cash-transfer-cct-program-fighting-poverty

      Be well…

  5. Yes, that's the Mexican project I was referring to. It used a RCT methodology to asess the impact (i.e., difference between program and control group) of the conditional cash transfer program.

  6. Thank you Jindra for this thoughtful analysis and for colleagues' participation in this discussion. While I can appreciate the strengths and weaknesses of research organisations that are mostly "single-design", i.e experimental, and the limitations of those designs in social contexts, e.g. poor external validity, my main concern is with what I perceive to be a lack of care, I would say in come cases intellectual rigour, in the use of terminology. For example I would argue that what 3ie does for the most part is not evaluation, understood as the determination of the value, worth or merit of things (Scriven). Nor are uses of the term impact sufficiently and appropriately well defined, i.e. referenced in an explicit and cogent conceptual framework that itself is grounded in explicit and cogent theoretical and philosophical approaches, to meet what I consider to be basic standards of intellectual, research and evaluation rigour. This said, considering the use of these terms through a political economy lens applied to the inception and development of organisations such as 3ie or JPAL, gives us a more sensible understanding of the various interests and implicit ideologies at play as they jockey for influence and money, i.e. power. The question then is what of the less powerful who are the intended beneficiaries of the "treatment". 

    • Ian, I am delighted to see you on our blog…. Yes, I too felt the terminological disconnect as well as the different ideologies. Could you tell me where have you seen the less powerful subjects taking the reins in their own hands and designing projects themselves for themselves? Also in your wide experiences, where have you seen our participants and partners evaluate us, rather than us them?

      The only example I have found, really, is this one from a grassroots Rwandan project (http://www.irinnews.org//report/99696/call-for-shake-up-in-africa-nutrition-research) that decreased (stunting) malnutrition by 18% in five years which is very fast by “‘setting up an almost universal community-based health insurance scheme… with the help of [each village determining its own way to tackle malnutrition… and not packaged interventions provided by donors” said Fidele Ngabo, director of Maternal Child Health. The article says “the Rwandan model could be used in other African countries, where foreign donor-driven initiatives tend to focus on treatment and technical solutions…. Change will only come when nutrition research is led by Africa, and interventions are designed to meet a country’s priorities”

  7. Jindra, it is great that you ask how often development interventions are evaluated after completion. And I guess there aren't many such studies. As Claudia wrote, there are longer-term evaluations in many other fields like education. In international development, there are some macro-economic studies on aid (see http://www.ingentaconnect.com/content/aea/jep/2003/00000017/00000003/art00002) but follow-ups of individual interventions seem to be rare and I agree with your main point that there should be more of them – not for every project (that would be way too expensive) but there is a lot to be said for carefully planned follow-ups of selected projects and larger projects.

    Some of the heat in this argument however comes I think from disagreement about definitions.

    "Ex-post" versus "ex-ante" I think most often means "retrospective" rather than "prospective". "Ex-post" as a bunch of methods just suggests that there wasn't an adequate baseline, so some other method has to be found to look at effects. So any evaluation which looks backwards from, say, 1 year into a project would be ex-post. More rarely, people use "ex-post" in the way you seem to, meaning "after project completion". I guess people can use the term either way, but don't be surprised to find some team saying they did an ex-post evaluation before the end of a project.

    Similarly, impact. There are at least three things this term may or may not imply:

    – difference made to important variables compared with what would have happened without the intervention. 
    – including negative as well as positive, unintended as well as intended consequences. 
    – longer-term consequences, often long after an intervention has finished – this is strongly related to the idea of sustainability.

    I think we can all agree that all of these three ideas are important things which are worth investigating but let's not get into a fight about which combination of them someone means when they use the word "impact". So if someone has published a retrospective study one year after project start and calls it an ex-post study of impact, that is fine. There is no policy dispute, just different ways of using two key terms. What we can agree on is that there should be more studies on the longer-term effects of development interventions, intended and unintended.

Trackbacks/Pingbacks

  1. What happens after the project ends?  Lessons about Funding, Assumptions and Fears (Part 3) | Valuing Voices - […] and USAID’s Food For Peace that commission such studies. Rare is the initiative such as 3ie that has research…
  2. Missing Link: Sustained and Emerging Impact Evaluation (reblog from BetterEvaluation.org) | Valuing Voices - […] views surrounding what constitutes a sustained impact in these organisations are important (read theValuing Voices blog post on the…

Submit a Comment

Your email address will not be published. Required fields are marked *

*

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Translate »