How ‘new’ are our projects… and who is aiming at the right outcomes?

» Posted by on Apr 6, 2017 in Aid effectiveness, Better Evaluation, Evaluation, ex-post evaluation, Faster Forward Fund, International aid, M&E, methodologies, outcomes, post-project evaluation, SEIE, Sustainable development, Sustained and Emerging Impacts Evaluation | 4 comments

How 'new' are our projects… and who is aiming at the right outcomes?

Valuing Voices exciting news is we have received research grant funding from the esteemed evaluator, Michael Scriven's Faster Forward Fund. We're looking into the value-added of (ex-post) Sustained and Emerging Impacts Evaluations (SEIEs) and we are doing the research now. We will be documenting methods used and discuss how best to evaluate such sustained impacts after project close-out. Very exciting stuff in this staggeringly 'new' field of evaluation.

During this research, a senior international development evaluation expert told us that they can't return to evaluate now-closed projects because they aren't the same projects anymore (after closeout) and we are no longer responsible for the results. That took my breath away.

All new projects come from old projects… we recycle old project design most of the time, occasionally making substantive changes in targeting or design but much of how we design and implement remains the same. And while we thoroughly evaluate them during implementation, learning ex-post is a key missing link which all projects in the future can benefit from as we do similar interventions and track similar outcomes year after year but we rarely know which ones were sustained or emerged anew. There absolutely are aspects that get adapted but there are only so many ways to heal the sick, improve crop growth, save money, learn to read and so on, and there is a world we need to learn about what enabled some to be sustained and even morph into new results!

This excellent article, Do NGOs (non-governmental organization) help?, notes that "due to donor pressure [NGOs] are increasingly forced to respond with a discrete project with x number of deliverable outcomes."  It goes on to cite D. Sriskandarajah, the secretary-general of Civicus, a global network of civil society organizations and activists, wrote: ‘We have become a part of the problem rather than the solution… Since demonstrating bang for your buck has become all-important, we divide our work into neat projects, taking on only those endeavours that can produce easily quantifiable outcomes. Reliant on funding to service our own sizeable organizations, we avoid approaches or issues that might threaten our brand or upset our donors. We trade in incremental change.’"

We also settle for results while we control them, and don't ask unpopular questions about who is to sustain these results, with what resources, and for goodness sake, why sustained impact was not funded, designed, implemented and monitored/ evaluated from the very onset in our rush to measurable results?

DontSettleForLess

As this great NGO article by Dinyar Godrej goes on to say, "most media scrutiny of NGO accountability is of how they use funds, their accountability to donors. But what of their accountability towards the recipients of their interventions?" They have no lobbyists to persuade our funders they would like this but nto that, and often such lobbying for their needs falls to the very NGOs that have won these large contracts and tasked with implementing a dizzying array of mandatory input, output, outcome and some impact indicators. We do care deeply about results!  US State Department/ USAID has a "Standard Foreign Assistance Master Indicator List" of 2,300 lines in an excel spreadsheet. (There are more indicators still– custom and cross-cutting indicators, the mind boggles).

Wow. But are we asking the right questions? Are we asking what was sustained after all this hard work was done and ended? Rarely. Who should be?! "It is perhaps unrealistic to expect such large structural changes to be delivered by NGOs when governments don’t tackle them either."

For the rub is this. When we take development over from national governments, largely do not involve country nationals in the funding, design and M&E of projects, then how sustained can these projects still be after we go? Millions are invested, then disappear… Last year, at local debrief at the end of one SEIE Valuing Voices did, the state of affairs became crystal clear when a government official asked us "Can you ever find some funds to fund us to do our own independent evaluations? Even if it is not the projects that they did themselves? We would be happy to get that support…”

When are we no longer responsible for doing great, sustained work? Valuing Voices will let you know what we found regarding the best ways to do SEIEs more. Stay tuned.

What do you think?

P.S. This blog topic prompted me to look for statistics on the number or percentage of funded projects that were renewed. Nothing.  Does anyone know how many or what % of projects were extended/ funded again after showing good results? (Often this happens in the form that a successful project in one area of the country gets either funded again or repeated elsewhere in the country or in the world, as have two of our own SEIEs, Niger and Ethiopia). For that matter, what made them so excellent to be replicated? What can we learn? 

29 years of listening to participants in Africa, Latin America, the Balkans, Europe and the US. I Value their Voices. Let’s have sustained impact!

Twitter 

4 Comments

  1. Congrtulations on the research grant. Fascinating comments on the donor stress on evaluations before or at project end as opposed to sustained and emerging impacts evaluations. As a co-founder of a small NGO (and frequent donor driven project evaluator), I notice that in fact, results may utlimately go in a different but still positive direction. For example, we had a water project for slum dwellers. It was well functioning at project end but a year later many of the taps were no longer functioning despite the participation of the dwellers in the entire proces. People they selected from among themselves were also trained on maintenance. Interestingly, however, about 6 months later we found that they had used their skills to lay simple pipes from a few of the water points directly to the dwellings. So they did not need all those individual water points and focused on maintaining a few of them and the pipes. In other wrds, we have to be aware that results may utlimately go in a totally unexpected direction which donors may not even like. 

    • Mei – thank you for your great comment… YES! That is precisely why we do such SEIEs (especially the Emerging bit) – as "we have to be aware that results may utlimately go in a totally unexpected direction".  Tell us more about why you went back, how you did it (we are researching methodologies :)) and I'm so intrigued by how you used this information that (unexpectedly) "they had used their skills to lay simple pipes from a few of the water points directly to the dwellings…[they] focused on maintaining a few of them and the pipes." What did you do differently next time?

      Warmly! Jindra

      • This is my own NGO that I co-founded with my sister in India. The Dutch NGO SIMAVI provided the funding for the water project. I keep in touch and have visited back regularly that is why I know. But international NGOs have told me similar stories when I come back to do end-of-project evaluations. Some mention their frustrations about such evaluations as they give me examples of how things ultimately went in a different direction but also with positive benefits. The point is that sometimes the results and sustainability potenial look good at project end but 6 months later things turned out to be less positive. Then later down the line there are good but different results after all. Maybe it is worthwhile to do the admitedly less scientific approach of going back now to agencies and ask them to relate ex-post facto incidents where this occured. I know of several instances where, for example, child labor monitoring committees stopped functioning after the project but child labor decreased in the community anyway. It seems that this is due, at least in part, to increased awareness that has sunk in and child "beneficaries" themselves becoming older and spreading the message. So the situation is obviously complex. A difficult research sitaution.

  2. Mei, there are untold riches of learning out there about sustained and emerging (!) impact, as you say, even if anecdotal stories. Yet one of the things we are trying to do with the FFF grant is bring a rigor to the study of what our participants and partners really did afterwards and learn how our assistance has changed their lives for the long term and what we should do differently. And as you say, there can be SO MUCH GOOD we find when we look; it just may not what our logical-frameworks expected it would be :). Thank you!

Submit a Comment

Your email address will not be published. Required fields are marked *

*

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Translate »