Public and Private paths to Sustained Global Development Impacts

Public and Private paths to Sustained Global

Development Impacts

(Reposted from: https://medium.com/@jindracekan/public-and-private-paths-to-sustained-global-development-impacts-9b7523891fce)

Six years. That’s how long ago I began researching proof of sustained impact(s) through its ex-post project evaluation. Until now Valuing Voices has focused on aid donors. We are expanding to the private sector.

 

In my PhD I was sure it was a lack of researched and shared proof of successful prevention of famine that led to inaction. In Valuing Voices’ research on ex-post project evaluation, I again felt “if only they knew, they would act”. I pulled together a variety of researchers and consultants who (often pro-bono, or for limited fees) researched the shockingly rare field evaluations of what was sustained after projects closed, why, and what participants and partners did themselves to sustain impacts.

 

Sustaining the outcomes and achieving impacts, are, after all, what global development projects promise. These ‘sustainable development’ results are at the top (or far-right, below) of our ‘logical frameworks’. We promise the country-level partners, our taxpayers and donors, that we will achieve them, yet…

 

 

 

 

 

 

 

 

 

 

 

 

 

 

We have done six post-project Sustained and Emerging Impacts Evaluations. We have created checklists on ex-post project evaluability thanks to a Faster Forward Fund grant by esteemed evaluator Michael Scriven. We have created preliminary guidance on Sustained and Emerging Impacts Evaluations (SEIE) and shared 25 such ex-post closure evaluations that we found returned to ask participants 2-15 years after close-out in (one of?) the only database on such evaluations in the world. We have drawn valuable lessons from the evaluations throughout nearly 60 blogs and presented at 10 conferences. We have found that results at the end of project are dynamic, that there can be greater failure – or sometimes greater success – than we would ever expect in our project assumptions. We have found that communities can create ‘emerging’ outcomes, adapting the activities to succeed onward with no further donor funding, and that when we design for long-term sustainability with our partners, then remarkable success can ensue.  So many lessons for programming that we need to learn from, including partnering with country-nationals, focusing on youth, questioning assumptions at exit, etc.

 

We have applied to many grants for support, unsuccessfully, and have applied to evaluate a handful more ex-post sustainability evaluations which other consultants have won – while we were disappointed, in equal measure we are happy others are learning to do this, as we share our resources freely to promote exactly such practices across hundreds of thousands unevaluated projects! We are currently doing an ex-post project evaluation of an agriculture value chain in Tanzania, yet there are a handful done per year. At one conference, our discussant Michael Bamberger joked we were lucky not to be found dead under a bridge for taking on such a dangerous topic. We remain undeterred, and delight in colleagues we promote such work and thanking us for ours.

 

At the same time, several things have become apparent:

 

 

 

  • Vital lessons for how aid can do better remain unexplored, and true accountability to our country-national participants and partners ends when fixed-time, fixed deliverable project resources are spent and proof of accountability for money and results that donors want are filed away. Sadly, while capacity building is done throughout implementation, knowledge management about results is abysmal as ‘our projects’ data almost always dies quietly in donor and implementer computer hard drives after close-out, rather than being accessible in-country for further learning. Go partner!

 

  • We hardly ever return after all our evaluations to share with communities which speaks to ‘partnerships’ not being with the participants, and we often ‘exit’ without giving ample time to handover so that things can be sustained, e.g. local partners found, local and other international funding harnessed, etc. Learn together!

 

  • There is a real need to fund systematized methods for such evaluations, mandate access to quality baseline, midterm and final evaluations, and mandate that all projects above a certain funding level (e.g. $1mil) include funding for such evaluation and learning 2-10 years later. Many so-called ex-post evaluations are in fact either delayed final evaluations, desk studies without any fieldwork, rather methodologically flawed comparisons or with fieldwork which doesn’t talk to the intended ‘beneficiaries’ for such pivotal ground-level feedback. Innovate by listening!

 

  • It is unclear to us how any organization that has done an ex-post sustainability evaluation has learned from it and changed their systems, although we have been told some are ‘looking for a successful project to evaluate’, and that after a failed one, they are discontinued. We know of some (I)NGOs who are putting ex-posts into their new strategies, and two INGOs who are researching exits more – good. Be brave!

 

  • Recently, we are delighted some new NGOs are dipping their feet into their first evaluations of sustainability, they do so bravely. The tension between accountability and learning is heightened at the prospect that implementers and donors have failed to create sustained impact. But why judge them when all the design and systems in place are to reward success while projects are running (and even those don’t always show much) so that they all get congratulations and more funding for very similar projects? Who knows who is focused on sustaining impacts with funding capacities, partnerships and country-led design, implementing with feedback loops and adjusting for the long-term, helping communities evaluate us rather than how well they are fulfilling our targets, etc. Sustaining impacts will win you funding!

 

  • Logically, here are many indications among ex-post sustainability evaluations that profitable, but low-risk and diversified agriculture, microenterprise/ business projects are better sustained (Niger, Ethiopia, Tanzania, Nepal, etc.). This does not mean that all projects need to be profitable, but cost-covering projects even in the health and education/ vocational training/ sectors is important as many of us know. Self-funding!

 

So rather than giving up on sustained impacts, we are adding another branch to the Valuing Voices tree.

 

My partners and I have extensively researched the need for and co-founded Impact Guild. We will work alongside NGOs and impact investors to foster:

 

  1. FUNDING: The money available from development aid donors is shrinking in volume + value, while development financing is scaling up exponentially.

 

 

 

 

 

 

 

 

 

 

 

 

The SDGs and the Paris Agreement are prompting a massive scale-up of development financing from billions to trillions of dollars into ‘sustainable development’, yet with rare Scandinavian and Foundation exceptions, donors appear to be switching from longer-term development to humanitarian aid. Further, despite decades of experience, international and national nonprofit development implementers are mostly absent in the conversation around scaling-up the flow of capital to achieve and sustain development goals. Exceptions are some in the International NGO Impact Investing Network (AMPLIFY)

2. RESULTS: Funding for projects that can show great results (e.g. Social Impact Bonds/ Development Impact Bonds, which are in fact pay-for-performance instruments), even sustained impacts from partnering with local small and medium enterprises, national level ministries, and local NGOs. Far too long, implementers have been able to get funding for projects with mediocre results; impact investors are raising the bar and even donors are helping hedge risk. This includes M&E ‘impact’ value that rigorously tracks results (savvy private-sector donors require counterfactual/ control group data, isolating results from that intervention).

3. LEARNING: Impact Investors have a lot to learn from non-profits and aid donors as well.

 

  • They talk about impact but too often that is synonymous with generic results, while International and National nonprofits (NGOs) have detailed, grassroots systems in place;
  • Most seem to be content – for now – to invest in the 17 Sustainable Development Goal areas (e.g. vetting investable projects by screening criteria of not only getting a financial return, but also by broad sectoral investments, e.g. poverty, hunger, climate etc.). Many claim they have affected change, without data to prove it. The SDGs are slowly creating indicators to address this, and investors also need to be brought along to differentiate between corporate efficiency activities for their operations and those that affect change at the output, outcome and impact levels in communities;
  • There are still large leaps of logic and claims among investors and some know that data is lacking to claim good grassroots targeting and actual results that prove they are changing hunger, poverty and other sectors in Africa, Asia, Latin Americ. Good development professionals would see that the very design would make results accessible only to the elite of that country (e.g. $1 nutrition bars are inaccessible to most of a country’s population living on income of $2.00 a day)
  • We will bring with us all we know about great potential sustained impacts programming, such as Theory of Sustainability, looking for emerging results alongside planned early onlearning from failure for success, partnering successfully for country-led development, etc.

    So keep watching these ‘spaces’: www.ValuingVoices.com and www.ImpactGuild.org for updates on bridging these worlds, hopefully for ever-greater sustained impacts. Let us know if you would like to partner!

How ‘new’ are our projects… and who is aiming at the right outcomes?

 

How ‘new’ are our projects… and who is aiming at the right outcomes?

 

Valuing Voices exciting news is we have received research grant funding from the esteemed evaluator, Michael Scriven’s Faster Forward Fund. We’re looking into the value-added of (ex-post) Sustained and Emerging Impacts Evaluations (SEIEs) and we are doing the research now. We will be documenting methods used and discuss how best to evaluate such sustained impacts after project close-out. Very exciting stuff in this staggeringly ‘new’ field of evaluation.

During this research, a senior international development evaluation expert told us that they can’t return to evaluate now-closed projects because they aren’t the same projects anymore (after closeout) and we are no longer responsible for the results. That took my breath away.

All new projects come from old projects… we recycle old project design most of the time, occasionally making substantive changes in targeting or design but much of how we design and implement remains the same. And while we thoroughly evaluate them during implementation, learning ex-post is a key missing link which all projects in the future can benefit from as we do similar interventions and track similar outcomes year after year but we rarely know which ones were sustained or emerged anew. There absolutely are aspects that get adapted but there are only so many ways to heal the sick, improve crop growth, save money, learn to read and so on, and there is a world we need to learn about what enabled some to be sustained and even morph into new results!

This excellent article, Do NGOs (non-governmental organization) help?, notes that “due to donor pressure [NGOs] are increasingly forced to respond with a discrete project with x number of deliverable outcomes” [1]. It goes on to cite D. Sriskandarajah, the secretary-general of Civicus, a global network of civil society organizations and activists, wrote: ‘We have become a part of the problem rather than the solution… Since demonstrating bang for your buck has become all-important, we divide our work into neat projects, taking on only those endeavours that can produce easily quantifiable outcomes. Reliant on funding to service our own sizeable organizations, we avoid approaches or issues that might threaten our brand or upset our donors. We trade in incremental change’” [1].

We also settle for results while we control them, and don’t ask unpopular questions about who is to sustain these results, with what resources, and for goodness sake, why sustained impact was not funded, designed, implemented and monitored/ evaluated from the very onset in our rush to measurable results?

 

DontSettleForLess

 

As this great NGO article by Dinyar Godrej goes on to say, “most media scrutiny of NGO accountability is of how they use funds, their accountability to donors. But what of their accountability towards the recipients of their interventions” [1]? They have no lobbyists to persuade our funders they would like this but nto that, and often such lobbying for their needs falls to the very NGOs that have won these large contracts and tasked with implementing a dizzying array of mandatory input, output, outcome and some impact indicators. We do care deeply about results! US State Department/ USAID has a “Standard Foreign Assistance Master Indicator List” of 2,300 lines in an excel spreadsheet [2]. (There are more indicators still– custom and cross-cutting indicators, the mind boggles).

Wow. But are we asking the right questions? Are we asking what was sustained after all this hard work was done and ended? Rarely. Who should be?! “It is perhaps unrealistic to expect such large structural changes to be delivered by NGOs when governments don’t tackle them either.”

For the rub is this. When we take development over from national governments, largely do not involve country nationals in the funding, design and M&E of projects, then how sustained can these projects still be after we go? Millions are invested, then disappear… Last year, at local debrief at the end of one SEIE Valuing Voices did, the state of affairs became crystal clear when a government official asked us “Can you ever find some funds to fund us to do our own independent evaluations? Even if it is not the projects that they did themselves? We would be happy to get that support…”

When are we no longer responsible for doing great, sustained work? Valuing Voices will let you know what we found regarding the best ways to do SEIEs more. Stay tuned.

What do you think?

P.S. This blog topic prompted me to look for statistics on the number or percentage of funded projects that were renewed. Nothing.  Does anyone know how many or what % of projects were extended/ funded again after showing good results? (Often this happens in the form that a successful project in one area of the country gets either funded again or repeated elsewhere in the country or in the world, as have two of our own SEIEs, Niger and Ethiopia). For that matter, what made them so excellent to be replicated? What can we learn?

 

 

Sources:

[1] Godrej, D. (2014, December 1). NGOs – Do They Help? Retrieved from https://newint.org/features/2014/12/01/ngos-keynote/

[2] US Department of State. (n.d.). Standard Foreign Assistance Indicators. Retrieved 2017, from https://web.archive.org/web/20170404072145/https://www.state.gov/f/indicators/index.htm