Grow the .002% of all global development projects that are evaluated ex-post closure for sustainability

Grow the .002% of all global development projects that are evaluated ex-post closure for sustainability

It seems like ‘fake news’ that after decades of global development so few evaluations would have peered back in time to see what was sustained. While I was consulting to the Policy Planning and Learning Bureau at USAID, I asked the head of this M&E department who does ex-post sustainability evaluation as I knew USAID had done some in the 1980s, Cindy Clapp-Wincek answered ‘No one, there are no incentives to do it.’ (She later became our advisor.)

Disbelieving, I did a year of secondary keyword research before devoting my professional consulting life to advocating for and doing ex-post evaluations of sustained outcomes and impacts. I searched USAID, OECD, and other bilateral and later multilateral donors’ databases and found thousands of studies, most of which were inaccurately named ‘ex-post’ or ‘post-closure’ studies.  Some of the roughly 1,000 projects I looked at at USAID and OECD that came up under ‘ex-post’, ‘ex post’, ‘post closure’ were final evaluations that were slightly delayed, a few were evaluations that were at least one year after closure, but were desk studies without interviews. Surprisingly, the vast majority of final evaluations found were those that only recommended ex-post evaluation several years later to confirm projected sustainability.

 

 

 

 

 

 

 

 

In 2016 at the American Evaluation Association conference, a group of us did a presentation. In it, I cited these statistics from of 1st year of Valuing Voices’ research:

  • Of 900+ “ex-post” “ex post” “post closure” documents in USAID’s DEC database, there were only 12 actual post-project evaluations with fieldwork have been done in the last 20 years
  • Of 12,000 World Bank projects – only 33 post-project evaluations asked ‘stakeholders’ for input, and only 3 showed clearly they talked to participants
  • In 2010 Asian Development Bank conducted 491 desk reviews of completed projects, and returned to 18 actual field-based post-project evaluations that included participant voices; they have done only this 1 study.
  • We found no evaluations by recipient governments of aid projects’ sustainability

12 years of research, advocacy and fieldwork later, the ‘catalysts’ database on Valuing Voices now shows actual fieldwork-informed evaluations by 40 organizations that had actual ex-posts that returned to the field to ask participants and project partners what was sustained, highlighting 92 ex-posts.

How many ex-post project closure evaluations have been done? .002% of all projects. The 0.002% statistic looks at just public foreign development aid from 1960 (not even counting private funding such as foundations or gifts to organizations, which isn’t tracked in any publicly available database). Calculating aggregated OECD aid statistics (excluding private because it’s only recent data) over 62 years $5.6 trillion by 2022 (thanks to Rebecca Regan-Sachs for the updated #s).

I then estimated 3.000 actual ex-posts which comes from 2,500 JICA projects plus almost 500 other projects that I have either found looking through databases all across the spectrum from governments and multilaterals (almost 100 in our catalysts, and am assuming there must be 400 others done in the 1980s-2000 like USAID and the World Bank).

Without a huge research team it is improssible to aggregate data on the total number of projects by all donors. So I extrapolated from project activity disbursements of one year (2022) for Mali on the www.foreignassistance.gov page. In my 35 years of experience, Mali, where I did my doctoral research, typifies he average USAID aid recipient. They had 382 projects going in 2022. I rounded up to 400 projects x 70 years (since 1960 when OECD data began) x 100 countries by just one donor (of the 150 possible recipient countries, to be conservative). This comes to 2.8 million projects. So if we take 39 OECD countries as donors (given most have far less to give than US), in total 109 million publicly funded aid projects disbursed $5.6 trillion since 1960. While final evaluations are industry standard, only .002% is the estimated number of ex-post evaluations of projects the were evaluated with data from local participants and partners of the 109 million projects .

This became Valuing Voices focus, and we created an open-access database for learning, and conducted our own  My team and I identified 92 ex-posts that returned to ask locals what lasted, what didn’t, why, and what emerged from their own efforts. We also created evaluability checklists and created a new evaluation, Sustained and Emerging Impacts Evaluation that included examining not just what donors put in place to last, but also what emerged outcomes from local efforts to sustain results with more limited resources, partnerships, capacities and local ownership/motivation. These four drivers were found by Rogers and Coates for USAID’s food security exit study in 2015). We have done 15 ex-posts for 9 clients since 2006 and shared Adaptation Fund ex-post training materials in 2023.

 

Yet the public assumes we know our development is sustainable. 2015’s ‘Sustainable Development Goals‘ focused aid on 17 themes, which was to generate $12 trillion more in annual spending on SDG sectors than the;$21 trillion already being invested each year. Nonetheless, a recent UN report states that there is now a $4 trillion annual financing gap to achieve the SDGs. All this funding goes to projects that are currently implemented, not to evaluate what had been sustained from past projects that already closed. Such learning from what succeeded or failed, or what emerged from local efforts to keep activities and results going is pivotal to improving current and future programming is almost wholly missing from the dialogue; I know, I asked multiple SDG evaluation experts.

 

Why do we return to learn so rarely? There are many reasons, the most prosaic among them being administrative.

  • When aid funds are spent over 2-10 years, projects are closed, evaluated at the end, ‘handed over’ to national governments, and no additional funding exists to return ‘ex-post’ closure to learn.
  • Next is the push to continue to improve lives through implementation which means low rates of overhead allocated to M&E and learning during, much less after closure.
  • Another is the assumption that ‘old’ projects differ so much from new ones, but there are few differences. After all there are only so many ways to grow food, feed the malnourished, educate children; evaluating ‘old’ projects can teach ‘new’ projects.
  • A last major one, from Valuing Voices’ research of 12 years may be the largest: Fear of admitting failure. Please read Valuing Voices’ 2016 blog highlighted many Lessons about Funding, Assumptions and Fears (Part 3). One US aid lobbyist told me in 2017 that I must not share this lack of learning about sustained impacts because it could imperil US aid funding; I told her I had to tell people because lives were at stake.
  • Overall, there is much to learn; most ex-post evaluations show mixed results. None show 100% sustainability and while most show 30-60% sustainability, none are 0% sustained either. If we don’t learn to replicate what worked and cease what didn’t now, then future programming will be as flawed and successes, especially brilliant emerging locally designed ex-post outcomes such as Niger’s local funding of redesign of health incentives will remain hidden.

 

Occasionally donors invest in sets of ex-post learning evaluations such as USAID’s ‘global waters’ seven water/ sanitation evaluations linked to the E3 Bureau taking sustainability as a strategic goal. Yet the overall findings from USAID’s own staff of these ex-posts Drivers of WASH study were chilling. While 25 million gained access to drinking water and 18 million to basic sanitation, ‘they have largely not endured.’ But the good news in such research is that the donor learned that infrastructure fails when spare parts are not accessible and maintenance not funded or performed, which can be planned for and addressed during implementation by investing in resources and partnerships. They learned that relying on volunteers is unreliable and management needs to be bolstered, which can lead to some implementation funding to be focused on capacities and local ownership. We can plan better for sustainability by learning from ex-post and exit studies (see Valuing Voices’ checklists in this 2023 article on Fostering Values-Driven Sustainability).

 

And since 2019, three climate funds, the Adaptation Fund, the Global Environmental Facility, and the Climate Investment Funds have turned to ex-post evaluations to look at sustainability and longer-term resilience and even transformation, given environmental shocks may take years to affect the project sites. The Adaptation Fund has done four ex-posts, with more to come in 2024/25, and the CIF is beginning now. The GEF has done a Post-Completion Assessment Pilot for the Yellow Sea Region . Hopeful!

Assuming Sustainability and Impact is Dangerous to Development (+ OECD/DAC evaluation criteria)

 

Assuming Sustainability and Impact is Dangerous to Development
(+ OECD/ DAC evaluation criteria)

 

We all do it; well, I used to do it too. I used to assume that if I helped my field staff and partners target and design funded projects well enough, and try to ensure a high quality of implementation and M&E, then it would result in sustainable programming. I assumed we would have moved our participants and partners toward projected long-term, top-of-logical-framework’s aspirational impact such as “vibrant agriculture leading to no hunger”, “locally sustained maternal child health and nutrition”, “self-sustained ecosystems”.

INTRAC nicely differentiates between what is typically measured (“outputs can only ever be the deliverables of a project or programme…that are largely within the control of an agency”) and what is not: “impact as the lasting or significant changes in people’s lives brought about by an intervention or interventions” [1]. They continue: “as few organisations are really judged on their impact, the OECD DAC impact definition (“positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended“) allows for long-term changes in institutional capacity or policy change to be classed as impact” [1]. Do we do this? Virtually never. 99% of the time we only evaluate what happened while the project and its results is under the control of the aid implementer. Yet the five OECD/DAC evaluation criteria asks us to evaluate relevance, effectiveness, efficiency (fair enough, this is important to know if a project was good) and also impact and sustainability. So in addition to the prescription to evaluate ‘long-term effects’ (impact), evaluators are to measure “whether the benefits of an activity are likely to continue after donor funding has been withdrawn… [including being] environmentally as well as financially sustainable” [2]. 

How do we know we are getting to sustained outcomes and impacts? We ask people on the receiving end ideally after projects end. It is dangerous to assume sustainability and impact, and assume positive development trajectories (Sridharan) unless we consistently do “ex-post” project evaluations such as these from our research or catalytic organizations that have done at least one ex-post. At very minimum we should evaluate projected sustainability at end of project with those tasked to sustain it before the same project is repeated. Unfortunately we rarely do so and the assumed sustainability is so often not borne out, as I presented at the European Evaluation Society conference Sustainability panel two weeks ago along with AusAid’s DFAT, the World Bank, University College London and UNFEM.

 

 

Will we ever know if we have gotten to sustained impacts? Not unless the OECD/DAC criteria are drastically updated and organizations evaluate most projects ex-post (not just good ones :)), learn from the results and fund and implement for country-led sustainability with the country nationals. We must, as Sanjeev Sridharan tells us in a forthcoming paper embed sustainability into our Theories of Change from the onset (“Till time (and poor planning) do us part: Programs as dynamic systems — Incorporating planning of sustainability into theories of change” (Canadian Journal of Program Evaluation, 2018).*

There are remarkable assumptions routinely made. Many projects put sustainability into the proposal, yet most close out projects in the last 6 months. Rarely do projects take the time to properly phase down or phase over (unlike CRS Niger); many exit ceremonially ‘handing over’ projects to country-nationals, disposing project assets, and leaving only a final report behind. Alternatively, this USAID Uganda CDCS Country Transition Plan which looks over 20 years in the future by when it assumes to have accomplished sustained impact for exit [3]. Maybe they will measure progress towards that goal and orient programs toward handover, as in the new USAID “Journey to Self-Reliance” – we hope! Truly, we can plan to exit, but only when data bears out our sustained impact, not when the money or political will runs out.

As OXFAM’s blog today on the evaluation criteria says, “Sustainability is often treated as an assessment of whether an output is likely to be sustained after the end of the project. No one, well, hardly anyone, ever measures sustainability in terms of understanding whether we are meeting the needs of the present, without compromising the ability of future generations to meet their own need” and “too often in development we evaluate a project or programme and claim impact in a very narrow sense rather than the broader ecology beyond project or programme parameters” [4]. In fact, most ‘impact evaluations’ actually test effectiveness rather than long-term impact. Too rarely do we test impact assumptions by returning 2-10 years later and gather proof of what impacted locals’ lives sustainably, much less – importantly – what emerged from their own efforts once we left (SEIEs)! Oh, our hubris.

if you’re interested in the European Evaluation Society’s DAC criteria update discussion, see flagship discussion and Zenda Offir’s blog which stresses the need for better design that include ownership, inclusivity, empowerment [5][6]. These new evaluation criteria need to be updated, including Florence Etta’s and AGDEN‘s additional criteria participation, non-discrimination and accountability!

 

 

We can no longer afford to spend resources without listening to our true clients – those tasked with sustaining the impacts after we pack up – our partners and participants. We can no longer fund what cannot be proven to be sustained that is impactful. We talk about effectiveness and country ownership (which is paramount for sustainability and long-term impact), with an OECD report (2018) found “increases [in[ aid effectiveness by reducing transaction costs and improving recipient countries ownership” [7]. Yet donor governments who ‘tie’ aid to their own country national’s contracts benefit a staggering amount from ‘aid’ given. “Australia and the United Kingdom both reported … 93 percent and 90 percent of the value of their contracts respectively went to their own firms” [7]. It is not so different in the USA where aid is becoming bureaucratically centralized in the hands of a few for-profit contractors and centralized hundreds of millions in a handful of contracts. We must Do Development Differently. We can’t be the prime beneficiaries of our own aid; accountability must be to our participants; is it their countries, not our projects, and we cannot keep dangerously assuming sustained impact. Please let us know what you think…

 

 

Footnotes:

[*] This paper is now available at https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/53055

 

Sources:

[1] Simister, N. (2015). Monitoring and Evaluation Series: Outcomes Outputs and Impact. Retrieved from https://www.intrac.org/wpcms/wp-content/uploads/2016/06/Monitoring-and-Evaluation-Series-Outcomes-Outputs-and-Impact-7.pdf

[2] OECD. (n.d.). DAC Criteria for Evaluating Development Assistance. Retrieved September, 2018, from https://web.archive.org/web/20180919035910/http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm

[3] USAID. (2016, December 6). USAID Uganda Country Development Cooperation Strategy 2016-2021. Retrieved October, 2018, from https://www.usaid.gov/uganda/cdcs

[4] Porter, S. (2018, October 18). DAC Criteria: The Hand That Rocks the Cradle. Retrieved from https://views-voices.oxfam.org.uk/2018/10/dac-criteria-the-hand-that-rocks-the-cradle/

[5] European Evaluation Society Biennial Conference: Flagship Symposia. (2018). Retrieved from http://www.ees2018.eu/1539782596-flagship-symposia.htm

[6] Ofir, Z. (2018, October 13). Updating the DAC Criteria, Part 11 (FINAL). From Evaluation Criteria to Design Principles. Retrieved from https://zendaofir.com/dac-criteria-part-11/

[7] OECD. (2018, June 11). 2018 Report On The DAC Untying Recommendation. Retrieved from http://www.oecd.org/dac/financing-sustainable-development/development-finance-standards/DCD-DAC(2018)12-REV2.en.pdf

 

Setting a higher bar: Sustained Impacts are about All of us

Setting a higher bar: Sustained Impacts are about All of us
Global development aid has a problem which may already affect impact investing as well.

It is that we think it’s really all about us (individuals, wealthy donors and INGO implementers) not all of us (you, me, and project participants, their partners and governments). It’s also about us for a short time.

 

All too often, the measurable results we in global development aid and Corporate Social Responsibility (CSR) funded projects that last 1-5 years track and report data for two reasons:

1) Donors have Compliance for grantees to meet (money spent, not lost, and results met by fixed deadlines of 1-5 years – look at some of the European Commission Contracting rules) and

2) Fund recipients and the participants they serve are accountable to ‘our’ donors and implementers who take what happened through their philanthropic grants as ‘their’ results.

Both can skew how sustainably we get to create impacts. An example of such strictures on sustainability from USAID.  As respected CGDev Elliot and Dunning researchers found in 2016 when assessing the ‘US Feed the Future Initiative: A New Approach to Food Security?‘ the $10.15 billion leveraged $20 billion from other funders for disbursement over three years (2013-16). “We are concerned that pressure to demonstrate results in the short term may undermine efforts to ensure any impact is sustainable…. Unfortunately, the pressure to show immediate results can encourage pursuit of agricultural investments unlikely to be sustained. For example, a common response to low productivity is to subsidize or facilitate access to improved inputs… it can deliver a quick payoff… however, if the subsidies become too expensive and are eliminated or reduced, fertilizer use and yields often fall…..

With so much focus on reporting early and often about the progress in implementing the initiative, there is a risk that it increases the pressure to disburse quickly and in ways that may not produce sustainable results. For example, for 2014, Feed the Future reports that nearly 7 million farmers applied “improved technologies or management practices as a result of U.S. Government assistance,” but only 1,300 received “long-term agricultural sector productivity.” Are the millions of others that are using improved inputs or management practices because of subsidies likely to have these practices sustained? And how likely are they to continue using improved practices once the project ends?”

 

3) Impact investors stick to the same two paths-to-results and add a new objective: market-competitive financial  returns. They also need to show short-term results to their investors, albeit with social, environmental and governance results like non-profits (future blog).

4) Altruists create things we want ‘beneficiaries’ (our participants) to have. For instance a plethora of apps for refugees cropped up in recent years, over 5,000 it is estimated, which can be appropriate, nor not so helpful. Much like #2 above, ‘we’re’ helping ‘them’ but again, it seems to be a ‘give a man a fish’… and my fish is cool sort of solution… but do our participants want/ need this?

 

How often is our work-for-change mostly about us/by us/ for us... when ideally it is mostly about ‘them’ (OK, given human self-interest, shouldn’t changes we want at least be about all of us?).

All too often we want to be the solution but really, our ‘grassroots’ clients who are our true customers need to generate their own solution. Best if we listen and we design for long-term sustainability together?

 

As the Brilliant Sidekick Manifesto stated in two of its ten steps:
a)I will step out of the spotlight: Sustainable solutions to poverty come from within are bottom-up, and flow from local leaders who are taking the risks of holding their politicians accountable and challenging the status quo.”

b)I will read “To Hell with Good Intentions” again and again: Politicians, celebrities and billionaire philanthropists will tell me that I can be a hero. I cannot. The poor are not powerless or waiting to be saved. Illich will check my delusions of grandeur.”

 

We have examples of where we have stepped away and participants had to fend for themselves. At Valuing Voices, we’ve done post project-exit evaluations 2-15 years afterward. What did participants value so much that they sustained it themselves (all about them, literally)? These Sustained and Emerging Impacts Evaluations (SEIE) also give us indications of Sustained ROI (Sustained Return on Investment (SusROI) is a key missing metric. As respected evaluator Ricardo Wilson-Grau said in an email, “I think calculating cost-effectiveness of an intervention’s outcomes would be a wonderful challenge for a financial officer searching for new challenges — if not a Nobel prize in economics!”)

Most of these evaluations are pretty bad news mixed with some good news about what folks could sustain after we left, couldn’t and why not. (These are the ones folks expect to have great results, otherwise they wouldn’t share them!)  While most clients are understandably interested in what of ‘theirs’ was still standing, and it was interesting disentangling where the results were attributable by implementation or design or partnership flaws or something else, what was mesmerizing was what came from ‘them’.

The key is looking beyond ‘unexpected’ results to look at emerging impacts that are about ‘them’ (aka what we didn’t expect that was a direct result of our project, e.g. spare parts were no longer available to fix the water well pump once we left or a drought rehabilitation water project that decreased violence against women), to what emerging results are attributable not to use but only to our participants and partners who took over after our projects closed.  One example is a Nepalese project ended yet the credit groups of empowered women spawned groups of support groups for battered women. Another is a child maternal health project changed how it worked as women reverted to birthing at home after NGOs left; community leaders punished both parents with incarceration in the health clinic for a week if they didn’t given birth there (wow did that work to sustain behavior change of both parents!).

Many of us at Valuing Voices are shocked that funders don’t seem that interested in this, as this is where they not only take over (viz picture, sustaining the project themselves), but they are making it theirs, not oursImagine assuming the point of development is to BE SUSTAINABLE.

Source: Community Life Competence

Our participants and national stakeholder partners are our true clients, yet… Feedback Labs tell us Americans alone gave $358 billion to charities (equivalent to the 2014 GDP of 20 countries) – in 2014 but how much of this was determined by what ‘beneficiaries’ want? Josh Woodard, a development expert, suggests a vouchers approach where our true clients, our participants, who would “purchase services from those competing organizations… [such an] approach to development would enable us all to see what services people actually value and want. And when we asked ourselves what our clients want, we would really mean the individuals in the communities we are in the business of working with and serving. Otherwise we’d be out of business pretty quickly.”

This opens the door to client feedback – imagine if participants could use social media to rate the sustained impacts on them of the projects they benefited from? A customer support expert wrote in Forbes, “Today, every customer has, or feels she has, a vote in how companies do business and treat customers. This is part of a new set of expectations among customers today that will only grow ... you can’t control product ratings, product discussions or much else in the way of reviews, except by providing the best customer experience possible and by being proactive in responding to negative trends that come to the surface in your reviews and ratings stronger.”

So how well are we working with our participants for ‘development’ to be about them?

What do you think?

Can’t wait to learn from post-project sustainability evaluation? If not why?

 

Can’t wait to learn from post-project sustainability evaluation? If not why?

 

A colleague who has been promoting ex-post sustainability evaluation in her organization questioned my claim that doing them had “benefits” for future programming. It was an “untested assumption that there will be sufficient, strong enough evidence to apply to future programming…  [and] the need to have evidence to cite for future work is not pressing enough.”

If you are on aid’s receiving end, what you care about is that good results are sustained, and you are able to live better, longer. You might want to show others evidence of what was sustained, rather than only what worked while external investments were there but stopped since. Absolutely, aid donors need to have evidence that something designed, funded, implemented, and monitored & evaluated showed good results, but we assume our results will be sustained after we have closed out and moved on. How well have we done? Let’s see.

At the American Evaluation Association meetings this month, several post-project evaluations were presented. Some came from Valuing Voices research, some from Social Impact, PLAN and World Vision and some others [1].

 

Results 3-5 years post close-out were, shall we say unexpected, from CRS Madagascar:

[2]

Nigeria,

[3]

and Honduras.

[4]

While there were some successes, including Niger

[5]

and Burkina Faso, where MCC/ PLAN found that three years post project “BRIGHT still had a significant positive impact—6.0 percentage points for children between ages 6 and 22—on self-reported enrollment. The impacts are smaller than estimated impacts on enrollment at 7 and 3 years after the start of the program,” they were rare [6].

If we don’t wonder why things didn’t work or why they did, and don’t return to find out if it happened again and what to do/ not to do again?, Often we continue to do very similar programming elsewhere, again assuming great results. How can we close our eyes and not do post project, Sustained and Emerging Impacts Evaluations (SEIE) and see, learn, do better?  How can we continue to do very similar water/ sanitation, health, food security, and education programs and projects (with potentially similar results), and call ourselves sustainable development professionals? Shouldn’t we always ask not how effective is our aid when it’s there, but after its gone?

If you want more data, see a presentation we did at USAID. What do you think?

 

 

Sources:

[1] Cekan, J., PhD. (2014, April 7). Evaluation of ERCS/Tigray’s “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity”. Retrieved from https://valuingvoices.com/wp-content/uploads/2017/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf

[2] Madagascar Rural Access To New Opportunities For Health And Prosperity (RANO-HP) Ex-Post Evaluation. (2017, June 1). USAID. Retrieved from https://www.globalwaters.org/resources/assets/madagascar-rural-access-new-opportunities-health-and-prosperity-rano-hp-ex-post-0

[3] The World Bank. (2014, June 26). Project Performance Assessment Report, Nigeria: Second National Fadama Development Project. Retrieved from http://ieg.worldbankgroup.org/sites/default/files/Data/reports/Nigeria_Fadama2_PPAR_889580PPAR0P060IC0disclosed07070140_0.pdf

[4] Rogers, B. L., Sanchez, L., & Fierstein, J. (n.d.). Exit Strategies Study: Honduras. Retrieved from https://www.fsnnetwork.org/sites/default/files/fanta_exit_strategies_presentation_honduras_-_final_-_feb_5_0.pdf

[5] Cekan, J., PhD, Kagendo, R., & Towns, A. (2016). Participation by All: The Keys to Sustainability of a CRS Food Security Project in Niger. Retrieved from https://www.crs.org/our-work-overseas/research-publications/participation-all

[6] Davis, M., Ingwersen, N., & Kazianga, H. (2016, August 29). Ten-Year Impacts of Burkina Faso’s BRIGHT Program. Retrieved from https://www.mathematica.org/our-publications-and-findings/publications/ten-year-impacts-of-burkina-fasos-bright-program

 

Building the Evidence Base for Post Project Evaluation: A report to the Faster Forward Fund

 

Building the Evidence Base for Post Project Evaluation:
A report to the Faster Forward Fund

 

We are delighted to share Valuing Voices’ report on the value added of post-project evaluation, which compares findings from eight end-of-project and subsequent post project evaluations [1].  Many of you are aware of how rarely post project evaluations are undertaken.  As a result, there is little real evidence about project impact on long-term sustainability. Valuing Voices received a grant from Michael Scriven’s Faster Forward Fund to begin to address this gap.

Our findings show that post project evaluations can contribute to better understanding of sustainability impacts, and reveal unexpected and emerging outcomes years after project close. They also indicate ways in which we can design and implement for sustainability.

Finding suitable projects for this review was difficult because so few post project evaluations are done, fewer are publically available, and fewer still had comparable final evaluations and included local voices.  Agencies that fund post project evaluations offer a range of reasons for doing so: to learn, to promote a success, to inform replication or scale, to provide justification for future funding, to promote accountabilities.  However, many funding agencies consider post project evaluation a luxury or not necessary.  JICA and OECD are notable exceptions in this regard.

Highlights include:

  • The review highlights the range of methods that have been used in post project evaluations, and point to the advantages of planning for sustainability measurement from the outset of the project.
  • The cases reviewed in the study highlight the (sometime dramatic) difference between the anticipated trajectory of a project, what is happening as the project ends, and what actually continued, was adapted, ceased or changed course after close out.
  • Taxonomies, knowledge management about evaluation, data retrieval/ retention, analysis, use and dissemination are elements of sustained impact evaluation that require attention.
  • Little documentation is available about how post project evaluations have actually informed and influenced organizational learning, sectoral dialogue or future programming.
  • Post project evaluations shed particularly interesting light on what emerged post-project that was entirely due to the efforts and resources of participants and partners after project investments stopped. More on these Sustained and Emerging Impacts Evaluations (SEIEs) at Better Evaluation.

As part of this report, Valuing Voices created an evaluability checklist for assessing whether a post project evaluation is viable, as well as a checklist for measuring sustainability starting at the beginning of the project cycle [2].

We welcome your comments on this report and checklists, and encourage you to share it in your networks and get us feedback on their use.  Please use the report and findings to advocate for more post project sustainability impact evaluations which will contribute to greater evidence-based learning about project sustainability.  Valuing Voices is among a handful of organizations who do post-project evaluations and we can either conduct one or refer you to another who does.

 

Thank you,

Laurie Zivetz, MPH, PhD and Jindra Cekan, PhD, with Kate Robins, MPH, PhD of Valuing Voices

 

The full report is available here:

https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf

 

Sources:

[1] Zivetz, L., Cekan, J., & Robbins, K. (2017, May). Building the Evidence Base for Post-Project Evaluation: Case Study Review and Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf

[2] Zivetz, L., & Cekan, J. (n.d.). Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2017/08/Valuing-Voices-Checklists.pdf
 

Living in a Well-meaning Lie: Valuing all Voices? – The Solutions Journal

Living in a Well-meaning Lie: Valuing all Voices?

By Jindra Cekan


Solar panels in Cap Haitien, Haiti.

Poor villagers like Edith, Aminata, Rituu, and Juan don’t appear much on the nightly news.  You might never know they exist unless you stop and read your mail from some charity asking you to help them.  On the brochures, they can look scared or sad; maybe surrounded by their thin children, with a parched land or dying animals behind them. Our foreign aid programs should be helping them, but are they?

 

I have met these people while they are working in their fields, growing corn and peas, millet and coffee, raising their chickens and goats. I have talked to them outside of health centers where they have brought their babies to be vaccinated or their parents for medical care.  I got to know them when I interviewed them under the big tree in the middle of their village, or in empty school rooms, asking them what they need from us, and how we can design foreign aid projects to better help them.

 

I have worked in international development as a technical expert in project design and monitoring and evaluation for international non-profits such as Catholic Relief Services, Save the Children, the Red Cross, and many others including the Bill and Melinda Gates Foundation and the US Agency for International Development (USAID). I estimate that I have designed and evaluated over 200 projects in 28 countries over the last 29 years. I have felt lucky to do this work, and foreign aid does achieve some very good work while we are there: helping farmers to farm better, or helping men and women to care for their family’s health, wealth, and future with new knowledge, tools, and items they need for daily living. My colleagues do wonderful work as well, in hard conditions, within countries with few resources, and for donors with unrealistic expectations of how much can be done well in short timeframes.  In 2010, USAID stated that they would aim for 30 percent of funding to be spent by national partners under USAID Forward.1 This is an excellent step toward the country-led development that the Paris Accords promised, yet as of 2016, there is no list of local partners, other than a handful of examples.2 The only ‘country partners’ list posted to the USAID website includes 80 organizations doing programming via USAID in Afghanistan alone, 55 of which are American firms, four US agencies, nine Afghan government-affiliated organizations, six foreign governments, six UN agencies, and two MENA firms.3 Not quite the national civil-society-NGO partners we envisioned in 2010. Under the new U.S. administration, these are likely to shrink even more as the 0.5 percent of our GNP we allocate to foreign aid is redirected inward—that much more reason to make it as sustained as possible.  European aid as well as other rising world nations need this approach just as much.

Large parts of international aid system remain broken. We design too many projects outside of the countries themselves. We have fixed funding and leave in pre-set times rather than when participants are actually ready to take over. We ‘handover’ without partnering throughout the whole project so that partners can determine what they are able to sustain. Even worse, we leave and do not look back to learn from our Ediths and Juans after our projects have closed.  Sometimes, we disparage their knowledge, and at other times we don’t make enough time to ask but wish we did. Mostly, our aid industry is designed around measuring success while we intervene, and then abruptly leave because funding ended. Yet development is, as international evaluator Ian Davies says, “A process, not a result.”

 

Our policies say we are doing “sustainable development”, that we are helping our ‘beneficiaries’ (really our partners and participants) feed themselves over the long-term, that our projects are almost all successful, and that all we need to do is to scale up the great projects out there.  But the numbers prove we are not, in fact, achieving sustainable development. Nine times out of ten, we rarely go back to talk to our participants and partners after our project end, and we move on.

 

The numbers are staggering:

  • Of the US$5 trillion dollars of international foreign aid spent since 1945, we have evaluated the long-term sustainability far less than one percent of the time.
  • Since 2000, for example, USAID and the Millennium Challenge Corporation have only done three such evaluations apiece, yet they spent well over US$300 billion.
  • The EU evaluated only a few dozen of its projects and programs, in spite of spending US$1.5 trillion in the last 15 years. The United Nations Development Program may do up to six a year, and the World Bank more, but how often do any of them talk to project participants and design anew based on what we learned that succeeded and failed?
  • The Japan International Cooperation Agency, and to some degree EU bilateral countries (through the Organization for Economic Cooperation and Development), have evaluated the sustainability of over 300 projects.
  • Despite this, tens of thousands of new projects are launched every year.
Cekan 2
Satellites atop homes in a slum in Tigray, Ethopia.

 

This is why I founded Valuing Voices—to analyze what little we know to make development better.4 Not to destroy international development, but to change how we fund, design, and implement it. We need to design for sustainability of the activities by the country nationals themselves, rather than designing for results we can show to get more funding. We also must jointly implement, monitor, and evaluate our projects so countries can continue after we leave. Smaller organizations can do even simple activities, designing projects based on what the participants feel they can self-sustain, and partnering with those who will take over while they are still there.

 

Having spread the word for the last three years, to mostly little response, I now turn to you, readers.  Our analysis – and a wonderful 2012 book, Time to Listen by Mary Anderson and Dayna Brown, shows that sometimes when our projects partner with country nationals, their people become – and stay – better off.5They want to be engaged, yet our very structure of delivering aid prevents this. Often we are not there long enough to make a lasting difference, or we invest scarce time on untested innovations that work in some places but don’t in others. Even worse, sometimes we design activities so badly that villages are left with irrelevant technology and trainings, wasted funding, and lost hopes. At other times, there are successes as well, but not returning robs us of the chance to replicate those. We do ‘impact evaluations,’ but only on successes during project implementation, and not on what people can self-sustain after we leave. Our vision is so limited. Our well-meaning self-interest blinds us.

 

Across the board, our development projects make one massive and incorrect assumption that once we ‘handover’ the project, the local government, community, and households have the means to sustain our multi-million dollar investments.6 We assume that technical knowledge will still be locally available to the villagers, that inputs like seeds and tools, data and vaccines will be accessible both physically and financially, that the government staff have the means to get to villages or that new NGOs and donors will appear to fill the gaps. The Huffington Post has stated, “as long-term projects and action-plans are established, more investment must go into financing locally designed solutions and projects that ensures ownership is placed back into local communities.”7 While more project have begun using feedback loops of listening to participants during implementation, virtually all good work stops when project funding stops.

 

Don’t we want development to be sustained after resources leave, and the opinions of these aid recipients to be heard? Don’t we want the next project to address the needs better?  Don’t we, as taxpayers, want to demand that agencies using our tax dollars learn what is really sustainable and what is not?  And shouldn’t we demand that all projects costing more than US$1 million over the past 10 years be examined now for lessons learned by sector (agriculture, health, credit, education, etc) and region? Shouldn’t post-project sustainability evaluations be included in all new projects?  Don’t our participants and partners deserve the dignified futures they hope for, our creating channels for their voices that enable them to evaluate us and teach us how we can help them to be successful?

 

In fact, a radical Foreign Aid Transparency Act was just passed in the U.S. in June 2016.8 The bill calls for the President, within 18 months of enactment, to “set forth guidelines…for the establishment of measurable goals, performance metrics, and monitoring and evaluation plans that can be applied with reasonable consistency to covered United States foreign assistance.” These include ‘ex-post’ (sustainability) evaluations, and “can have enormous value when it comes to making programming and budgeting decisions.”  Yet while there is a call for guidance to be developed, no funding came along with this bill.  Without the funds to make this happen, this may be more ‘window dressing’ for sustainable development than excellent policy.

Cekan 3
A small enterprise in Cap Haitien, Haiti.

 

There is some hope coming from the corporate sector. While impact investors are often more focused on return from emerging economies than fostering sustainable development, corporate social responsibility is building bridges in lovely ways.  Tsikululu Social Investment of South Africa has thought about what advice to give to the companies they advise on such investments, as well as exiting from them.9

 

We argue that our budgeting needs a basic business metric: Return on Investment. In a time of huge demands on our resources worldwide from refugee flows, terrorism and climate change, we currently do little or no analysis of:

  • How much actual investment: What percent of allocated funds went to the activities that benefitted the partners and participants themselves, rather than being used as overhead for operations?
  • How much return: What is the value of what remains used 3-10 years after we leave? What was the value-added that communities and other funders (including the national governments themselves) who followed catalyzed based on our earlier investments? What were unexpected new results that emerged?

 

We envision a beautiful future, one where Edith, Aminata, Rituu, and Juan and their local partners are at the center of development.  We imagine a world in which we listen to what people in need can sustain for themselves. Through these approaches data is shared widely on what has worked best and why; aid projects invest in country-systems and staff that boost their ability to self-sustain; and only sustainable projects are designed and funded that foster country-led development. The global adoption of the Sustainable Development Goals finally puts the focus on all we can do to foster sustainability of our work.

 

By valuing voices and focusing on sustainable solutions for excellent impacts, this will promote truly sustainable development from our aid organizations, government and non-governmental alike. We have much to learn, and there is not a moment to waste before we start Valuing Voices of those we serve and partner with on country-led development.

 

References

  1. USAID Forward. USAID [online] (2017). https://www.usaid.gov/usaidforward.
  2. In-country Partners. USAID [online] (2016). https://www.usaid.gov/partnership-opportunities/in-country-partners.
  3. Implementing Partners. USAID [online] (2017). https://www.usaid.gov/afghanistan/implementing-partners.
  4. Valuing Voices [online]. https://valuingvoices.com/.
  5. Jacobs, A. Time to Listen by Dayna Brown and Mary B Anderson. NGO Performance [online] (December 4, 2012). https://ngoperformance.org/2012/12/04/time-to-listen-by-dayna-brown-and-….
  6. Cekan, J. What happens after the project ends? Lessons about funding, assumptions and fears (Part 3). Valuing Voices [online] (February 29, 2016). https://valuingvoices.com/what-happens-after-the-project-ends-lessons-abo….
  7. Zuabi, V. Investing in Locally Designed Solutions for Syria and the Middle East. The Huffington Post [online] (May 24, 2016). http://www.huffingtonpost.com/vanessa-zuabi/investing-in-locally-desi_b_….
  8. Ingram, G, Miles, C & Veillette, C. The Foreign Aid Transparency and Accountability Act is Law! Now What? Modernizing Foreign Assistance Network [online] (August 2, 2016). http://modernizeaid.net/2016/08/foreign-aid-transparency-accountability-….
  9. Cekan, J. Towards responsible donor exiting strategies and practices: Reblog from Tshikululu. Valuing Voices [online] (October 8, 2016). https://valuingvoices.com/towards-responsible-donor-exiting-strategies-an….

Source: Living in a Well-meaning Lie: Valuing all Voices? – The Solutions Journal