Grow the .002% of all global development projects that are evaluated ex-post closure for sustainability

Grow the .002% of all global development projects that are evaluated ex-post closure for sustainability

It seems like ‘fake news’ that after decades of global development so few evaluations would have peered back in time to see what was sustained. While I was consulting to the Policy Planning and Learning Bureau at USAID, I asked the head of this M&E department who does ex-post sustainability evaluation as I knew USAID had done some in the 1980s, Cindy Clapp-Wincek answered ‘No one, there are no incentives to do it.’ (She later became our advisor.)

Disbelieving, I did a year of secondary keyword research before devoting my professional consulting life to advocating for and doing ex-post evaluations of sustained outcomes and impacts. I searched USAID, OECD, and other bilateral and later multilateral donors’ databases and found thousands of studies, most of which were inaccurately named ‘ex-post’ or ‘post-closure’ studies.  Some of the roughly 1,000 projects I looked at at USAID and OECD that came up under ‘ex-post’, ‘ex post’, ‘post closure’ were final evaluations that were slightly delayed, a few were evaluations that were at least one year after closure, but were desk studies without interviews. Surprisingly, the vast majority of final evaluations found were those that only recommended ex-post evaluation several years later to confirm projected sustainability.

 

 

 

 

 

 

 

 

In 2016 at the American Evaluation Association conference, a group of us did a presentation. In it, I cited these statistics from of 1st year of Valuing Voices’ research:

  • Of 900+ “ex-post” “ex post” “post closure” documents in USAID’s DEC database, there were only 12 actual post-project evaluations with fieldwork have been done in the last 20 years
  • Of 12,000 World Bank projects – only 33 post-project evaluations asked ‘stakeholders’ for input, and only 3 showed clearly they talked to participants
  • In 2010 Asian Development Bank conducted 491 desk reviews of completed projects, and returned to 18 actual field-based post-project evaluations that included participant voices; they have done only this 1 study.
  • We found no evaluations by recipient governments of aid projects’ sustainability

12 years of research, advocacy and fieldwork later, the ‘catalysts’ database on Valuing Voices now shows actual fieldwork-informed evaluations by 40 organizations that had actual ex-posts that returned to the field to ask participants and project partners what was sustained, highlighting 92 ex-posts.

How many ex-post project closure evaluations have been done? .002% of all projects. The 0.002% statistic looks at just public foreign development aid from 1960 (not even counting private funding such as foundations or gifts to organizations, which isn’t tracked in any publicly available database). Calculating aggregated OECD aid statistics (excluding private because it’s only recent data) over 62 years $5.6 trillion by 2022 (thanks to Rebecca Regan-Sachs for the updated #s).

I then estimated 3.000 actual ex-posts which comes from 2,500 JICA projects plus almost 500 other projects that I have either found looking through databases all across the spectrum from governments and multilaterals (almost 100 in our catalysts, and am assuming there must be 400 others done in the 1980s-2000 like USAID and the World Bank).

Without a huge research team it is improssible to aggregate data on the total number of projects by all donors. So I extrapolated from project activity disbursements of one year (2022) for Mali on the www.foreignassistance.gov page. In my 35 years of experience, Mali, where I did my doctoral research, typifies he average USAID aid recipient. They had 382 projects going in 2022. I rounded up to 400 projects x 70 years (since 1960 when OECD data began) x 100 countries by just one donor (of the 150 possible recipient countries, to be conservative). This comes to 2.8 million projects. So if we take 39 OECD countries as donors (given most have far less to give than US), in total 109 million publicly funded aid projects disbursed $5.6 trillion since 1960. While final evaluations are industry standard, only .002% is the estimated number of ex-post evaluations of projects the were evaluated with data from local participants and partners of the 109 million projects .

This became Valuing Voices focus, and we created an open-access database for learning, and conducted our own  My team and I identified 92 ex-posts that returned to ask locals what lasted, what didn’t, why, and what emerged from their own efforts. We also created evaluability checklists and created a new evaluation, Sustained and Emerging Impacts Evaluation that included examining not just what donors put in place to last, but also what emerged outcomes from local efforts to sustain results with more limited resources, partnerships, capacities and local ownership/motivation. These four drivers were found by Rogers and Coates for USAID’s food security exit study in 2015). We have done 15 ex-posts for 9 clients since 2006 and shared Adaptation Fund ex-post training materials in 2023.

 

Yet the public assumes we know our development is sustainable. 2015’s ‘Sustainable Development Goals‘ focused aid on 17 themes, which was to generate $12 trillion more in annual spending on SDG sectors than the;$21 trillion already being invested each year. Nonetheless, a recent UN report states that there is now a $4 trillion annual financing gap to achieve the SDGs. All this funding goes to projects that are currently implemented, not to evaluate what had been sustained from past projects that already closed. Such learning from what succeeded or failed, or what emerged from local efforts to keep activities and results going is pivotal to improving current and future programming is almost wholly missing from the dialogue; I know, I asked multiple SDG evaluation experts.

 

Why do we return to learn so rarely? There are many reasons, the most prosaic among them being administrative.

  • When aid funds are spent over 2-10 years, projects are closed, evaluated at the end, ‘handed over’ to national governments, and no additional funding exists to return ‘ex-post’ closure to learn.
  • Next is the push to continue to improve lives through implementation which means low rates of overhead allocated to M&E and learning during, much less after closure.
  • Another is the assumption that ‘old’ projects differ so much from new ones, but there are few differences. After all there are only so many ways to grow food, feed the malnourished, educate children; evaluating ‘old’ projects can teach ‘new’ projects.
  • A last major one, from Valuing Voices’ research of 12 years may be the largest: Fear of admitting failure. Please read Valuing Voices’ 2016 blog highlighted many Lessons about Funding, Assumptions and Fears (Part 3). One US aid lobbyist told me in 2017 that I must not share this lack of learning about sustained impacts because it could imperil US aid funding; I told her I had to tell people because lives were at stake.
  • Overall, there is much to learn; most ex-post evaluations show mixed results. None show 100% sustainability and while most show 30-60% sustainability, none are 0% sustained either. If we don’t learn to replicate what worked and cease what didn’t now, then future programming will be as flawed and successes, especially brilliant emerging locally designed ex-post outcomes such as Niger’s local funding of redesign of health incentives will remain hidden.

 

Occasionally donors invest in sets of ex-post learning evaluations such as USAID’s ‘global waters’ seven water/ sanitation evaluations linked to the E3 Bureau taking sustainability as a strategic goal. Yet the overall findings from USAID’s own staff of these ex-posts Drivers of WASH study were chilling. While 25 million gained access to drinking water and 18 million to basic sanitation, ‘they have largely not endured.’ But the good news in such research is that the donor learned that infrastructure fails when spare parts are not accessible and maintenance not funded or performed, which can be planned for and addressed during implementation by investing in resources and partnerships. They learned that relying on volunteers is unreliable and management needs to be bolstered, which can lead to some implementation funding to be focused on capacities and local ownership. We can plan better for sustainability by learning from ex-post and exit studies (see Valuing Voices’ checklists in this 2023 article on Fostering Values-Driven Sustainability).

 

And since 2019, three climate funds, the Adaptation Fund, the Global Environmental Facility, and the Climate Investment Funds have turned to ex-post evaluations to look at sustainability and longer-term resilience and even transformation, given environmental shocks may take years to affect the project sites. The Adaptation Fund has done four ex-posts, with more to come in 2024/25, and the CIF is beginning now. The GEF has done a Post-Completion Assessment Pilot for the Yellow Sea Region . Hopeful!

Trust is both a belief and a bond: Why is Trust for outsiders declining in communities? (Reblog from APEA and Rituu Nanda)

Following on from the localization blog by PLAN, APEA has shared why trust is declining in local communities. Note the learn not give lesson 🙂

Reblog of Asia Pacific Evaluation Association / Rituu Nanda: https://asiapacificeval.org/trust-is-both-a-belief-and-a-bond/

Trust is both a belief and a bond: Why is Trust for outsiders declining in communities? (Insights from 13 countries across four continents)

A group of 16 Participants of the Community Ownership in MEL and Research Group of APEA from Benin, Bangladesh, France, India, Indonesia, Nepal, Nigeria, New Zealand, Nigeria, Pakistan, Sri Lanka, Tunis, UK came together on 15th March’24 to discuss issues around Trust. (Ahmed, Amol Shaila Suresh, Anita Cheria, Bhuban Bajracharya, Chinenye Mercy Morka, Fiona Cram, Jhank, Lipika Das Gupta, Luc Barrière-Constantin, Maroof, Pakeeza Arif, Randika De Mel, Rituu B Nanda, Sarah Gharbi, Sushila Pandit, Visvalingam Muralithas.)

(This post is compiled by Rituu B Nanda based on the inputs of the discussion)

Trust is a basic need It helps anyone survive, and maintain relationships including with self, immediate family members, society. Trust helps in connecting one with others and just being yourself.

What do we mean by trust?

  • The absence of doubt about other’s intention…
  • Being transparent and acknowledging that we know less about them, and seek to build understanding.
  • Recognizing that our actions may only play a small role in people’s lives and approaching with a sense of modesty and humility.
  •  Standing with the communities, being one of them.
  • Convince the communities about our sincere intentions
  • Trust is about visiting with people

Decline in trust-internal and external

We observe a loss of trust both towards outsiders and within the communities themselves. Communities are losing trust in the system, implementation agencies, evaluation professionals and researchers. Due to longstanding practices followed by outsiders, the younger generation exhibits less trust in them compared to older generations, as their expectations remain unfulfilled. Moreover, trust within communities has eroded not only within groups in communities and neighbourhoods but also amongst family members.

Why is community’s Trust eroding in outsiders?

  • Evaluators’ approach is often perceived as overly theoretical by communities. Community members feel that stakeholders treat them merely as means to achieve their own agenda. Outsiders, driven by predefined targets, tend to impose objectives on communities, exacerbating mistrust.
  • Tribal or indigenous communities prefer  isolation and often resist mainstreaming efforts , feeling unheard and excluded from decision-making processes that affect them.
  • Repeated service delivery projects in communities foster dependency rather than empowerment. Additionally, implementation agencies lack transparency and come with pre-planned projects without involving community in design.
  • The decline in trust is compounded by a sense of fatigue, as projects often address surface-level issues without tackling systemic problems. Consequently, expectations remain unmet, fuelling disillusionment among community members.
  • Communities express frustration with organizations that promise resources but vanish after a few years, leaving behind unfulfilled promises. This perpetuates the erosion of trust in development professionals.
Example from north Sri Lanka: Post-war Reconciliation A range of complex challenges can affect the well-being and social cohesion of residents. The northern part of Sri Lanka has been deeply affected by the decades-long civil conflict, and issues of reconciliation between different ethnic and religious groups remain. Rebuilding trust and fostering dialogue among communities is essential for long-term peace and stability.

How can we restore Trust?

Government authorities, civil society organizations, local communities, and other stakeholders have to prioritize inclusive development, social cohesion, and respect for diversity to build trust:

  • Trust is fragile and rather than imposing external definition, we need to understand how communities define trustthemselves.
  • Mistrust within a community often stems from past episodes, usually rooted in bitter experiences. It is the responsibility of outsiders to delve into these causes and initiate the rebuilding process.
  • Without trust from the community, a project is likely to fail. So, engagement with communities must commence with trust-building.
  • Cultivate competent communities which can address certain issues independently. For example, in rural Nepal, where a small percentage of households face extreme poverty, local government and communities can collaborate to address their needs.
  • Transparency regarding project objectives is paramount. Instead of devising plans and then presenting them to the community, they should be involved in shaping the work plan from the start.
  • Outsiders need to come to learn, not to give” – We need to be learners ourselves.  If we position ourselves solely as experts, communities may be less inclined to engage with us. Knowledge exchange should flow both ways.  Building trust necessitates authenticity and genuine connections.  To restore trust – we have to be non-judgemental, by being ourselves
  • Consistency is fundamental in trust-building. Move away from short term funding. Long-term engagement demonstrates commitment, gradually fostering trust over time.
Rebuilding Trust-from deficit based to strength-based approach After the Ebola epidemic, Constellation was invited to Guinea and Liberia to rebuild trust  in communities toward government health services. Employing the SALT approach, communities took ownership and initiated dialogue with health officials, thereby strengthening trust and partnership between both stakeholders. https://the-constellation.org/opening-safe-conversations-to-restore-trust-after-ebola/  

Conclusion Participants agreed that trust is fundamental for survival, and therefore, growing mistrust is worrying as the world becomes increasingly fragmented. Trustworthy relationships are crucial for sustaining long-term partnerships and ensuring the effectiveness of evaluations. Through trust, we can foster a safe environment for ourselves and others.

 

REBLOG: COMMENTARY A USAID localization model finally emerges by Justin Fugle October 6, 2023

NOTE: Am reposting a great blog (plus attended the webinar) on localization of aid (getting US foreign assistance directly to local organizations), as we know, what is local tends to be sustained over the long-term, and NGOs can design, implement, monitor & evaluate with sustainability in mind, as they’re on the ground over the long-term. Original post: https://www.brookings.edu/articles/a-usaid-localization-model-finally-emerges

A USAID localization model finally emerges (reblog)

More than a decade ago, the efforts of the Centers for Disease Control (CDC) to directly fund its local implementers accelerated substantially, showing that localization could be successfully implemented within the rules and constraints of the U.S. government.  At the same time, the U.S. Agency for International Development (USAID)’s localization initiative failed to move the needle, with direct funding to local entities at 4.2% in 2012 and merely 4.4% in 2018. USAID’s troubles with localization were so systemic at the time that, according to an article earlier this year by former senior U.S. officials, the Agency “declined to adopt” an approach shifting its resources to local organizations, “despite agreeing to the policy by signing the agreement” with the State Department’s Office of the Global AIDS Coordinator, better known as PEPFAR.

US implementers transferred leadership to local orgs in just four years

 By contrast, the CDC followed its PEPFAR agreement to advance localization, transforming grants with U.S. implementers into “terminal transition awards,” mandating that the American organizations would have just four years to transfer full responsibility for all activities to local entities “without any drop off in the quality or coverage of services” to the population. As a result, PEPFAR’s budget flowing directly through the CDC to local organizations and governments reached fully 67% by 2012!

This was a result of CDC transitioning its Antiretroviral Therapy programs in 13 countries from U.S.-based organizations and grantees to Ministries of Health and indigenous organizations. Critically, studies found that program service delivery by those local entities was comparable to that of its U.S. partners, demonstrating 11 years ago that localization could be achieved while delivering results and safeguarding taxpayer dollars. Based on this success, PEPFAR went on to set and largely achieve a goal that a whopping 70% of its funding would be awarded directly to local organizations and governments.

With this in mind, it must be satisfying for USAID staff to see their own in-house localization model finally emerging with direct local funding increasing to more than 10% in FY22. Of course, this shift would need to accelerate considerably to meet or even get close to Administrator Samantha Power’s goal of 25% local funding by 2025; however, there now seems to be a path that Bureaus and Missions across USAID could follow to fulfill that commitment. It builds on the PEPFAR-CDC model and USAID’s ongoing procurement and staffing reforms.

USAID’s direct local funding for HIV/AIDS programs jumped 81% in just four years

Two recent peer-reviewed journal articles by USAID document that between FY18 and FY22, USAID’s PEPFAR-funded HIV/AIDS programs expanded annual direct funding “to local partners by $345 million, or an 81% increase.” The data shows that this major expansion was accomplished incrementally across the Missions. For example, local direct funding rose from $452 million in FY18 to $600 million in FY20 to $797 million in FY22. Thus, the 81% increase was accomplished through steady, widespread, and manageable progress. It should be noted here that third-party assessments of USAID localization data have raised doubts, both in terms of methodology and the inclusion of some international organizations. Still, it seems clear that the increases were rapid and significant. This sharp jump also gains credibility from the fact that it replicated the increases achieved by the CDC in the 2010s.

The USAID authors stressed that to qualify as local, partners had to be “locally incorporated, registered, and have a majority of local staff, including at senior levels,” so their local partners would all qualify as local entities under USAID’s current definition. They added that based on the available program assessments, local partners again “displayed quality of service comparable to international partners.” So, after struggling for years to move the needle at all, how was USAID’s HIV/AIDS team able to achieve an 81% jump in just four years? The authors cite six aspects of USAID’s emerging localization model. None of these seem to be exclusive to HIV programming, suggesting they could be widely adopted across the Global Health Bureau and the Agency.

Six key factors made it possible

  1. An ambitious goal (PEPFAR’s 70% local funding commitment) resulting in country-specific strategies that include local funding targets based on the Missions’ specific context and procurement plans.
  2. Strong data systems to monitor progress toward both the direct local funding target and the program performance of the local partners.
  3. Strengthening local partner organizational and financial capacity. Critically, the capacity strengthening efforts have short timelines and focus on preparing the local partners to become prime awardees, as in terminal transition awards. This may not completely align with USAID’s new Local Capacity Strengthening approach.
  4. Bolstering USAID’s capacity to manage local awards. As noted in a blog last year, this point recognizes that USAID sometimes lacks the capacity in its operating systems and organizational culture to work with local organizations, and must accept responsibility for improving. In the case of USAID’s HIV programs, 98 new positions (funded by PEPFAR) were approved across 16 Missions. These included new staff in Global Health as well as Acquisition and Assistance and Financial Management. Some Missions also hired a local transition or local capacity adviser. Therefore, current requests by Missions and Bureaus for similar positions should be prioritized.
  5. Changing the way USAID does business, including expedited procurement approaches and building a wider network of local partners through personal and online outreach and by convening local partner conferences.
  6. USAID leadership at headquarters and within country teams made transitioning to local partners a top priority.

Model is transferable to other bureaus 

As to whether these results were replicable beyond HIV programs, then Acting Administrator for Global Health, Jennifer Adams, wrote that USAID HIV had developed the “largest local partner funding footprint’ across any Agency program with “significant experience and lessons learned” to share with their colleagues within the Global Health Bureau and through the Agency about successful direct partnerships with local organizations. Reflecting on the same results, USAID’s former Chief of Staff testified before Congress this year that the experience was widely applicable, asserting that “every large” cooperative agreement and contract should include “mandatory Transition Awards to local organizations/local entities for the vast majority of the substantive work by the end of the period of performance.”

Key reforms in A&A staffing and partnering

The second set of breakthroughs at the heart of USAID’s emerging localization model have been reform to its Acquisition and Assistance (A&A) practices and staffing. This builds from recognition in Congress and the Front Office that USAID’s business practices are perhaps the single largest barrier to advancing locally-led development. To address these issues and improve aid effectiveness, USAID’s new Acquisition and Assistance (A&A) strategy was launched six months ago. It sets out a path to achieve the 25% local funding goal, expand and equip the A&A workforce, and acquire a more diverse set of partners for locally-led development solutions.

One key personnel innovation was to recognize USAID’s local staff as an overlooked resource. The new A&A strategy explicitly recognizes that the local Contracting Officer corps is underutilized with just 10% having warrants to obligate and manage funds on behalf of the U.S. government. After years of ‘slow walking’ the idea, USAID has again made rapid progress, moving from 19 local staff with these administrative warrants in FY22 to 40 now, exceeding its ambitious target of doubling the number in just one year. This shows that there are many well-qualified local staff as well as pent-up demand, so hopefully USAID will continue to expand their ranks in FY24.

A key effort to expand USAID’s local partner base is a new public-facing A&A website, WorkwithUsaid.org. WorkwithUSAID.org helps introduce USAID to prospective partners in civil society and has seen a good amount of traffic and engagement, with more than 5,000 organizations registered, over 60% of which are considered local entities.

Another important shift in the A&A strategy is the effort to improve local partners’ ability to recover their costs of winning and implementing awards. The current “de minimus” overhead recovery rate of just 10% underfunds the core and proposal-writing expenses of local entities. When compared to the 20-40% overhead rates received by USAID’s traditional implementers through NICRA, the current 10% rate emerges as a glaring disadvantage and disincentive for local partners to accept the risk of working with USAID. Thankfully, new draft guidance from OMB has opened the door to raising the “de minimus” overhead rate to 15%. Finalization of this rule would allow USAID to more fairly compensate its local partners and break what has been called the “starvation cycle,” of unrecovered overhead costs by local entities.

Successful localization at CDC and USAID HIV have blazed a trail

The rapid growth of direct funding to local entities by the CDC’s PEPFAR-funded programs a decade ago and by USAID’s PEPFAR-funded HIV programs more recently demonstrate that USAID can still reach or get close to the 25% direct local funding target by FY25. One key aspect would be the adoption of this internal USAID model by other technical sectors. That would be aided by the innovations of the A&A strategy, facilitating changes to USAID’s business practices while also reducing the costs for local entities to become its partners. It’s now fair to say that the localization trail has been blazed with PEPFAR, CDC, and USAID HIV as its pioneers. It’s reasonable for localization’s bipartisan supporters in Congress to expect other parts of USAID to adopt similar approaches.

AUTHORS