When Funders Move On (Originally published by Stanford Social Innovation Review 03/15)

When Funders Move OnDonors and nonprofits need to learn more about how to help program participants keep progressing after the support ends.

Imagine standing in Detroit or South-Central Los Angeles. A team of experts has come to help you out of grinding poverty. Some of these experts specialize in credit issues, others in education or health or gardening. They have funding for three years, so they set up offices, create participant lists and prioritize problems to tackle. They give you seeds and loans, and advice. And you—and others from your neighborhood—begin creating small businesses and home gardens. You and other adults learn about infant nutrition; children who live in your area get free school materials; teachers at your local schools receive extra training. Everyone begins to do better.

A year and a half passes. Another expert arrives to find out how things are going relative to the team’s projections. Some businesses are succeeding, others have faltered; some gardens are flourishing, others are neglected. You participate in a focus group, and you answer questions optimistically.

At the three-year mark, many of your neighbors are participating in this project, and tangible successes appear to be spreading. But suddenly, the experts are packing their boxes. The project office closes. The initiative has supposedly been “handed over” to the community. No one who worked for the project comes back.

 No one comes back. This is the state of affairs for too many so-called “sustainable international development” initiatives around the world, and it has to change.

As Gugelev and Stern brilliantly note in a recent SSIR article, we must be transparent about our “endgame”: Too many international development projects are bound to fixed endings and “fail to reckon with the gap between what the nonprofit can achieve and what the problem actually requires.” Due to fixed funding requirements, donors often leave when the calendar tells them to, whether or not a project has achieved the desired impact. And according to Valuing Voices research, they don’t even go back to assess the outcomes of their work or consider what (if anything) might help progress continue!

Since 2000, the US government has spent more than $280 billion on bilateral and multilateral assistance; the EU has spent $1.4 trillion. Just in 2002, US foundations, businesses, and NGOs spent more than $34 billion overseasAnd while most taxpayers believe that this spending supports “sustainable development,” our research shows that 99 percent of the nonprofit grant and for-profit contracted projects these funds enabled were not evaluated after the funding concluded. Unfortunately, this continues: In 2014, the US spent $20 billion and the EU spent $80 billion on program assistance without any plan for post-project evaluation. This does not mean the projects are not sustainable; we simply do not know.

In fact, the United States Agency for International Development (USAID)—once considered the leader in post-project evaluations assessing relevance, effectiveness, efficiency, and sustainability—has managed only one post-project evaluation in 30 years (due later this year). And although thousands of documents appear in multilateral donor database searches as evaluations, most are “desk studies”—conducted remotely and not based on new fieldwork. Of these, only a few include feedback from program participants (leading the way are Japan’s International Cooperation Agency and the UN’s Organization for Economic Co-operation and Development, which have systematically done post-project evaluations).

ValuingVoicesArrowGraphic

There is usually terrific monitoring and evaluation during project implementation (red) and evaluation at start-mid-end (green), virtually no one returns afterwards (blue).

Why don’t we do a better job of following up? Are we afraid of the possibility of seeing poor results? If so, it’s time to face that fear. In some cases, we will surely see good results or even unanticipated positive impact—we’re missing that too.

Hewlett Foundation’s Fay Twersky implored nonprofits to “systematically solicit feedback from intended beneficiaries” in an SSIR podcast on Monitoring and Evaluation (M&E). We agree. We need to know more than whether those participants’ situations are improving while a program is in full swing, and we need to know what it will take for things to continue to improve after the funding goes away. Imagine the cost efficiencies we would gain by replicating activities that they could sustain. Imagine the cost-efficiencies and productivity if we prioritized activities with the largest sustainable return on investment (ROI). Now that is something impact investors could buy into.

There are positive signs on the horizon. According to Keystone Accountability, an increasing number of nonprofit organizations are committing to “making governments, NGOs, and donors more responsive to the needs of their constituents.” And funding that supports the idea of using participant feedback to improve programs and make them sustainable is on the rise, as Center for Effective Philanthropy’s “Hearing from Those We Seek to Help” and the Fund for Shared Insight have noted. All of this could yield a hugely different array of endgames that are sustainable by communities that can perhaps later even select development aid offers based on past effectiveness.

But there is still much to do. We offer the following recommendations to project implementers and donors, based on our own experience, and on our observations of several initiatives where we have seen project participants independently continuing and adapting work that was begun with external support:

  • Shift the development model from what donors and implementers think would be best to what the intended participants think is best. Design projects with them, and mandate that request for proposals (RFP) design involves communities.
  • Document, share, and discuss what was most sustained and what participating communities and local partners can do to sustain projects, and how we can sustainably support them through design and implementation to make them more effective.
  • Require a plan for transitioning to sustainability after projects close for any project that uses more than $1 million dollars. This should include handover plans to local nonprofits, with training and financial support; training for communities on how to manage the sustainable activities it prioritizes; financing mechanisms for those activities; and report sharing in IATI open-data format, with project data saved and stored in the cloud for global access.
  • Do post-project sustainability evaluations on all these projects, and discuss the results widely with other funders, the government, and the private sector, including how to feed back lessons into future design.
  • Advocate for participatory input in all evaluations. This input should make up 30 percent of future evaluation findings (now far less).
  • Consistently solicit feedback from local communities through national evaluators, both during and after projects, to better understand how the program you’re running or supporting from afar is working on the ground. Invest in building the national capacity and systems needed to make that feedback helpful for all stakeholders, including national governments.
  • Advocate for extensive civil society input into the United Nations’ Sustainable Development Goals so they serve our participants’ visions for the world they want.

 

As Peter Kimeu of Catholic Relief Services said to me, “It will be sustainable development if the people at community level are involved in designing and delivering their own dreams of development.”

 **************************************

Jindra Cekan (@WhatWeValue) is founder of Valuing Voices, with 28 years in international development design, monitoring, and evaluation. She has a doctorate from the Fletcher School of Law and Diplomacy; was a University of Cambridge Fellow; and works with foundation, nonprofit, and for-profit clients.

Learning from the Past… for Future Sustainability

Learning from the Past… for Future Sustainability

Heading up Food Security for Catholic Relief Services (CRS) was my first international development job in 1995-1999 and I have watched this organization grow in its commitment to program quality and learning/ knowledge management ever since.  At the time I oversaw 17 of of CRS' USAID/ Food for Peace (FFP) programs.  So I was delighted that not only has CRS done an ex-post evaluation and used the findings for programming (e.g. the effectiveness of investing in a particular sector—for example, the importance of supporting girls’ education within a food security program) and also for advocacy (e.g. evaluation lessons from Rwandan peace-building projects seven years after the genocide informed CRS’ evolving approach to peace and justice strategies), but I get to celebrate FFP learning too.  In addition to having consulted to USAID/PPL (Policy, Planning and Learning) and the FANTA project, all featured below, I went to Tufts University’s Fletcher School. Super to see great organizations learning about sustainability!

CRS’ 2007 project package guidance for implementation and guidance (ProPackII), described ex-post evaluation/sustainable impact evaluation’s aim “to determine which project interventions have been continued by project participants on their own [which] may contribute to future program design…. it is fair to say that NGOs rarely evaluate what remains following the withdrawal of project funding [which] is unfortunate [as] important lessons can be generated regarding factors that help to ensure project sustainability.

A 2004 Catholic Relief Services excellent ex-post evaluation in Ethiopia was featured: “Looking at the past for better programming: dap I Ex-Post Assessment Report”. It assessed sustainability of Agriculture Natural Resource Management, and Food-Assisted Child Survival/Community Based Health Care programming, done as an internal evaluation by CRS staff and partners with document review, partner, government and community interviews. Results were mixed.

  • Some activities generated enough food and income that households could eat throughout the year and have some savings, making them more resilient against drought
  • Almost 100% of cropland bunding and irrigation practices for improved crop production were still being applied and buffered them during a subsequent drought
  • Health practices had also continued (e.g. trained traditional birth attendants had continued to provide services with high levels of enthusiasm and commitment, and increased levels of health care-seeking behaviors existed).

However, many other benefits and services had severely deteriorated:

  • Nearly all water committees had dissolved, fee collection was irregular or had been discontinued
  • Many water schemes were not operational
  • The centrally managed [tree] nurseries had been abandoned (given the existing management capacity of communities and government).

CRS/Ethiopia and its partners came to see that:

  • “The potable water strategy had over-focused on the technical aspects (“hardware”) while not paying enough attention to the community organizing dimensions and support by existing government services (“software”).
  • Even limited post-project follow-up by partners and government staff might have gone a long way towards mitigating the deterioration of project benefits and services.

What was terrific was that they “went on to use these findings and lessons learned from this ex-post evaluation to inform the design of similar projects in Ethiopia… while also raising awareness of these issues among partner staff”. The ex-post recommended increased planning for sustainability, setting up village management for post-project and incentive maintenance.  Great learning, yet we have found few ex-posts at CRS or elsewhere.  Our industry needs to explore issues such as those the evaluators posed: Was the lack of sustainability due to technical, institutional or financial faults in the programming? In other words, was the lack of self-sustainability due to the design/ aim of the activity itself or how it was implemented?

In 2013, USAID’s Food For Peace commissioned fascinating research on Exit Strategies.  Tufts University went to Bolivia, Honduras, India and Kenya which were phasing out of Title II food aid to look at how to “ensure that the benefits of interventions are sustained after they end, [as] there is little rigorous evidence on the effectiveness of different types of exit strategies used in USAID Office of Food for Peace Title II development food aid prog

rams.” The research is to “assess the extent to which the programs’ impacts were maintained or improved and to help understand factors of success or failure in the specific exit strategies that were used.” They have made the important discernment that the effectiveness of Title II programs depends on both short-term impact and long-term sustainability.

FANTA_Exit_Strategies_Bolivia.pdf

The FANTA project (contractor) made the following preliminary results available:

  1. Impact assessment at exit does not consistently predict sustained impact two years later…. It can be misleading.
  2. Many activities, practices, and impacts across sectors declined over the two years after exit. These declines are related to inadequate design and implementation of sustainability strategies and exit processes.
  3. There are specific ways to increase the likelihood of sustainability: Sustaining service provision and beneficiary utilization of services and practices depends on three critical factors: Resources, Technical and Management Capacity, Motivation
  4. Withdrawal of free food rations or any other free input (as incentive) jeopardizes sustainability without consideration of substitute incentives. For instance,

     

     

    • Withdrawal of food was a disincentive for participation in and provision of [child] growth monitoring…. Resources and health system linkages are needed to sustain health activities
    • Motivation, capacity and resources are all needed to maintain water systems
    • Agriculture and Natural Resource Management suffered greatly when resource incentives disappeared

Their main recommendations are that sustainability should be built into the design from the beginning, program cycles are longer and exit is gradual.

CRS found the same issue of incentives as a barrier, as they did technical and (institutional) capacity/ motivation/ management issues.  We have much to learn… at least we’ve started Valuing Voices and asking… and eventually designing for sustainability!