What happens after the project ends?  Lessons from post-project sustained impacts evaluations (Part 1)

by | Feb 13, 2016 | Aid effectiveness, Bill and Melinda Gates Foundation, Catholic Relief Services (CRS), Central Asia, Ethiopia, Ethiopian Red Cross, Evaluation, Federation of the Red Cross, Lutheran World Relief, Mercy Corps, Niger, PACT, Partners for Democratic Change, post-project evaluation, Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation, USAID

 

What happens after the project ends?
Lessons from post-project sustained impacts evaluations (Part 1)

 

We talk a lot about impact of our interventions, but far less is analyzed about the sustained impact of our work in the years after projects close out. We take for granted that successful strategies will continue after projects shut down.  Do they always? Maybe they do but we don’t know. Maybe there are innovations and impacts to be learned from…

To answer these question Valuing Voices spent 2 ½ years looking for and analyzing ‘post-project’ evaluations of projects undertaken 2-10 years after projects ended.  The result: in our $137 billon international development industry, some 99% of projects remain unevaluated after project close out [1].  Only 17 agencies we have found so far have publically available post-project evaluations; most of them have one, while the OECD and JICA have dozens.  Hundreds of studies recommend such learning that is missing from our industry’s program cycle (green slice).

 

What_have_we_learned_from_postproject_sustainability_impact_Pt1and2_0216_docx

 

Six decades on, this astonishing finding raises serious questions about stewardship of resources and commitment to learning—particularly learning from participant and partner stakeholders for whom sustainability matters most, and who are tasked with it over the long-term.

A review of post project evaluations generate food for thought about good program design and illustrate the value added of post project perspectives. This rapid review of select ‘ex-post’ evaluations points to three early lessons:

How we do it matters for great results
Expect unexpected results
Who takes over? Country Nationals

First, organizations go back to see how well their projects results were sustained. What we learned was that how well they used participatory processes in how they implemented and handed over mattered a lot for sustainability and that we must expect unexpected results.

 

How we do it matters for great results

1. Catholic Relief Services/ USAID PROSAN food security project in Niger

Valuing Voices evaluated this food security (agriculture, health and resilience project in 2015 which ran from 2007-2012 (report forthcoming). The $32 million project was implemented in a consortium of CRS, CARE and Helen Keller but this evaluation focused only on CRS areas.

  • Interviews with over 500 participants found that three years after project closeout, 80% of project activities still continued, as did many village committees and there were a variety of community innovations [2]
  • On average, households can feed themselves through their own production or purchase of food for 8-12 months three years after closeout compared to 6-9 months at closeout three years earlier. Such impact was unexpected. [2]
  • 91% of respondents reported improved household health, hygiene, and nutrition [2]
  • A 2.5-year participatory exit process from CRS to country stakeholders (local government, an array of local and international NGOs and the private sector ensured continuity and boosted local ownership [2]
  • 20% of the activities did not continue, mostly food-assisted NRM and resilience-related [2]
  • Youth make up 50% of the population and need to be engaged during the project for long term sustainability to occur. [2]

 

2. PartnersGlobal (formerly Partners for Democratic Change)

Their mission is to “to build sustainable capacity to advance civil society and a culture of change and conflict management worldwide” uses an approach that is “bottom-up, locally-led rather than foreign-led, based on the belief that change comes from sustainable efforts led by local people, organizations and institutions invested in their own long-term future.” They went back and to review 55 case studies of projects through 22 centers they founded in central and eastern Europe from 1989-2011. They found:

  • In 80% of cases, there was advancement of good governance by influencing the participation of civil society working with government [3]
  • In 50% of the cases there was increased access to justice and managing and resolving disputes/conflicts, thereby strengthening civil society [3]
  • 18 of 22 of the centers that had been established still exist today (82%) [3].

 

3. Mercy Corps

They did a post-project evaluation in Central Asia in 2007, one and three years after two conflict resolution projects ended which were worth $18 million. These complex community mobilization programs with aims “to empower communities to work together in a participatory manner to address the infrastructure and social needs [while] developing sustainable skills [and] empowers communities to identify and utilize existing resources within the communities and not to depend only on external assistance.”

  • 72% of youth report that they continue to use at least one skill they learned during the programs (e.g. teamwork and communication, and skills such as sewing, construction, roofing, journalism and cooking) [4]
  • 68% of community members witnessed local government becoming more involved in community activities after the end of the programs as compared with before the programs [4]
  • 57% of the communities studied continuing to use one or more of the decision-making practices promoted during the program [4]
  • 42% of members, representing 70% of communities, reported that the community had worked collectively on new projects or repairs to existing infrastructure. Participants and partners had implemented almost 100 infrastructure projects by themselves independent of donor funds. [4]

 

 

These are terrific expected results.
We also learned to Expect unexpected results

4. Federation of the Red Cross and Ethiopian Red Cross

Valuing Voices combined a final evaluation of “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity” that had funding over $3 million in 2009 from the Swedish Red Cross with an assessment of projected sustainability.  It was an IFRC/ERCS collaboration with the Ethiopian government to provide credit for food security inputs to 2,259 households, which were to be repaid in cash over time as well water and agriculture/ seedlings for environmental resilience.  We answered the DAC criteria for evaluation and found the project overall to be quite good, albeit with weak data tracking systems.

In terms of sustainability, we used participatory methods to learn about what people felt they could self-sustain once the project left their area, so we could shape a similar follow-on project design to be moved elsewhere in Tigray, particularly around the credit for animals.

  • While 87% of the loans had been promoted by the government and given for large animals (oxen and cows), and 13% was for small animals (sheep, goats, chickens)…
  • But project participants we interviewed, strongly preferred the small animals in terms of being able to sustain them on their own. They felt they could afford these smaller amounts of credit as well as the feed to sustain them, without taking the risk of animal death leaving them with large debt. This was especially true for women, who preferred poultry to all other animals 15:1.

In our quest for fast results, are we asking participants to bear too much risk? As one of our Valuing Voices team asks, Who is responsible for sustainability?

 

5. Lutheran World Relief

From 2005-2007 Lutheran World Relief intervened in Niger, the world’s poorest country, with a $500,000 Pastoralist Survival and Recovery Programme (ARVIP) drought rehabilitation project funded by the Bill and Melinda Gates Foundation. There were numerous outcomes from targeting sheep, wells and animal fodder to 600 of the poorest women in 10 communities in northern Niger, among them:

  • Women’s share of household income increased from 5% to 25% in some households. This was due to the value of the sheep grants, as well as time-savings used for income generation. Access to wells in five of the villages saved women a staggering 7-10 hours every other day from not having to go fetch water 3.5 hours away each way for household and animal needs and were free to weave mats or cook food for sale [5]
  • Many said they didn’t have to resort to worse survival strategies during the next hungry season after they received the sheep [5]

What was not expected were these results:

  • Many women in several villages reported an impact that was completely unexpected to the implementer and donor which was “our husbands don’t beat us anymore” [5]. This was thanks to both increased respect and income from the sheep as well as access to well-water which led to cleanliness and their ability to be home for their husbands, children and mothers-in-law, rather than fetching water whole days 2-3 times a week. The same was found in PACT’s WORTH empowerment and village banking project in Nepal that wrote, “one in 10 reported that WORTH has actually helped “change her life” because of its impact on domestic violence” [6].
  • We defined success too narrowly. Many interviewees were content with the project even though prospects for project-expected drought resilience or sustained food security were less likely. Some women sold the sheep to buy food, pay their children’s school fees or their daughters’ dowries, while some had their sheep sold by their husbands who used them to buy other animals, pay for ceremonies or other expenses.  Participants saw the project as bringing them resources and considered it a success. Spending assets on immediate needs is not at all illogical for a community who can feed itself only 4 months a year; for some households, their pressing needs far outweighed the luxury to wait and buffer seasonal food insecurity far down the line.

 

We hope you agree that allocating funds and attention to post-project sustained impacts evaluations is necessary for the remaining 99% of international development projects as it offers a fantastic learning opportunity about how to ‘do development’ well now and for the future. Without returning to look for what participants and partners valued enough to continue on their own, without returning to learn about unexpected sustained impacts, we rob ourselves of pivotal learning needed for success.

In part two, we look at ownership onward, planning for handover and lessons from who takes over? Country Nationals. In part three, we focus on Funding, Assumptions and Fears.

Please join us in advocating for this! Please think about your own projects… and whether you have considered these things, or need our help. We’re listening!

 

 

Sources:

[1] OECD. (2015, December 22). Detailed final 2014 aid figures released by OECD/DAC. Retrieved from http://www.oecd.org/dac/stats/final2014oda.htm

[2] Cekan, J., PhD, Kagendo, R., Towns, A. (2016). Participation by All: The Keys to Sustainability of a CRS Food Security Project in Niger. Retrieved from https://www.crs.org/our-work-overseas/research-publications/participation-all

[3] Carstarphen, N., PhD. (2013, November). Sustainable Investment in Local Capacity for Democracy and Peace: A Global Evaluation of Partners for Democratic Change. Retrieved from https://www.partnersglobal.org/resource/sustainable-investment-in-local-capacity-for-democracy-and-peace/

[4] Westerman, B., & Sheard, S. (2007, December). Sustainability Field Study – Understanding What Promotes Lasting Change at the Community Level. Retrieved from https://reliefweb.int/report/world/sustainability-field-study-understanding-what-promotes-lasting-change-community-level

[5] Cekan, J. (2013). Increasing women’s incomes, increasing peace: Unexpected lessons from Niger. Participatory Learning and Action, (66), 75-82. Retrieved from https://pubs.iied.org/G03661/

[6] Mayoux, L. & Valley Research Group (2008, June). Women Ending Poverty: The WORTH Program in Nepal – Empowerment through Literacy, Banking and Business 1999-2007. Retrieved from https://www.findevgateway.org/case-study/2008/06/women-ending-poverty-worth-program-nepal-empowerment-through-literacy-banking

 

Partners For Democratic Change: Promote Capacity Building to Achieve Self-Sustainability!

by | Feb 27, 2014 | Evaluation, Governance, Local Participants, Participation, Results, Sustainability

Catalyst organizations are those whose focus is on implementing programs with community level involvement during projects and local feedback loops to inform post-project evaluations for impact self-sustainability. An excellent example of this is Partners for Democratic Change, whose stated mission is, “to build sustainable capacity to advance civil society and a culture of change and conflict management worldwide,” focusing on initiating democratic practices through an approach called Sustainable Impact Investing. The goal of this approach is to foster the capacity of in-country organizations to “deliver systematic change,” with a focus on development that, “is bottom-up, locally-led rather than foreign-led, based on the belief that change comes from sustainable efforts led by local people, organizations and institutions invested in their own long-term future.”

To implement this progressive and participatory vision for sustainable development, Partners’ founded 22 Centers for Change and Conflict Management between 1989-2011, initially in the regions of Eastern and Central Europe. They later expanded to other regions struggling with democratic sociopolitical change. Partners’ conducted its own ex-post evaluation that averaged the results of 55 case studies that led to positive significant outcomes. The takeaway resulted in three main sustainability lessons:

  1. “The importance of investing in local partners and building their capacity to promote democratic change;
     
  2. the most pressing development challenges facing the world need to be addressed in a participatory manner with the input and shared commitment of government, businesses and civil society, which requires local leaders with sophisticated skills in change and conflict management;
     
  3. and finally, the work of social entrepreneurs to make a difference in their own countries is strengthened and legitimated by technical and relational support from an international network of like-minded professionals facing similar challenges.”

With these objectives in mind, the greatest positive outcome that was observed occurred in almost 90% of the 55 stories. This outcome was that development and participation of civil society is most commonly achieved through, “education, training, mentoring, coaching, partnerships and coalition building, organizational development and capacity building, and creating and enabling environment that supports civil society development, such as passing NGO laws.” Further, in 80% of cases, there was advancement of good governance by influencing the participation of civil society working with government on the issues listed above, specifically free and fair elections, human rights protection, etc. Another 50% of the cases increased access to justice and managing and resolving disputes/conflicts, thereby strengthening civil society, and about 40% of the stories focused on promoting inclusive societies, improving majority-minority relations, and increasing leadership capacity for women and youth as agents for social change. The overall result of Partners’ efforts resulted in substantial impacts. Since 1991, the Centers have trained around 15,000 mediators and worked directly with more than 300,000 participants, benefitting an estimated total of 17.5 million people – and these are even considered to be conservative estimates. 22 total Centers had been established, and 18 still exist today with a success rate of 82%.

Yet the Centers still faced challenges, most notably in, “institutionalizing the processes they used to achieve results so that impact can be maximized and sustainable.” While the Centers effectively managed to implement collaborative and participatory methods to attain these successful outcomes, without the ability to institutionalize these processes in local communities and government institutions, the likelihood of sustainability is threatened.

Herein lies the importance of valuing local voices and participation, as it is clear that successful development initiatives depend on working with the community rather than on behalf of the community. Collaborative efforts between local participants and the international organizations that aim to enhance socioeconomic development in their communities results in both farther reaching and more sustainable outcomes than projects that ignore local feedback. Partners also does a great job bridging the objectives of building organizational capacity to sustain programming while also ValuingVoices of participants regarding how that capacity will be beneficial to them. There is an obvious need throughout the development community to follow the good examples made by Partners for Democratic Change in order to promote greater levels of participation on the path to sustainable development.