What happens after the project ends? Lessons from post-project sustained impacts evaluations (Part 1)
What happens after the project ends?
Lessons from post-project sustained impacts evaluations (Part 1)
We talk a lot about impact of our interventions, but far less is analyzed about the sustained impact of our work in the years after projects close out. We take for granted that successful strategies will continue after projects shut down. Do they always? Maybe they do but we don’t know. Maybe there are innovations and impacts to be learned from…
To answer these question Valuing Voices spent 2 ½ years looking for and analyzing ‘post-project’ evaluations of projects undertaken 2-10 years after projects ended. The result: in our $137 billon international development industry, some 99% of projects remain unevaluated after project close out [1]. Only 17 agencies we have found so far have publically available post-project evaluations; most of them have one, while the OECD and JICA have dozens. Hundreds of studies recommend such learning that is missing from our industry’s program cycle (green slice).
Six decades on, this astonishing finding raises serious questions about stewardship of resources and commitment to learning—particularly learning from participant and partner stakeholders for whom sustainability matters most, and who are tasked with it over the long-term.
A review of post project evaluations generate food for thought about good program design and illustrate the value added of post project perspectives. This rapid review of select ‘ex-post’ evaluations points to three early lessons:
How we do it matters for great results
Expect unexpected results
Who takes over? Country Nationals
First, organizations go back to see how well their projects results were sustained. What we learned was that how well they used participatory processes in how they implemented and handed over mattered a lot for sustainability and that we must expect unexpected results.
How we do it matters for great results
1. Catholic Relief Services/ USAID PROSAN food security project in Niger
Valuing Voices evaluated this food security (agriculture, health and resilience project in 2015 which ran from 2007-2012 (report forthcoming). The $32 million project was implemented in a consortium of CRS, CARE and Helen Keller but this evaluation focused only on CRS areas.
- Interviews with over 500 participants found that three years after project closeout, 80% of project activities still continued, as did many village committees and there were a variety of community innovations [2]
- On average, households can feed themselves through their own production or purchase of food for 8-12 months three years after closeout compared to 6-9 months at closeout three years earlier. Such impact was unexpected. [2]
- 91% of respondents reported improved household health, hygiene, and nutrition [2]
- A 2.5-year participatory exit process from CRS to country stakeholders (local government, an array of local and international NGOs and the private sector ensured continuity and boosted local ownership [2]
- 20% of the activities did not continue, mostly food-assisted NRM and resilience-related [2]
- Youth make up 50% of the population and need to be engaged during the project for long term sustainability to occur. [2]
2. PartnersGlobal (formerly Partners for Democratic Change)
Their mission is to “to build sustainable capacity to advance civil society and a culture of change and conflict management worldwide” uses an approach that is “bottom-up, locally-led rather than foreign-led, based on the belief that change comes from sustainable efforts led by local people, organizations and institutions invested in their own long-term future.” They went back and to review 55 case studies of projects through 22 centers they founded in central and eastern Europe from 1989-2011. They found:
- In 80% of cases, there was advancement of good governance by influencing the participation of civil society working with government [3]
- In 50% of the cases there was increased access to justice and managing and resolving disputes/conflicts, thereby strengthening civil society [3]
- 18 of 22 of the centers that had been established still exist today (82%) [3].
3. Mercy Corps
They did a post-project evaluation in Central Asia in 2007, one and three years after two conflict resolution projects ended which were worth $18 million. These complex community mobilization programs with aims “to empower communities to work together in a participatory manner to address the infrastructure and social needs [while] developing sustainable skills [and] empowers communities to identify and utilize existing resources within the communities and not to depend only on external assistance.”
- 72% of youth report that they continue to use at least one skill they learned during the programs (e.g. teamwork and communication, and skills such as sewing, construction, roofing, journalism and cooking) [4]
- 68% of community members witnessed local government becoming more involved in community activities after the end of the programs as compared with before the programs [4]
- 57% of the communities studied continuing to use one or more of the decision-making practices promoted during the program [4]
- 42% of members, representing 70% of communities, reported that the community had worked collectively on new projects or repairs to existing infrastructure. Participants and partners had implemented almost 100 infrastructure projects by themselves independent of donor funds. [4]
These are terrific expected results.
We also learned to Expect unexpected results
4. Federation of the Red Cross and Ethiopian Red Cross
Valuing Voices combined a final evaluation of “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity” that had funding over $3 million in 2009 from the Swedish Red Cross with an assessment of projected sustainability. It was an IFRC/ERCS collaboration with the Ethiopian government to provide credit for food security inputs to 2,259 households, which were to be repaid in cash over time as well water and agriculture/ seedlings for environmental resilience. We answered the DAC criteria for evaluation and found the project overall to be quite good, albeit with weak data tracking systems.
In terms of sustainability, we used participatory methods to learn about what people felt they could self-sustain once the project left their area, so we could shape a similar follow-on project design to be moved elsewhere in Tigray, particularly around the credit for animals.
- While 87% of the loans had been promoted by the government and given for large animals (oxen and cows), and 13% was for small animals (sheep, goats, chickens)…
- But project participants we interviewed, strongly preferred the small animals in terms of being able to sustain them on their own. They felt they could afford these smaller amounts of credit as well as the feed to sustain them, without taking the risk of animal death leaving them with large debt. This was especially true for women, who preferred poultry to all other animals 15:1.
In our quest for fast results, are we asking participants to bear too much risk? As one of our Valuing Voices team asks, Who is responsible for sustainability?
From 2005-2007 Lutheran World Relief intervened in Niger, the world’s poorest country, with a $500,000 Pastoralist Survival and Recovery Programme (ARVIP) drought rehabilitation project funded by the Bill and Melinda Gates Foundation. There were numerous outcomes from targeting sheep, wells and animal fodder to 600 of the poorest women in 10 communities in northern Niger, among them:
- Women’s share of household income increased from 5% to 25% in some households. This was due to the value of the sheep grants, as well as time-savings used for income generation. Access to wells in five of the villages saved women a staggering 7-10 hours every other day from not having to go fetch water 3.5 hours away each way for household and animal needs and were free to weave mats or cook food for sale [5]
- Many said they didn’t have to resort to worse survival strategies during the next hungry season after they received the sheep [5]
What was not expected were these results:
- Many women in several villages reported an impact that was completely unexpected to the implementer and donor which was “our husbands don’t beat us anymore” [5]. This was thanks to both increased respect and income from the sheep as well as access to well-water which led to cleanliness and their ability to be home for their husbands, children and mothers-in-law, rather than fetching water whole days 2-3 times a week. The same was found in PACT’s WORTH empowerment and village banking project in Nepal that wrote, “one in 10 reported that WORTH has actually helped “change her life” because of its impact on domestic violence” [6].
- We defined success too narrowly. Many interviewees were content with the project even though prospects for project-expected drought resilience or sustained food security were less likely. Some women sold the sheep to buy food, pay their children’s school fees or their daughters’ dowries, while some had their sheep sold by their husbands who used them to buy other animals, pay for ceremonies or other expenses. Participants saw the project as bringing them resources and considered it a success. Spending assets on immediate needs is not at all illogical for a community who can feed itself only 4 months a year; for some households, their pressing needs far outweighed the luxury to wait and buffer seasonal food insecurity far down the line.
We hope you agree that allocating funds and attention to post-project sustained impacts evaluations is necessary for the remaining 99% of international development projects as it offers a fantastic learning opportunity about how to ‘do development’ well now and for the future. Without returning to look for what participants and partners valued enough to continue on their own, without returning to learn about unexpected sustained impacts, we rob ourselves of pivotal learning needed for success.
In part two, we look at ownership onward, planning for handover and lessons from who takes over? Country Nationals. In part three, we focus on Funding, Assumptions and Fears.
Please join us in advocating for this! Please think about your own projects… and whether you have considered these things, or need our help. We’re listening!
Sources:
[1] OECD. (2015, December 22). Detailed final 2014 aid figures released by OECD/DAC. Retrieved from http://www.oecd.org/dac/stats/final2014oda.htm
[2] Cekan, J., PhD, Kagendo, R., Towns, A. (2016). Participation by All: The Keys to Sustainability of a CRS Food Security Project in Niger. Retrieved from https://www.crs.org/our-work-overseas/research-publications/participation-all
[3] Carstarphen, N., PhD. (2013, November). Sustainable Investment in Local Capacity for Democracy and Peace: A Global Evaluation of Partners for Democratic Change. Retrieved from https://www.partnersglobal.org/resource/sustainable-investment-in-local-capacity-for-democracy-and-peace/
[4] Westerman, B., & Sheard, S. (2007, December). Sustainability Field Study – Understanding What Promotes Lasting Change at the Community Level. Retrieved from https://reliefweb.int/report/world/sustainability-field-study-understanding-what-promotes-lasting-change-community-level
[5] Cekan, J. (2013). Increasing women’s incomes, increasing peace: Unexpected lessons from Niger. Participatory Learning and Action, (66), 75-82. Retrieved from https://pubs.iied.org/G03661/
[6] Mayoux, L. & Valley Research Group (2008, June). Women Ending Poverty: The WORTH Program in Nepal – Empowerment through Literacy, Banking and Business 1999-2007. Retrieved from https://www.findevgateway.org/case-study/2008/06/women-ending-poverty-worth-program-nepal-empowerment-through-literacy-banking
