Learning from the Past… for Future Sustainability

by valuingvoicesjin | Feb 16, 2015 | Bolivia, Catholic Relief Services (CRS), Ethiopia, ex-post evaluation, Exit strategies, Food for Peace (FFP), Food security, Honduras, India, International aid, Kenya, post-project evaluation, Project design, self-sustainability, Sustainable development

Learning from the Past… for Future Sustainability

Heading up Food Security for Catholic Relief Services (CRS) was my first international development job in 1995-1999 and I have watched this organization grow in its commitment to program quality and learning/ knowledge management ever since.  At the time I oversaw 17 of of CRS' USAID/ Food for Peace (FFP) programs.  So I was delighted that not only has CRS done an ex-post evaluation and used the findings for programming (e.g. the effectiveness of investing in a particular sector—for example, the importance of supporting girls’ education within a food security program) and also for advocacy (e.g. evaluation lessons from Rwandan peace-building projects seven years after the genocide informed CRS’ evolving approach to peace and justice strategies), but I get to celebrate FFP learning too.  In addition to having consulted to USAID/PPL (Policy, Planning and Learning) and the FANTA project, all featured below, I went to Tufts University’s Fletcher School. Super to see great organizations learning about sustainability!

CRS’ 2007 project package guidance for implementation and guidance (ProPackII), described ex-post evaluation/sustainable impact evaluation’s aim “to determine which project interventions have been continued by project participants on their own [which] may contribute to future program design…. it is fair to say that NGOs rarely evaluate what remains following the withdrawal of project funding [which] is unfortunate [as] important lessons can be generated regarding factors that help to ensure project sustainability.”

A 2004 Catholic Relief Services excellent ex-post evaluation in Ethiopia was featured: “Looking at the past for better programming: dap I Ex-Post Assessment Report”. It assessed sustainability of Agriculture Natural Resource Management, and Food-Assisted Child Survival/Community Based Health Care programming, done as an internal evaluation by CRS staff and partners with document review, partner, government and community interviews. Results were mixed.

• Some activities generated enough food and income that households could eat throughout the year and have some savings, making them more resilient against drought
• Almost 100% of cropland bunding and irrigation practices for improved crop production were still being applied and buffered them during a subsequent drought
• Health practices had also continued (e.g. trained traditional birth attendants had continued to provide services with high levels of enthusiasm and commitment, and increased levels of health care-seeking behaviors existed).

However, many other benefits and services had severely deteriorated:

• Nearly all water committees had dissolved, fee collection was irregular or had been discontinued
• Many water schemes were not operational
• The centrally managed [tree] nurseries had been abandoned (given the existing management capacity of communities and government).

CRS/Ethiopia and its partners came to see that:

• “The potable water strategy had over-focused on the technical aspects (“hardware”) while not paying enough attention to the community organizing dimensions and support by existing government services (“software”).
• Even limited post-project follow-up by partners and government staff might have gone a long way towards mitigating the deterioration of project benefits and services.

What was terrific was that they “went on to use these findings and lessons learned from this ex-post evaluation to inform the design of similar projects in Ethiopia… while also raising awareness of these issues among partner staff”. The ex-post recommended increased planning for sustainability, setting up village management for post-project and incentive maintenance.  Great learning, yet we have found few ex-posts at CRS or elsewhere.  Our industry needs to explore issues such as those the evaluators posed: Was the lack of sustainability due to technical, institutional or financial faults in the programming? In other words, was the lack of self-sustainability due to the design/ aim of the activity itself or how it was implemented?

In 2013, USAID’s Food For Peace commissioned fascinating research on Exit Strategies.  Tufts University went to Bolivia, Honduras, India and Kenya which were phasing out of Title II food aid to look at how to “ensure that the benefits of interventions are sustained after they end, [as] there is little rigorous evidence on the effectiveness of different types of exit strategies used in USAID Office of Food for Peace Title II development food aid prog

rams.” The research is to “assess the extent to which the programs’ impacts were maintained or improved and to help understand factors of success or failure in the specific exit strategies that were used.” They have made the important discernment that the effectiveness of Title II programs depends on both short-term impact and long-term sustainability.

The FANTA project (contractor) made the following preliminary results available:

1. Impact assessment at exit does not consistently predict sustained impact two years later…. It can be misleading.
2. Many activities, practices, and impacts across sectors declined over the two years after exit. These declines are related to inadequate design and implementation of sustainability strategies and exit processes.
3. There are specific ways to increase the likelihood of sustainability: Sustaining service provision and beneficiary utilization of services and practices depends on three critical factors: Resources, Technical and Management Capacity, Motivation
4. Withdrawal of free food rations or any other free input (as incentive) jeopardizes sustainability without consideration of substitute incentives. For instance,

    

    

   • Withdrawal of food was a disincentive for participation in and provision of [child] growth monitoring…. Resources and health system linkages are needed to sustain health activities
   • Motivation, capacity and resources are all needed to maintain water systems
   • Agriculture and Natural Resource Management suffered greatly when resource incentives disappeared

Their main recommendations are that sustainability should be built into the design from the beginning, program cycles are longer and exit is gradual.

CRS found the same issue of incentives as a barrier, as they did technical and (institutional) capacity/ motivation/ management issues.  We have much to learn… at least we’ve started Valuing Voices and asking… and eventually designing for sustainability!