Assuming Sustainability and Impact is Dangerous to Development (+ OECD/DAC evaluation criteria)

 

Assuming Sustainability and Impact is Dangerous to Development
(+ OECD/ DAC evaluation criteria)

 

We all do it; well, I used to do it too. I used to assume that if I helped my field staff and partners target and design funded projects well enough, and try to ensure a high quality of implementation and M&E, then it would result in sustainable programming. I assumed we would have moved our participants and partners toward projected long-term, top-of-logical-framework’s aspirational impact such as “vibrant agriculture leading to no hunger”, “locally sustained maternal child health and nutrition”, “self-sustained ecosystems”.

INTRAC nicely differentiates between what is typically measured (“outputs can only ever be the deliverables of a project or programme…that are largely within the control of an agency”) and what is not: “impact as the lasting or significant changes in people’s lives brought about by an intervention or interventions” [1]. They continue: “as few organisations are really judged on their impact, the OECD DAC impact definition (“positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended“) allows for long-term changes in institutional capacity or policy change to be classed as impact” [1]. Do we do this? Virtually never. 99% of the time we only evaluate what happened while the project and its results is under the control of the aid implementer. Yet the five OECD/DAC evaluation criteria asks us to evaluate relevance, effectiveness, efficiency (fair enough, this is important to know if a project was good) and also impact and sustainability. So in addition to the prescription to evaluate ‘long-term effects’ (impact), evaluators are to measure “whether the benefits of an activity are likely to continue after donor funding has been withdrawn… [including being] environmentally as well as financially sustainable” [2]. 

How do we know we are getting to sustained outcomes and impacts? We ask people on the receiving end ideally after projects end. It is dangerous to assume sustainability and impact, and assume positive development trajectories (Sridharan) unless we consistently do “ex-post” project evaluations such as these from our research or catalytic organizations that have done at least one ex-post. At very minimum we should evaluate projected sustainability at end of project with those tasked to sustain it before the same project is repeated. Unfortunately we rarely do so and the assumed sustainability is so often not borne out, as I presented at the European Evaluation Society conference Sustainability panel two weeks ago along with AusAid’s DFAT, the World Bank, University College London and UNFEM.

 

 

Will we ever know if we have gotten to sustained impacts? Not unless the OECD/DAC criteria are drastically updated and organizations evaluate most projects ex-post (not just good ones :)), learn from the results and fund and implement for country-led sustainability with the country nationals. We must, as Sanjeev Sridharan tells us in a forthcoming paper embed sustainability into our Theories of Change from the onset (“Till time (and poor planning) do us part: Programs as dynamic systems — Incorporating planning of sustainability into theories of change” (Canadian Journal of Program Evaluation, 2018).*

There are remarkable assumptions routinely made. Many projects put sustainability into the proposal, yet most close out projects in the last 6 months. Rarely do projects take the time to properly phase down or phase over (unlike CRS Niger); many exit ceremonially ‘handing over’ projects to country-nationals, disposing project assets, and leaving only a final report behind. Alternatively, this USAID Uganda CDCS Country Transition Plan which looks over 20 years in the future by when it assumes to have accomplished sustained impact for exit [3]. Maybe they will measure progress towards that goal and orient programs toward handover, as in the new USAID “Journey to Self-Reliance” – we hope! Truly, we can plan to exit, but only when data bears out our sustained impact, not when the money or political will runs out.

As OXFAM’s blog today on the evaluation criteria says, “Sustainability is often treated as an assessment of whether an output is likely to be sustained after the end of the project. No one, well, hardly anyone, ever measures sustainability in terms of understanding whether we are meeting the needs of the present, without compromising the ability of future generations to meet their own need” and “too often in development we evaluate a project or programme and claim impact in a very narrow sense rather than the broader ecology beyond project or programme parameters” [4]. In fact, most ‘impact evaluations’ actually test effectiveness rather than long-term impact. Too rarely do we test impact assumptions by returning 2-10 years later and gather proof of what impacted locals’ lives sustainably, much less – importantly – what emerged from their own efforts once we left (SEIEs)! Oh, our hubris.

if you’re interested in the European Evaluation Society’s DAC criteria update discussion, see flagship discussion and Zenda Offir’s blog which stresses the need for better design that include ownership, inclusivity, empowerment [5][6]. These new evaluation criteria need to be updated, including Florence Etta’s and AGDEN‘s additional criteria participation, non-discrimination and accountability!

 

 

We can no longer afford to spend resources without listening to our true clients – those tasked with sustaining the impacts after we pack up – our partners and participants. We can no longer fund what cannot be proven to be sustained that is impactful. We talk about effectiveness and country ownership (which is paramount for sustainability and long-term impact), with an OECD report (2018) found “increases [in[ aid effectiveness by reducing transaction costs and improving recipient countries ownership” [7]. Yet donor governments who ‘tie’ aid to their own country national’s contracts benefit a staggering amount from ‘aid’ given. “Australia and the United Kingdom both reported … 93 percent and 90 percent of the value of their contracts respectively went to their own firms” [7]. It is not so different in the USA where aid is becoming bureaucratically centralized in the hands of a few for-profit contractors and centralized hundreds of millions in a handful of contracts. We must Do Development Differently. We can’t be the prime beneficiaries of our own aid; accountability must be to our participants; is it their countries, not our projects, and we cannot keep dangerously assuming sustained impact. Please let us know what you think…

 

 

Footnotes:

[*] This paper is now available at https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/53055

 

Sources:

[1] Simister, N. (2015). Monitoring and Evaluation Series: Outcomes Outputs and Impact. Retrieved from https://www.intrac.org/wpcms/wp-content/uploads/2016/06/Monitoring-and-Evaluation-Series-Outcomes-Outputs-and-Impact-7.pdf

[2] OECD. (n.d.). DAC Criteria for Evaluating Development Assistance. Retrieved September, 2018, from https://web.archive.org/web/20180919035910/http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm

[3] USAID. (2016, December 6). USAID Uganda Country Development Cooperation Strategy 2016-2021. Retrieved October, 2018, from https://www.usaid.gov/uganda/cdcs

[4] Porter, S. (2018, October 18). DAC Criteria: The Hand That Rocks the Cradle. Retrieved from https://views-voices.oxfam.org.uk/2018/10/dac-criteria-the-hand-that-rocks-the-cradle/

[5] European Evaluation Society Biennial Conference: Flagship Symposia. (2018). Retrieved from http://www.ees2018.eu/1539782596-flagship-symposia.htm

[6] Ofir, Z. (2018, October 13). Updating the DAC Criteria, Part 11 (FINAL). From Evaluation Criteria to Design Principles. Retrieved from https://zendaofir.com/dac-criteria-part-11/

[7] OECD. (2018, June 11). 2018 Report On The DAC Untying Recommendation. Retrieved from http://www.oecd.org/dac/financing-sustainable-development/development-finance-standards/DCD-DAC(2018)12-REV2.en.pdf

 

What happens after the project ends?  Lessons from post-project sustained impacts evaluations (Part 1)

 

What happens after the project ends?
Lessons from post-project sustained impacts evaluations
(Part 1)

 

We talk a lot about impact of our interventions, but far less is analyzed about the sustained impact of our work in the years after projects close out. We take for granted that successful strategies will continue after projects shut down.  Do they always? Maybe they do but we don’t know. Maybe there are innovations and impacts to be learned from…

To answer these question Valuing Voices spent 2 ½ years looking for and analyzing ‘post-project’ evaluations of projects undertaken 2-10 years after projects ended.  The result: in our $137 billon international development industry, some 99% of projects remain unevaluated after project close out [1]Only 17 agencies we have found so far have publically available post-project evaluations; most of them have one, while the OECD and JICA have dozens.  Hundreds of studies recommend such learning that is missing from our industry’s program cycle (green slice).

 

What_have_we_learned_from_postproject_sustainability_impact_Pt1and2_0216_docx

 

Six decades on, this astonishing finding raises serious questions about stewardship of resources and commitment to learning—particularly learning from participant and partner stakeholders for whom sustainability matters most, and who are tasked with it over the long-term.

A review of post project evaluations generate food for thought about good program design and illustrate the value added of post project perspectives. This rapid review of select ‘ex-post’ evaluations points to three early lessons:

How we do it matters for great results
Expect unexpected results
Who takes over? Country Nationals

First, organizations go back to see how well their projects results were sustained. What we learned was that how well they used participatory processes in how they implemented and handed over mattered a lot for sustainability and that we must expect unexpected results.

 

How we do it matters for great results

1. Catholic Relief Services/ USAID PROSAN food security project in Niger

Valuing Voices evaluated this food security (agriculture, health and resilience project in 2015 which ran from 2007-2012 (report forthcoming). The $32 million project was implemented in a consortium of CRS, CARE and Helen Keller but this evaluation focused only on CRS areas.

  • Interviews with over 500 participants found that three years after project closeout, 80% of project activities still continued, as did many village committees and there were a variety of community innovations [2]
  • On average, households can feed themselves through their own production or purchase of food for 8-12 months three years after closeout compared to 6-9 months at closeout three years earlier. Such impact was unexpected. [2]
  • 91% of respondents reported improved household health, hygiene, and nutrition [2]
  • A 2.5-year participatory exit process from CRS to country stakeholders (local government, an array of local and international NGOs and the private sector ensured continuity and boosted local ownership [2]
  • 20% of the activities did not continue, mostly food-assisted NRM and resilience-related [2]
  • Youth make up 50% of the population and need to be engaged during the project for long term sustainability to occur. [2]

 

2. PartnersGlobal (formerly Partners for Democratic Change)

Their mission is to “to build sustainable capacity to advance civil society and a culture of change and conflict management worldwide” uses an approach that is “bottom-up, locally-led rather than foreign-led, based on the belief that change comes from sustainable efforts led by local people, organizations and institutions invested in their own long-term future.” They went back and to review 55 case studies of projects through 22 centers they founded in central and eastern Europe from 1989-2011. They found:

  • In 80% of cases, there was advancement of good governance by influencing the participation of civil society working with government [3]
  • In 50% of the cases there was increased access to justice and managing and resolving disputes/conflicts, thereby strengthening civil society [3]
  • 18 of 22 of the centers that had been established still exist today (82%) [3].

 

3. Mercy Corps

They did a post-project evaluation in Central Asia in 2007, one and three years after two conflict resolution projects ended which were worth $18 million. These complex community mobilization programs with aims “to empower communities to work together in a participatory manner to address the infrastructure and social needs [while] developing sustainable skills [and] empowers communities to identify and utilize existing resources within the communities and not to depend only on external assistance.”

  • 72% of youth report that they continue to use at least one skill they learned during the programs (e.g. teamwork and communication, and skills such as sewing, construction, roofing, journalism and cooking) [4]
  • 68% of community members witnessed local government becoming more involved in community activities after the end of the programs as compared with before the programs [4]
  • 57% of the communities studied continuing to use one or more of the decision-making practices promoted during the program [4]
  • 42% of members, representing 70% of communities, reported that the community had worked collectively on new projects or repairs to existing infrastructure. Participants and partners had implemented almost 100 infrastructure projects by themselves independent of donor funds. [4]

 

 

These are terrific expected results.
We also learned to Expect unexpected results

4. Federation of the Red Cross and Ethiopian Red Cross

Valuing Voices combined a final evaluation of “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity” that had funding over $3 million in 2009 from the Swedish Red Cross with an assessment of projected sustainability.  It was an IFRC/ERCS collaboration with the Ethiopian government to provide credit for food security inputs to 2,259 households, which were to be repaid in cash over time as well water and agriculture/ seedlings for environmental resilience.  We answered the DAC criteria for evaluation and found the project overall to be quite good, albeit with weak data tracking systems.

In terms of sustainability, we used participatory methods to learn about what people felt they could self-sustain once the project left their area, so we could shape a similar follow-on project design to be moved elsewhere in Tigray, particularly around the credit for animals.

  • While 87% of the loans had been promoted by the government and given for large animals (oxen and cows), and 13% was for small animals (sheep, goats, chickens)…
  • But project participants we interviewed, strongly preferred the small animals in terms of being able to sustain them on their own. They felt they could afford these smaller amounts of credit as well as the feed to sustain them, without taking the risk of animal death leaving them with large debt. This was especially true for women, who preferred poultry to all other animals 15:1.

In our quest for fast results, are we asking participants to bear too much risk? As one of our Valuing Voices team asks, Who is responsible for sustainability?

 

5. Lutheran World Relief

From 2005-2007 Lutheran World Relief intervened in Niger, the world’s poorest country, with a $500,000 Pastoralist Survival and Recovery Programme (ARVIP) drought rehabilitation project funded by the Bill and Melinda Gates Foundation. There were numerous outcomes from targeting sheep, wells and animal fodder to 600 of the poorest women in 10 communities in northern Niger, among them:

  • Women’s share of household income increased from 5% to 25% in some households. This was due to the value of the sheep grants, as well as time-savings used for income generation. Access to wells in five of the villages saved women a staggering 7-10 hours every other day from not having to go fetch water 3.5 hours away each way for household and animal needs and were free to weave mats or cook food for sale [5]
  • Many said they didn’t have to resort to worse survival strategies during the next hungry season after they received the sheep [5]

What was not expected were these results:

  • Many women in several villages reported an impact that was completely unexpected to the implementer and donor which was “our husbands don’t beat us anymore” [5]. This was thanks to both increased respect and income from the sheep as well as access to well-water which led to cleanliness and their ability to be home for their husbands, children and mothers-in-law, rather than fetching water whole days 2-3 times a week. The same was found in PACT’s WORTH empowerment and village banking project in Nepal that wrote, “one in 10 reported that WORTH has actually helped “change her life” because of its impact on domestic violence” [6].
  • We defined success too narrowly. Many interviewees were content with the project even though prospects for project-expected drought resilience or sustained food security were less likely. Some women sold the sheep to buy food, pay their children’s school fees or their daughters’ dowries, while some had their sheep sold by their husbands who used them to buy other animals, pay for ceremonies or other expenses.  Participants saw the project as bringing them resources and considered it a success. Spending assets on immediate needs is not at all illogical for a community who can feed itself only 4 months a year; for some households, their pressing needs far outweighed the luxury to wait and buffer seasonal food insecurity far down the line.

 

We hope you agree that allocating funds and attention to post-project sustained impacts evaluations is necessary for the remaining 99% of international development projects as it offers a fantastic learning opportunity about how to ‘do development’ well now and for the future. Without returning to look for what participants and partners valued enough to continue on their own, without returning to learn about unexpected sustained impacts, we rob ourselves of pivotal learning needed for success.

In part two, we look at ownership onward, planning for handover and lessons from who takes over? Country Nationals. In part three, we focus on Funding, Assumptions and Fears.

Please join us in advocating for this! Please think about your own projects… and whether you have considered these things, or need our help. We’re listening!

 

 

Sources:

[1] OECD. (2015, December 22). Detailed final 2014 aid figures released by OECD/DAC. Retrieved from http://www.oecd.org/dac/stats/final2014oda.htm

[2] Cekan, J., PhD, Kagendo, R., Towns, A. (2016). Participation by All: The Keys to Sustainability of a CRS Food Security Project in Niger. Retrieved from https://www.crs.org/our-work-overseas/research-publications/participation-all

[3] Carstarphen, N., PhD. (2013, November). Sustainable Investment in Local Capacity for Democracy and Peace: A Global Evaluation of Partners for Democratic Change. Retrieved from https://www.partnersglobal.org/resource/sustainable-investment-in-local-capacity-for-democracy-and-peace/

[4] Westerman, B., & Sheard, S. (2007, December). Sustainability Field Study – Understanding What Promotes Lasting Change at the Community Level. Retrieved from https://reliefweb.int/report/world/sustainability-field-study-understanding-what-promotes-lasting-change-community-level

[5] Cekan, J. (2013). Increasing women’s incomes, increasing peace: Unexpected lessons from Niger. Participatory Learning and Action, (66), 75-82. Retrieved from https://pubs.iied.org/G03661/

[6] Mayoux, L. & Valley Research Group (2008, June). Women Ending Poverty: The WORTH Program in Nepal – Empowerment through Literacy, Banking and Business 1999-2007. Retrieved from https://www.findevgateway.org/case-study/2008/06/women-ending-poverty-worth-program-nepal-empowerment-through-literacy-banking