ReBlog Ex-Post Eval Week Tips for Conducting an Ex-Post Evaluation by Wendi Bevins

Repost:

Reblog: Eval Week: Tips for Conducting an Ex-Post Evaluation by Wendi Bevins

Wendi Bevins
Wendi Bevins

I am Wendi Bevins, and when my boss told me in 2017 that we were going to conduct an ex-post evaluation, I was thrilled. At the time I was the Monitoring and Evaluation Manager at Lutheran World Relief[1] (LWR).

I was thrilled to go back to a closed project and evaluate it against our original theory of change, test our assumptions, and see if our approaches were sustainable beyond the end of a project. We could really learn some valuable lessons. And all those things were true, but they came alongside some useful lessons about human nature, which I offer here as suggestions to anyone who wants to conduct an ex-post.

Hot Tip: Manage Expectations

Ex-post results, like any evaluation’s results, will not be entirely clear, and some will be disappointing. In my own experience, I pinned my hopes on an assumption and was disappointed when the result was murky. These kinds of realities do not mean that an ex-post is a bad investment, but they do mean that stakeholders should be prepared early in the planning stage to manage their expectations. Then stakeholders are more likely to use the findings effectively where they do point to a clear (if unexpected) path forward and they aren’t tempted to dismiss all the findings out of hand. While that is true for nearly all evaluations, it is especially true for ex-posts because the stakes feel higher.

Lessons Learned: How you apply findings matters almost as much as the findings themselves

Your ex-post will definitely yield findings. You will not know ahead of time what those findings are, and as I learned, you should resist the urge to guess what they will be. However, having a way to apply the findings is important. The value of the ex-post is tied up in an organization’s ability to make use of its findings. In the case of our first ex-post evaluation, we applied what we learned to an organization-wide strategy that was being developed as the ex-post ended. Like other evaluation findings, ex-post findings can also be valuable in the development of new tools or the design of new programming, but even better because they give a longer time frame for reflection.

Rad Resources: Practical Tips and Examples

In partnership with World Vision US, LWR compiled a set of practical tips for planning and carrying out an ex-post evaluation. It reflects our lessons learned over 11 ex-post evaluations across the two organizations, such as selecting the right project; hiring an evaluator; and staying future-oriented. LWR also condensed the most transferrable lessons from our two ex-post evaluations into user-friendly summaries: the Tanzanian Grape Value Chain project and Nicaraguan Gender-Sensitive Food Security project.

I hope if you are fortunate enough to participate in an ex-post evaluation that these tips give you a good head start.


[1] Lutheran World Relief and IMA World Health have since created Corus International, an ensemble of for-profit and non-profit organizations with social impact missions.

What happens after the project ends?  Lessons from post-project sustained impacts evaluations (Part 1)

 

What happens after the project ends?
Lessons from post-project sustained impacts evaluations
(Part 1)

 

We talk a lot about impact of our interventions, but far less is analyzed about the sustained impact of our work in the years after projects close out. We take for granted that successful strategies will continue after projects shut down.  Do they always? Maybe they do but we don’t know. Maybe there are innovations and impacts to be learned from…

To answer these question Valuing Voices spent 2 ½ years looking for and analyzing ‘post-project’ evaluations of projects undertaken 2-10 years after projects ended.  The result: in our $137 billon international development industry, some 99% of projects remain unevaluated after project close out [1]Only 17 agencies we have found so far have publically available post-project evaluations; most of them have one, while the OECD and JICA have dozens.  Hundreds of studies recommend such learning that is missing from our industry’s program cycle (green slice).

 

What_have_we_learned_from_postproject_sustainability_impact_Pt1and2_0216_docx

 

Six decades on, this astonishing finding raises serious questions about stewardship of resources and commitment to learning—particularly learning from participant and partner stakeholders for whom sustainability matters most, and who are tasked with it over the long-term.

A review of post project evaluations generate food for thought about good program design and illustrate the value added of post project perspectives. This rapid review of select ‘ex-post’ evaluations points to three early lessons:

How we do it matters for great results
Expect unexpected results
Who takes over? Country Nationals

First, organizations go back to see how well their projects results were sustained. What we learned was that how well they used participatory processes in how they implemented and handed over mattered a lot for sustainability and that we must expect unexpected results.

 

How we do it matters for great results

1. Catholic Relief Services/ USAID PROSAN food security project in Niger

Valuing Voices evaluated this food security (agriculture, health and resilience project in 2015 which ran from 2007-2012 (report forthcoming). The $32 million project was implemented in a consortium of CRS, CARE and Helen Keller but this evaluation focused only on CRS areas.

  • Interviews with over 500 participants found that three years after project closeout, 80% of project activities still continued, as did many village committees and there were a variety of community innovations [2]
  • On average, households can feed themselves through their own production or purchase of food for 8-12 months three years after closeout compared to 6-9 months at closeout three years earlier. Such impact was unexpected. [2]
  • 91% of respondents reported improved household health, hygiene, and nutrition [2]
  • A 2.5-year participatory exit process from CRS to country stakeholders (local government, an array of local and international NGOs and the private sector ensured continuity and boosted local ownership [2]
  • 20% of the activities did not continue, mostly food-assisted NRM and resilience-related [2]
  • Youth make up 50% of the population and need to be engaged during the project for long term sustainability to occur. [2]

 

2. PartnersGlobal (formerly Partners for Democratic Change)

Their mission is to “to build sustainable capacity to advance civil society and a culture of change and conflict management worldwide” uses an approach that is “bottom-up, locally-led rather than foreign-led, based on the belief that change comes from sustainable efforts led by local people, organizations and institutions invested in their own long-term future.” They went back and to review 55 case studies of projects through 22 centers they founded in central and eastern Europe from 1989-2011. They found:

  • In 80% of cases, there was advancement of good governance by influencing the participation of civil society working with government [3]
  • In 50% of the cases there was increased access to justice and managing and resolving disputes/conflicts, thereby strengthening civil society [3]
  • 18 of 22 of the centers that had been established still exist today (82%) [3].

 

3. Mercy Corps

They did a post-project evaluation in Central Asia in 2007, one and three years after two conflict resolution projects ended which were worth $18 million. These complex community mobilization programs with aims “to empower communities to work together in a participatory manner to address the infrastructure and social needs [while] developing sustainable skills [and] empowers communities to identify and utilize existing resources within the communities and not to depend only on external assistance.”

  • 72% of youth report that they continue to use at least one skill they learned during the programs (e.g. teamwork and communication, and skills such as sewing, construction, roofing, journalism and cooking) [4]
  • 68% of community members witnessed local government becoming more involved in community activities after the end of the programs as compared with before the programs [4]
  • 57% of the communities studied continuing to use one or more of the decision-making practices promoted during the program [4]
  • 42% of members, representing 70% of communities, reported that the community had worked collectively on new projects or repairs to existing infrastructure. Participants and partners had implemented almost 100 infrastructure projects by themselves independent of donor funds. [4]

 

 

These are terrific expected results.
We also learned to Expect unexpected results

4. Federation of the Red Cross and Ethiopian Red Cross

Valuing Voices combined a final evaluation of “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity” that had funding over $3 million in 2009 from the Swedish Red Cross with an assessment of projected sustainability.  It was an IFRC/ERCS collaboration with the Ethiopian government to provide credit for food security inputs to 2,259 households, which were to be repaid in cash over time as well water and agriculture/ seedlings for environmental resilience.  We answered the DAC criteria for evaluation and found the project overall to be quite good, albeit with weak data tracking systems.

In terms of sustainability, we used participatory methods to learn about what people felt they could self-sustain once the project left their area, so we could shape a similar follow-on project design to be moved elsewhere in Tigray, particularly around the credit for animals.

  • While 87% of the loans had been promoted by the government and given for large animals (oxen and cows), and 13% was for small animals (sheep, goats, chickens)…
  • But project participants we interviewed, strongly preferred the small animals in terms of being able to sustain them on their own. They felt they could afford these smaller amounts of credit as well as the feed to sustain them, without taking the risk of animal death leaving them with large debt. This was especially true for women, who preferred poultry to all other animals 15:1.

In our quest for fast results, are we asking participants to bear too much risk? As one of our Valuing Voices team asks, Who is responsible for sustainability?

 

5. Lutheran World Relief

From 2005-2007 Lutheran World Relief intervened in Niger, the world’s poorest country, with a $500,000 Pastoralist Survival and Recovery Programme (ARVIP) drought rehabilitation project funded by the Bill and Melinda Gates Foundation. There were numerous outcomes from targeting sheep, wells and animal fodder to 600 of the poorest women in 10 communities in northern Niger, among them:

  • Women’s share of household income increased from 5% to 25% in some households. This was due to the value of the sheep grants, as well as time-savings used for income generation. Access to wells in five of the villages saved women a staggering 7-10 hours every other day from not having to go fetch water 3.5 hours away each way for household and animal needs and were free to weave mats or cook food for sale [5]
  • Many said they didn’t have to resort to worse survival strategies during the next hungry season after they received the sheep [5]

What was not expected were these results:

  • Many women in several villages reported an impact that was completely unexpected to the implementer and donor which was “our husbands don’t beat us anymore” [5]. This was thanks to both increased respect and income from the sheep as well as access to well-water which led to cleanliness and their ability to be home for their husbands, children and mothers-in-law, rather than fetching water whole days 2-3 times a week. The same was found in PACT’s WORTH empowerment and village banking project in Nepal that wrote, “one in 10 reported that WORTH has actually helped “change her life” because of its impact on domestic violence” [6].
  • We defined success too narrowly. Many interviewees were content with the project even though prospects for project-expected drought resilience or sustained food security were less likely. Some women sold the sheep to buy food, pay their children’s school fees or their daughters’ dowries, while some had their sheep sold by their husbands who used them to buy other animals, pay for ceremonies or other expenses.  Participants saw the project as bringing them resources and considered it a success. Spending assets on immediate needs is not at all illogical for a community who can feed itself only 4 months a year; for some households, their pressing needs far outweighed the luxury to wait and buffer seasonal food insecurity far down the line.

 

We hope you agree that allocating funds and attention to post-project sustained impacts evaluations is necessary for the remaining 99% of international development projects as it offers a fantastic learning opportunity about how to ‘do development’ well now and for the future. Without returning to look for what participants and partners valued enough to continue on their own, without returning to learn about unexpected sustained impacts, we rob ourselves of pivotal learning needed for success.

In part two, we look at ownership onward, planning for handover and lessons from who takes over? Country Nationals. In part three, we focus on Funding, Assumptions and Fears.

Please join us in advocating for this! Please think about your own projects… and whether you have considered these things, or need our help. We’re listening!

 

 

Sources:

[1] OECD. (2015, December 22). Detailed final 2014 aid figures released by OECD/DAC. Retrieved from http://www.oecd.org/dac/stats/final2014oda.htm

[2] Cekan, J., PhD, Kagendo, R., Towns, A. (2016). Participation by All: The Keys to Sustainability of a CRS Food Security Project in Niger. Retrieved from https://www.crs.org/our-work-overseas/research-publications/participation-all

[3] Carstarphen, N., PhD. (2013, November). Sustainable Investment in Local Capacity for Democracy and Peace: A Global Evaluation of Partners for Democratic Change. Retrieved from https://www.partnersglobal.org/resource/sustainable-investment-in-local-capacity-for-democracy-and-peace/

[4] Westerman, B., & Sheard, S. (2007, December). Sustainability Field Study – Understanding What Promotes Lasting Change at the Community Level. Retrieved from https://reliefweb.int/report/world/sustainability-field-study-understanding-what-promotes-lasting-change-community-level

[5] Cekan, J. (2013). Increasing women’s incomes, increasing peace: Unexpected lessons from Niger. Participatory Learning and Action, (66), 75-82. Retrieved from https://pubs.iied.org/G03661/

[6] Mayoux, L. & Valley Research Group (2008, June). Women Ending Poverty: The WORTH Program in Nepal – Empowerment through Literacy, Banking and Business 1999-2007. Retrieved from https://www.findevgateway.org/case-study/2008/06/women-ending-poverty-worth-program-nepal-empowerment-through-literacy-banking