Reblog: Eval Week: Tips for Conducting an Ex-Post Evaluation by Wendi Bevins
Published on the American Evaluation Association AEA365blog series https://aea365.org/blog/ex-post-eval-week-tips-for-conducting-an-ex-post-evaluation-by-wendi-bevins/ January 19, 2021
I am Wendi Bevins, and when my boss told me in 2017 that we were going to conduct an ex-post evaluation, I was thrilled. At the time I was the Monitoring and Evaluation Manager at Lutheran World Relief (LWR).
I was thrilled to go back to a closed project and evaluate it against our original theory of change, test our assumptions, and see if our approaches were sustainable beyond the end of a project. We could really learn some valuable lessons. And all those things were true, but they came alongside some useful lessons about human nature, which I offer here as suggestions to anyone who wants to conduct an ex-post.
Hot Tip: Manage Expectations
Ex-post results, like any evaluation’s results, will not be entirely clear, and some will be disappointing. In my own experience, I pinned my hopes on an assumption and was disappointed when the result was murky. These kinds of realities do not mean that an ex-post is a bad investment, but they do mean that stakeholders should be prepared early in the planning stage to manage their expectations. Then stakeholders are more likely to use the findings effectively where they do point to a clear (if unexpected) path forward and they aren’t tempted to dismiss all the findings out of hand. While that is true for nearly all evaluations, it is especially true for ex-posts because the stakes feel higher.
Lessons Learned: How you apply findings matters almost as much as the findings themselves
Your ex-post will definitely yield findings. You will not know ahead of time what those findings are, and as I learned, you should resist the urge to guess what they will be. However, having a way to apply the findings is important. The value of the ex-post is tied up in an organization’s ability to make use of its findings. In the case of our first ex-post evaluation, we applied what we learned to an organization-wide strategy that was being developed as the ex-post ended. Like other evaluation findings, ex-post findings can also be valuable in the development of new tools or the design of new programming, but even better because they give a longer time frame for reflection.
Rad Resources: Practical Tips and Examples
In partnership with World Vision US, LWR compiled a set of practical tips for planning and carrying out an ex-post evaluation. It reflects our lessons learned over 11 ex-post evaluations across the two organizations, such as selecting the right project; hiring an evaluator; and staying future-oriented. LWR also condensed the most transferrable lessons from our two ex-post evaluations into user-friendly summaries: the Tanzanian Grape Value Chain project and Nicaraguan Gender-Sensitive Food Security project.
I hope if you are fortunate enough to participate in an ex-post evaluation that these tips give you a good head start.
 Lutheran World Relief and IMA World Health have since created Corus International, an ensemble of for-profit and non-profit organizations with social impact missions.
What happens after the project ends?
Lessons from post-project sustained impacts evaluations (Part 1)
We talk a lot about impact of our interventions, but far less is analyzed about the sustained impact of our work in the years after projects close out. We take for granted that successful strategies will continue after projects shut down. Do they always? Maybe they do but we don’t know. Maybe there are innovations and impacts to be learned from…
To answer these question Valuing Voices spent 2 ½ years looking for and analyzing ‘post-project’ evaluations of projects undertaken 2-10 years after projects ended. The result: in our $137 billon international development industry, some 99% of projects remain unevaluated after project close out . Only 17 agencies we have found so far have publically available post-project evaluations; most of them have one, while the OECD and JICA have dozens. Hundreds of studies recommend such learning that is missing from our industry’s program cycle (green slice).
A review of post project evaluations generate food for thought about good program design and illustrate the value added of post project perspectives. This rapid review of select ‘ex-post’ evaluations points to three early lessons:
How we do it matters for great results Expect unexpected results Who takes over? Country Nationals
First, organizations go back to see how well their projects results were sustained. What we learned was that how well they used participatory processes in how they implemented and handed over mattered a lot for sustainability and that we must expect unexpected results.
Valuing Voices evaluated this food security (agriculture, health and resilience project in 2015 which ran from 2007-2012 (report forthcoming). The $32 million project was implemented in a consortium of CRS, CARE and Helen Keller but this evaluation focused only on CRS areas.
Interviews with over 500 participants found that three years after project closeout, 80% of project activities still continued, as did many village committees and there were a variety of community innovations 
On average, households can feed themselves through their own production or purchase of food for 8-12 months three years after closeout compared to 6-9 months at closeout three years earlier. Such impact was unexpected. 
91% of respondents reported improved household health, hygiene, and nutrition 
A 2.5-year participatory exit process from CRS to country stakeholders (local government, an array of local and international NGOs and the private sector ensured continuity and boosted local ownership 
20% of the activities did not continue, mostly food-assisted NRM and resilience-related 
Youth make up 50% of the population and need to be engaged during the project for long term sustainability to occur. 
Their mission is to “to build sustainable capacity to advance civil society and a culture of change and conflict management worldwide” uses an approach that is “bottom-up, locally-led rather than foreign-led, based on the belief that change comes from sustainable efforts led by local people, organizations and institutions invested in their own long-term future.” They went back and to review 55 case studies of projects through 22 centers they founded in central and eastern Europe from 1989-2011. They found:
In 80% of cases, there was advancement of good governance by influencing the participation of civil society working with government 
In 50% of the cases there was increasedaccess to justice and managing and resolving disputes/conflicts, thereby strengthening civil society 
18 of 22 of the centers that had been established still exist today (82%) .
They did a post-project evaluation in Central Asia in 2007, one and three years after two conflict resolution projects ended which were worth $18 million. These complex community mobilization programs with aims “to empower communities to work together in a participatory manner to address the infrastructure and social needs [while] developing sustainable skills [and] empowers communities to identify and utilize existing resources within the communities and not to depend only on external assistance.”
72% of youth report that they continue to use at least one skill they learned during the programs (e.g. teamwork and communication, and skills such as sewing, construction, roofing, journalism and cooking) 
68% of community members witnessed local government becoming more involved in community activities after the end of the programs as compared with before the programs 
57% of the communities studied continuing to use one or more of the decision-making practices promoted during the program 
42% of members, representing 70% of communities, reported that the community had worked collectively on new projects or repairs to existing infrastructure. Participants and partners had implemented almost 100 infrastructure projects by themselves independent of donor funds. 
These are terrific expected results. We also learned to Expect unexpected results
Valuing Voices combined a final evaluation of “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity” that had funding over $3 million in 2009 from the Swedish Red Cross with an assessment of projected sustainability. It was an IFRC/ERCS collaboration with the Ethiopian government to provide credit for food security inputs to 2,259 households, which were to be repaid in cash over time as well water and agriculture/ seedlings for environmental resilience. We answered the DAC criteria for evaluation and found the project overall to be quite good, albeit with weak data tracking systems.
In terms of sustainability, we used participatory methods to learn about what people felt they could self-sustain once the project left their area, so we could shape a similar follow-on project design to be moved elsewhere in Tigray, particularly around the credit for animals.
While 87% of the loans had been promoted by the government and given for large animals (oxen and cows), and 13% was for small animals (sheep, goats, chickens)…
But project participants we interviewed, strongly preferred the small animals in terms of being able to sustain them on their own. They felt they could afford these smaller amounts of credit as well as the feed to sustain them, without taking the risk of animal death leaving them with large debt. This was especially true for women, who preferred poultry to all other animals 15:1.
From 2005-2007 Lutheran World Relief intervened in Niger, the world’s poorest country, with a $500,000 Pastoralist Survival and Recovery Programme (ARVIP) drought rehabilitation project funded by the Bill and Melinda Gates Foundation. There were numerous outcomes from targeting sheep, wells and animal fodder to 600 of the poorest women in 10 communities in northern Niger, among them:
Women’s share of household income increased from 5% to 25% in some households. This was due to the value of the sheep grants, as well as time-savings used for income generation. Access to wells in five of the villages saved women a staggering 7-10 hours every other day from not having to go fetch water 3.5 hours away each way for household and animal needs and were free to weave mats or cook food for sale 
Many said they didn’t have to resort to worse survival strategies during the next hungry season after they received the sheep 
What was not expected were these results:
Many women in several villages reported an impact that was completely unexpected to the implementer and donor which was “our husbands don’t beat us anymore” . This was thanks to both increased respect and income from the sheep as well as access to well-water which led to cleanliness and their ability to be home for their husbands, children and mothers-in-law, rather than fetching water whole days 2-3 times a week. The same was found in PACT’s WORTH empowerment and village banking project in Nepal that wrote, “one in 10 reported that WORTH has actually helped “change her life” because of its impact on domestic violence” .
We defined success too narrowly. Many interviewees were content with the project even though prospects for project-expected drought resilience or sustained food security were less likely. Some women sold the sheep to buy food, pay their children’s school fees or their daughters’ dowries, while some had their sheep sold by their husbands who used them to buy other animals, pay for ceremonies or other expenses. Participants saw the project as bringing them resources and considered it a success. Spending assets on immediate needs is not at all illogical for a community who can feed itself only 4 months a year; for some households, their pressing needs far outweighed the luxury to wait and buffer seasonal food insecurity far down the line.
We hope you agree that allocating funds and attention to post-project sustained impacts evaluations is necessary for the remaining 99% of international development projects as it offers a fantastic learning opportunity about how to ‘do development’ well now and for the future. Without returning to look for what participants and partners valued enough to continue on their own, without returning to learn about unexpected sustained impacts, we rob ourselves of pivotal learning needed for success.
Jindra Cekan, Ph.D. has used participatory methods for 30 years to connect with participants, ranging from villagers in Africa, Central/ Latin America and the Balkans to policy makers and Ministers around the world for her international clients. Their voices have informed the new Sustained and Emerging Impacts Evaluation, other M&E, stakeholder analysis, strategic planning, knowledge management and organizational learning.
If you don’t find what you are looking for via the search, categories, or posts above, you can go to the Blog page, scroll to the bottom, and click “previous posts” to go through all of the posts (newest–>oldest).