Ex-post Eval Week: Measuring sustainability post-program –go in and stay for the learning! By Holta Trandafili
Reblog from AEA: https://aea365.org/blog/ex-post-eval-week-measuring-sustainability-post-program-go-in-and-stay-for-the-learning-by-holta-trandafili/ January 21, 2021
Holta Trandafili
Greetings, I am Holta Trandafili, a researcher and evaluator captivated by sustainability theories and the sustainment of results. I believe that a thoughtful, systematic inquiry of what happens after an intervention ends adds value to what we know about sustainability. Since 2015 I have co-led ten post-program evaluations (also known as ex-posts) in Uganda, Kenya, Sri Lanka, India, Myanmar, and Bolivia. Their findings point to questions and issues of theory, measurement, and sustainability expectations relevant to any program:
To what do we compare results to judge success? Is it that 60% of community groups or water points being operational three years after closeout a good result? Should it be 87% or 90%? Why? Should we use the end-line as the measuring yardstick, especially as contexts change? Whose view of success counts?
How long should we expect results, or community groups left behind, or activities to continue post-program? Two or ten years or Forever? Why?
Is going back once enough to make a judgment on sustainability? What would we find if we went back in 2020 where we evaluated ex-post in 2015 or even 2019?
Lessons Learned:Here are my reflections and resources on sustainability:
To the enthusiastic evaluators ready to start ex-posts
Lesson learned: Organizations often carry out ex-posts for accountability. However, greater wealth lays in learning. Make learning part of your evaluation objectives. It took my organization 5 years from the first ex-post to have more open conversations and share our sustainability learning on what to improve: how we design, transition, and measure programs’ impact. Now we are genuinely more accountable.
Get involved: Don’t lose heart if your first ex-posts prove difficult to conduct or have mixed results or unearth new questions and insights on sustainability. You are not alone. Find another evaluator that has gone through an ex-post experience and ask them to write a blog, present at a conference, write guidelines, attend a course, or merely meet to vent and dream.
To those already fighting to mainstream ex-post measurement in their organizations or their clients
Hot Tips:
Mainstreaming ex-post evaluations is commendable for any institution. In this process we should start making the case to pilot longitudinal ex-post measurements (i.e., going back not once but several points in time). We can truly unpack the issues of temporality and longevity for sustainment of results. See JICA’s example on ex-post monitoring.
Invest in theory-driven evaluations like Realist Evaluation to unpack the hidden mechanisms behind which different types of outcomes are sustained, asking: among whom, in what contexts, how, why?
Reblog: Eval Week: Tips for Conducting an Ex-Post Evaluation by Wendi Bevins
Published on the American Evaluation Association AEA365blog series https://aea365.org/blog/ex-post-eval-week-tips-for-conducting-an-ex-post-evaluation-by-wendi-bevins/ January 19, 2021
Wendi Bevins
I am Wendi Bevins, and when my boss told me in 2017 that we were going to conduct an ex-post evaluation, I was thrilled. At the time I was the Monitoring and Evaluation Manager at Lutheran World Relief[1] (LWR).
I was thrilled to go back to a closed project and evaluate it against our original theory of change, test our assumptions, and see if our approaches were sustainable beyond the end of a project. We could really learn some valuable lessons. And all those things were true, but they came alongside some useful lessons about human nature, which I offer here as suggestions to anyone who wants to conduct an ex-post.
Hot Tip: Manage Expectations
Ex-post results, like any evaluation’s results, will not be entirely clear, and some will be disappointing. In my own experience, I pinned my hopes on an assumption and was disappointed when the result was murky. These kinds of realities do not mean that an ex-post is a bad investment, but they do mean that stakeholders should be prepared early in the planning stage to manage their expectations. Then stakeholders are more likely to use the findings effectively where they do point to a clear (if unexpected) path forward and they aren’t tempted to dismiss all the findings out of hand. While that is true for nearly all evaluations, it is especially true for ex-posts because the stakes feel higher.
Lessons Learned: How you apply findings matters almost as much as the findings themselves
Your ex-post will definitely yield findings. You will not know ahead of time what those findings are, and as I learned, you should resist the urge to guess what they will be. However, having a way to apply the findings is important. The value of the ex-post is tied up in an organization’s ability to make use of its findings. In the case of our first ex-post evaluation, we applied what we learned to an organization-wide strategy that was being developed as the ex-post ended. Like other evaluation findings, ex-post findings can also be valuable in the development of new tools or the design of new programming, but even better because they give a longer time frame for reflection.
Rad Resources: Practical Tips and Examples
In partnership with World Vision US, LWR compiled a set of practical tips for planning and carrying out an ex-post evaluation. It reflects our lessons learned over 11 ex-post evaluations across the two organizations, such as selecting the right project; hiring an evaluator; and staying future-oriented. LWR also condensed the most transferrable lessons from our two ex-post evaluations into user-friendly summaries: the Tanzanian Grape Value Chain project and Nicaraguan Gender-Sensitive Food Security project.
I hope if you are fortunate enough to participate in an ex-post evaluation that these tips give you a good head start.
[1] Lutheran World Relief and IMA World Health have since created Corus International, an ensemble of for-profit and non-profit organizations with social impact missions.
Wishing for Ex-Post Evaluation Christmas Lights
Rather than Needles in Haystacks
This is what the life of most ex-post evaluation researchers looks like, mostly without the counting congratulator:
I recently spent three days looking for ex-post evaluations for a client across nearly a dozen organizations. I was hard-pressed to find 16 actual ones. Sorting through ‘impact evaluations’ that were done in the middle of implementation does not tell us anything about what was sustained after we left, nor do delayed final evaluations that happen to be done after closure. While these (rightly) focus on cost-effectiveness, relevance, and efficiency, measures of sustained impact are projections, not actual measures of what outcomes and impacts stood the test of time. I weeded out some desk studies that did not return to ask anyone who participated. Others titled ‘ex-post’ were barely midterms (I can only gather they misconstrued ‘ex-post’ as after-starting implementation?), and a few more reports only recommended doing ex-post evaluation after this final evaluation. For more lessons on how random and misconstrued ex-posts can be, see Valuing Voices’ research for Scriven. None of these 16 actual ex-posts even told us anything about what emerged (as we look at during Sustained and Emerging Impacts Evaluations) from local efforts in the years after assistance ended [1].
This is what I wish my ex-post haystack would look like, bountiful treasures of numerous ex-post-project evaluations, as numerous as these Christmas lights here in Tabor, Czech Republic.
If we had more ex-posts to learn from, we could learn from what lasted. What could locals sustain? Why? Why not? How can we do better next time? We could compare across sectors and countries, and we could see what conditions and processes during implementation supported sustainability -and, importantly – why some failed, so we don’t repeat those mistakes.
We could move from our current orange slices that end at closure to green sustainability of the project cycle:
I will be adding the ones I found to our Catalysts list soon, but when my client asked me who held databases of ex-post evaluations, I had to say only Valuing Voices and Japan’s JICA (since 1993, who even differentiates the ex-posts between Technical Grants and ODA Loans). This is not to say some cannot be found by trawling the OECD or the World Bank, but this is Needle-in-Haystack work again, and so there are only two databases to learn from. Isn’t that shocking?
Now JICA has really upped the illumination ante, so to speak: They are now doing what they are calling ‘JICA’s Ex-post Monitoring’, which was like Christmas come early [2]! Returning to learn at least 7 years after the ex-post, which was 1-3 years after closure, such as in this case of ex-post monitoring and learning from 10 projects (2007). They have done ex-post monitoring for a total of 91 cases, evaluating the sustained impacts of results, see if JICA’s recommendations to their partners had been implemented, how they had adapted to changes over a decade post-closure, and find learning for new programming. “Ex-post monitoring is undertaken 7 years after a project was completed in principle in order to determine whether or not the expected effects and impacts continue to be generated, to check that there are no sustainability-related problems with the technical capacities, systems, and finances of the executing agency nor with the operation and management of developed facilities, etc., and to ascertain what action has been taken vis-a-vis the lessons learned and recommendations gleaned during the ex-post evaluation.” While it was unclear why these specific projects were selected, it is amazing they are doing 5-10 per year.
What we can learn from returning again is illustrated by one of JICA’s water project loans in RSA, which ended in 2003, had an ex-post in 2006, followed by monitoring of sustainability in 2013 [3]. While the report included issues of data access and evaluators expressed caution in attributing causation of positive changes to the project, but it not only continued functioning, the government of South Africa (RSA) solved barriers found at the ex-post:
“Data for the supply and demand of water pertaining to the Kwandebele region could not be obtained. However, considering the calculation from the water-supplied population and supplied volume and the result from the DWAF interview, water shortage could not be detected in the four municipalities studied by this project…” [3]
“The ex-post evaluation indicated that the four components were not in the state to be operated and managed effectively. Currently, the components are operated and managed effectively and are operating under good condition [and] concerning sustainability, improvement can be seen from the time of ex-post evaluation. Shortage of employees and insufficient technical knowledge has been resolved…” [3]
“Compared to the time of ex-post evaluation, improvement was seen in the under-five mortality and life expectancy. However, since the components implemented by this project are limited in comparison with the scope of the project, it is impossible to present a clear causal relationship” [3].
In another, from Indonesia’s air quality testing labs which involved capacity building and equipment maintenance 6 years after the ex-post, they mostly found training and use continued despite organizational changes and maintenance challenges: [4]
“After the ex-post evaluation, many of the target laboratories changed their affiliation from the Ministry of Public Works (MOPW) and MOH to provincial governments. While the relocation of equipment has been carried out in a handful of provinces, in other provinces, equipment is still located at the laboratories where it was originally installed, and these laboratories still have the right of use” [4]
In spite of some irregularities,”As the Ministry of Environment (MOE) still has ownership of the equipment, some laboratories have inappropriate audit results that show the allocation of O&M budget to equipment which is not included in their accounting…” [4]
“Out of 20 laboratories where the questionnaire survey confirmed that equipment still remained, 15 laboratories replied that spare parts for equipment are still available but are difficult to obtain…It takes several months to one year to obtain spare parts, occasionally out of Indonesia, even if a repair service is available” [4].
In this case, there were lessons learned for JICA and Indonesia’s Ministry of the Environment programs about ownership and the right use of the equipment and retiring obsolete equipment. Talk about a commitment to learning from the ongoing success or failure of one’s projects!
As you have read here on Valuing Voices for more than 6 years unless we include post-project sustainability that asks our participants and partners how sustained their lives and livelihoods could be and even resilient to shocks like political or climate change, we cannot say we are doing Sustainable Development. We need such lessons about what could be sustained and why.
We can prepare better to foster sustainability. In the coming months, we are working on checklists to consider during funding, design, implementation, M&E pre-and post-exit to foster sustainability. Will keep you posted, but as World Vision also found: “Measuring sustainability through ex-posts requires setting clear benchmarks to measure success prior to program closure, including timelines for expected sustainment.”
And as my gift to you this Holiday Season, let me share WV’s Learning Brief about Sustainability, with wise and provocative questions to ponder about dynamic systems, benchmarking, continuous learning, attribution, and managing expectations [5]. World Vision shares how infrastructure and community groups and social cohesion fared well, yet lessons circled back to the need for JICA-like ‘monitoring’ and mirror rich ex-post lessons from FFP/Tufts (Rogers, Coates) and Hiller et al. that explains why we do ex-posts at all: “Project impact at the time of exit does not consistently predict sustainability“ [6].
Now my gift: a few big lessons from the 6 years of researching sustainability across the development spectrum. I have found no evaluations that were only positive OR ONLY NEGATIVE. Most results trended downwards, a few held steady, and all were mixed. We cannot assume the sustainability of results at closure nor optimistic projections, as we’ve seen in the climate arena.
Please consider:
Designing with our participants and partners so what we do,
Implementing with partners far longer to make sure things still work,
Adapting exit based on benchmarks to see how well the resources, partnerships, capacities, and ownership have been transferred,
Using control or comparison groups to make sure ‘success’ was due to you and being careful about attributing results to your projects while considering how you contributed to a larger whole of ongoing country progress or stagnation,
Being willing to jettison what is unlikely to be sustained and learn from what we designed and implemented poorly (due to our design, their implementation, and external conditions),
Given climate change, learning fast, adaptively, and revising fast given changing conditions,
Without knowing what has been sustained, we cannot replicate nor scale up,
Sharing lessons with your leaders – for people’s lives depend on our work,
Learning from what emerged as our participants and partners refashioned implementation in new ways could sustain it (without the millions we brought),
Refocusing ‘success’ from how much we have spent to how much was sustained.
Please make our next Christmas merry. Do MANY ex-post evaluations, Learn TONS, Share WIDELY WHAT WORKED AND FAILED TO WORK (you will be praised!), and let’s CHANGE HOW WE DO SUSTAINABLE DEVELOPMENT.
May 2020 bring health, happiness, and to all of us a more sustainable world!
Jindra Cekan, Ph.D. has used participatory methods for 30 years to connect with participants, ranging from villagers in Africa, Central/ Latin America and the Balkans to policy makers and Ministers around the world for her international clients. Their voices have informed the new Sustained and Emerging Impacts Evaluation, other M&E, stakeholder analysis, strategic planning, knowledge management and organizational learning.
If you don’t find what you are looking for via the search, categories, or posts above, you can go to the Blog page, scroll to the bottom, and click “previous posts” to go through all of the posts (newest–>oldest).
