Ex-post Eval Week: Measuring sustainability post-program –go in and stay for the learning! By Holta Trandafili
Reblog from AEA: https://aea365.org/blog/ex-post-eval-week-measuring-sustainability-post-program-go-in-and-stay-for-the-learning-by-holta-trandafili/ January 21, 2021
Greetings, I am Holta Trandafili, a researcher and evaluator captivated by sustainability theories and the sustainment of results. I believe that a thoughtful, systematic inquiry of what happens after an intervention ends adds value to what we know about sustainability. Since 2015 I have co-led ten post-program evaluations (also known as ex-posts) in Uganda, Kenya, Sri Lanka, India, Myanmar, and Bolivia. Their findings point to questions and issues of theory, measurement, and sustainability expectations relevant to any program:
To what do we compare results to judge success? Is it that 60% of community groups or water points being operational three years after closeout a good result? Should it be 87% or 90%? Why? Should we use the end-line as the measuring yardstick, especially as contexts change? Whose view of success counts?
How long should we expect results, or community groups left behind, or activities to continue post-program? Two or ten years or Forever? Why?
Is going back once enough to make a judgment on sustainability? What would we find if we went back in 2020 where we evaluated ex-post in 2015 or even 2019?
Lessons Learned:Here are my reflections and resources on sustainability:
To the enthusiastic evaluators ready to start ex-posts
Lesson learned: Organizations often carry out ex-posts for accountability. However, greater wealth lays in learning. Make learning part of your evaluation objectives. It took my organization 5 years from the first ex-post to have more open conversations and share our sustainability learning on what to improve: how we design, transition, and measure programs’ impact. Now we are genuinely more accountable.
Get involved: Don’t lose heart if your first ex-posts prove difficult to conduct or have mixed results or unearth new questions and insights on sustainability. You are not alone. Find another evaluator that has gone through an ex-post experience and ask them to write a blog, present at a conference, write guidelines, attend a course, or merely meet to vent and dream.
To those already fighting to mainstream ex-post measurement in their organizations or their clients
Mainstreaming ex-post evaluations is commendable for any institution. In this process we should start making the case to pilot longitudinal ex-post measurements (i.e., going back not once but several points in time). We can truly unpack the issues of temporality and longevity for sustainment of results. See JICA’s example on ex-post monitoring.
Invest in theory-driven evaluations like Realist Evaluation to unpack the hidden mechanisms behind which different types of outcomes are sustained, asking: among whom, in what contexts, how, why?
Reblog: Eval Week: Tips for Conducting an Ex-Post Evaluation by Wendi Bevins
Published on the American Evaluation Association AEA365blog series https://aea365.org/blog/ex-post-eval-week-tips-for-conducting-an-ex-post-evaluation-by-wendi-bevins/ January 19, 2021
I am Wendi Bevins, and when my boss told me in 2017 that we were going to conduct an ex-post evaluation, I was thrilled. At the time I was the Monitoring and Evaluation Manager at Lutheran World Relief (LWR).
I was thrilled to go back to a closed project and evaluate it against our original theory of change, test our assumptions, and see if our approaches were sustainable beyond the end of a project. We could really learn some valuable lessons. And all those things were true, but they came alongside some useful lessons about human nature, which I offer here as suggestions to anyone who wants to conduct an ex-post.
Hot Tip: Manage Expectations
Ex-post results, like any evaluation’s results, will not be entirely clear, and some will be disappointing. In my own experience, I pinned my hopes on an assumption and was disappointed when the result was murky. These kinds of realities do not mean that an ex-post is a bad investment, but they do mean that stakeholders should be prepared early in the planning stage to manage their expectations. Then stakeholders are more likely to use the findings effectively where they do point to a clear (if unexpected) path forward and they aren’t tempted to dismiss all the findings out of hand. While that is true for nearly all evaluations, it is especially true for ex-posts because the stakes feel higher.
Lessons Learned: How you apply findings matters almost as much as the findings themselves
Your ex-post will definitely yield findings. You will not know ahead of time what those findings are, and as I learned, you should resist the urge to guess what they will be. However, having a way to apply the findings is important. The value of the ex-post is tied up in an organization’s ability to make use of its findings. In the case of our first ex-post evaluation, we applied what we learned to an organization-wide strategy that was being developed as the ex-post ended. Like other evaluation findings, ex-post findings can also be valuable in the development of new tools or the design of new programming, but even better because they give a longer time frame for reflection.
Rad Resources: Practical Tips and Examples
In partnership with World Vision US, LWR compiled a set of practical tips for planning and carrying out an ex-post evaluation. It reflects our lessons learned over 11 ex-post evaluations across the two organizations, such as selecting the right project; hiring an evaluator; and staying future-oriented. LWR also condensed the most transferrable lessons from our two ex-post evaluations into user-friendly summaries: the Tanzanian Grape Value Chain project and Nicaraguan Gender-Sensitive Food Security project.
I hope if you are fortunate enough to participate in an ex-post evaluation that these tips give you a good head start.
 Lutheran World Relief and IMA World Health have since created Corus International, an ensemble of for-profit and non-profit organizations with social impact missions.
Netherlands’ ex-post project evaluations: 14 Jan 2021
In June-August 2020, Preston Stewart, our Valuing Voices intern, conducted through government databases of the four Nordic countries – Norway, Finland, Sweden, and Denmark – and the Netherlands to identify ex-post evaluations of government-sponsored projects. The findings and recommendations for action are detailed in a four-part white paper series, beginning with a paper called The Search.
We would like to invite you to a presentation of our findings, and a discussion of what can be learned about:
1) the search process,
2) how ex-post evaluations are defined and categorized,
3) what was done well by each country’s ex-posts,
4) sustainability-related findings and lessons, and
5) what M&E experts in each country can improve on ex-post evaluation practices.
One big finding is that there were only 32 evaluations that seemed to be ex-post project, and only 1/2 of them actually were at least 2 years after project closure.
We have many more lessons about conflicting definitions, that ex-post evaluation is not the norm in the evaluations processes of the five governments, that development programs could, if committed to ex-post evaluation, learn about sustainability by engaging with the findings from many more such evaluations, and to increase accountability to the public and for transparent learning, ex-post evaluations should be shared in public, easy-to-access online repositories.
Your ticket purchase entitles you to the webinar, its meeting recording, associated documents, and online Sustainability Network membership for resources and discussion. Payment on a sliding, pay-as-you-can scale.
Even as evaluation specialists, rarely do we get the chance to carry out ex-post evaluations. We recently carried out an ex-post evaluation of Catholic Relief Services’ (CRS) Expanding Financial Inclusion (EFI) programme and believe we’ve found some key lessons that make the case for more ex-post evaluations.
We’ll be sharing learning from the evaluation alongside CRS colleagues at the Savings Led Working Group session on Members Day of the SEEP Annual Conference – so pop by if you would like to learn more.
What is an ex-post evaluation?
Ex-post evaluations are (by definition) done after the project has closed. There is no hard and fast rule on exactly when an ex-post evaluation should be done but as the aim of an ex-post evaluations is to assess the sustainability of results and impacts, usually some time will need to have passed to make this assessment.
A little bit about EFI
EFI was a Mastercard Foundation-funded program in Burkina Faso, Senegal, Zambia and Uganda whose core goal was to ensure that vulnerable households experienced greater financial inclusion. Within EFI, Private Service Providers (PSPs) formed and facilitated savings groups using CRS’ Savings and Internal Lending Communities (SILC) methodology, with the SILC groups responsible for paying the PSP a small fee for the services that they provide.
This payment is intended to improve sustainability by incentivising the groups’ facilitators to form and train new groups, as well as providing continued support to existing groups, beyond the end of the project.
A little bit about the evaluation
So, if the aim of the PSP model is sustainability, you need an evaluation that can test this! Evaluation at the end of project implementation can assess indications of results that might be sustained into the future. However, if you wait until some time has passed after activities have ended, then there is much clearer evidence on which activities and results are ongoing – and how likely these are to continue. Uganda was also a great test case for the evaluation because CRS hadn’t provided any follow-on support.
Our evaluation set out to assess the extent to which the EFI-trained PSPs and their SILC groups were still functioning 19 months after the programme ended and the extent to which the PSP model had contributed to the sustainability of activities and results.
What the ex-post evaluation found
We found a handful of findings that were only possible because it was an ex-post evaluation:
There were 56% more reported groups among the sampled PSPs at the time of data collection than there were at the end of the project.
Half of the PSP networks established within the sample are still functioning (to some extent).
PSPs continued to receive remuneration for the work that they did, 19 months after project closure. However, there were inconsistencies in frequency and scale of remuneration, as well as variation in strategies to sensitize communities on the need to pay.
This only covers a fraction of the findings but we were able to conclude that the PSP model appeared to be highly sustainable. The evaluation also found that there were challenges to sustainability which could be addressed in future delivery of the PSP model. Significantly, the PSP model was designed with sustainability in mind – and this evaluation provides good evidence that PSPs were still operating 19 months after the end of the project.
What made the evaluation possible
We get it. It isn’t always easy to do ex-post evaluations. Evaluations are usually included in donor-implementer contracts, which end shortly after the project ends, leaving implementers without the resources to go back and evaluate 18 months later. This often results in a lack of funding and an absence of project staff. This is also combined with new projects starting up, obscuring opportunities for project-specific findings and learning as it’s not possible to attribute results to a specific project.
In many ways, we were lucky. Itad implements the Mastercard Foundations Savings Learning Lab, a six-year initiative that supports learning among the Foundation’s savings sector portfolio programmes – including EFI. EFI closed in the Learning Lab’s second year and with support from the Foundation and enthusiasm from CRS, we set aside some resource to continue this learning post-project. So, we had funding!
We also worked with incredibly motivated ex-EFI, CRS staff who made time to actively engage in the evaluation process and facilitate links to the PSP network, PSPs and SILC group members. So, we had the people!
And, no-one had implemented a similar PSP model in supported districts of Uganda since the end of EFI. So, we were also able to attribute!
Why we should strive to do more ex-post evaluations
Despite these challenges, and recognising it isn’t always easy, doesn’t mean it is not possible. And with projects like EFI where sustainability was central to its model, we would say it’s essential to assess whether the programme worked and how the model can be improved.
Unfortunately, practitioners and evaluators can shout all we like but the onus is on funders. We need funders to carve out dedicated resource for ex-post evaluations. This is even more important for programmes that have the development of replicable and sustainable models at their core. For some projects, this can be anticipated – and planned for – at project design stage. Other projects may show promise for learning on sustainability, unexpectedly, during implementation. Dedicated funding pots or call-down contracts for ex-post evaluations are just a couple of ways donors might be able to resource ex-post evaluations when there is a clear need for additional learning on the sustainability project results.
This learning should lead to better decision making, more effective use of donor funds and ultimately, more sustainable outcomes for beneficiaries.”
RESOURCES: “Finding 1.iii. PSPs continue to receive remuneration for the work that they do; however, there are inconsistencies in frequency and scale of remuneration, as well as variety in strategies to sensitize communities on the need to pay.”
CAPACITIES: “Finding 3.ii. All networks included a core function of “collaboration, information-sharing and problemsolving”; however, networks were not sufficiently supported or incentivized to fulfill complex functions, such as PSP quality assurance or consumer protection, and their coverage area and late implementation limited the continued functioning of networks.”
PARTNERSHIP: “Finding 2.i. Only four of the 24 groups are clearly linked with other stakeholders and two were supported by EFI to create these linkages.”
What makes it Difficult to Evaluate Projects Ex-post: Lessons from WWF
The World Wildlife Fund (WWF) generously shared lessons from two ex-posts and in particular what they felt should not have been done in terms of organizational learning. They originally wanted to see whether ”the work done is sustainable and so to provide lessons for the implementing office, donor offices and the wider WWF network about designing and implementing projects to deliver outcomes that are sustained beyond WWF’s involvement.”
They used six criteria for country/ case selection:
There were three types of interesting results (refer to Table 1).
Methodological problemsin comparability to final evaluation and impact data as well as choice of project: problems with site/ case and lack of water data comparability as well as the entry of a new NGO garnering benefits (so limited contribution/ isolation of results). WWF tried to mitigate it with good design guidance and ongoing review processes during the ex-post.
The emergence of new data ex-post about the benefits of the local user association for conflict management and emerging new catchment management
There were ex-post limitations intrinsic to the species and ecosystem resources on which they rely, specifically species and water. Other problems of not tracking impacts on animals, not learning from evaluations, and not having saved documents might benefit other projects’ quality.
Table 1. WWF Case 1
Some learning might be attributable to the difficulties of evaluating ecosystems or it might provide important cautionary lessons for others considering such ecosystem evaluations.
But overall, it shows that in contexts where one of these criteria are present:
1. original programme having design flaws,
2. poor collection of evidence of progress against the chosen indicators of success,
3. poor documentation retention,
Then, there is limited value to be gained from doing an ex-post evaluation.
WWF Case 2:
The excellent news is that ex-post demonstrated that all three kinds of impact indicators, namely species numbers, forest cover, and conservation area improved (note, data are confidential, thereby not cited below) and that the XX Conservation Area is still managed by the community (Table 2). Also, the team gathered information that living standards had improved, as had, impressively a country-level seed fund and active participation in managing and scientifically learning from their resource.
The second case also highlights typical ex-post methodological evaluation considerations, including choice of sites/ timing:
not applying a standard monitoring process for the whole programme period, which caused confusion about end of project versus ex-post impact progress
the ex-post evaluation was planned predicated on the project finishing and staff exit from the area, but there was ongoing financial support of the CSO from the central office for a further two years which meant that this was not a true ex-post evaluation as it happened before the two-year minimum timeframe
measurement led to ‘anecdotal’ evidence and comparisons made to national statistics rather than change from project evaluations which limits ex-post learning particularly
there were negative findings of sustainability how well the community appreciated the XXCA and insufficient economic return on investment expectations that some costs such as trash removal were not covered by anticipated tourist revenues.
It also highlights that ecosystem ex-post evaluation is complex, as wildlife number was difficult to methodologically confirm, due to progression in monitoring technologies. A valuable question that WWF shared was “how to assess [continued] benefits to people from natural resource management?”
Table 2: WWF Case 2
WWF found and shared important and appropriate lessons about what these ex-posts brought with them. Most fell into aforementioned categories of Choice of projects, sites, timing, and importantly, the value of considering Methodological considerations regarding data access, quality from project onset.
What was new to us at Valuing Voices was that Organizational considerations including interest in learning by WWF-UK and their partnerswhich is something we have not seen before. Equally, the importance of a solid rationale for asking a country office to participate in such an evaluation, and not imposing an ex-post on a less than enthusiastic program for whom sustainability may not be a priority. This links these cases together, as given limited budgets, WWF and their partners could choose to prioritize design spending, perhaps above final evaluations, especially if local NGOs know donors will leave at project’s end, so funding to improve their internal capacity strengthening might be a larger priority. Knowing the project has an immutable closeout can affect interest at the headquarters, as it can be quite difficult to find funding to evaluate ex-post, and more pressing current work competes. Also, implementing local NGOs/ CBOs could actually know about sustainability far more than donors, hereby eliminating their interest, as an M&E officer recounted. Both organizations could have other priorities, including new projects with new staff, who had less interest in the old project. Staff turnover is an additional, important consideration in whether to do an ex-post.
On June 24th under GLocal’s UNConference, “Co-creating our future stories of hope and action”, Jindra Cekan, Holta Trandafili, and Isabella Jean presented their work on sustainability evaluations and exit strategies via local voices. We chaired a 2-hour discussion session on the following topics:
Sustainability of global development projects and exit from them,
The importance of valuing local partners’ and participants’ voices,
How to embed ex-post evaluation of sustainability into the project cycle,
How expectations, benchmarking, and early joint planning in exit strategies, as well as considering long-term ownership & relevance will support projects to be sustained locally, including questioning who will maintain results,
Considering power dynamics between donors and ultimate ‘beneficiaries’ and
the value of the impact of the project from a variety of perspectives, and more
· Fear of learning about failure in our global development industry – INGOs are “waiting for a successful enough project” to commission ex-post project sustainability evaluation
· It needs to be a culture of learning, not a culture of success.
· Lack of transparency in sharing program evaluation results with communities and local government is widespread, and even more rarely do we come back after many years and share learnings
· Participatory approaches are vital, listening to participants about sustained impacts is key
· It is never attribution, always contribution. To isolate impacts, we need to look for project sites that haven’t had multiple other organizations overlapping through all phases.
· Building sustainability planning throughout the project cycle is key – but often doesn’t happen
· Sustainability needs to be planned to be researched, including evaluating why or why not were project elements sustained, and why? What has the project done to enable communities to sustain improvements?
· Expectations of sustainability need to be more modest (as most results are mixed good/bad)
· We need to ask: How are you defining and measuring sustainability – for how long should the results last? Among how many participants? Have you set benchmarks for success?
· We should expand your toolbox on methodology to investigate sustainability. Stories of success are one of a myriad of methods used, including mixed-methods, cost-benefit, etc.
· Start with the need for learning not [just] accountability
· Sustainability investigation/evaluation/learning should be mindful that this is NOT about projects. It is about people.
· We have a system that focuses on gaps and needs to be filled vs. existing capacities’ structures to be reinforced. How can our work on measuring sustainability bring this to light and call it out, so that we change the norm?
· Planning for sustainability requires the insight to integrate resources and experiences of outsiders with the assets and capacities of insiders to develop context-appropriate strategies for change
If you would like to discuss this with any of us, please send us comments and we’re happy to respond. Thanks again to the G-Local UnConference team!
Below please find our bios:
Jindra Cekan/ova has worked in global development for 33 years focused on participatory design and M&E for global non-profits. She founded Valuing Voices 7 years ago. For details, see: Valuing Voices Founder
Holta Trandafili is the Research, Learning, and Analytics Manager with World Vision US and has been leading field research, monitoring, and evaluation since 2007. She has led sustainability measurement studies for World Vision programs in Uganda, Kenya, Sri Lanka, India, Burma, and Bolivia. Her areas of expertise and interest include program and community groups’ sustainability measurements; agency-level measurements; empowerment approaches to development; integrated programming; local capacities for peace; gender analysis; and outcome monitoring. Currently, Mrs. Trandafili serves as an Advisory Committee member for InterAction’s Effectiveness and Program Evaluation Working Group and chairs one of the sub-working groups under The Movement for Community-led Development.
Isabella Jean supports international and local organizations and funders to document promising practices, facilitate learning and strengthen capacities for conflict sensitivity, peacebuilding and humanitarian effectiveness. She has facilitated action research, collaborative learning and advisory engagements in over 25 countries, and serves as an advisor to policymakers, senior leadership and program teams. Isabella co-authored the book, Time to Listen: Hearing People on the Receiving End of Aid and developed practical guidance to support accountability to communities, listening and feedback loops, and responsible INGO exits. She teaches graduate-level courses on aid effectiveness, program strategies and M&E of peacebuilding at Brandeis University’s Heller School for Social Policy and Management. Previously, Isabella directed training at a community organizing network and conducted policy research for the Institute for Responsive Education, UNDP, and Coexistence International.
Jindra Cekan, Ph.D. has used participatory methods for 30 years to connect with participants, ranging from villagers in Africa, Central/ Latin America and the Balkans to policy makers and Ministers around the world for her international clients. Their voices have informed the new Sustained and Emerging Impacts Evaluation, other M&E, stakeholder analysis, strategic planning, knowledge management and organizational learning.
If you don’t find what you are looking for via the search, categories, or posts above, you can go to the Blog page, scroll to the bottom, and click “previous posts” to go through all of the posts (newest–>oldest).