Wishing for ex-post evaluation Christmas Lights rather than Needles in Haystacks

 

Wishing for Ex-Post Evaluation Christmas Lights
Rather than Needles in Haystacks

 

This is what life of most ex-post evaluation researchers looks like, mostly without the counting congratulator:

 

I recently spent three days looking for ex-post evaluations for a client across nearly a dozen organizations. I was hard-pressed to find 16 actual ones. Sorting through ‘impact evaluations’ that were done in the middle of implementation does not tell us anything about what was sustained after we leave, nor do delayed final evaluations that happen to be done after closure. While these (rightly) focus on cost-effectiveness, relevance and efficiency, measures of sustained impact are projections, not actual measures of what outcomes and impacts stood the test of time. I weeded out some desk studies that did not return to ask anyone who participated. Others titled ‘ex-post’ were barely midterms (I can only gather they misconstrued ‘ex-post’ as after-starting implementation?) and a few more reports only recommended doing ex-post evaluation after this final evaluation. For more lessons on how random and misconstrued ex-posts can be, see Valuing Voices’ research for Scriven. None of these 16 actual ex-posts even told us anything about what emerged (as we look at during Sustained and Emerging Impacts Evaluations) from local efforts in the years after assistance ended [1].

This is what I wish my ex-post haystack would look like, bountiful treasures of numerous ex-post-project evaluations, as numerous as these Christmas lights here in Tabor, Czech Republic.

 

If we had more ex-posts to learn from, we could learn from what lasted. What could locals sustain? Why? Why not? How can we do better next time? We could compare across sectors and countries, and we could see what conditions and processes during implementation supported sustainability -and importantly – why some failed, so we don’t repeat those mistakes.

We could move from our current orange slices that ends at closure to green sustainability of the project cycle:

I will be adding the ones I found to our Catalysts list soon, but when my client asked me who held databases of ex-post evaluations, I had to say only Valuing Voices and Japan’s JICA (since 1993 who even differentiates the ex-posts between Technical Grants and ODA Loans). This is not to say some cannot be found by trawling the OECD or the World Bank, but this is Needle-in-Haystack work again and so there are only 2 databases to learn from. Isn’t that shocking?

Now JICA has really upped the illumination ante, so to speak: They are now doing what they are calling JICA’s Ex-post Monitoring’ which was like Christmas come early [2]! Returning to learn at least 7 years after the ex-post which was 1-3 years after closure, such as among this case of ex-post monitoring and learning from 10 projects (2007). They have done ex-post monitoring for a total of 91 cases, evaluating the sustained impacts of results, see if JICA’s recommendations to their partners had been implemented, how they had adapted to changes over a decade post-closure, and find learning for new programming. “Ex-post monitoring is undertaken 7 years after a project was completed in principle in order to determine whether or not the expected effects and impacts continue to be generated, to check that there are no sustainability-related problems with the technical capacities, systems and finances of the executing agency nor with the operation and management of developed facilities, etc., and to ascertain what action has been taken vis-a-vis the lessons learned and recommendations gleaned during the ex-post evaluation.” While it was unclear why these specific projects were selected, it is amazing they are doing 5-10 per year.

They are my ex-post gods/ goddesses and I fawned over two JICA evaluators at the last European Evaluation Society Conference. Why do I fawn? JICA lists 2273 results under ex-post evaluations of Technical Cooperation, Grant Aid, ODA loans! They are literally the only organization I know whose searched reports are actually ‘ex-post’.

What we can learn from returning again is illustrated by one of JICA’s water project loans in RSA, which ended in 2003, had an ex-post in 2006, followed by monitoring of sustainability in 2013 [3]. While the report included issues of data access and evaluators expressed caution in attributing causation of positive changes to the project, but it not only continued functioning, the government of South Africa (RSA) solved barriers found at the ex-post:

  • “Data for the supply and demand of water pertaining to the Kwandebele region could not be obtained. However, considering the calculation from the water supplied population and supplied volume and the result from the DWAF interview, water shortage could not be detected in the four municipalities studied by this project…” [3]
  • “The ex-post evaluation indicated that the four components were not in the state to be operated and managed effectively. Currently, the components are operated and managed effectively and are operating under good condition [and] concerning sustainability, improvement can be seen from the time of ex-post evaluation. Shortage of employees and insufficient technical knowledge has been resolved…” [3]
  • “Compared to the time of ex-post evaluation, improvement was seen in the under-five mortality and life expectancy. However, since the components implemented by this project are limited in comparison with the scope of the project, it is impossible to present a clear causal relationship” [3].

In another, from Indonesia’s air quality testing labs which involved capacity building and equipment maintenance 6 years after the ex-post, they mostly found training and use continued despite organizational changes and maintenance challenges: [4]

  • “After the ex-post evaluation, many of the target laboratories changed their affiliation from the Ministry of Public Works (MOPW) and MOH to provincial governments. While the relocation of equipment has been carried out in a handful of provinces, in other provinces equipment is still located at the laboratories where it was originally installed and these laboratories still have the right of use” [4]
  • In spite of some irregularities ”As the Ministry of Environment (MOE) still has ownership of the equipment, some laboratories have inappropriate audit results that show allocation of O&M budget to equipment which is not included in their accounting…” [4]
  • “Out of 20 laboratories where the questionnaire survey confirmed that equipment still remained, 15 laboratories replied that spare parts for equipment are still available but are difficult to obtain…It takes several months to one year to obtain spare parts, occasionally out of Indonesia, even if a repair service is available” [4].

In this case, there were lessons learned for JICA and Indonesia’s Ministry of the Environment programs about ownership and the right use of the equipment and retiring obsolete equipment. Talk about a commitment to learning from the ongoing success or failure of one’s projects!

 

As you have read here on Valuing Voices for more than six years, unless we include post-project sustainability that asks our participants and partners how sustained their lives and livelihoods could be, and even resilient to shocks like political or climate change, we cannot say we are doing Sustainable Development. We need such lessons about what could be sustained and why.

We can prepare better to foster sustainability. In the coming months we are working on checklists to consider during funding, design, implementation, M&E pre-and post-exit, to foster sustainability. Will keep you posted, but as World Vision also found: “Measuring sustainability through ex-posts requires setting clear benchmarks to measure success prior to program closure, including timelines for expected sustainment.”

And as my gift to you this Holiday Season, let me share WV’s Learning Brief about Sustainability, with wise and provocative questions to ponder about dynamic systems, benchmarking, continuous learning, attribution, and managing expectations [5]. World Vision shares how infrastructure and community groups and social cohesion fared well, yet lessons circled back to the need for JICA-like ‘monitoring’ and mirror rich ex-post lessons from FFP/Tufts (Rogers, Coates) and Hiller et al. that explains why we do ex-posts at all: “Project impact at the time of exit does not consistently predict sustainability“ [6].

 

Now my gift: a few big lessons from  the six years of researching sustainability across the development spectrum.  I have found no evaluations that were only positive. Most results trended downwards, a few held steady, and all were mixed. We cannot assume the sustainability of results at closure, nor optimistic projections as we’ve seen in the climate arena.

Please consider:

  • Designing with our participants and partners so what we do,
  • Implementing with partners far longer to make sure things still work,
  • Adapting exit based on benchmarks to see how well the resources, partnerships, capacities, and ownership have been transferred,
  • Using control or comparison groups to make sure ‘success’ was due to you and being careful about attributing results to your projects while considering how you contributed to a larger whole of ongoing country progress or stagnation,
  • Being willing to jettison what is unlikely to be sustained and learn from what we designed and implemented poorly (due to our design, their implementation, external conditions),
  • Given climate-change, learning fast, adaptively and revising fast given changing conditions,
  • Without knowing what has been sustained we cannot replicate nor scale-up,
  • Sharing lessons with your leaders – for people’s lives depend on our work,
  • Learning from what emerged as our participants and partners refashioned implementation in new ways could sustain it (without the millions we brought),
  • Refocusing ‘success’ from how much we have spent, to how much was sustained.

 

Please make our next Christmas merry. Do MANY ex-post evaluations, Learn TONS, Share WIDELY WHAT WORKED AND FAILED TO WORK (you will be praised!), and let’s CHANGE HOW WE DO SUSTAINABLE DEVELOPMENT.

 

May 2020 bring health, happiness, and to all of us a more sustainable world!

 

 

Sources:

[1] Cekan, J., Zivetz, L., & Rogers, P. (2016). Sustained and Emerging Impacts Evaluation (SEIE). Retrieved from https://www.betterevaluation.org/en/themes/SEIE

[2] JICA. (n.d.). Ex-post Monitoring. Retrieved December, 2019, from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/index.html

[3] Matsuyama, K. (2012). Ex-Post Monitoring of Japanese ODA Loan Project: South Africa, Kwandebele Region Water Augmentation Project. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/c8h0vm000001rdlp-att/2012_full_03.pdf

[4] Kobayashi, N. (2009, August). Ex-post Monitoring of Completed ODA Loan Project: Indonesia, The Bepedal Regional Monitoring Capacity Development Project. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/c8h0vm000001rdlp-att/indonesia2008_01.pdf

[5] Trandafili, H. (2019). Learning Brief: What does sustainability look like post-program? Retrieved from https://valuingvoices.com/wp-content/uploads/2019/12/Sustainability-Learning-Brief_final_WV-icons.pdf

[6] Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fsnnetwork.org/ffp-sustainability-and-exit-strategies-study-synthesis-report

 

Public and Private paths to Sustained Global Development Impacts

Public and Private paths to Sustained Global

Development Impacts

(Reposted from: https://medium.com/@jindracekan/public-and-private-paths-to-sustained-global-development-impacts-9b7523891fce)

Six years. That’s how long ago I began researching proof of sustained impact(s) through its ex-post project evaluation. Until now Valuing Voices has focused on aid donors. We are expanding to the private sector.

 

In my PhD I was sure it was a lack of researched and shared proof of successful prevention of famine that led to inaction. In Valuing Voices’ research on ex-post project evaluation, I again felt “if only they knew, they would act”. I pulled together a variety of researchers and consultants who (often pro-bono, or for limited fees) researched the shockingly rare field evaluations of what was sustained after projects closed, why, and what participants and partners did themselves to sustain impacts.

 

Sustaining the outcomes and achieving impacts, are, after all, what global development projects promise. These ‘sustainable development’ results are at the top (or far-right, below) of our ‘logical frameworks’. We promise the country-level partners, our taxpayers and donors, that we will achieve them, yet…

 

 

 

 

 

 

 

 

 

 

 

 

 

 

We have done six post-project Sustained and Emerging Impacts Evaluations. We have created checklists on ex-post project evaluability thanks to a Faster Forward Fund grant by esteemed evaluator Michael Scriven. We have created preliminary guidance on Sustained and Emerging Impacts Evaluations (SEIE) and shared 25 such ex-post closure evaluations that we found returned to ask participants 2-15 years after close-out in (one of?) the only database on such evaluations in the world. We have drawn valuable lessons from the evaluations throughout nearly 60 blogs and presented at 10 conferences. We have found that results at the end of project are dynamic, that there can be greater failure – or sometimes greater success – than we would ever expect in our project assumptions. We have found that communities can create ‘emerging’ outcomes, adapting the activities to succeed onward with no further donor funding, and that when we design for long-term sustainability with our partners, then remarkable success can ensue.  So many lessons for programming that we need to learn from, including partnering with country-nationals, focusing on youth, questioning assumptions at exit, etc.

 

We have applied to many grants for support, unsuccessfully, and have applied to evaluate a handful more ex-post sustainability evaluations which other consultants have won – while we were disappointed, in equal measure we are happy others are learning to do this, as we share our resources freely to promote exactly such practices across hundreds of thousands unevaluated projects! We are currently doing an ex-post project evaluation of an agriculture value chain in Tanzania, yet there are a handful done per year. At one conference, our discussant Michael Bamberger joked we were lucky not to be found dead under a bridge for taking on such a dangerous topic. We remain undeterred, and delight in colleagues we promote such work and thanking us for ours.

 

At the same time, several things have become apparent:

 

 

 

  • Vital lessons for how aid can do better remain unexplored, and true accountability to our country-national participants and partners ends when fixed-time, fixed deliverable project resources are spent and proof of accountability for money and results that donors want are filed away. Sadly, while capacity building is done throughout implementation, knowledge management about results is abysmal as ‘our projects’ data almost always dies quietly in donor and implementer computer hard drives after close-out, rather than being accessible in-country for further learning. Go partner!

 

  • We hardly ever return after all our evaluations to share with communities which speaks to ‘partnerships’ not being with the participants, and we often ‘exit’ without giving ample time to handover so that things can be sustained, e.g. local partners found, local and other international funding harnessed, etc. Learn together!

 

  • There is a real need to fund systematized methods for such evaluations, mandate access to quality baseline, midterm and final evaluations, and mandate that all projects above a certain funding level (e.g. $1mil) include funding for such evaluation and learning 2-10 years later. Many so-called ex-post evaluations are in fact either delayed final evaluations, desk studies without any fieldwork, rather methodologically flawed comparisons or with fieldwork which doesn’t talk to the intended ‘beneficiaries’ for such pivotal ground-level feedback. Innovate by listening!

 

  • It is unclear to us how any organization that has done an ex-post sustainability evaluation has learned from it and changed their systems, although we have been told some are ‘looking for a successful project to evaluate’, and that after a failed one, they are discontinued. We know of some (I)NGOs who are putting ex-posts into their new strategies, and two INGOs who are researching exits more – good. Be brave!

 

  • Recently, we are delighted some new NGOs are dipping their feet into their first evaluations of sustainability, they do so bravely. The tension between accountability and learning is heightened at the prospect that implementers and donors have failed to create sustained impact. But why judge them when all the design and systems in place are to reward success while projects are running (and even those don’t always show much) so that they all get congratulations and more funding for very similar projects? Who knows who is focused on sustaining impacts with funding capacities, partnerships and country-led design, implementing with feedback loops and adjusting for the long-term, helping communities evaluate us rather than how well they are fulfilling our targets, etc. Sustaining impacts will win you funding!

 

  • Logically, here are many indications among ex-post sustainability evaluations that profitable, but low-risk and diversified agriculture, microenterprise/ business projects are better sustained (Niger, Ethiopia, Tanzania, Nepal, etc.). This does not mean that all projects need to be profitable, but cost-covering projects even in the health and education/ vocational training/ sectors is important as many of us know. Self-funding!

 

So rather than giving up on sustained impacts, we are adding another branch to the Valuing Voices tree.

 

My partners and I have extensively researched the need for and co-founded Impact Guild. We will work alongside NGOs and impact investors to foster:

 

  1. FUNDING: The money available from development aid donors is shrinking in volume + value, while development financing is scaling up exponentially.

 

 

 

 

 

 

 

 

 

 

 

 

The SDGs and the Paris Agreement are prompting a massive scale-up of development financing from billions to trillions of dollars into ‘sustainable development’, yet with rare Scandinavian and Foundation exceptions, donors appear to be switching from longer-term development to humanitarian aid. Further, despite decades of experience, international and national nonprofit development implementers are mostly absent in the conversation around scaling-up the flow of capital to achieve and sustain development goals. Exceptions are some in the International NGO Impact Investing Network (AMPLIFY)

2. RESULTS: Funding for projects that can show great results (e.g. Social Impact Bonds/ Development Impact Bonds, which are in fact pay-for-performance instruments), even sustained impacts from partnering with local small and medium enterprises, national level ministries, and local NGOs. Far too long, implementers have been able to get funding for projects with mediocre results; impact investors are raising the bar and even donors are helping hedge risk. This includes M&E ‘impact’ value that rigorously tracks results (savvy private-sector donors require counterfactual/ control group data, isolating results from that intervention).

3. LEARNING: Impact Investors have a lot to learn from non-profits and aid donors as well.

 

  • They talk about impact but too often that is synonymous with generic results, while International and National nonprofits (NGOs) have detailed, grassroots systems in place;
  • Most seem to be content – for now – to invest in the 17 Sustainable Development Goal areas (e.g. vetting investable projects by screening criteria of not only getting a financial return, but also by broad sectoral investments, e.g. poverty, hunger, climate etc.). Many claim they have affected change, without data to prove it. The SDGs are slowly creating indicators to address this, and investors also need to be brought along to differentiate between corporate efficiency activities for their operations and those that affect change at the output, outcome and impact levels in communities;
  • There are still large leaps of logic and claims among investors and some know that data is lacking to claim good grassroots targeting and actual results that prove they are changing hunger, poverty and other sectors in Africa, Asia, Latin Americ. Good development professionals would see that the very design would make results accessible only to the elite of that country (e.g. $1 nutrition bars are inaccessible to most of a country’s population living on income of $2.00 a day)
  • We will bring with us all we know about great potential sustained impacts programming, such as Theory of Sustainability, looking for emerging results alongside planned early onlearning from failure for success, partnering successfully for country-led development, etc.

    So keep watching these ‘spaces’: www.ValuingVoices.com and www.ImpactGuild.org for updates on bridging these worlds, hopefully for ever-greater sustained impacts. Let us know if you would like to partner!

Face our fears! Learn from failure…

Face our fears! Learn from failure…

Global Giving has a nice example about getting participant feedback success/ failrure of a project their fundraising funded. The organization failed which was sorrowful to the players in West Africa and funders worldwide (I too am a soccer-mom). Yes, it failed. It is hard to read those words and all involved faced it and they learned from it. Thanks to a sprited discussion on Pelican Platform for Evidence-based Learning about a plethora of unintended impacts, I learned about the organization committed to learning from failures: Admitting Failure.  

 

Maybe some of you have heard of FailFests, such as the one in Raleigh N.C US this year that answered "Why are we doing this?" with "We’re on a mission to erase the stigma around failure. The more we talk about it, the more we can learn from it. Failure doesn’t have to be fatal!" You may know Engineers without Borders Canada has been an early proponent of "openly acknowledging failure is often a catalyst for innovation that takes our work from good to great."  Such examples are heartening, especially for participatory sustainability evaluation.

Admitting_Failure___Engineers_Without_Borders_Canada

In 2014 a met a fellow evaluator at the American Evaluation Association Conference where I was presenting on Post-Project Sustainability Evaluation. They told me their organization was waiting for a really successful project to end so they could evaluate its sustainability. Another evaluator told me of a post-project evaluation that was hidden away, unpublished, after findings were negative. And that's where the problem lies: our international development industry's neurosis about presenting anything unsuccessful.

 

No wonder so few projects have been evaluated post-project, as we fear:


* What if activities and outcomes aren't sustained (note: we didn't design them that way, often we opt for quicker wins achievable only with large resource inflows of funds and technical help)?

 

* What if our funders find out that funds had limited impact or partners had no means to continue good programming?

 

* What if we had entirely unintended impacts that favored some over others (well beyond our expected logical frameworks and Theories of Change)?

 

Hallelujah 🙂 Why can't we see this as good news that propels us to improve our projects by listening to what worked? To learn not to do what didn't work again?! To design in locally sustainable ways now that we have learned?


* What if we learn that our resources and empowerment led them to succeed on their own terms in ways we couldn't imagine, far exceeding the planned impacts we had expected?

 

* What if we find that unexpected outcomes showcased ways groups within communities stood on their feet, making development work for them on their terms?

 

Would we design and implement, monitor and evaluate projects differently? We'll see. Organizations such as USAID’s Food for Peace funded a four-country study on exit-strategies (forthcoming 2015), are looking at success and failure. Catholic Relief Services hired Valuing Voices to do sustainability evaluation in Africa. Others may follow…

 

Post-project sustainability evaluations expose us to learning the full range. Let's be brave and ask, learn and innovate, making aid and philanthropy more effective, and learn from failure for success!