Wishing for ex-post evaluation Christmas Lights rather than Needles in Haystacks

by | Dec 11, 2019 | capacity, ex-post evaluation, Failure, Indonesia, JICA, Monitoring and Evaluation, RSA South Africa, water, World Vision

 

Wishing for Ex-Post Evaluation Christmas Lights
Rather than Needles in Haystacks

 

This is what the life of most ex-post evaluation researchers looks like, mostly without the counting congratulator:

 

I recently spent three days looking for ex-post evaluations for a client across nearly a dozen organizations. I was hard-pressed to find 16 actual ones. Sorting through ‘impact evaluations’ that were done in the middle of implementation does not tell us anything about what was sustained after we left, nor do delayed final evaluations that happen to be done after closure. While these (rightly) focus on cost-effectiveness, relevance, and efficiency, measures of sustained impact are projections, not actual measures of what outcomes and impacts stood the test of time. I weeded out some desk studies that did not return to ask anyone who participated. Others titled ‘ex-post’ were barely midterms (I can only gather they misconstrued ‘ex-post’ as after-starting implementation?), and a few more reports only recommended doing ex-post evaluation after this final evaluation. For more lessons on how random and misconstrued ex-posts can be, see Valuing Voices’ research for Scriven. None of these 16 actual ex-posts even told us anything about what emerged (as we look at during Sustained and Emerging Impacts Evaluations) from local efforts in the years after assistance ended [1].

This is what I wish my ex-post haystack would look like, bountiful treasures of numerous ex-post-project evaluations, as numerous as these Christmas lights here in Tabor, Czech Republic.

 

If we had more ex-posts to learn from, we could learn from what lasted. What could locals sustain? Why? Why not? How can we do better next time? We could compare across sectors and countries, and we could see what conditions and processes during implementation supported sustainability -and, importantly – why some failed, so we don’t repeat those mistakes.

We could move from our current orange slices that end at closure to green sustainability of the project cycle:

I will be adding the ones I found to our Catalysts list soon, but when my client asked me who held databases of ex-post evaluations, I had to say only Valuing Voices and Japan’s JICA (since 1993, who even differentiates the ex-posts between Technical Grants and ODA Loans). This is not to say some cannot be found by trawling the OECD or the World Bank, but this is Needle-in-Haystack work again, and so there are only two databases to learn from. Isn’t that shocking?

Now JICA has really upped the illumination ante, so to speak: They are now doing what they are calling ‘JICA’s Ex-post Monitoring’, which was like Christmas come early [2]! Returning to learn at least 7 years after the ex-post, which was 1-3 years after closure, such as in this case of ex-post monitoring and learning from 10 projects (2007). They have done ex-post monitoring for a total of 91 cases, evaluating the sustained impacts of results, see if JICA’s recommendations to their partners had been implemented, how they had adapted to changes over a decade post-closure, and find learning for new programming. “Ex-post monitoring is undertaken 7 years after a project was completed in principle in order to determine whether or not the expected effects and impacts continue to be generated, to check that there are no sustainability-related problems with the technical capacities, systems, and finances of the executing agency nor with the operation and management of developed facilities, etc., and to ascertain what action has been taken vis-a-vis the lessons learned and recommendations gleaned during the ex-post evaluation.” While it was unclear why these specific projects were selected, it is amazing they are doing 5-10 per year.

They are my ex-post gods/ goddesses, and I fawned over two JICA evaluators at the last European Evaluation Society Conference. Why do I fawn? JICA lists 2,273 results under ex-post evaluations of Technical Cooperation, Grant Aid, ODA loans! They are literally the only organization I know whose searched reports are actually ‘ex-post’.

What we can learn from returning again is illustrated by one of JICA’s water project loans in RSA, which ended in 2003, had an ex-post in 2006, followed by monitoring of sustainability in 2013 [3]. While the report included issues of data access and evaluators expressed caution in attributing causation of positive changes to the project, but it not only continued functioning, the government of South Africa (RSA) solved barriers found at the ex-post:

  • “Data for the supply and demand of water pertaining to the Kwandebele region could not be obtained. However, considering the calculation from the water-supplied population and supplied volume and the result from the DWAF interview, water shortage could not be detected in the four municipalities studied by this project…” [3]
  • “The ex-post evaluation indicated that the four components were not in the state to be operated and managed effectively. Currently, the components are operated and managed effectively and are operating under good condition [and] concerning sustainability, improvement can be seen from the time of ex-post evaluation. Shortage of employees and insufficient technical knowledge has been resolved…” [3]
  • “Compared to the time of ex-post evaluation, improvement was seen in the under-five mortality and life expectancy. However, since the components implemented by this project are limited in comparison with the scope of the project, it is impossible to present a clear causal relationship” [3].

In another, from Indonesia’s air quality testing labs which involved capacity building and equipment maintenance 6 years after the ex-post, they mostly found training and use continued despite organizational changes and maintenance challenges: [4]

  • “After the ex-post evaluation, many of the target laboratories changed their affiliation from the Ministry of Public Works (MOPW) and MOH to provincial governments. While the relocation of equipment has been carried out in a handful of provinces, in other provinces, equipment is still located at the laboratories where it was originally installed, and these laboratories still have the right of use” [4]
  • In spite of some irregularities,”As the Ministry of Environment (MOE) still has ownership of the equipment, some laboratories have inappropriate audit results that show the allocation of O&M budget to equipment which is not included in their accounting…” [4]
  • “Out of 20 laboratories where the questionnaire survey confirmed that equipment still remained, 15 laboratories replied that spare parts for equipment are still available but are difficult to obtain…It takes several months to one year to obtain spare parts, occasionally out of Indonesia, even if a repair service is available” [4].

In this case, there were lessons learned for JICA and Indonesia’s Ministry of the Environment programs about ownership and the right use of the equipment and retiring obsolete equipment. Talk about a commitment to learning from the ongoing success or failure of one’s projects!

 

As you have read here on Valuing Voices for more than 6 years unless we include post-project sustainability that asks our participants and partners how sustained their lives and livelihoods could be and even resilient to shocks like political or climate change, we cannot say we are doing Sustainable Development. We need such lessons about what could be sustained and why.

We can prepare better to foster sustainability. In the coming months, we are working on checklists to consider during funding, design, implementation, M&E pre-and post-exit to foster sustainability. Will keep you posted, but as World Vision also found: “Measuring sustainability through ex-posts requires setting clear benchmarks to measure success prior to program closure, including timelines for expected sustainment.”

And as my gift to you this Holiday Season, let me share WV’s Learning Brief about Sustainability, with wise and provocative questions to ponder about dynamic systems, benchmarking, continuous learning, attribution, and managing expectations [5]. World Vision shares how infrastructure and community groups and social cohesion fared well, yet lessons circled back to the need for JICA-like ‘monitoring’ and mirror rich ex-post lessons from FFP/Tufts (Rogers, Coates) and Hiller et al. that explains why we do ex-posts at all: “Project impact at the time of exit does not consistently predict sustainability“ [6].

 

Now my gift: a few big lessons from the 6 years of researching sustainability across the development spectrum.  I have found no evaluations that were only positive OR ONLY NEGATIVE. Most results trended downwards, a few held steady, and all were mixed. We cannot assume the sustainability of results at closure nor optimistic projections, as we’ve seen in the climate arena.

Please consider:

  • Designing with our participants and partners so what we do,
  • Implementing with partners far longer to make sure things still work,
  • Adapting exit based on benchmarks to see how well the resources, partnerships, capacities, and ownership have been transferred,
  • Using control or comparison groups to make sure ‘success’ was due to you and being careful about attributing results to your projects while considering how you contributed to a larger whole of ongoing country progress or stagnation,
  • Being willing to jettison what is unlikely to be sustained and learn from what we designed and implemented poorly (due to our design, their implementation, and external conditions),
  • Given climate change, learning fast, adaptively, and revising fast given changing conditions,
  • Without knowing what has been sustained, we cannot replicate nor scale up,
  • Sharing lessons with your leaders – for people’s lives depend on our work,
  • Learning from what emerged as our participants and partners refashioned implementation in new ways could sustain it (without the millions we brought),
  • Refocusing ‘success’ from how much we have spent to how much was sustained.

 

Please make our next Christmas merry. Do MANY ex-post evaluations, Learn TONS, Share WIDELY WHAT WORKED AND FAILED TO WORK (you will be praised!), and let’s CHANGE HOW WE DO SUSTAINABLE DEVELOPMENT.

 

May 2020 bring health, happiness, and to all of us a more sustainable world!

 

 

Sources:

[1] Cekan, J., Zivetz, L., & Rogers, P. (2016). Sustained and Emerging Impacts Evaluation (SEIE). Retrieved from https://www.betterevaluation.org/en/themes/SEIE

[2] JICA. (n.d.). Ex-post Monitoring. Retrieved December, 2019, from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/index.html

[3] Matsuyama, K. (2012). Ex-Post Monitoring of Japanese ODA Loan Project: South Africa, Kwandebele Region Water Augmentation Project. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/c8h0vm000001rdlp-att/2012_full_03.pdf

[4] Kobayashi, N. (2009, August). Ex-post Monitoring of Completed ODA Loan Project: Indonesia, The Bepedal Regional Monitoring Capacity Development Project. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/c8h0vm000001rdlp-att/indonesia2008_01.pdf

[5] Trandafili, H. (2019). Learning Brief: What does sustainability look like post-program? Retrieved from https://valuingvoices.com/wp-content/uploads/2019/12/Sustainability-Learning-Brief_final_WV-icons.pdf

[6] Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fsnnetwork.org/ffp-sustainability-and-exit-strategies-study-synthesis-report

 

Learning from a river of ex-post project evaluations, tools and guidance… Thanks USAID!

by | Nov 10, 2019 | Aid effectiveness, civil society, ex-post evaluation, Ghana, Impact, impact evaluation, Indonesia, Maternal Child Health, MCHN, NGOs, post-project evaluation, sanitation & hygiene, Senegal, Sustainable development, USAID, water/ sanitation

Learning from a river of ex-post project evaluations and tools… Thanks USAID!

Dear ex-post aficionados. It’s raining ex-post project evaluations. Here’s hoping learning from such evaluations in water/ sanitation, maternal/child health and even capacity building/ peacekeeping, and their number increases!

 

1. WATER/ SANITATION & HYGIENE:

USAID has a series of six ex-post evaluations of the water/ sanitation and hygiene sectors since 2017! What is exciting is that they are also looking to the future. These evaluations will “provide insight into what happens after an activity ends, and how to mitigate challenges in future programming, potentially. The series will inform USAID’s WASH activity design and implementation and contribute to a larger sector discussion on achieving sustainability.”

The E3 water division (Water CKM ) took sustainability on as their strategy and have made great strides these last two years. They have done five ex-post project evaluations, cited below, and MSI has completed one more wat/san/ hygiene ex-post evaluations, specifically:

Madagascar Rural Access to New Opportunities for Health and Prosperity (RANO-HP) – Published June 2017
The first evaluation in the series explores the sustainability of the sanitation and hygiene components of the RANO-HP activity, implemented in 26 communes from 2009–2013.

Indonesia Environmental Services Program (ESP) – Published August 2017
The second evaluation in the series examines the sustainability of water utility capacity building, microcredit, and financial outcomes associated with the ESP activity, which was implemented from 2004–2010.

Ethiopia Millennium Water Alliance (MWA-EP) – Published May 2018
The third evaluation in the series examines the long-term sustainability of outcomes related to rural water point construction, rehabilitation, and management, as well as participatory sanitation and hygiene education and construction related to the MWA-EP activity, implemented in 24 rural districts between 2004–2009.

Financial Institutions Reform and Expansion–Debt and Infrastructure (FIRE-D) – Published September 2018
This evaluation is the fourth in the series. It examines how urban water and sanitation services in India have changed since FIRE-D closed and to what extent policies, practices, and financing mechanisms introduced through FIRE-D have been sustained.

Millennium Water and Sanitation Program in Senegal (PEPAM/USAID) – Published July 2019
The fifth ex-post evaluation in the series looks at the PEPAM project (Programme d’Eau Potable et d’Assainissement du Millénaire au Sénégal), implemented from 2009–2014 to improve sustainable access to WASH in four regions of Senegal.

 

USAID-funded by MSI: USAID/Ghana’s Water Access, Sanitation, and Hygiene for Urban Poor (WASH-UP)– published Nov 2018

Also USAID and Rotary International developed a WASH Sustainability Index Tool, “to assess a WASH activity’s likelihood to be sustainable according to the following factors: availability of finance for sanitation; local capacity for construction and maintenance of latrines; the influence of social norms; and governance.” This is similar to what we learned from USAID/ FFP/ Tufts/ FHI360 12 ex-posts that resources, capacities, motivation and linkages (aka partnerships, including governance) are vital to sustaining outcomes and impacts.

 

It will be interesting to see whether they examine the other ex-posts for excellent lessons, as they have the Senegalese evaluation:

  • “Whether or not to subsidize sanitation access …Based on this evaluation’s findings and exploration of the literature, subsidies can help improve the quality of household latrines, but increasing use of those latrines remains a challenge.
  • In contrast, CLTS (a nonsubsidized approach) is often credited with increasing use of unimproved latrines, but serious questions linger about quality and long-term sustainability of the latrines built after CLTS triggering, particularly as it relates to moving up the sanitation ladder. This evaluation… provides the opportunity to examine the potential value of a hybrid approach….
  • The handwashing results suggest that low-cost, low-quality handwashing stations such as tippy taps do not lead to sustained behavior change. It may be worth considering hygiene investments that reduce the behavior change burden on targeted beneficiaries.

2. MATERNAL/ CHILD HEALTH & NUTRITION:

 “Sustainability of a Community-Based CHOICE Program to Improve the Health and Nutrition Status of Mothers and Infants in Indonesia,” The report focused on whether the USAID-funded CHOICE program had left sustainable impacts: improving the health and nutrition status of children under the age of five, as well as the health status of pregnant and lactating women and mothers or caretakers of young children in the Pandeglang District of Indonesia. “After examining the data collected from the PSS, the researchers found that there were significant improvements in many indicators—such as births attended by skilled personnel, the treatment of diarrhea, and the nutritional quality of food fed to infants—in the six years after the CHOICE program ended. However, despite these improvements, the researchers found no significant statistical differences between villages that received the CHOICE program interventions and comparison villages, which did not. This speaks to using such a comparison methodology to focus on actual contribution and rule out the “rising tide lifts all boats” phenomenon.

 

3. CAPACITY DEVELOPMENT & PEACEBUILDING:

USAID’s Regional Office in Thailand evaluated its capacity building and peacebuilding program 1.5 years ex-post.  While civil society was strengthened and there were inroads made on peacebuilding, “many interventions initiated during Sapan did not continue post-Sapan, although some did remain. For example, “stakeholders cite evidence of continuing to use some governance tools in local governance related to service delivery [although] because of limited financial resources after Sapan ended, they had to change some of their interventions and reduce the range of people they could include. There are lessons for whose capacities are built, two-way feedback loops with local partners, using local organizations such as universities to sustain training, planning sufficient time for partners to internalize training lessons, etc.

 

4. USAID FUNDED GUIDANCE:

‘Impact Evaluations’ have a new focus on long-term impact, rather than effectiveness during implementation (which was at least the original intent of impact evaluation in the 1980s)! In September 2018, USAID and Notre Dame issued a Guide for Planning Long-term Impact Evaluations as part of the Utilizing the Expertise of the ERIE Program Consortium. The guide covers the difference between traditional impact evaluation designs and data collection methods and how to apply them to long-term impact evaluations (LTIE). It also shares examples across a range of sectors, including later evaluating past impact evaluations, which ended before final evaluation.

Finally, in new 2018 USAID guidance, ex-post evaluation is clarified as the source of the sustainability of services and benefits. USAID clarifies that “questions about the sustainability of project services and benefits can be asked at any stage, but must usually be adjusted to take evaluation timing into account. Thus, for example, in a mid-term evaluation, a question about the existence of a sustainability plan and early action on that plan might be appropriate. An end-of-project evaluation could address questions about how effective a sustainability plan seems to be, and early evidence concerning the likely continuation of project services and benefits after project funding ends. Only an ex-post evaluation, however, can provide empirical data about whether a project’s services and benefits were sustained.”

Such richness that we can learn from. Keep the momentum going on the 99% of all global projects yet unevaluated ex-post, and change how we fund, design, implement, monitor and evaluate global development projects!