Sustainable Development Goals (SDGs), Funding and Accountability for sustainable projects?

Sustainable Development Goals (SDGs), Funding and Accountability for sustainable projects?

What are Sustainable Development Goals? ” the United Nations adopted the new post-2015 development agenda. The new proposals – to be achieved by 2030- set 17 new ‘sustainable’ development goals (SDGs) and 169 targets. Some, like Oxfam, see the SDGs as a country budgeting and prioritization as well as an international fundraising tool. They cite that “government revenue currently funds 77% of spending…aligned with government priorities, balanced between investment and recurrent and easy to implement than donor-funded spending…” National investments are vital, but how much has the world used the SDGs to target investments and foster sustainable results?

Using results data such as that of the sectoral SDGs, countries can also ensure accountability for the policies implemented to reduce global and local inequities, but we must learn from the data. Over halfway to the goal, data is being collected, and while there is robust monitoring by countries who have built their M&E systems, other countries are faltering. “A recent report by Paris21 found even highly developed countries are still not able to report more than 40-50% of the SDG indicators” and “only 44% of SDG indicators have sufficient data for proper global and regional monitoring”. Further, there is very little evaluation or transparent accountability. Some of the data illuminate vitally need-to-know-for-better-programming. SDG data shows good news that Western and Asian countries have done better than most of the world 2015-19… but there is a lot of missing data while other data shows staggering inequities such as these:

  • In Vietnam, a child born into the majority Kinh, or Viet, ethnic group is three and a half times less likely to die in his or her first five years than a child from other Vietnamese ethnic groups.
  • In the United States, a black woman is four times more likely to die in childbirth than a white woman.

So are we using the SDG data to better target funding and improve design? This is the kind of evaluative learning (or at least sharing by those that are doing it :)) that is missing. As my colleague and friend Sanjeev Sridharan writes on Rethinking Evaluation, “As a field we need to more clearly understand evaluation’s role in addressing inequities and promoting inclusion” including “Promoting a Culture of Learning for Evaluation – these include focus on utilization and integration of evaluation into policy and programs.” How well learning is integrating is unknown.

As a big picture update on the progress of the Sustainable Development Goals (SDGs) in 2021, with only nine years left to the goal: It’s not looking good. The scorecards show COVID-19 has slowed down or wiped out many achievements, with 100 million people pushed into extreme poverty, according to the IMF. Pre-Covid, our blog on sectoral SDG statistics on health, poverty, hunger, and climate, was already showing very mixed results and a lack of mutual accountability.

The private sector is ever-being pushed to fund more of such development costs, only marginally successfully, as public sector expenditures are squeezed. Yet the G20 estimates that $2.5 TRILLION is needed every year to meet the SDG goals. As we have seen at Impact Guild, the push to incentivize private commitments is faltering. “To ensure its sustainability, the private sector has specific interests in securing long-term production along commodity supply chains, while reducing their environmental and social impacts and mitigating risks… The long-term economic impacts of funding projects that support the sustainability agenda are, thus, clearly understood. However, additional capital needs to flow into areas that address the risks appropriately. For example, much remains to be done to factor climate change as a risk variable into emerging markets that face the largest financing gap in achieving the SDGs.” Further, if decreased funding trends continue, by 2030, at minimum 400 million people will still live on less than $1.25 a day; around 650 million people will be undernourished, and nearly 1 billion people will be without energy access. So we’re not meeting the SDGs, they’re being derailed by COVID in places, and we aren’t beginning to cost out the need to address climate change and its effects on global development…. so now what?

From: https://www.g20-insights.org/policy_briefs/incentivizing-the-private-sector-to-support-the-united-nations-sustainable-development-goals/

To ensure that giving everyone a fair chance in life is more than just a slogan; accountability is crucial. This should include a commitment from world leaders to report on progress on “leaving no one behind” in the SDG follow-up and review framework established for the post-2015 agenda and for the private sector to loudly track their investments across the SDGs. For as The Center for American Progress wrote, money and results are key: We must “measure success in terms of outcomes for people, rather than in inputs—such as the amount of money spent on a project—as well as in terms of national or global outcomes” and that “policymakers at the global level and in each country should task a support team of researchers with undertaking an analysis of each commitment.”

A further concern. While we seem to measure the statistics periodically and see funding allocated to SDG priorities, but there are few causal links drawn between intensity in investment in any SDG goal and sustained results. To what degree are the donations/ investments into the SDGs linked to improvements? Without measuring causality or attribution, it could be a case of “A rising tide lifts all boats” as economies improve or, as Covid-related economic decline wiped out 20 years of development gains as Bill Gates noted last year. We need proof that trillions of dollars of international “Sustainable development” programs have any sustained impact beyond the years of intervention.

We must do more evaluation and learn from SDG data for better targeting of investments and do ex-post sustainability evaluations to see what was most sustained, impactful, and relevant. Donors should raise more funds to meet needs and consider only funding what could be sustained locally. Given the still uncounted demands on global development funding, we can no longer hope or wait for global mobilization of trillions given multiple crises pushing more of the world into crisis. Let’s focus now.

Wishing for ex-post evaluation Christmas Lights rather than Needles in Haystacks

 

Wishing for Ex-Post Evaluation Christmas Lights
Rather than Needles in Haystacks

 

This is what life of most ex-post evaluation researchers looks like, mostly without the counting congratulator:

 

I recently spent three days looking for ex-post evaluations for a client across nearly a dozen organizations. I was hard-pressed to find 16 actual ones. Sorting through ‘impact evaluations’ that were done in the middle of implementation does not tell us anything about what was sustained after we leave, nor do delayed final evaluations that happen to be done after closure. While these (rightly) focus on cost-effectiveness, relevance and efficiency, measures of sustained impact are projections, not actual measures of what outcomes and impacts stood the test of time. I weeded out some desk studies that did not return to ask anyone who participated. Others titled ‘ex-post’ were barely midterms (I can only gather they misconstrued ‘ex-post’ as after-starting implementation?) and a few more reports only recommended doing ex-post evaluation after this final evaluation. For more lessons on how random and misconstrued ex-posts can be, see Valuing Voices’ research for Scriven. None of these 16 actual ex-posts even told us anything about what emerged (as we look at during Sustained and Emerging Impacts Evaluations) from local efforts in the years after assistance ended [1].

This is what I wish my ex-post haystack would look like, bountiful treasures of numerous ex-post-project evaluations, as numerous as these Christmas lights here in Tabor, Czech Republic.

 

If we had more ex-posts to learn from, we could learn from what lasted. What could locals sustain? Why? Why not? How can we do better next time? We could compare across sectors and countries, and we could see what conditions and processes during implementation supported sustainability -and importantly – why some failed, so we don’t repeat those mistakes.

We could move from our current orange slices that ends at closure to green sustainability of the project cycle:

I will be adding the ones I found to our Catalysts list soon, but when my client asked me who held databases of ex-post evaluations, I had to say only Valuing Voices and Japan’s JICA (since 1993 who even differentiates the ex-posts between Technical Grants and ODA Loans). This is not to say some cannot be found by trawling the OECD or the World Bank, but this is Needle-in-Haystack work again and so there are only 2 databases to learn from. Isn’t that shocking?

Now JICA has really upped the illumination ante, so to speak: They are now doing what they are calling JICA’s Ex-post Monitoring’ which was like Christmas come early [2]! Returning to learn at least 7 years after the ex-post which was 1-3 years after closure, such as among this case of ex-post monitoring and learning from 10 projects (2007). They have done ex-post monitoring for a total of 91 cases, evaluating the sustained impacts of results, see if JICA’s recommendations to their partners had been implemented, how they had adapted to changes over a decade post-closure, and find learning for new programming. “Ex-post monitoring is undertaken 7 years after a project was completed in principle in order to determine whether or not the expected effects and impacts continue to be generated, to check that there are no sustainability-related problems with the technical capacities, systems and finances of the executing agency nor with the operation and management of developed facilities, etc., and to ascertain what action has been taken vis-a-vis the lessons learned and recommendations gleaned during the ex-post evaluation.” While it was unclear why these specific projects were selected, it is amazing they are doing 5-10 per year.

They are my ex-post gods/ goddesses and I fawned over two JICA evaluators at the last European Evaluation Society Conference. Why do I fawn? JICA lists 2273 results under ex-post evaluations of Technical Cooperation, Grant Aid, ODA loans! They are literally the only organization I know whose searched reports are actually ‘ex-post’.

What we can learn from returning again is illustrated by one of JICA’s water project loans in RSA, which ended in 2003, had an ex-post in 2006, followed by monitoring of sustainability in 2013 [3]. While the report included issues of data access and evaluators expressed caution in attributing causation of positive changes to the project, but it not only continued functioning, the government of South Africa (RSA) solved barriers found at the ex-post:

  • “Data for the supply and demand of water pertaining to the Kwandebele region could not be obtained. However, considering the calculation from the water supplied population and supplied volume and the result from the DWAF interview, water shortage could not be detected in the four municipalities studied by this project…” [3]
  • “The ex-post evaluation indicated that the four components were not in the state to be operated and managed effectively. Currently, the components are operated and managed effectively and are operating under good condition [and] concerning sustainability, improvement can be seen from the time of ex-post evaluation. Shortage of employees and insufficient technical knowledge has been resolved…” [3]
  • “Compared to the time of ex-post evaluation, improvement was seen in the under-five mortality and life expectancy. However, since the components implemented by this project are limited in comparison with the scope of the project, it is impossible to present a clear causal relationship” [3].

In another, from Indonesia’s air quality testing labs which involved capacity building and equipment maintenance 6 years after the ex-post, they mostly found training and use continued despite organizational changes and maintenance challenges: [4]

  • “After the ex-post evaluation, many of the target laboratories changed their affiliation from the Ministry of Public Works (MOPW) and MOH to provincial governments. While the relocation of equipment has been carried out in a handful of provinces, in other provinces equipment is still located at the laboratories where it was originally installed and these laboratories still have the right of use” [4]
  • In spite of some irregularities ”As the Ministry of Environment (MOE) still has ownership of the equipment, some laboratories have inappropriate audit results that show allocation of O&M budget to equipment which is not included in their accounting…” [4]
  • “Out of 20 laboratories where the questionnaire survey confirmed that equipment still remained, 15 laboratories replied that spare parts for equipment are still available but are difficult to obtain…It takes several months to one year to obtain spare parts, occasionally out of Indonesia, even if a repair service is available” [4].

In this case, there were lessons learned for JICA and Indonesia’s Ministry of the Environment programs about ownership and the right use of the equipment and retiring obsolete equipment. Talk about a commitment to learning from the ongoing success or failure of one’s projects!

 

As you have read here on Valuing Voices for more than six years, unless we include post-project sustainability that asks our participants and partners how sustained their lives and livelihoods could be, and even resilient to shocks like political or climate change, we cannot say we are doing Sustainable Development. We need such lessons about what could be sustained and why.

We can prepare better to foster sustainability. In the coming months we are working on checklists to consider during funding, design, implementation, M&E pre-and post-exit, to foster sustainability. Will keep you posted, but as World Vision also found: “Measuring sustainability through ex-posts requires setting clear benchmarks to measure success prior to program closure, including timelines for expected sustainment.”

And as my gift to you this Holiday Season, let me share WV’s Learning Brief about Sustainability, with wise and provocative questions to ponder about dynamic systems, benchmarking, continuous learning, attribution, and managing expectations [5]. World Vision shares how infrastructure and community groups and social cohesion fared well, yet lessons circled back to the need for JICA-like ‘monitoring’ and mirror rich ex-post lessons from FFP/Tufts (Rogers, Coates) and Hiller et al. that explains why we do ex-posts at all: “Project impact at the time of exit does not consistently predict sustainability“ [6].

 

Now my gift: a few big lessons from  the six years of researching sustainability across the development spectrum.  I have found no evaluations that were only positive. Most results trended downwards, a few held steady, and all were mixed. We cannot assume the sustainability of results at closure, nor optimistic projections as we’ve seen in the climate arena.

Please consider:

  • Designing with our participants and partners so what we do,
  • Implementing with partners far longer to make sure things still work,
  • Adapting exit based on benchmarks to see how well the resources, partnerships, capacities, and ownership have been transferred,
  • Using control or comparison groups to make sure ‘success’ was due to you and being careful about attributing results to your projects while considering how you contributed to a larger whole of ongoing country progress or stagnation,
  • Being willing to jettison what is unlikely to be sustained and learn from what we designed and implemented poorly (due to our design, their implementation, external conditions),
  • Given climate-change, learning fast, adaptively and revising fast given changing conditions,
  • Without knowing what has been sustained we cannot replicate nor scale-up,
  • Sharing lessons with your leaders – for people’s lives depend on our work,
  • Learning from what emerged as our participants and partners refashioned implementation in new ways could sustain it (without the millions we brought),
  • Refocusing ‘success’ from how much we have spent, to how much was sustained.

 

Please make our next Christmas merry. Do MANY ex-post evaluations, Learn TONS, Share WIDELY WHAT WORKED AND FAILED TO WORK (you will be praised!), and let’s CHANGE HOW WE DO SUSTAINABLE DEVELOPMENT.

 

May 2020 bring health, happiness, and to all of us a more sustainable world!

 

 

Sources:

[1] Cekan, J., Zivetz, L., & Rogers, P. (2016). Sustained and Emerging Impacts Evaluation (SEIE). Retrieved from https://www.betterevaluation.org/en/themes/SEIE

[2] JICA. (n.d.). Ex-post Monitoring. Retrieved December, 2019, from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/index.html

[3] Matsuyama, K. (2012). Ex-Post Monitoring of Japanese ODA Loan Project: South Africa, Kwandebele Region Water Augmentation Project. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/c8h0vm000001rdlp-att/2012_full_03.pdf

[4] Kobayashi, N. (2009, August). Ex-post Monitoring of Completed ODA Loan Project: Indonesia, The Bepedal Regional Monitoring Capacity Development Project. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/oda_loan/monitoring/c8h0vm000001rdlp-att/indonesia2008_01.pdf

[5] Trandafili, H. (2019). Learning Brief: What does sustainability look like post-program? Retrieved from https://valuingvoices.com/wp-content/uploads/2019/12/Sustainability-Learning-Brief_final_WV-icons.pdf

[6] Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fsnnetwork.org/ffp-sustainability-and-exit-strategies-study-synthesis-report