Valuing Voices Better Ex-Post Evaluation White Paper: Nordics & the Netherlands

Donors often talk about #SustainableDevelopment, and we know they try. But do they return #expost to #evaluate it? Here are lessons from #Nordic (#Norway#Finland#Sweden & #Denmark) and the #Netherlands in our #ValuingVoices #WhitePapers. Let us know what you think… Thanks, Jindra & Preston Stewart..

The document(s) cover:
1) the search process,
2) how ex-post evaluations are defined and categorized,
3) what was done well by each country’s ex-posts,
4) sustainability-related findings and lessons, and
5) what M&E experts in each country can improve on ex-post evaluation practices.
6) Case studies of the Netherlands, Norway, and Finland
One big finding is that there were only 32 evaluations that seemed to be ex-post project, and only 16 of them actually were at least 2 years after project closure.

Executive Summary/ Abstract of the set of White Papers

Ex-post completion or ex-post project evaluations are surprisingly rare given widespread commitments to ‘sustainable development’. Given the well-known commitment of the Nordic countries of Norway, Finland, Sweden, and Denmark as well as the Netherlands, Valuing Voices, a consulting firm specializing in ex-post project evaluation, chose these countries as case studies. We hired an intern from Harvard College, who searched the national databases (in English only) and presented these results. The findings were surprising. There were far fewer ex-posts in total than expected. Those that were included a surprising number that was co-evaluated alongside final evaluations, although final evaluation illuminates relevance, effectiveness, efficiency (and the new OECD criteria of coherence), while ex-post project evaluation illuminates sustainability and impact two or more years after closure.  There were lessons for Ministries about the searchability as well as the quality (and in the case of Denmark and in part, Sweden), the dearth of ex-posts at all. Overall, the recommendations for the five countries researched for this paper include better and more standardized definitions of ex-post project, greater absolute numbers done, transparent sharing of those done in publicly searchable manners,  methodologically clear comparisons (baseline and midterm), and clarity in differentiating different evaluations.

 

There are caveats to these papers. This research was privately by Valuing Voices so the samples featured from the five countries were limited to public sources about ex-post and ex-post evaluations.  While we reached out to Ministry evaluative staff in all countries, only two made themselves available for consultation (the Netherlands and Finland) and provided a handful of additional ex-post evaluations.  This paper focuses on what such research yielded, not definitive findings of programs or multi-year country strategies that are funded for 20-30 years continuously, nor projects funded by country-level embassies which did not feature on the Ministry site. We focus on project bilateral project evaluations, not multilateral funding of sectors. We also presented these findings in a 2021 webinar, during which we received input that Sweden’s EBA has a (non-project) portfolio of ‘country evaluations’ which looked back over 10 or even 20-year time horizons.. Still, we found very few ex-post evaluations at EBA (see cite in Search, below).  No input was received by Norway or Denmark.

 

There are four papers in this series combined in this White Paper. First, the search across databases of all five Nordics (Norway, Finland, Sweden, and Denmark) and the Netherlands (pages 1-21). This is followed by country-specific, detailed, papers for those with ex-posts: The Netherlands (pages 22-37), Norway (pages 38-47), and Finland (pages 48-59).  We look forward to your feedback. Jindra@ValuingVoices.com for the Valuing Voices team.

Let us know what you think… Thanks, Jindra & Preston Stewart

“What IS Sustainability?” It depends on whom you ask: OECD, the UN, or Harvard Business School

“What IS Sustainability?” It depends on whom you ask: OECD, the UN, or Harvard Business School

Recently I’ve had conversations where I had to define which sustainability we were talking about. Was it:

  1. ex-post-project sustainability of outcomes and impacts,
  2. environmental sustainability, or
  3. business sustainability?

 

Since I spend most of my time evaluating the ex-post sustained and emerging impacts of foreign aid projects years after projects close, or at least advocate for it, let’s start there.

  1. The Organisation for Economic Co-operation and Development (OECD) is a “forum and knowledge hub for data and analysis, exchange of experiences, best-practice sharing, and advice on public policies and international standard-setting.” Regarding evaluation specifically, the OECD has “established common definitions for six evaluation criteria – relevance, coherence, effectiveness, efficiency, impact, and sustainability – to support consistent, high-quality evaluation”. Focusing on long-term sustainability, their evaluation guidance is:

 

 

Source: OECD, Better Criteria for Better Evaluation, 2019

 

The good news is that in this recent publication on Applying Evaluation Criteria Thoughtfully (2021), OECD keeps the updated definition but inches towards recommending actual ex-post project sustainability evaluation, rather than just projected (and assumed “likely to continue” sustainability). For this, “likely” is the most significant reason evaluators for donors and implementers have assumed, rather than evaluated, sustainability for decades. Further, positive, ‘sustained’ trajectories are also assumed at close-out/ exit, but rarely tested ex-post.

The OECD criteria give not evaluating it as an option. I far prefer “net benefits of the intervention continue” as it is a marching order: Prove that results were sustained. In this evolution, this 2021 report states, “After the completion of the intervention, and evaluation of sustainability would look at whether or not the benefits did continue, this time drawing on data and evidence from the intervention’s actual achieved benefits.”

 

OECD even goes on to recommend implementing and monitoring for sustainability. The new piece de resistance is: “Sustainability should be considered at each point of the results chain and the project cycle of an intervention”:

  1. “The sustainability of inputs (financial or otherwise) after the end of the intervention and the sustainability of impacts in the broader context of the intervention…. as well as whether there was willingness and capacity to sustain financing (resources) at the end of the intervention
  2. For example, an evaluation could assess whether an intervention considered partner capacities
  3. Built ownership at the beginning of the implementation period…. And
  4. In general, evaluators can examine the conditions for sustainability that were or were not created in the design of the intervention and by the intervention activities and whether there was adaptation where required.”

Yes, Valuing Voices highlighted this at the American Evaluation Association presentation “Barking up a Better Tree: Lessons about SEIE Sustained and Emerging Impact Evaluation” in 2016, and we have developed this into 2020’s Exiting for Sustainability trainings and checklists. Wonderful to see implementing for sustainability in guidance by the OECD!

 

Moreover, while the 2019 OECD report mentioned resilience in passing, related to sustainability, “encourages analysis of potential trade-offs, and of the resilience of capacities/ systems underlying the continuation of benefits”. Such resilience and continuation of benefits evaluation involve examining huge systems (the financial, economic, social, environmental, and institutional capacities) that projects and programs are implemented within, whose stability is needed to sustain net benefits over time. Yes, for ex-post sustainability questions for evaluators to consider should include: “To what extent did the intervention contribute to strengthening the resilience of particularly disadvantaged or vulnerable groups” on which the sustained impacts of so much of our “Leave No One Behind” myth of Sustainable Development rely.

 

However, OECD makes suggestions to evaluate even broader, overwhelming what is feasible: “…this involves analyses of resilience, risks, and potential trade-offs.” Whose? All stakeholders, from participants to local partners and national and international implementers, and international donors? How far back and how far forward? What a huge undertaking. Further, the OECD points evaluators to define resilience, but as I learned in my Famine Early Warning System research and a current ex-post evaluation process for the Adaptation Fund, that involves creating evaluable boundaries by determining resilient to what kinds of shocks? Vital questions current industry monitoring and evaluation budgets for all evaluations, much less (too-rare) ex-post project evaluations, are insufficient for as they hover around 3-5% of total costs.

 

  1. Slight progress at OECD is being made by acknowledging environmental sustainability first brought up by the Brundtland Report, “Our Common Future” back in 1987. This linchpin report highlighted that “critical global environmental problems were primarily the result of the enormous poverty of the South and the non-sustainable patterns of consumption and production in the North. It called for a strategy that united development and the environment – described by the now-common term “sustainable development”… that meets the needs of the present without compromising the ability of future generations to meet their own needs.”

 

While an OECD brief in 2008 considers the environmental aspects of our thinking about sustainability, it argues that sustainability primarily about “using economic development to foster a fairer society while respecting ecosystems and natural resources.” The 2021 Applying Evaluation Criteria Thoughtfully rather unhelpfully mostly ignores the environment’s role in sustainability: “Confusion can arise between sustainability in the sense of the continuation of results, and environmental sustainability or the use of resources for future generations. While environmental sustainability is a concern (and maybe examined under several criteria, including relevance, coherence, impact, and sustainability), the primary meaning of the criteria is not about environmental sustainability as such; when describing sustainability, evaluators should be clear on how they are interpreting the criterion.” Given rapid climate change, I would argue that any sustained and emerging outcomes and impacts of projects that does not include an evaluation of the environmental context will fail to foster sustained resilience. Yet donors’ fixed funding timeframes that set completion to disbursement without evaluating sustainability or resilience continue to be huge barriers.

 

  1. Finally, business sustainability brings together these impacts on communities and society along with impacts on the environment. These are called ESG (Environmental, Social, and Governance) criteria. A Harvard Business School brief defines sustainability as “doing business without negatively impacting the environment, community, or society as a whole. “Where applied well, the aspiration is that “beyond helping curb global challenges, sustainability can drive business success.” While Harvard Business Review highlights “What Works’ in Calculating the Value of Impact Investing, they are, like almost all of global development ‘while- we-are-there’-measures. There is one mention of ‘terminal value’, 5 years after close of ownership, and they estimate social return on investments. This is a good, step, but as insufficient as foreign aid – for these are projected, not actual results.

At Valuing Voices, we have found hopeful examples such as IKEA as well as where ‘impact investing’ hype does not match the claims. Nonetheless, increasingly businesses are trying to consider circular economy systemic principles of “economic development designed to benefit businesses, society, and the environment.” This is regenerative, aims to decouple growth from the consumption of finite resources, not generate excess waste that cannot be reused and actuals seem to be measured at least during investments. As Harvard notes, “this leads investors to look at factors such as a company’s carbon footprint, water usage (both Environment), community development efforts (Social), and board diversity (Governance).” We encourage them to measure long-term/ longitudinally. A current Harvard Business Review sobering article on the ineffectiveness so far of measuring environmental sustainability and ESG. “…reporting is not a proxy for progress. Measurement is often nonstandard, incomplete, imprecise, and misleading. And headlines touting new milestones in disclosure and socially responsible investment are often just fanciful ‘greenwishing’”.

Australia’s RMIT defines business sustainability as comprising 4 pillars: Human, Social, Economic, and Environmental which combines a) “Human sustainability focuses on the importance of anyone directly or indirectly involved in the making of products, or provision of services or broader stakeholders;… b) Social sustainability focuses on maintaining and improving social quality with concepts such as cohesion, reciprocity and honesty and the importance of relationships amongst people;… c) Economic sustainability aims to improve the standard of living [and] the efficient use of assets to maintain company profitability over time;… d) Environmental sustainability places emphasis on how business can achieve positive economic outcomes without doing any harm, in the short- or long-term, to the environment.” But how well measured?!

Would ESG success be sustained over the long-term rather than short-term shareholder profit cycles? Will the OECD start to recommend extensive ex-post evaluation? Will they develop guidance to incorporate environmental concerns in evaluation for our common good? I do not yet know, but I implore these silos to start talking. No time to waste!

As my colleague and  collaborator Susan Legro commented, we need to:

1) Continue to seek clarity and specificity in the terminology that we use, ensuring that it is clear to all stakeholders and beneficiaries; and

2) Find ways to study projects and initiatives over the longer term, which is the only way to study the designation of “sustainable” for any initiatives seeking that label.

What are your thoughts?

 

Upcoming Jan Webinar: Lessons from Nordic / the Netherlands’ ex-post project evaluations: 14 Jan 2021

Upcoming Webinar: Lessons from Nordic / the

Netherlands’ ex-post project evaluations: 14 Jan 2021

In June-August 2020, Preston Stewart, our Valuing Voices intern, conducted through government databases of the four Nordic countries – Norway, Finland, Sweden, and Denmark – and the Netherlands to identify ex-post evaluations of government-sponsored projects. The findings and recommendations for action are detailed in a four-part white paper series, beginning with a paper called The Search.

We would like to invite you to a presentation of our findings, and a discussion of what can be learned about:

1) the search process,

2) how ex-post evaluations are defined and categorized,

3) what was done well by each country’s ex-posts,

4) sustainability-related findings and lessons, and

5) what M&E experts in each country can improve on ex-post evaluation practices.

One big finding is that there were only 32 evaluations that seemed to be ex-post project, and only 1/2 of them actually were at least 2 years after project closure.

 

We have many more lessons about conflicting definitions, that ex-post evaluation is not the norm in the evaluations processes of the five governments, that development programs could, if committed to ex-post evaluation, learn about sustainability by engaging with the findings from many more such evaluations, and to increase accountability to the public and for transparent learning, ex-post evaluations should be shared in public, easy-to-access online repositories.

Join us!

REGISTER: https://www.eventbrite.com/e/webinar-lessons-from-nordic-the-netherlands-ex-post-project-evaluations-tickets-132856426147 

Your ticket purchase entitles you to the webinar, its meeting recording, associated documents, and online Sustainability Network membership for resources and discussion. Payment on a sliding, pay-as-you-can scale.

Interview repost: Why Measure & Evaluate Corporate Sustainability Projects?

Interview repost: Why Measure & Evaluate Corporate Sustainability Projects?

A CSR firm in Central Europe asked me to talk about sustainability, evaluation, and Corporate Social Responsibility. We had a terrific interview – please click thru to: https://www.besmarthead.com/cs/media/post/smartheadtalks-2-jindra-cekan-eng

 

 

The topics we discussed include:

* Sustainability in CSR involves value creation through benefits-creation for both companies and consumers

* Projects feeding into the United Nations Sustainable Development Goals may not be sustained over the long-term and we must return to evaluate what could be sustained and what emerged ex-post

* Scale measurable and meaningful impact investment through the transformation of development nonprofits’ programming + approach

* Three activities in sustainability that companies can start doing right away, that you would recommend to any company anywhere in the world

And Very Happy Holidays to all… may our lives and world be more sustainable in 2021….