by Jindra Cekan | Apr 9, 2021 | business, environment, ESG, ex-post evaluation, foreign aid, Impact, impact investing, OECD, Sustainable development
“What IS Sustainability?” It depends on whom you ask: OECD, the UN, or Harvard Business School
Recently I’ve had conversations where I had to define which sustainability we were talking about. Was it:
- ex-post-project sustainability of outcomes and impacts,
- environmental sustainability, or
- business sustainability?
Since I spend most of my time evaluating the ex-post sustained and emerging impacts of foreign aid projects years after projects close, or at least advocate for it, let’s start there.
- The Organisation for Economic Co-operation and Development (OECD) is a “forum and knowledge hub for data and analysis, exchange of experiences, best-practice sharing, and advice on public policies and international standard-setting.” Regarding evaluation specifically, the OECD has “established common definitions for six evaluation criteria – relevance, coherence, effectiveness, efficiency, impact, and sustainability – to support consistent, high-quality evaluation”. Focusing on long-term sustainability, their evaluation guidance is:
Source: OECD, Better Criteria for Better Evaluation, 2019
The good news is that in this recent publication on Applying Evaluation Criteria Thoughtfully (2021), OECD keeps the updated definition but inches towards recommending actual ex-post project sustainability evaluation, rather than just projected (and assumed “likely to continue” sustainability). For this, “likely” is the most significant reason evaluators for donors and implementers have assumed, rather than evaluated, sustainability for decades. Further, positive, ‘sustained’ trajectories are also assumed at close-out/ exit, but rarely tested ex-post.
The OECD criteria give not evaluating it as an option. I far prefer “net benefits of the intervention continue” as it is a marching order: Prove that results were sustained. In this evolution, this 2021 report states, “After the completion of the intervention, and evaluation of sustainability would look at whether or not the benefits did continue, this time drawing on data and evidence from the intervention’s actual achieved benefits.”
OECD even goes on to recommend implementing and monitoring for sustainability. The new piece de resistance is: “Sustainability should be considered at each point of the results chain and the project cycle of an intervention”:
- “The sustainability of inputs (financial or otherwise) after the end of the intervention and the sustainability of impacts in the broader context of the intervention…. as well as whether there was willingness and capacity to sustain financing (resources) at the end of the intervention
- For example, an evaluation could assess whether an intervention considered partner capacities
- Built ownership at the beginning of the implementation period…. And
- In general, evaluators can examine the conditions for sustainability that were or were not created in the design of the intervention and by the intervention activities and whether there was adaptation where required.”
Yes, Valuing Voices highlighted this at the American Evaluation Association presentation “Barking up a Better Tree: Lessons about SEIE Sustained and Emerging Impact Evaluation” in 2016, and we have developed this into 2020’s Exiting for Sustainability trainings and checklists. Wonderful to see implementing for sustainability in guidance by the OECD! Many of the elements his research cites overlap with our Exiting for Sustainability checklislts which includes ownership, resources, capacities, partnerships, and implementation/ M&E process.
Moreover, while the 2019 OECD report mentioned resilience in passing, related to sustainability, “encourages analysis of potential trade-offs, and of the resilience of capacities/ systems underlying the continuation of benefits”. Such resilience and continuation of benefits evaluation involve examining huge systems (the financial, economic, social, environmental, and institutional capacities) that projects and programs are implemented within, whose stability is needed to sustain net benefits over time. Yes, for ex-post sustainability questions for evaluators to consider should include: “To what extent did the intervention contribute to strengthening the resilience of particularly disadvantaged or vulnerable groups” on which the sustained impacts of so much of our “Leave No One Behind” myth of Sustainable Development rely.
However, OECD makes suggestions to evaluate even broader, overwhelming what is feasible: “…this involves analyses of resilience, risks, and potential trade-offs.” Whose? All stakeholders, from participants to local partners and national and international implementers, and international donors? How far back and how far forward? What a huge undertaking. Further, the OECD points evaluators to define resilience, but as I learned in my Famine Early Warning System research and a current ex-post evaluation process for the Adaptation Fund, that involves creating evaluable boundaries by determining resilient to what kinds of shocks? Vital questions current industry monitoring and evaluation budgets for all evaluations, much less (too-rare) ex-post project evaluations, are insufficient for as they hover around 3-5% of total costs.
- Slight progress at OECD is being made by acknowledging environmental sustainability first brought up by the Brundtland Report, “Our Common Future” back in 1987. This linchpin report highlighted that “critical global environmental problems were primarily the result of the enormous poverty of the South and the non-sustainable patterns of consumption and production in the North. It called for a strategy that united development and the environment – described by the now-common term “sustainable development”… that meets the needs of the present without compromising the ability of future generations to meet their own needs.”
While an OECD brief in 2008 considers the environmental aspects of our thinking about sustainability, it argues that sustainability primarily about “using economic development to foster a fairer society while respecting ecosystems and natural resources.” The 2021 Applying Evaluation Criteria Thoughtfully rather unhelpfully mostly ignores the environment’s role in sustainability: “Confusion can arise between sustainability in the sense of the continuation of results, and environmental sustainability or the use of resources for future generations. While environmental sustainability is a concern (and maybe examined under several criteria, including relevance, coherence, impact, and sustainability), the primary meaning of the criteria is not about environmental sustainability as such; when describing sustainability, evaluators should be clear on how they are interpreting the criterion.” Given rapid climate change, I would argue that any sustained and emerging outcomes and impacts of projects that does not include an evaluation of the environmental context will fail to foster sustained resilience. Yet donors’ fixed funding timeframes that set completion to disbursement without evaluating sustainability or resilience continue to be huge barriers.
In 2022 a new resource on Sustainability from the media perspective questions the OECD. Sustainability: Going Beyond the Buzzword groups sustainability described in various media as: 1) Economic viability, 2) Social Sustainability, 3) Institutional Sustainability and 4) technical sustainability. But we’ve learned in 2022 from consulting work that the technology piece mentioned in this Buzzwords study regarding ‘baking in’ sustainability via the infrastructure/ assets created (The Adaptation Fund’s ex-post and resilience evaluations) and incorporating INGOs’ considerations of buying-local, seeing what local spare parts and technicians will be available in the future is a great added consideration. So too is promoting locally-led development through the very participatory way projects are implemented, with locals leading. Among the best resources is CDA/ Peace Direct/ Search for Common Ground’s Stopping as Success studies for USAID.
- Finally, business sustainability brings together these impacts on communities and society along with impacts on the environment. These are called ESG (Environmental, Social, and Governance) criteria. A Harvard Business School brief defines sustainability as “doing business without negatively impacting the environment, community, or society as a whole. “Where applied well, the aspiration is that “beyond helping curb global challenges, sustainability can drive business success.” While Harvard Business Review highlights “What Works’ in Calculating the Value of Impact Investing, they are, like almost all of global development ‘while- we-are-there’-measures. There is one mention of ‘terminal value’, 5 years after close of ownership, and they estimate social return on investments. This is a good, step, but as insufficient as foreign aid – for these are projected, not actual results.
At Valuing Voices, we have found hopeful examples such as IKEA as well as where ‘impact investing’ hype does not match the claims. Nonetheless, increasingly businesses are trying to consider circular economy systemic principles of “economic development designed to benefit businesses, society, and the environment.” This is regenerative, aims to decouple growth from the consumption of finite resources, not generate excess waste that cannot be reused and actuals seem to be measured at least during investments. As Harvard notes, “this leads investors to look at factors such as a company’s carbon footprint, water usage (both Environment), community development efforts (Social), and board diversity (Governance).” We encourage them to measure long-term/ longitudinally. A current Harvard Business Review sobering article on the ineffectiveness so far of measuring environmental sustainability and ESG. “…reporting is not a proxy for progress. Measurement is often nonstandard, incomplete, imprecise, and misleading. And headlines touting new milestones in disclosure and socially responsible investment are often just fanciful ‘greenwishing’”.
Australia’s RMIT defines business sustainability as comprising 4 pillars: Human, Social, Economic, and Environmental which combines a) “Human sustainability focuses on the importance of anyone directly or indirectly involved in the making of products, or provision of services or broader stakeholders;… b) Social sustainability focuses on maintaining and improving social quality with concepts such as cohesion, reciprocity and honesty and the importance of relationships amongst people;… c) Economic sustainability aims to improve the standard of living [and] the efficient use of assets to maintain company profitability over time;… d) Environmental sustainability places emphasis on how business can achieve positive economic outcomes without doing any harm, in the short- or long-term, to the environment.” But how well measured?!
Would ESG success be sustained over the long-term rather than short-term shareholder profit cycles? Will the OECD start to recommend extensive ex-post evaluation? Will they develop guidance to incorporate environmental concerns in evaluation for our common good? I do not yet know, but I implore these silos to start talking. No time to waste!
As my colleague and collaborator Susan Legro commented, we need to:
1) Continue to seek clarity and specificity in the terminology that we use, ensuring that it is clear to all stakeholders and beneficiaries; and
2) Find ways to study projects and initiatives over the longer term, which is the only way to study the designation of “sustainable” for any initiatives seeking that label.
3) I’ll add spread the word, as I just found out Harvard’s Extension School course on Sustainability features this blog in its curriculum 🙂
What are your thoughts?
by Jindra Cekan | Dec 18, 2018 | Sustainable development
There are six bright Hallelujahs on the sustained impact journey this Holiday. I’ve paired each of them by a wish, so appropriate this Holiday Season.
1. Netherlands Greatness! The government of theNetherlands has recently started talking with their implementing agencies about how to track and prove sustainability six years after exit, as one Dutch colleague told me at the European Evaluation Society Conference this Oct.
WOW.
As we know, demanding evaluation of sustainability after project closure is rarer than vegan turkey on Thanksgiving, and the Dutch Ministry of ForeignAffairs is tacking the issues of a) how will they pay for it after project funding has ended (hint: see 3ie), b) how will the data disseminated and influence which decision-makers? Note: Japan’s development agency JICA is the only other bilateral to have this as a mandated aspect of their ongoing evaluations. We congratulate the Dutch for joining Japan in sustained impact evaluation leadership!
My Wish: For the other bilaterals and multilaterals for whom such evaluations are exotically occasional learnings to see the value from learning about the successes and failures of how(un)sustained your results are. Also to fund such evaluations for at least 10% of all programming per year, and do better programming focused on helping locals sustain the results they want. See, that wasn’t such a huge wish was it?
…
2. USAID Transformation to Self-Resilience is a promising analysis USAID is doing under the leadership of AdministratorMark Green. This Transformation “Journey to Self-Reliance” is “reorienting the way it does business to focus on supporting our partner countries… to solve their own development challenges.” It consists of a variety of policies, from using more private sector money for development to two exit-related initiatives including Financing Self-Reliance “to strengthen support for our host country partners in their efforts to finance their own development journey”and includes continuing to build local capacity and Partnering and Procurement such as using adaptive approaches and “principles of supporting locally-led development and long-term sustainability.”
Will USAID really follow the SDG aim to Leave No one Behind in terms of focusing on the poor? Metrics of USAID’s Country Roadmaps, range from national economic growth and democracy/governance to sectoral measures in health, food security etc. Maybe this is not just a way to use national data to justify exit of aid programs due to shrinking funding, but instead is a way for USAID to determine who is actually self-reliant among their recipient countries and focus more resources on those who aren’t on more needs-based criteria than politically strategic bases than in the past.Maybe it will look hard at assumptions rather than Uganda’s CDCS which projects sustainability in 30 years but will program on until then. OXFAM’sAid Localization may be a better, faster route.
My Wish: that bilateral donors like USAID test how well their current aid projects support self-reliance and how well they’re furthering sustainability. This includes learning what has been self-sustained and not renewing what hasn’t. We offer our database of 25 catalytic organizations’ ex-post project evaluations at Valuing Voices which were informed by local voices of our participants and partners. These fascinating evaluations need to inform new programming as sustainability is only high occasionally, unlike what we assume. We ask commissioners of such evaluations to tell us what they have learned and how they have done things differently (e.g. have they funding differently, chosen what projects are selected as implementers differently, and while adaptive monitoring is a terrific addition, how has evaluation changed -other than pending OECD DAC Criteria and Ofir’s blog commentary plus the WorldBank’s sustainability blog) and how has exit, sustainability planning and handover to national government using data about sustainability and impact changed and people really can sustain themselves? NOTE: this may lead to less money being spent over a longer time, more flexibly and locally-directed. We donor nations may benefit less as locals take their own ‘development’ in hand. Hallelujah.
…
3. Recently I did a webinar for InterAction webinar with dear sustained impact aficionados from Catholic Relief Services and World Vision on “How Sustained and Resilient are our Impacts?” Lessons about accountability for locally sustainable development was front and center. We all shared that ex-posts project evaluations have, shall we say, mixed results of success and failure. This can be difficult for agencies to absorb as the projects selected are not only typically selected to showcase success, but normally these are selected to be showcased as the best projects. Also, unsustained results can question the quality of funder’s design the agency’s and implementation, risking reputation and funding for both. But hiding failure is worse. A recent Evidence Action statement admitting failure of a promising intervention, promising to evaluate why and learn from it only gained kudos. Conversely, great results can show where future focus could be, or much can be (rightly?) attributed to unexpected new conditions which led it in very different directions.
My Wish: My CRS colleague was challenged by a younger American colleague who asked him “are we liable if we don’t deliver sustainable development?” Astonished, he answered “yes.”. Let’s get over our infallibility and inclination to promise something only the countries themselves can deliver. Yes, we are liable for wasting money and locals time and hopes. Yes, we can claim success when partnerships go well. May we all learn from failures, celebrate successes –especially those locals consider successes, and replicate only those.
…
4. Impact Investment is slowing inching forward in helping move more of the Sustainable Development Goals (SDGs) toward fruition. The pressure on many global funders and countries alike to meet the 17 SDG targets set in 2015 is rising. Funding needs to accomplish the goals is $4 trillion a year, yes, trillion, compared to $150 billion available now in foreign aid. Accessing private capital is a wise investment if such development could unleash up to $12 trillion in growth. Repeatedly people like myself and Impact Guild read that “demand for SDG-aligned investment products outstrips supply“ which is true not in terms of the actual business and civil society non-profits trying to generate Social, Environmental, Governmental and financial Return on Investment (SROI and ESG), but in terms of profitable and stable investments. Investors tend to be risk-averse yet there are a range instruments that cover the range of development to humanitarian needs. It is not a mechanism to ignore as for-profit funding will sharply increase in coming years with up to $8 trillion of impact investments moving into global disasters recovery alone and vital “climate finance” investments only going to grow, as $100 billion are year is being raised for poor countries to stem the rising effects of climate change. Combined with a $10.3 billion private Green Climate Fund that privately funds climate resilience projects all over the world to meet SDG13, we can no longer rely on diminishing foreign aid alone.
While patient pessimism is needed to overcome the hype that now private capital can replace philanthropy, INGOs are – cautiously innovating. As this month’s Amplify report from 40 international NGOs dipping their toes into impact investment collaborations notes, there are a wide array of collaborations. Some are investing their own private donors funds – or those leveraging other donors’ and investors’ funds in Development Impact Bonds (DIBs) or Social Impact Bonds (SIBs), and other instruments to the tune of $545.1million in assets. These are mainly in profit-low-hanging fruit development sectors of agriculture, livelihoods, and financial inclusion. Yet some of the INGO’s greatest strengths are least appreciated and what the relatively risk-averse and ‘due-diligence-focused’ impact investors do not yet know they need: ”their deep knowledge of local environments, programs, and technical solutions; their long-standing networks and sophistication in partnering with multiple actors; their financial sophistication from their wide-ranging donor relationships; and their experience in complex, multi-year measurement of impact.“
My Wish: that these two industries listen to one another. Non-profits can be mired in a can’t spend-can’t-risk mindset while being queasy about generating revenue, while impact investors can be torn between profits being king and claims of ‘impact’ from just investing in a sector without monitoring actual effects on the ground. There is simply too much good to be done.
…
5. Brazil/ France’s Corporate Social Responsibility (CSR) – The French renewable energy company Voltalia has created their first CSR-funded Social Projects for Social Responsibility. It is using some excellent sustainability principles as well as leveraging Brazilian Bank BNDES social sub-credit seed financing. I met Voltalia’s small CSR team at the EES conference. They started working in impoverished northeast Brazil, using great non-profit processes of communities identifying needed projects and processes that overlapped with what CSR could offer, over half of which were in education, health and ‘social responsibility’ (aka livelihoods linked to water access and use such as fisheries etc). They invested substantial funds into these grassroots projects, alongside interesting Social Management Tools that included cost management tracking of local co-investment, great collaborative communications plans, sustainability reporting, including Social Return on Investment (SROI). Like many non-profits before them, it has been hard, but also enormously rewarding and their work has led to greater employee engagement and was recognized by global headquarters – kudos! – interested? Contact them.
My Wish: Walking in the shoes of our partners and participants teaches us about their capacities and needs, and how to adapt what we can offer to be of greatest use. We need more companies with ‘dust on their shoes’, as my PhD advisor called my fieldwork. This epitomizes the Sidekick Manifesto approach where we offer local leaders help, rather than ‘solving’ poverty, hunger, ill-health, un-representation for them (which our efforts only accomplish for a short-term anyway, see #1, 2).
…
6. Climate Conference COP24 corralled nearly 200 nations to create technical targets and measurements to try to keep us below IPCC’s dire projections that we are not on target to limit the temperature increase to 1.5C degrees, and that it will cost billions to the economies of the world, not to mention jeopardize past development gains. You may wonder why this is a Hallelujah. It is because for those I most care about – my children and future generations-especially those in the ‘less developed countries’ who have least-caused yet are least able to mitigate climate change effects- are finally starting to be seen and heard. Conferences such as COP24, as well as a huge range of ‘non-state actors’, including corporations, non-profits, philanthropies, tribes, even individuals like you and me. Whether it is our self-interest or expenditure-avoidance which propels us to decrease our emissions and increase our use of clean technologies, or our altruism toward the millions of species on earth, we have globally begun turning to sustainability of the ecological kind. Scientists are proposing remarkable inventiveness to capture already present CO2, studies are showing how we can be more energy efficient, buy more sustainable fashion, to eat more kindly, etc.
My Wish: We have begun changing our consumption and carbon emission and our youth are pushing us to conserve our planet. I am hopeful that we will come together to address this threat to all species. Is it far enough? Is it fast enough? Santa – help! So since it’s my holiday hallelujah, read Hans Rosling on how it’s all getting better anyway.
Happy Holidays and New Year everyone!
