Reblogged from the journal EVALUATION,Volume: 28 issue: 1, page(s): 7-35, Article first published online: January 24, 2022; Issue published: January 1, 2022
“Leading evaluation practitioners were asked about lessons from the recent 26th Conference of the Parties (COP26) for evaluation practice. Contributors emphasize the importance of evaluating equity between rich and poor countries and other forms of climate injustice. The role of the evaluation is questioned: what can evaluation be expected to do on its own and what requires collaboration across disciplines, professions and civil society – and across generations? Contributors discuss the implications of the post-Glasgow climate ‘pact’ for the continued relevance of evaluation. Should evaluators advocate for the marginalized and become activists on behalf of sustainability and climate justice – as well as advocates of evidence? Accountability-driven and evidence-based evaluation is needed to assess the effectiveness of investments in adaptation and mitigation. Causal pathways in different settings and ‘theories of no-change’ are needed to understand gaps between stakeholder promises and delivery. Evaluators should measure unintended consequences and what is often left unmeasured, and be sensitive to failure and unanticipated effects of funded actions. Evaluation timescales and units of analysis beyond particular programmes are needed to evaluate the complexities of climate change, sustainability and to take account of natural systems. The implications for evaluation commissioning and funding are discussed as well as the role of evaluation in programme-design and implementation.”
Here is my article on sustainability, measurement and reporting:
“Promises made and promises unfulfilled: Focusing evaluations after COP26
Like many evaluators reading this, I am not a climate specialist but an international political economist, a Czech-American. Both my countries have polluted more than our fair share. Maybe like you, I feel responsible for those who polluted less but suffer more. Professionally, I focus on grassroots sustainability of ex-post project evaluations, including those funded by the Adaptation Fund, and consult on environmental, social and governance ‘impact’. I worry that aid impacts sustained through ingenious local efforts will not hold up to climate shocks for which our aid was not designed, and funding is insufficient.
Where to focus? Knowing what aspect of evaluation interests you, shapes which aspect of the COP26 juggernaut to examine. Results gaps between promises made versus actual change accomplished are an evaluator’s daily bread. Were I an evaluator of environmental processes such as deforestation, ocean acidification/ biodiversity, CO2 emissions, I could evaluate along the lines of the recent publication edited by Juha Uitto (2021)Evaluating Environment in International Development. Abel Gbala, an Ivoirien monitoring and evaluation expert, offers a range of roles evaluators take on, and two for climate change are
Evaluator as a ‘judge’ (following Scriven) to investigate and justify the value of an evaluand, supported by both empirical facts and probative reasoning;
Evaluator as ‘activist’, as argued by Bitar (2019), and Montrosse-Moorhead, et al. (2019: Chapter 3, 33) advocating for social justice and addressing the needs and interests of the vulnerable and disadvantaged.
Following the money and focusing on the centrality of justice and equity between rich and poorer/‘developing’ countries involves judging and being an activist with sharing results. This includes measuring how well the Global North has helped the Global South deal with the inequity in adapting to, mitigating or addressing the devastation of a range of climate change disproportionately caused by the Global North over two centuries. Notably, only small proportions of all financing get to indigenous and local communities (see Rainforest Foundation Norway, 2021; USAID, 2021). Evaluating to whom funding goes is vital for sustainable results.
Many promises are unfulfilled
The 2015 Paris Agreement (United Nations Framework convention on Climate Change, 2016) promised fewer climate-harming emissions, yet the 60 biggest banks have invested 3.8 trillion in fossil fuels (Project Regeneration, 2020). Paris signatories promised US$100 billion climate funding a year, but the COP26 showed massive shortfalls. Not only have an insufficient US$55–US$80 billion a year been given since 2013 (Timperley, 2021), but in a recent Financial Times article by Hook and Kao (2021), Amar Bhattacharya, of the Brookings Institute, stated, ‘In terms of real impact of climate finance, and efficacy across different donors, there has been no development impact or climate impact study done to date’. A German climate watchdog confirms a massive gap if and how US$80 billion in 2019 has been spent, noting, ‘The absence of a detailed, publicly available account of this financing . . . risks all sorts of omissions: donors mis-labeling their funding [as “significant” [impactful], or money being misspent, or an under-estimation of the true volume of money required’ (Subramanian, 2021). Evaluators and auditors are needed to confirm that funding was allocated, disbursed and had an impact on climate change needs.
Needs are tenfold more
India and African countries state they need US$1–US$1.3 trillion in finance by 2030 (Rathi and Chaudhary, 2021). This is not unreasonable, given that ‘developing’ countries ‘are currently shouldering approximately $70 billion per year costs of adapting to climate change’ themselves. The Global South also wants funds to be more evenly split between adaptation (now 25%) to help them deal with sea-level rise and extreme weather events and mitigation (now 75%) (Pontecorvo, 2021). Why the imbalance towards mitigation? Because mitigation is remunerative to investors, companies and banks, who offer loans for countries to switch to clean energy or sell ‘carbon-offsets’. As noted by Timperley (2021), ‘just $20 billion went to adaptation projects in 2019’ versus UN-estimated needs of US$300 billion. It is also essential to measure the effectiveness of finance once it arrives and help those in the climate field see how such investments’ efficacy can be improved.
Where to go to measure costs and finance? One priority for evaluators is to know where to look for data on finance and costs on sustainability and adaptation. This is spread over many national and international databases and reports, and across private and public institutions. Burmeister et al. (2019) have a useful table that summarizes the many finance sources that could be used by evaluators when trying to track actual expenditures and investments on adaptation.
Proof of promises is key
Oxfam’s Climate Finance Shadow Report 2020 (Carty et al., 2020) helps judges and activists see that while donors reported giving US$59.5 billion in 2017 and 2018, ‘the true value. . . may be as little as $19-22.5 billion per year once loan repayments, interest, and other forms of over-reporting are stripped out’. Eighty percent was primarily given as loans, and a further 50 percent of this was non-concessional, requiring higher repayments from emerging countries. In short, our climate ‘largesse’ is increasing their indebtedness. Another watchdog looks at the recipient side. Climate Governance by Transparency International (2021) traces in-country corruption of the funds received. The International Financial Reporting Standards Foundation’s International Sustainability Standards Board questions corporate ‘greenwashing’. Other evaluations remind multilateral and bilateral donors not to claim what they cannot substantiate. Aid promises ‘sustainable development’. Climate funds such as the GEF could be delivering, but Čekan/ová and Legro (2022) examined the GEF’s 2019 report claim that 84 percent were sustainable post-project. ‘Can We Assume Sustained Impact? Verifying the Sustainability of Climate Change Mitigation Results’ showed no proof of ex-post project fieldwork or research to substantiate it. Worse, ‘in the absence of sufficient information regarding project sustainability, determining post-project greenhouse gas emission reductions is not possible, because these are dependent on the continuation of project benefits following project closure’.
It is vital to monitor and evaluate the gaps between promises made and actual change. Gaps include between the finance needed by developing/poorer countries and what is delivered; provable measurements of the impacts and effectiveness of finance given; and the knock-on effects of support for climate action, including indebtedness. As evaluators, we need champions willing to listen, for no one has to listen to evaluators, but much like years of the climate-science IPCC, perseverance and public interest, plus our collective survival on the line, measurements increasingly matter and drive imperative change. Our planet, institutions and many promises and fewer results need all of us.”
I also encourage readers to see the other 13 authors” fascinating submissions in Evaluation post-COP26. Also, many thanks to Elliot Stern, editor, for his support and for making this issue open-access for global learning.
I am quoting liberally and highlighting our work from the Adaptation Fund’s website where their commitment to learning from what lasts is clear. “Ex post evaluations are a key element of the AF-TERG FY21-FY23 strategy and work programme, originating from the request of the Adaptation Fund Board to develop post-implementation learning for Fund projects and programmes and provide accountability of results financed by the Fund. They intend to evaluate aspects of both sustainability of outcomes and climate resilience, and over time feed into ex-post-evaluation-informed adjustments within the Fund’s Monitoring Evaluation and Learning (MEL) processes.”
How are we defining sustainability’s path to evaluate it? Here is a flowchart from our training:
There are four phases from 0 to 3: Phase 0 Foundational Review: Not only was this work preceded by months of background research on both evaluability of their young portfolio (e.g., under 20 of the 100 projects funded were closed at least three years, a selection criteria we had) and secondary research on evidence of ex-post sustainability evaluation in climate change/ resilience across the Adaptation Fund’s sectors.
Phase 1 Framework and Pilots Shortlist: Our Phase 1 report from mid-2021 provided an overview of the first stage of ex-post evaluations, outlining methods and identifying a list of potential projects for ex-post evaluation pilots from the Fund’s 17 completed, evaluated projects. The framework presented in the report introduced possible methods to evaluate the sustainability of project outcomes, considering the characteristics, strengths, and weaknesses of the Fund portfolio. It also presents an analysis tool to assess climate resilience, bearing in mind that this area is pivotal to climate change adaptation yet has rarely been measured.
Vetting and pilot selection, revised design for evaluating sustained outcomes related to resilience to climate change. Key aspects are: 1) Timing (3-5 years since closure or projects at least 4 years long within the last 5 years and seasonality matches the final evaluation) and 2) Good quality of implementation and M&E with measurable outputs and outcomes traceable to impact(s) and 3) Safety to do fieldwork re: Covid, civil peace, etc.
We (my so-clever colleagues Meg Spearman and Dennis Bours) introduced a new resilience analysis tool that includes consideration of the climate disturbances, the human and natural systems (and their nexus) affected by and affecting project outcomes. This includes five characteristics of resilience in the outcomes (presence of feedback loops, at scale, plus being diverse, dynamic, and redundant) and means/actions to support outcomes. Resilience can be identified via a clear summary of the structures (S) and functions (F) that typify Resistance, Resilience and Transformation showing where a project is and is moving towards. It is a typology of resistance-resilience-transformation (RRT) onto which the overall project can be mapped based on how actions are designed to maintain or change existing structures and functions. That was integrated into the Adaptation Fund resilience evaluation approach.
Phase 2 Methods Testing and Ex-post Field-testing: Training of national evaluators and piloting two ex-post evaluations per year includes selecting among these methods to evaluate sustainability ex-post plus the RRT and resilience measures above. In the first ex-post in Samoa’s “Enhancing Resilience of Samoa’s Coastal Communities to Climate Change” (UNDP) happening December 21, it is through qualitative evaluation of wall-infrastructure. The second, Ecuador’s “Enhancing resilience of communities to the adverse effects of climate change on food security, in Pichincha Province and the Jubones River basin “(WFP) has training completed and fieldwork should be from January 22, likely be of food security assets and methods TBD.
Phase 3 Evaluations continue, with MEL Capacity Building: Two more years of ex-post pilot evaluations (2 per year) with lesson informing integration into the MEL of the Adaptation Fund. We are already finding out lessons of rigor, of knowledge management, of unexpected benefits of returning years after closure, including indications of sustainability and resilience of the assets, with much more learning to come.
Innovations include “the relative novelty of climate change adaptation portfolios and the limited body of work on ex post evaluation for adaptation, it presents possible methods that will be piloted in field-tested ex post evaluations in fiscal year 2022 (FY22).” This includes piloting shockingly rare evaluation of oft-promised resilience. In the update to AF’s Board three months ago, it transparently outlined shortlisting of five completed projects as potential candidates for the pilots, of which two projects were selected for ex post evaluations. It outlined our process of co-creating the evaluation with national partners to prioritize their learning needs while building national capacity to assess sustainability and resilience of project outcomes in the field onward.
Also, training materials for ex post pilots are being shared to foster country and industry learning, focusing on evaluating projects at ex-post and emerging sustainability and resilience, as well as presenting and adapting methods to country and project realities.
The training had three sessions (which could not have happened without colleague Caroline’s expertise):
Part A: Understanding ex-post & resilience evaluations. Introduce and understand ex-post evaluations of sustainability and resilience, especially in the field of climate change adaptation
Part B: Discussing country-specific outcome priorities and co-creating learning with stakeholders. Discuss the project and its data more in-depth to understand and select what outcome(s) will be evaluated at ex-post
Part C: Developing country-specific methods and approaches. Discuss range of methods with the national evaluator and M&E experts to best evaluate the selected outcome(s) and impact(s)
So in 2018 I Wished for Sustained Outcomes to be explored and in 2019 I Wished again for more Ex-post Evaluation than Needles in Haystacks. In 2021 some of these wishes are becoming fulfilled! Seasonal Happiness for me is learning about resilience to climate change, diminished vulnerability and searching for proof of sustainability and emerging outcomes and impact(s) and I am grateful to the Adaptation Fund for its commitment to sustainability.
Accompany us on this path, cheer us on, and do your own ex-post sustainability and resilience evaluations! Happy holidays from the Czech Republic!
5 Ways to Foster Sustainability and Resilience to Climate Change
by Omar Abdou, M.A. and Jindra Cekan/ova, Ph.D
In the global climate change graphic, the blue are countries that contribute very few emissions to #climatechange. They are also those who suffer the most from climate change effects and who have most #globalaid projects which are most at risk of no longer being #sustained:
Omar Abdou, a food security and M&E expert from Niger, lives a vulnerability to climate change there is indisputable. Sahelien production systems suffer the full brunt of the negative effects. Indeed, the essential rural activities, namely agriculture, and livestock which occupy at least 80% of the population, are becoming more and more uncertain as they rely on two pillars: rainfall and the exploitation of natural resources (soil, vegetation, other water sources). But, in recent decades, we have witnessed a drastic reduction in rainfall. In pastoral zones, this decrease is reflected in the degradation of the vegetation cover and the deficits in livestock-vital fodder. For instance, the fodder assessment from 2000 to 2020 for the Tahoua Region has been in deficit for 15 years. More and more recurrent and frequent crises seriously affect the main productive capital of households, which is the herd.
Emergency sales of increasingly emaciated animals at a low price are growing to pay for consumer goods and to buy fodder to feed a few animals. These are kept for breeding after the crisis, but what used to be seasonal crisis (dry-season) has become chronic.
In agro-pastoral zones, agricultural production in the rainy season, which has changed its duration and strength, becomes uncertain to feed the household members. The harvest is often exhausted at most by the 6th month of the year for many households. Thus, there is no more food surplus to supply the area’s food needs where cereals are not grown. Suddenly, cereals become expensive and inaccessible for many families. Their coping strategies are diverse but not always effective. In pastoral settings, parents are forced to take their children out of school to go further south in search of food and pasture. Sometimes more sedentary agro-pastoralist farmers pledge or sell their land to pay for food. Those left behind who have nothing to sell, are forced to cut trees and sell the wood for charcoal, further degrading the environment. Unfortunately, Niger’s neighbors are now mostly at war, so these young men run the risk of being recruited by armed groups or are killed. In these conditions, there is serious instability where the means of production (natural and human) are sold for survival and overexploited natural resources decrease. Increasingly, it is practically impossible to talk about sustainability. It is, therefore, necessary to think of consistent and innovative projects to enable households to produce sustainably while preserving the ecosystem for future generations. Outlets such as migration of the able-bodied workers to neighboring countries to provide labor in the lean season work now but are unsustainable.
Jindra notes that 31 Oct the UN Climate Change Conference (COP26) begins. Climatescience documents life-threatening effects such as Omar and billions of others are already experiencing around the ‘developing’ world. The IPCC Working Group report shows that “the world will probably reach or exceed 1.5 degrees C (2.7 degrees F) of warming within just the next two decades. Whether we limit warming to this level and prevent the most severe climate impacts depends on actions taken this decade. Only with ambitious emissions cuts can the world keep global temperature rise to 1.5 degrees C, the limit scientists say is necessary for preventing the worst climate impacts. Under a high-emissions scenario, the IPCC finds the world may warm by 4.4 degrees C by 2100 — with catastrophic results.” Erratic rainfall, greater heatwaves will affect 70% of all farmers and herders who rely on rain for their production and a new report on Adaptation to climate across West African shows aridity increasing, which must inform global development.
What can we do:
1- LISTEN AND ACT GLOBALLY: There are excellent resources to track demands that the Global North divest and stop subsidizing fossil fuels. For instance, “In 2017/18, the G7 collectively spent more on fossil fuel finance than on climate finance, according to analysis by Oil Change International. They allocated nearly $40bn that year.” Hypocrisy abounds as this Climate Change News article goes on to note that “Since the Paris Agreement was signed, G20 countries’ export credit agencies have provided 14 times as much support for fossil fuels as clean energy “to $64 billion a year, and investors still prioritize profit over planet & people.
2- LISTEN AND PRIORITIZE GLOBAL SOUTH VOICES:Whilefunding demands for over $100billion to LDCs is a demand of COP26, the need to support local research and activist organizations who show measured results is mostly missing (more from Sweden here). Other resources include WRI’s Allied for Climate Transformation by 2025 podcast series among the most Climate Change vulnerable nations. “ACT2025 consortium to ensure voices from countries most exposed to climate change are heard, empowered, mobilized, and adequately supported in international climate negotiations.“ The series includes how countries such as Niger adapt to climate change. If nothing else, consider the tragedy of the loss of $1.5 billion in aid invested in Niger, not to mention the ensuing suffering global Northern consumption is unleashing on the global South.
3- DESIGN CLIMATE SMARTLY: Global Development needs to do much more climate-smart programming as 50 years of investments in foreign aid to ‘developing’ countries has already started to be threatened. What can be done? Fromsequestering the carbon content of soil via farming to fostering climate-smart agriculture, which, according to the UN’s FAO means “agriculture that sustainably increases productivity, enhances resilience (adaptation), reduces/removes GHGs (mitigation) where possible, and enhances achievement of national food security and development goals”. Importantly this involves productivity plus adaptation and mitigation against changes wrought by climate change.
4- MEASURE SUSTAINABILITY: Exciting measurements are beginning!Scotland is valuing its nature, and there are calls for The Rights Of Nature In Evaluation Of Environmental Sustainability. Jindra is consulting to a wonderful team at the Adaptation Fund. We are evaluating what could be sustained, and what withstood climatic changes is ex-post project sustainability and resilience evaluation. The Adaptation Fund on evaluating projects’ longer-term (3-5 year ex-post project closure) sustainability and, importantly their resilience to climate change. We are piloting these rare kinds of evaluations in six countries, starting with Samoa and Ecuador in 2021. As the Fund has funded over 100 projects in 100 countries, with almost 30 million participants, we have a rich array to choose from. Such a learning opportunity about what lasts and has withstood disturbances is vital. As the Fund notes: “Climate change is predicted to greatly affect the poorest people in the world, who are often hardest hit by weather catastrophes, desertification, and rising sea levels, but who have contributed the least to the problem of global warming. In some parts of the world, climate change has already contributed to worsening food security, reduced the predictable availability of freshwater, and exacerbated the spread of disease and other threats to human health. Helping the most vulnerable countries and communities is an increasing challenge and imperativefor the international community, especially because climate adaptation requires significant resources beyond what is already needed to achieve international development objectives.”
5- TAKE PERSONAL RESPONSIBILITY: For 8 years, Valuing Voices has pushed back to evaluate the ‘Sustainable Development” we have promised for 50 years. We must be accountable primarily to our participants and partners, rather than donors, and only by listening to these country nationals, we can learn what was sustained not, why or why not, and what emerged from their efforts. In 2019, we wrote all of us are accountable to improve the SDGs. It is wonderful to have climate change leading organizations invest in ex-post evaluation that adds resilience, prioritizing organizational learning and accountability. We all must increase our awareness that curb our consumption, for the sake of Niger, small island states, and many other parts of our planet can no longer stand our business-as-usual. Limiting climate change begins with us, from changing our food consumption to consumer choices to political advocacy to seeing the Earth as indivisible from us and us from the Earth. Let us start to invest in global sustainability today.
Do you see any bright spots on the horizon? Do share!
“What IS Sustainability?” It depends on whom you ask: OECD, the UN, or Harvard Business School
Recently I’ve had conversations where I had to define which sustainability we were talking about. Was it:
ex-post-project sustainability of outcomes and impacts,
environmental sustainability, or
business sustainability?
Since I spend most of my time evaluating the ex-post sustained and emerging impacts of foreign aid projects years after projects close, or at least advocate for it, let’s start there.
The Organisation for Economic Co-operation and Development (OECD) is a “forum and knowledge hub for data and analysis, exchange of experiences, best-practice sharing, and advice on public policies and international standard-setting.” Regarding evaluation specifically, the OECD has “established common definitions for six evaluation criteria – relevance, coherence, effectiveness, efficiency, impact, and sustainability – to support consistent, high-quality evaluation”. Focusing on long-term sustainability, their evaluation guidance is:
The good news is that in this recent publication on Applying Evaluation Criteria Thoughtfully (2021), OECD keeps the updated definition but inches towards recommending actual ex-post project sustainability evaluation, rather than just projected (and assumed “likely to continue” sustainability). For this, “likely” is the most significant reason evaluators for donors and implementers have assumed, rather than evaluated, sustainability for decades. Further, positive, ‘sustained’ trajectories are also assumed at close-out/ exit, but rarely tested ex-post.
The OECD criteria give not evaluating it as an option. I far prefer “net benefits of the intervention continue” as it is a marching order: Prove that results were sustained. In this evolution, this 2021 report states, “After the completion of the intervention, and evaluation of sustainability would look at whether or not the benefits did continue, this time drawing on data and evidence from the intervention’s actual achieved benefits.”
OECD even goes on to recommend implementing and monitoring for sustainability. The new piece de resistance is: “Sustainability should be considered at each point of the results chain and the project cycle of an intervention”:
“The sustainability of inputs (financial or otherwise) after the end of the intervention and the sustainability of impacts in the broader context of the intervention…. as well as whether there was willingness and capacity to sustain financing (resources) at the end of the intervention
For example, an evaluation could assess whether an intervention considered partner capacities
Built ownership at the beginning of the implementation period…. And
In general, evaluators can examine the conditions for sustainability that were or were not created in the design of the intervention and by the intervention activities and whether there was adaptation where required.”
Moreover, while the 2019 OECD report mentioned resilience in passing, related to sustainability, “encourages analysis of potential trade-offs, and of the resilience of capacities/ systemsunderlying the continuation of benefits”. Such resilience and continuation of benefits evaluation involve examining huge systems (the financial, economic, social, environmental, and institutional capacities) that projects and programs are implemented within, whose stability is needed to sustain net benefits over time. Yes, for ex-post sustainability questions for evaluators to consider should include: “To what extent did the intervention contribute to strengthening the resilience of particularly disadvantaged or vulnerable groups” on which the sustained impacts of so much of our “Leave No One Behind” myth of Sustainable Development rely.
However, OECD makes suggestions to evaluate even broader, overwhelming what is feasible: “…this involves analyses of resilience, risks, and potential trade-offs.” Whose? All stakeholders, from participants to local partners and national and international implementers, and international donors? How far back and how far forward? What a huge undertaking. Further, the OECD points evaluators to define resilience, but as I learned in my Famine Early Warning System research and a current ex-post evaluation process for the Adaptation Fund, that involves creating evaluable boundaries by determining resilient to what kinds of shocks?Vital questions current industry monitoring and evaluation budgets for all evaluations, much less (too-rare) ex-post project evaluations, are insufficient for as they hover around 3-5% of total costs.
Slight progress at OECD is being made by acknowledging environmental sustainability first brought up by the Brundtland Report, “Our Common Future” back in 1987. This linchpin report highlighted that “critical global environmental problems were primarily the result of the enormous poverty of the South and the non-sustainable patterns of consumption and production in the North. It called for a strategy that united development and the environment – described by the now-common term “sustainable development”… that meets the needs of the present without compromising the ability of future generations to meet their own needs.”
While an OECD brief in 2008 considers the environmental aspects of our thinking about sustainability, it argues that sustainability primarily about “using economic development to foster a fairer society while respecting ecosystems and natural resources.” The 2021 Applying Evaluation Criteria Thoughtfully rather unhelpfully mostly ignores the environment’s role in sustainability: “Confusion can arise between sustainability in the sense of the continuation of results, and environmental sustainability or the use of resources for future generations. While environmental sustainability is a concern (and maybe examined under several criteria, including relevance, coherence, impact, and sustainability), the primary meaning of the criteria is not about environmental sustainability as such; when describing sustainability, evaluators should be clear on how they are interpreting the criterion.” Given rapid climate change, I would argue that any sustained and emerging outcomes and impacts of projects that does not include an evaluation of the environmental context will fail to foster sustained resilience.Yet donors’ fixed funding timeframes that set completion to disbursement without evaluating sustainability or resilience continue to be huge barriers.
In 2022 a new resource on Sustainability from the media perspective questions the OECD. Sustainability: Going Beyond the Buzzword groups sustainability described in various media as: 1) Economic viability, 2) Social Sustainability, 3) Institutional Sustainability and 4) technical sustainability. But we’ve learned in 2022 from consulting work that the technology piece mentioned in this Buzzwords study regarding ‘baking in’ sustainability via the infrastructure/ assets created (The Adaptation Fund’s ex-post and resilience evaluations) and incorporating INGOs’ considerations of buying-local, seeing what local spare parts and technicians will be available in the future is a great added consideration. So too is promoting locally-led development through the very participatory way projects are implemented, with locals leading. Among the best resources is CDA/ Peace Direct/ Search for Common Ground’s Stopping as Success studies for USAID.
Finally, business sustainability brings together these impacts on communities and society along with impacts on the environment. These are called ESG (Environmental, Social, and Governance) criteria. A Harvard Business School brief defines sustainability as “doing business without negatively impacting the environment, community, or society as a whole. “Where applied well, the aspiration is that “beyond helping curb global challenges, sustainability can drive business success.” While Harvard Business Review highlights “What Works’ in Calculating the Value of Impact Investing, they are, like almost all of global development ‘while- we-are-there’-measures. There is one mention of ‘terminal value’, 5 years after close of ownership, and they estimate social return on investments. This is a good, step, but as insufficient as foreign aid – for these are projected, not actual results.
At Valuing Voices, we have found hopeful examples such as IKEA as well as where ‘impact investing’ hype does not match the claims. Nonetheless, increasingly businesses are trying to consider circular economy systemic principles of “economic development designed to benefit businesses, society, and the environment.” This is regenerative, aims to decouple growth from the consumption of finite resources, not generate excess waste that cannot be reused and actuals seem to be measured at least during investments. As Harvard notes, “this leads investors to look at factors such as a company’s carbon footprint, water usage (both Environment), community development efforts (Social), and board diversity (Governance).” We encourage them to measure long-term/ longitudinally. A current Harvard Business Review sobering article on the ineffectiveness so far of measuring environmental sustainability and ESG. “…reporting is not a proxy for progress. Measurement is often nonstandard, incomplete, imprecise, and misleading. And headlines touting new milestones in disclosure and socially responsible investment are often just fanciful ‘greenwishing’”.
Australia’s RMIT defines business sustainability as comprising 4 pillars: Human, Social, Economic, and Environmental which combines a) “Human sustainability focuses on the importance of anyone directly or indirectly involved in the making of products, or provision of services or broader stakeholders;… b) Social sustainability focuses on maintaining and improving social quality with concepts such as cohesion, reciprocity and honesty and the importance of relationships amongst people;… c) Economic sustainability aims to improve the standard of living [and] the efficient use of assets to maintain company profitability over time;… d) Environmental sustainability places emphasis on how business can achieve positive economic outcomes without doing any harm, in the short- or long-term, to the environment.” But how well measured?!
Would ESG success be sustained over the long-term rather than short-term shareholder profit cycles? Will the OECD start to recommend extensive ex-post evaluation? Will they develop guidance to incorporate environmental concerns in evaluation for our common good? I do not yet know, but I implore these silos to start talking. No time to waste!
As my colleague and collaborator Susan Legro commented, we need to:
1) Continue to seek clarity and specificity in the terminology that we use, ensuring that it is clear to all stakeholders and beneficiaries; and
2) Find ways to study projects and initiatives over the longer term, which is the only way to study the designation of “sustainable” for any initiatives seeking that label.
3) I’ll add spread the word, as I just found out Harvard’s Extension School course on Sustainability features this blog in its curriculum 🙂
Ex-post Eval Week: Exiting For Sustainability by Jindra Cekan
Reblogged from AEA: https://aea365.org/blog/ex-post-eval-week-exiting-for-sustainability-by-jindra-cekan-2/ January 22, 2021
Jindra Cekan
Hello. My name is Jindra Cekan, and I am the Founder and Catalyst of Valuing Voices at Cekan Consulting LLC. Our research, evaluation and advocacy network have been working on post-project (ex-post) evaluations since 2013. I have loved giraffes for decades and fund conservation efforts (see pix).
Our planet is in trouble as are millions of species, including these twiga giraffes and billions of homo-sapiens. Yet in global development we evaluate projects based on their sectoral, e.g. economic, social, educational, human rights etc., results, with barely a glance at the natural systems on which they rest. IDEAS Prague featured Andy Rowe and Michael Quinn Patton who showed that I too have been blind to this aspect of sustainability.
I have argued ad nauseum that the OECD’s definition of projected sustainability and impact don’t give a hoot about sustaining lives and livelihoods.. If we did, we would not just claim we do ‘sustainable development’ and invest in ‘Sustainable Development Goals’ but go about proving how well, for how long, by whom, after closeout.
After hearing Rowe, I added to my Sustained Exit Checklists new elements about how we must evaluate Risks to Sustainability and Resilience to Shocks that included the natural environment. I added Adaptation to Implementation based on feedback on how much implementation would need to change based in part on climatic changes.
Source: Author’s Sustained Exit training (December 2020)
Lessons Learned:
Yet new evaluation thinking by Rowe, Michael Quinn Patton, Astrid Brouselle/ Jim McDavid take us a quantum leap beyond. We must ask how can any intervention be sustained without evaluating the context in which it operates. Is it resilient to environmental threats? Can participants adapt to shocks,? Have we assessed and mitigated the environmental impacts of our interventions? As Professor Brouselle writes, “changing our way of thinking about interventions when designing and evaluating them…. away from our many exploitation systems that lead to exhaustion of resources and extermination of many species.”
This 2020 new thinking includes ascertaining:
(Andy Rowe) Ecosystems of biotic natural capital and abiotic natural capital (from trees to minerals) with effects on health, education, public safety/ climate risk and community development
(Astrid Brouselle and Jim McDavid) Human systems that affect our interventions, including: Power relations, prosperity, equity and we need to make trade-offs between environment and development goals clear.
We have miles to go of systems and values to change. Please read this and let’s start sustaining NOW.
Rad Resources:
Rowe, A. (2019). Sustainability-read evaluation: A call to action. In G. Jules (Ed.) Evaluating Sustainability: Evaluative Support for Managing Processes in the Public Interest. New Directions for Evaluation, 162, 29-48.
Michael Quinn Patton (2020) Evaluation Criteria for Evaluating Transformation: Implications for the Coronavirus Pandemic and the Global Climate Emergency. In the American Journal of Evaluationhttps://journals.sagepub.com/doi/abs/10.1177/1098214020933689
This week, AEA365 is celebrating Ex-post Eval Week during which blog authors share lessons from project exits and ex-post evaluations. Am grateful to the American Evaluation Association that we could share these resources….
Jindra Cekan, Ph.D. has used participatory methods for 30 years to connect with participants, ranging from villagers in Africa, Central/ Latin America and the Balkans to policy makers and Ministers around the world for her international clients. Their voices have informed the new Sustained and Emerging Impacts Evaluation, other M&E, stakeholder analysis, strategic planning, knowledge management and organizational learning.
If you don’t find what you are looking for via the search, categories, or posts above, you can go to the Blog page, scroll to the bottom, and click “previous posts” to go through all of the posts (newest–>oldest).
