I am quoting liberally and highlighting our work from the Adaptation Fund’s website where their commitment to learning from what lasts is clear. “Ex post evaluations are a key element of the AF-TERG FY21-FY23 strategy and work programme, originating from the request of the Adaptation Fund Board to develop post-implementation learning for Fund projects and programmes and provide accountability of results financed by the Fund. They intend to evaluate aspects of both sustainability of outcomes and climate resilience, and over time feed into ex-post-evaluation-informed adjustments within the Fund’s Monitoring Evaluation and Learning (MEL) processes.”
How are we defining sustainability’s path to evaluate it? Here is a flowchart from our training:
There are four phases from 0 to 3: Phase 0 Foundational Review: Not only was this work preceded by months of background research on both evaluability of their young portfolio (e.g., under 20 of the 100 projects funded were closed at least three years, a selection criteria we had) and secondary research on evidence of ex-post sustainability evaluation in climate change/ resilience across the Adaptation Fund’s sectors.
Phase 1 Framework and Pilots Shortlist: Our Phase 1 report from mid-2021 provided an overview of the first stage of ex-post evaluations, outlining methods and identifying a list of potential projects for ex-post evaluation pilots from the Fund’s 17 completed, evaluated projects. The framework presented in the report introduced possible methods to evaluate the sustainability of project outcomes, considering the characteristics, strengths, and weaknesses of the Fund portfolio. It also presents an analysis tool to assess climate resilience, bearing in mind that this area is pivotal to climate change adaptation yet has rarely been measured.
Vetting and pilot selection, revised design for evaluating sustained outcomes related to resilience to climate change. Key aspects are: 1) Timing (3-5 years since closure or projects at least 4 years long within the last 5 years and seasonality matches the final evaluation) and 2) Good quality of implementation and M&E with measurable outputs and outcomes traceable to impact(s) and 3) Safety to do fieldwork re: Covid, civil peace, etc.
We (my so-clever colleagues Meg Spearman and Dennis Bours) introduced a new resilience analysis tool that includes consideration of the climate disturbances, the human and natural systems (and their nexus) affected by and affecting project outcomes. This includes five characteristics of resilience in the outcomes (presence of feedback loops, at scale, plus being diverse, dynamic, and redundant) and means/actions to support outcomes. Resilience can be identified via a clear summary of the structures (S) and functions (F) that typify Resistance, Resilience and Transformation showing where a project is and is moving towards. It is a typology of resistance-resilience-transformation (RRT) onto which the overall project can be mapped based on how actions are designed to maintain or change existing structures and functions. That was integrated into the Adaptation Fund resilience evaluation approach.
Phase 2 Methods Testing and Ex-post Field-testing: Training of national evaluators and piloting two ex-post evaluations per year includes selecting among these methods to evaluate sustainability ex-post plus the RRT and resilience measures above. In the first ex-post in Samoa’s “Enhancing Resilience of Samoa’s Coastal Communities to Climate Change” (UNDP) happening December 21, it is through qualitative evaluation of wall-infrastructure. The second, Ecuador’s “Enhancing resilience of communities to the adverse effects of climate change on food security, in Pichincha Province and the Jubones River basin “(WFP) has training completed and fieldwork should be from January 22, likely be of food security assets and methods TBD.
Phase 3 Evaluations continue, with MEL Capacity Building: Two more years of ex-post pilot evaluations (2 per year) with lesson informing integration into the MEL of the Adaptation Fund. We are already finding out lessons of rigor, of knowledge management, of unexpected benefits of returning years after closure, including indications of sustainability and resilience of the assets, with much more learning to come.
Innovations include “the relative novelty of climate change adaptation portfolios and the limited body of work on ex post evaluation for adaptation, it presents possible methods that will be piloted in field-tested ex post evaluations in fiscal year 2022 (FY22).” This includes piloting shockingly rare evaluation of oft-promised resilience. In the update to AF’s Board three months ago, it transparently outlined shortlisting of five completed projects as potential candidates for the pilots, of which two projects were selected for ex post evaluations. It outlined our process of co-creating the evaluation with national partners to prioritize their learning needs while building national capacity to assess sustainability and resilience of project outcomes in the field onward.
Also, training materials for ex post pilots are being shared to foster country and industry learning, focusing on evaluating projects at ex-post and emerging sustainability and resilience, as well as presenting and adapting methods to country and project realities.
The training had three sessions (which could not have happened without colleague Caroline’s expertise):
Part A: Understanding ex-post & resilience evaluations. Introduce and understand ex-post evaluations of sustainability and resilience, especially in the field of climate change adaptation
Part B: Discussing country-specific outcome priorities and co-creating learning with stakeholders. Discuss the project and its data more in-depth to understand and select what outcome(s) will be evaluated at ex-post
Part C: Developing country-specific methods and approaches. Discuss range of methods with the national evaluator and M&E experts to best evaluate the selected outcome(s) and impact(s)
5 Ways to Foster Sustainability and Resilience to Climate Change
by Omar Abdou, M.A. and Jindra Cekan/ova, Ph.D
In the global climate change graphic, the blue are countries that contribute very few emissions to #climatechange. They are also those who suffer the most from climate change effects and who have most #globalaid projects which are most at risk of no longer being #sustained:
Omar Abdou, a food security and M&E expert from Niger, lives a vulnerability to climate change there is indisputable. Sahelien production systems suffer the full brunt of the negative effects. Indeed, the essential rural activities, namely agriculture, and livestock which occupy at least 80% of the population, are becoming more and more uncertain as they rely on two pillars: rainfall and the exploitation of natural resources (soil, vegetation, other water sources). But, in recent decades, we have witnessed a drastic reduction in rainfall. In pastoral zones, this decrease is reflected in the degradation of the vegetation cover and the deficits in livestock-vital fodder. For instance, the fodder assessment from 2000 to 2020 for the Tahoua Region has been in deficit for 15 years. More and more recurrent and frequent crises seriously affect the main productive capital of households, which is the herd.
Emergency sales of increasingly emaciated animals at a low price are growing to pay for consumer goods and to buy fodder to feed a few animals. These are kept for breeding after the crisis, but what used to be seasonal crisis (dry-season) has become chronic.
In agro-pastoral zones, agricultural production in the rainy season, which has changed its duration and strength, becomes uncertain to feed the household members. The harvest is often exhausted at most by the 6th month of the year for many households. Thus, there is no more food surplus to supply the area’s food needs where cereals are not grown. Suddenly, cereals become expensive and inaccessible for many families. Their coping strategies are diverse but not always effective. In pastoral settings, parents are forced to take their children out of school to go further south in search of food and pasture. Sometimes more sedentary agro-pastoralist farmers pledge or sell their land to pay for food. Those left behind who have nothing to sell, are forced to cut trees and sell the wood for charcoal, further degrading the environment. Unfortunately, Niger’s neighbors are now mostly at war, so these young men run the risk of being recruited by armed groups or are killed. In these conditions, there is serious instability where the means of production (natural and human) are sold for survival and overexploited natural resources decrease. Increasingly, it is practically impossible to talk about sustainability. It is, therefore, necessary to think of consistent and innovative projects to enable households to produce sustainably while preserving the ecosystem for future generations. Outlets such as migration of the able-bodied workers to neighboring countries to provide labor in the lean season work now but are unsustainable.
Jindra notes that 31 Oct the UN Climate Change Conference (COP26) begins. Climatescience documents life-threatening effects such as Omar and billions of others are already experiencing around the ‘developing’ world. The IPCC Working Group report shows that “the world will probably reach or exceed 1.5 degrees C (2.7 degrees F) of warming within just the next two decades. Whether we limit warming to this level and prevent the most severe climate impacts depends on actions taken this decade. Only with ambitious emissions cuts can the world keep global temperature rise to 1.5 degrees C, the limit scientists say is necessary for preventing the worst climate impacts. Under a high-emissions scenario, the IPCC finds the world may warm by 4.4 degrees C by 2100 — with catastrophic results.” Erratic rainfall, greater heatwaves will affect 70% of all farmers and herders who rely on rain for their production and a new report on Adaptation to climate across West African shows aridity increasing, which must inform global development.
What can we do:
1- LISTEN AND ACT GLOBALLY: There are excellent resources to track demands that the Global North divest and stop subsidizing fossil fuels. For instance, “In 2017/18, the G7 collectively spent more on fossil fuel finance than on climate finance, according to analysis by Oil Change International. They allocated nearly $40bn that year.” Hypocrisy abounds as this Climate Change News article goes on to note that “Since the Paris Agreement was signed, G20 countries’ export credit agencies have provided 14 times as much support for fossil fuels as clean energy “to $64 billion a year, and investors still prioritize profit over planet & people.
3- DESIGN CLIMATE SMARTLY: Global Development needs to do much more climate-smart programming as 50 years of investments in foreign aid to ‘developing’ countries has already started to be threatened. What can be done? Fromsequestering the carbon content of soil via farming to fostering climate-smart agriculture, which, according to the UN’s FAO means “agriculture that sustainably increases productivity, enhances resilience (adaptation), reduces/removes GHGs (mitigation) where possible, and enhances achievement of national food security and development goals”. Importantly this involves productivity plus adaptation and mitigation against changes wrought by climate change.
4- MEASURE SUSTAINABILITY: Exciting measurements are beginning!Scotland is valuing its nature, and there are calls for The Rights Of Nature In Evaluation Of Environmental Sustainability. Jindra is consulting to a wonderful team at the Adaptation Fund. We are evaluating what could be sustained, and what withstood climatic changes is ex-post project sustainability and resilience evaluation. The Adaptation Fund on evaluating projects’ longer-term (3-5 year ex-post project closure) sustainability and, importantly their resilience to climate change. We are piloting these rare kinds of evaluations in six countries, starting with Samoa and Ecuador in 2021. As the Fund has funded over 100 projects in 100 countries, with almost 30 million participants, we have a rich array to choose from. Such a learning opportunity about what lasts and has withstood disturbances is vital. As the Fund notes: “Climate change is predicted to greatly affect the poorest people in the world, who are often hardest hit by weather catastrophes, desertification, and rising sea levels, but who have contributed the least to the problem of global warming. In some parts of the world, climate change has already contributed to worsening food security, reduced the predictable availability of freshwater, and exacerbated the spread of disease and other threats to human health. Helping the most vulnerable countries and communities is an increasing challenge and imperativefor the international community, especially because climate adaptation requires significant resources beyond what is already needed to achieve international development objectives.”
5- TAKE PERSONAL RESPONSIBILITY: For 8 years, Valuing Voices has pushed back to evaluate the ‘Sustainable Development” we have promised for 50 years. We must be accountable primarily to our participants and partners, rather than donors, and only by listening to these country nationals, we can learn what was sustained not, why or why not, and what emerged from their efforts. In 2019, we wrote all of us are accountable to improve the SDGs. It is wonderful to have climate change leading organizations invest in ex-post evaluation that adds resilience, prioritizing organizational learning and accountability. We all must increase our awareness that curb our consumption, for the sake of Niger, small island states, and many other parts of our planet can no longer stand our business-as-usual. Limiting climate change begins with us, from changing our food consumption to consumer choices to political advocacy to seeing the Earth as indivisible from us and us from the Earth. Let us start to invest in global sustainability today.
Do you see any bright spots on the horizon? Do share!
“What IS Sustainability?” It depends on whom you ask: OECD, the UN, or Harvard Business School
Recently I’ve had conversations where I had to define which sustainability we were talking about. Was it:
ex-post-project sustainability of outcomes and impacts,
environmental sustainability, or
Since I spend most of my time evaluating the ex-post sustained and emerging impacts of foreign aid projects years after projects close, or at least advocate for it, let’s start there.
The Organisation for Economic Co-operation and Development (OECD) is a “forum and knowledge hub for data and analysis, exchange of experiences, best-practice sharing, and advice on public policies and international standard-setting.” Regarding evaluation specifically, the OECD has “established common definitions for six evaluation criteria – relevance, coherence, effectiveness, efficiency, impact, and sustainability – to support consistent, high-quality evaluation”. Focusing on long-term sustainability, their evaluation guidance is:
The good news is that in this recent publication on Applying Evaluation Criteria Thoughtfully (2021), OECD keeps the updated definition but inches towards recommending actual ex-post project sustainability evaluation, rather than just projected (and assumed “likely to continue” sustainability). For this, “likely” is the most significant reason evaluators for donors and implementers have assumed, rather than evaluated, sustainability for decades. Further, positive, ‘sustained’ trajectories are also assumed at close-out/ exit, but rarely tested ex-post.
The OECD criteria give not evaluating it as an option. I far prefer “net benefits of the intervention continue” as it is a marching order: Prove that results were sustained. In this evolution, this 2021 report states, “After the completion of the intervention, and evaluation of sustainability would look at whether or not the benefits did continue, this time drawing on data and evidence from the intervention’s actual achieved benefits.”
OECD even goes on to recommend implementing and monitoring for sustainability. The new piece de resistance is: “Sustainability should be considered at each point of the results chain and the project cycle of an intervention”:
“The sustainability of inputs (financial or otherwise) after the end of the intervention and the sustainability of impacts in the broader context of the intervention…. as well as whether there was willingness and capacity to sustain financing (resources) at the end of the intervention
For example, an evaluation could assess whether an intervention considered partner capacities
Built ownership at the beginning of the implementation period…. And
In general, evaluators can examine the conditions for sustainability that were or were not created in the design of the intervention and by the intervention activities and whether there was adaptation where required.”
Moreover, while the 2019 OECD report mentioned resilience in passing, related to sustainability, “encourages analysis of potential trade-offs, and of the resilience of capacities/ systemsunderlying the continuation of benefits”. Such resilience and continuation of benefits evaluation involve examining huge systems (the financial, economic, social, environmental, and institutional capacities) that projects and programs are implemented within, whose stability is needed to sustain net benefits over time. Yes, for ex-post sustainability questions for evaluators to consider should include: “To what extent did the intervention contribute to strengthening the resilience of particularly disadvantaged or vulnerable groups” on which the sustained impacts of so much of our “Leave No One Behind” myth of Sustainable Development rely.
However, OECD makes suggestions to evaluate even broader, overwhelming what is feasible: “…this involves analyses of resilience, risks, and potential trade-offs.” Whose? All stakeholders, from participants to local partners and national and international implementers, and international donors? How far back and how far forward? What a huge undertaking. Further, the OECD points evaluators to define resilience, but as I learned in my Famine Early Warning System research and a current ex-post evaluation process for the Adaptation Fund, that involves creating evaluable boundaries by determining resilient to what kinds of shocks?Vital questions current industry monitoring and evaluation budgets for all evaluations, much less (too-rare) ex-post project evaluations, are insufficient for as they hover around 3-5% of total costs.
Slight progress at OECD is being made by acknowledging environmental sustainability first brought up by the Brundtland Report, “Our Common Future” back in 1987. This linchpin report highlighted that “critical global environmental problems were primarily the result of the enormous poverty of the South and the non-sustainable patterns of consumption and production in the North. It called for a strategy that united development and the environment – described by the now-common term “sustainable development”… that meets the needs of the present without compromising the ability of future generations to meet their own needs.”
While an OECD brief in 2008 considers the environmental aspects of our thinking about sustainability, it argues that sustainability primarily about “using economic development to foster a fairer society while respecting ecosystems and natural resources.” The 2021 Applying Evaluation Criteria Thoughtfully rather unhelpfully mostly ignores the environment’s role in sustainability: “Confusion can arise between sustainability in the sense of the continuation of results, and environmental sustainability or the use of resources for future generations. While environmental sustainability is a concern (and maybe examined under several criteria, including relevance, coherence, impact, and sustainability), the primary meaning of the criteria is not about environmental sustainability as such; when describing sustainability, evaluators should be clear on how they are interpreting the criterion.” Given rapid climate change, I would argue that any sustained and emerging outcomes and impacts of projects that does not include an evaluation of the environmental context will fail to foster sustained resilience.Yet donors’ fixed funding timeframes that set completion to disbursement without evaluating sustainability or resilience continue to be huge barriers.
Finally, business sustainability brings together these impacts on communities and society along with impacts on the environment. These are called ESG (Environmental, Social, and Governance) criteria. A Harvard Business School brief defines sustainability as “doing business without negatively impacting the environment, community, or society as a whole. “Where applied well, the aspiration is that “beyond helping curb global challenges, sustainability can drive business success.” While Harvard Business Review highlights “What Works’ in Calculating the Value of Impact Investing, they are, like almost all of global development ‘while- we-are-there’-measures. There is one mention of ‘terminal value’, 5 years after close of ownership, and they estimate social return on investments. This is a good, step, but as insufficient as foreign aid – for these are projected, not actual results.
At Valuing Voices, we have found hopeful examples such as IKEA as well as where ‘impact investing’ hype does not match the claims. Nonetheless, increasingly businesses are trying to consider circular economy systemic principles of “economic development designed to benefit businesses, society, and the environment.” This is regenerative, aims to decouple growth from the consumption of finite resources, not generate excess waste that cannot be reused and actuals seem to be measured at least during investments. As Harvard notes, “this leads investors to look at factors such as a company’s carbon footprint, water usage (both Environment), community development efforts (Social), and board diversity (Governance).” We encourage them to measure long-term/ longitudinally. A current Harvard Business Review sobering article on the ineffectiveness so far of measuring environmental sustainability and ESG. “…reporting is not a proxy for progress. Measurement is often nonstandard, incomplete, imprecise, and misleading. And headlines touting new milestones in disclosure and socially responsible investment are often just fanciful ‘greenwishing’”.
Australia’s RMIT defines business sustainability as comprising 4 pillars: Human, Social, Economic, and Environmental which combines a) “Human sustainability focuses on the importance of anyone directly or indirectly involved in the making of products, or provision of services or broader stakeholders;… b) Social sustainability focuses on maintaining and improving social quality with concepts such as cohesion, reciprocity and honesty and the importance of relationships amongst people;… c) Economic sustainability aims to improve the standard of living [and] the efficient use of assets to maintain company profitability over time;… d) Environmental sustainability places emphasis on how business can achieve positive economic outcomes without doing any harm, in the short- or long-term, to the environment.” But how well measured?!
Would ESG success be sustained over the long-term rather than short-term shareholder profit cycles? Will the OECD start to recommend extensive ex-post evaluation? Will they develop guidance to incorporate environmental concerns in evaluation for our common good? I do not yet know, but I implore these silos to start talking. No time to waste!
As my colleague and collaborator Susan Legro commented, we need to:
1) Continue to seek clarity and specificity in the terminology that we use, ensuring that it is clear to all stakeholders and beneficiaries; and
2) Find ways to study projects and initiatives over the longer term, which is the only way to study the designation of “sustainable” for any initiatives seeking that label.
Ex-post Eval Week: Exiting For Sustainability by Jindra Cekan
Reblogged from AEA: https://aea365.org/blog/ex-post-eval-week-exiting-for-sustainability-by-jindra-cekan-2/ January 22, 2021
Hello. My name is Jindra Cekan, and I am the Founder and Catalyst of Valuing Voices at Cekan Consulting LLC. Our research, evaluation and advocacy network have been working on post-project (ex-post) evaluations since 2013. I have loved giraffes for decades and fund conservation efforts (see pix).
Our planet is in trouble as are millions of species, including these twiga giraffes and billions of homo-sapiens. Yet in global development we evaluate projects based on their sectoral, e.g. economic, social, educational, human rights etc., results, with barely a glance at the natural systems on which they rest. IDEAS Prague featured Andy Rowe and Michael Quinn Patton who showed that I too have been blind to this aspect of sustainability.
I have argued ad nauseum that the OECD’s definition of projected sustainability and impact don’t give a hoot about sustaining lives and livelihoods.. If we did, we would not just claim we do ‘sustainable development’ and invest in ‘Sustainable Development Goals’ but go about proving how well, for how long, by whom, after closeout.
After hearing Rowe, I added to my Sustained Exit Checklists new elements about how we must evaluate Risks to Sustainability and Resilience to Shocks that included the natural environment. I added Adaptation to Implementation based on feedback on how much implementation would need to change based in part on climatic changes.
Yet new evaluation thinking by Rowe, Michael Quinn Patton, Astrid Brouselle/ Jim McDavid take us a quantum leap beyond. We must ask how can any intervention be sustained without evaluating the context in which it operates. Is it resilient to environmental threats? Can participants adapt to shocks,? Have we assessed and mitigated the environmental impacts of our interventions? As Professor Brouselle writes, “changing our way of thinking about interventions when designing and evaluating them…. away from our many exploitation systems that lead to exhaustion of resources and extermination of many species.”
This 2020 new thinking includes ascertaining:
(Andy Rowe) Ecosystems of biotic natural capital and abiotic natural capital (from trees to minerals) with effects on health, education, public safety/ climate risk and community development
(Astrid Brouselle and Jim McDavid) Human systems that affect our interventions, including: Power relations, prosperity, equity and we need to make trade-offs between environment and development goals clear.
We have miles to go of systems and values to change. Please read this and let’s start sustaining NOW.
Rowe, A. (2019). Sustainability-read evaluation: A call to action. In G. Jules (Ed.) Evaluating Sustainability: Evaluative Support for Managing Processes in the Public Interest. New Directions for Evaluation, 162, 29-48.
This week, AEA365 is celebrating Ex-post Eval Week during which blog authors share lessons from project exits and ex-post evaluations. Am grateful to the American Evaluation Association that we could share these resources….
Ex-post Eval Week: Measuring sustainability post-program –go in and stay for the learning! By Holta Trandafili
Reblog from AEA: https://aea365.org/blog/ex-post-eval-week-measuring-sustainability-post-program-go-in-and-stay-for-the-learning-by-holta-trandafili/ January 21, 2021
Greetings, I am Holta Trandafili, a researcher and evaluator captivated by sustainability theories and the sustainment of results. I believe that a thoughtful, systematic inquiry of what happens after an intervention ends adds value to what we know about sustainability. Since 2015 I have co-led ten post-program evaluations (also known as ex-posts) in Uganda, Kenya, Sri Lanka, India, Myanmar, and Bolivia. Their findings point to questions and issues of theory, measurement, and sustainability expectations relevant to any program:
To what do we compare results to judge success? Is it that 60% of community groups or water points being operational three years after closeout a good result? Should it be 87% or 90%? Why? Should we use the end-line as the measuring yardstick, especially as contexts change? Whose view of success counts?
How long should we expect results, or community groups left behind, or activities to continue post-program? Two or ten years or Forever? Why?
Is going back once enough to make a judgment on sustainability? What would we find if we went back in 2020 where we evaluated ex-post in 2015 or even 2019?
Lessons Learned:Here are my reflections and resources on sustainability:
To the enthusiastic evaluators ready to start ex-posts
Lesson learned: Organizations often carry out ex-posts for accountability. However, greater wealth lays in learning. Make learning part of your evaluation objectives. It took my organization 5 years from the first ex-post to have more open conversations and share our sustainability learning on what to improve: how we design, transition, and measure programs’ impact. Now we are genuinely more accountable.
Get involved: Don’t lose heart if your first ex-posts prove difficult to conduct or have mixed results or unearth new questions and insights on sustainability. You are not alone. Find another evaluator that has gone through an ex-post experience and ask them to write a blog, present at a conference, write guidelines, attend a course, or merely meet to vent and dream.
To those already fighting to mainstream ex-post measurement in their organizations or their clients
Mainstreaming ex-post evaluations is commendable for any institution. In this process we should start making the case to pilot longitudinal ex-post measurements (i.e., going back not once but several points in time). We can truly unpack the issues of temporality and longevity for sustainment of results. See JICA’s example on ex-post monitoring.
Invest in theory-driven evaluations like Realist Evaluation to unpack the hidden mechanisms behind which different types of outcomes are sustained, asking: among whom, in what contexts, how, why?
Jindra Cekan, Ph.D. has used participatory methods for 30 years to connect with participants, ranging from villagers in Africa, Central/ Latin America and the Balkans to policy makers and Ministers around the world for her international clients. Their voices have informed the new Sustained and Emerging Impacts Evaluation, other M&E, stakeholder analysis, strategic planning, knowledge management and organizational learning.
If you don’t find what you are looking for via the search, categories, or posts above, you can go to the Blog page, scroll to the bottom, and click “previous posts” to go through all of the posts (newest–>oldest).