“What IS Sustainability?” It depends on whom you ask: OECD, the UN, or Harvard Business School
Recently I’ve had conversations where I had to define which sustainability we were talking about. Was it:
- ex-post-project sustainability of outcomes and impacts,
- environmental sustainability, or
- business sustainability?
Since I spend most of my time evaluating the ex-post sustained and emerging impacts of foreign aid projects years after projects close, or at least advocate for it, let’s start there.
- The Organisation for Economic Co-operation and Development (OECD) is a “forum and knowledge hub for data and analysis, exchange of experiences, best-practice sharing, and advice on public policies and international standard-setting.” Regarding evaluation specifically, the OECD has “established common definitions for six evaluation criteria – relevance, coherence, effectiveness, efficiency, impact, and sustainability – to support consistent, high-quality evaluation”. Focusing on long-term sustainability, their evaluation guidance is:
Source: OECD, Better Criteria for Better Evaluation, 2019
The good news is that in this recent publication on Applying Evaluation Criteria Thoughtfully (2021), OECD keeps the updated definition but inches towards recommending actual ex-post project sustainability evaluation, rather than just projected (and assumed “likely to continue” sustainability). For this, “likely” is the most significant reason evaluators for donors and implementers have assumed, rather than evaluated, sustainability for decades. Further, positive, ‘sustained’ trajectories are also assumed at close-out/ exit, but rarely tested ex-post.
The OECD criteria give not evaluating it as an option. I far prefer “net benefits of the intervention continue” as it is a marching order: Prove that results were sustained. In this evolution, this 2021 report states, “After the completion of the intervention, and evaluation of sustainability would look at whether or not the benefits did continue, this time drawing on data and evidence from the intervention’s actual achieved benefits.”
OECD even goes on to recommend implementing and monitoring for sustainability. The new piece de resistance is: “Sustainability should be considered at each point of the results chain and the project cycle of an intervention”:
- “The sustainability of inputs (financial or otherwise) after the end of the intervention and the sustainability of impacts in the broader context of the intervention…. as well as whether there was willingness and capacity to sustain financing (resources) at the end of the intervention
- For example, an evaluation could assess whether an intervention considered partner capacities
- Built ownership at the beginning of the implementation period…. And
- In general, evaluators can examine the conditions for sustainability that were or were not created in the design of the intervention and by the intervention activities and whether there was adaptation where required.”
Yes, Valuing Voices highlighted this at the American Evaluation Association presentation “Barking up a Better Tree: Lessons about SEIE Sustained and Emerging Impact Evaluation” in 2016, and we have developed this into 2020’s Exiting for Sustainability trainings and checklists. Wonderful to see implementing for sustainability in guidance by the OECD!
Moreover, while the 2019 OECD report mentioned resilience in passing, related to sustainability, “encourages analysis of potential trade-offs, and of the resilience of capacities/ systems underlying the continuation of benefits”. Such resilience and continuation of benefits evaluation involve examining huge systems (the financial, economic, social, environmental, and institutional capacities) that projects and programs are implemented within, whose stability is needed to sustain net benefits over time. Yes, for ex-post sustainability questions for evaluators to consider should include: “To what extent did the intervention contribute to strengthening the resilience of particularly disadvantaged or vulnerable groups” on which the sustained impacts of so much of our “Leave No One Behind” myth of Sustainable Development rely.
However, OECD makes suggestions to evaluate even broader, overwhelming what is feasible: “…this involves analyses of resilience, risks, and potential trade-offs.” Whose? All stakeholders, from participants to local partners and national and international implementers, and international donors? How far back and how far forward? What a huge undertaking. Further, the OECD points evaluators to define resilience, but as I learned in my Famine Early Warning System research and a current ex-post evaluation process for the Adaptation Fund, that involves creating evaluable boundaries by determining resilient to what kinds of shocks? Vital questions current industry monitoring and evaluation budgets for all evaluations, much less (too-rare) ex-post project evaluations, are insufficient for as they hover around 3-5% of total costs.
- Slight progress at OECD is being made by acknowledging environmental sustainability first brought up by the Brundtland Report, “Our Common Future” back in 1987. This linchpin report highlighted that “critical global environmental problems were primarily the result of the enormous poverty of the South and the non-sustainable patterns of consumption and production in the North. It called for a strategy that united development and the environment – described by the now-common term “sustainable development”… that meets the needs of the present without compromising the ability of future generations to meet their own needs.”
While an OECD brief in 2008 considers the environmental aspects of our thinking about sustainability, it argues that sustainability primarily about “using economic development to foster a fairer society while respecting ecosystems and natural resources.” The 2021 Applying Evaluation Criteria Thoughtfully rather unhelpfully mostly ignores the environment’s role in sustainability: “Confusion can arise between sustainability in the sense of the continuation of results, and environmental sustainability or the use of resources for future generations. While environmental sustainability is a concern (and maybe examined under several criteria, including relevance, coherence, impact, and sustainability), the primary meaning of the criteria is not about environmental sustainability as such; when describing sustainability, evaluators should be clear on how they are interpreting the criterion.” Given rapid climate change, I would argue that any sustained and emerging outcomes and impacts of projects that does not include an evaluation of the environmental context will fail to foster sustained resilience. Yet donors’ fixed funding timeframes that set completion to disbursement without evaluating sustainability or resilience continue to be huge barriers.
- Finally, business sustainability brings together these impacts on communities and society along with impacts on the environment. These are called ESG (Environmental, Social, and Governance) criteria. A Harvard Business School brief defines sustainability as “doing business without negatively impacting the environment, community, or society as a whole. “Where applied well, the aspiration is that “beyond helping curb global challenges, sustainability can drive business success.” While Harvard Business Review highlights “What Works’ in Calculating the Value of Impact Investing, they are, like almost all of global development ‘while- we-are-there’-measures. There is one mention of ‘terminal value’, 5 years after close of ownership, and they estimate social return on investments. This is a good, step, but as insufficient as foreign aid – for these are projected, not actual results.
At Valuing Voices, we have found hopeful examples such as IKEA as well as where ‘impact investing’ hype does not match the claims. Nonetheless, increasingly businesses are trying to consider circular economy systemic principles of “economic development designed to benefit businesses, society, and the environment.” This is regenerative, aims to decouple growth from the consumption of finite resources, not generate excess waste that cannot be reused and actuals seem to be measured at least during investments. As Harvard notes, “this leads investors to look at factors such as a company’s carbon footprint, water usage (both Environment), community development efforts (Social), and board diversity (Governance).” We encourage them to measure long-term/ longitudinally. A current Harvard Business Review sobering article on the ineffectiveness so far of measuring environmental sustainability and ESG. “…reporting is not a proxy for progress. Measurement is often nonstandard, incomplete, imprecise, and misleading. And headlines touting new milestones in disclosure and socially responsible investment are often just fanciful ‘greenwishing’”.
Australia’s RMIT defines business sustainability as comprising 4 pillars: Human, Social, Economic, and Environmental which combines a) “Human sustainability focuses on the importance of anyone directly or indirectly involved in the making of products, or provision of services or broader stakeholders;… b) Social sustainability focuses on maintaining and improving social quality with concepts such as cohesion, reciprocity and honesty and the importance of relationships amongst people;… c) Economic sustainability aims to improve the standard of living [and] the efficient use of assets to maintain company profitability over time;… d) Environmental sustainability places emphasis on how business can achieve positive economic outcomes without doing any harm, in the short- or long-term, to the environment.” But how well measured?!
Would ESG success be sustained over the long-term rather than short-term shareholder profit cycles? Will the OECD start to recommend extensive ex-post evaluation? Will they develop guidance to incorporate environmental concerns in evaluation for our common good? I do not yet know, but I implore these silos to start talking. No time to waste!
As my colleague and collaborator Susan Legro commented, we need to:
1) Continue to seek clarity and specificity in the terminology that we use, ensuring that it is clear to all stakeholders and beneficiaries; and
2) Find ways to study projects and initiatives over the longer term, which is the only way to study the designation of “sustainable” for any initiatives seeking that label.
What are your thoughts?
Sustained Exit? Prove it or Improve it!
(reposted from Medium https://jindracekan.medium.com/sustained-exit-prove-it-or-improve-it-702ac507e2a5)
Do we exit global development projects knowing our impacts are sustained? We hope so. As Professor Bea Rogers of Tufts said after evaluating 12 projects 2 years post-closure ( https://www.fsnnetwork.org/resource/exit-strategies-study), “ Hope is Not a Strategy”, yet too often that is what projects that assume sustainability does. They/we hope. But is this good enough? For me, confirming that hope means evaluating beyond exit to ex-post, at least 2 years after donor investments end. 99% of the time, donors & development practitioners don’t return to see what lasted, what didn’t, why nor what emerged from people’s own effort. Yet we implement similar programs over and over onward, not learning lessons from the past. Sigh.
We need to evaluate what we expected to remain from our implemented projects. We also need to learn from what evaluator Bob Williams calls, “the sustainability of the idea that underpinned the results (even if the results were no longer evident)”. This is often beneath what emerged: Our projects catalyzed the local’s desire to sustain activities: taking new ways, that are locally manageable (changing how the development idea is implemented onward) or even having entirely new initiatives emerge from the participant groupings — from their own priorities, not ours. (For more on emerging impacts: https://www.betterevaluation.org/en/themes/SEIE)
Evaluation leaders talk about power, they talk about the environment. After 7 years of researching and evaluating projects ex-post evaluations, I have found there are no brilliant 100% sustained + projects nor are there any 100% abjectly scorched earth ones either. Our results are middling at best. And therein lies the rub. Projects are what donors want to give. Sometimes that overlaps with what recipient countries want, sometimes not. Most of the time the resources to sustain our multimillion-dollar, -euro, -yen, etc., investments aren’t there. We can use incentives (e.g. food aid or cash) that can bolster short-term success while we spend, but once phased out, can lead to sustainability sharply falling off as early as 2 years after we exit. It’s because while ‘development’ is about ‘our’ spending on ‘our’ programs, about short-term success while we’re implementing, rather than our equal partners’ priorities and ability to sustain it. We misuse our power. We care about ourselves far more than the people we ostensibly went there to ‘save’.
And as esteemed evaluators Andy Rowe/ Michael Quinn Patton noted, given climate change we need to question even more assumptions about how sustained and resilient our programming can be, by evaluating the natural environment on which our programming relies pre, during & implementation, at exit and ex-post closure. (More on sustained environment: https://valuingvoices.com/sustaining-sustainable-development/)
It also means we need to talk to those to whom we will eventually hand over early on to make sure we’ve built-in resilience to the climatic, economic shocks we know of so far. I recommend my colleagues Holta Trandafili and Isabella Jean’s presentations on partnering we did a couple of months ago: https://valuingvoices.com/sustainability-ready-what-it-takes-to-support-measure-lasting-change-webinar/
Finally, I have come to see that to make sustainability more likely for years to come, we must fund, design, implement, and monitor/ evaluate For Sustainability throughout the project cycle. I have come to see that folks need guidance to help support their integration of sustainability throughout, including environment & resilience, benchmarks, and more. We can learn from what ex-posts teach. Join me please, to help craft more sustained development:
Upcoming Sustained Exit Webinar: 6 Nov 2020, 14:30–17 CEST, 8:30–11 EST
“Sustained Exit? Prove it or Improve it!” Interactive webinar discussion of ex-post sustainability evaluation lessons and how to integrate into ongoing #aid programs. On Zoom, participants get resources: checklists, slides, recording, Join us to #sustain #impacts! Register, sliding scale: https://sustainedexit.eventbrite.com
Sustaining “Sustainable Development”?
As a global development industry, we have almost no evidence of how (un)sustained the outcomes or impacts of 99% of our projects because we have never returned to evaluate them. But from early indications based on the ex-posts, we have evaluated 2-20 years after donor departure it is, learning from what was and was not sustained is vital before replication and assuming sustainability. Most results taper off quite quickly, showing 20-80% decreases as early as two years post-closure and donor exit. A few cases of good news also appear, but more trajectories falter and fail than rise or remain. Sustainability, then, is not a yes-no answer, but a how much, yet too few ask… hence if they were, resilient, they are less so, or even not at all, now.
At Valuing Voices we focus on the sustainability of projects after external support ends. Still, those projects are also dependent on the viability of the environment in which they are based. As Andy Rowe, an evaluator on the GEF’s Adaptation Fund board, noted at IDEAS’ Conference in Prague late 2019 , a need for sustainability-ready evaluation to help us know how viable the resources are on which so many of our projects rest . He states, “the evaluation we have today treats human and natural systems as unconnected and rarely considers the natural system”. He goes on to differentiate between biotic natural capital (air, water, plants, and trees) and abiotic natural capital sources (fossil fuels, minerals, and metals, wind, and solar).
How much are projects designed assuming those resources are and will remain plentiful? How often do we evaluate how much our projects drain or rely on these environmental elements? Many projects are required to do environmental compliance and safeguarding against damage at project onset . Others, such as agriculture and natural resource management or water/ sanitation, often focus on improving the environment on which those activities rely, e.g., improving soil or terrain (e.g., terraces, zais), planting seedlings, and improving access to potable water for humans and animals. Still, many projects ‘assume’ inputs like rainfall, tree cover, solar power, or do not consider the sustainability of natural resources for the communities in which they intervene. Examples are both those that rely on natural systems as well as those supposedly beyond them, e.g., enterprise development, education, safety nets, etc. Yet many enterprises, schools, safety nets do rely on a. viable environment in which their participants trade, learn, and live, and all are subject to the growing climate change disruptions.
Why is this urgent? The OECD/DAC reminds us that “Natural assets represent, on average 26% of the wealth of developing countries compared to 2% in OECD economies” . Unless we protect them and address the demand for natural resources, demand will far outstrip supply. “By 2030, an additional 1 billion people are expected to live in severely water-stressed areas, and global terrestrial biodiversity is expected to decline an additional 10%, leading to a loss of essential ecosystem services. By 2050, growing levels of dangerous air emissions from transport and industry will increase the global number of premature deaths linked to airborne particulate matter to 3.6 million people a year, more than doubling today’s levels. Failure to act could also lead to a 50% increase in global greenhouse gas emissions by 2050, and global mean temperature increases of 3-6°C by the end of the century, in turn contributing to more severe and sometimes more frequent natural disasters… [so] reconciling development with environmental protection and sustainable resource management is broadly agreed as a central concern for the post-2015 development agenda.”
When we return to projects that are a mix of behavior change and environment, we find a wide range of results:
- Some projects, such as JICA Vietnam’s water supply and irrigation infrastructure reached 80% of the final results two years later . And while the pilot projects were worse off (as low as 28% of irrigated hectares), longer-standing projects sustained as much as 72% of final results. While such agricultural development assumes continued water supply and access, does it evaluate it? No.
- Some can define what ex-post lessons are more narrowly as functioning mechanisms: New ex-posts of water/ sanitation showed better – but still mixed results, such as USAID Senegal’s . “While a majority (63 percent) of the water points remained functional, the performance varied significantly based on the technology used. Of the different technologies, the Erobon rope pumps performed poorly (27 percent functional), while the India Mark (74 percent functional) and mechanized pumps (70 percent functional) performed the best.”
- Some projects that include environmental considerations illustrate our point by only focusing on behavior change as this sanitation/ hygiene ex-post from Madagascar did, where results fell off precipitously three years ex-post but without considering water supply or quality much .
- There can be useful learning when one combines an evaluation of both types of sustainability (ex-post and environmental). A JICA irrigation project in Cambodia shows that when irrigation canals were mostly sustained over the five-years ex-post, they could serve increasing needs for land coverage and rice production . The area of irrigated fields at the national level in 2010 reached the target, and the irrigated field area has since continued to increase in most areas. Even the largest drop [in area irrigated] post-closure was only 11%. They reported that the unit yield of rice at the end-line survey in 2012 at 11 sites was 3.24t/ha (average) versus 3.11t/ha of unit yield of rice at the ex-post evaluation in 2017, which [almost] maintains the 2012 level. The ex-post showed that “continuous irrigation development in the said site can be considered as the main reason for the increase in land area. Securing an adequate amount of water is an important factor in continuously improving rice productivity.” The research also found that 81% of agricultural incomes as a result of the irrigation had increased, 11% stayed the same, and 8% had decreased. Again, this looks to be among the most resilient projects that, based on ex-post research, included environment which was also found to be as resilient as the livelihoods it was fostering.
- Sometimes more bad than good news is important when tracking environment and ex-post sustainability: Food for the Hungry, ADRA, and CARE Kenya found that unreliable water supply reduced the motivation to pay for water, threatening the resources to maintain the system . What improved prospects of sustainability understand why communities could not sustain water and sanitation results based on willingness-to-pay models, as well as water being unavailable. Further, a lesson the organizations ideally learned was that “gradual exit, with the opportunity for project participants to operate independently prior to project closure, made it more likely that activities would be continued without project support.” So the question remains, what was learned by these organizations to avoid similar bad results and improve good, resilient results in similar circumstances?
Neither sustainability nor environmental quality can be assumed to continue nor to have positive results. Both are extensively under-evaluated, and given climate change disruptions, and this must change. Rowe concludes: “Climate change is a major threat to the long-term sustainability both attacking the natural systems (e.g. lower rainfall or higher floods, worse soil quality, increasing pests attacking crops, disappearing fish stocks, microplastics in our air and water, increasing sea levels from melting glaciers, worsening public health etc.) and destabilizining our Earth’s regenerative capacity. Fortunately, technical barriers do not prevent us from starting to infuse sustainability into evaluation; the barriers are social and associated with the worldview and vision of evaluation.”
 IDEAS 2019 Global Assembly. (n.d.). Retrieved from https://2019.global-assembly.org/
 Rowe, A. (2019). Sustainability‐Ready Evaluation: A Call to Action. New Directions for Evaluation, 162, 29-48. Retrieved from https://www.researchgate.net/publication/333616139_Sustainability-Ready_Evaluation_A_Call_to_Action
 USAID. (2013, October 31). Environmental Compliance Procedures. Retrieved from https://www.usaid.gov/our_work/environment/compliance/pdf/216
 OECD. (2015). Element 4, Paper 1: Global and local environmental sustainability, development and growth. Retrieved from https://www.oecd.org/dac/environment-development/FINAL%20POST-2015%20global%20and%20local%20environmental%20sustainability.pdf
 Haraguchi, T. (2017). Socialist Republic of Viet Nam: FY 2017 Ex-Post Evaluation of Japanese ODA Loan Project “Small-Scale Pro Poor Infrastructure Development Project (III)”. Retrieved from https://www2.jica.go.jp/en/evaluation/pdf/2017_VNXVII-5_4.pdf
 Coates, J., Kegode, E., Galante, T., & Blau, A. (2016, February). Sustaining Development: Results from a Study of Sustainability and Exit Strategies among Development Food Assistance Projects: Kenya Country Study. USAID. Retrieved from https://www.globalwaters.org/resources/assets/ex-post-evaluation-senegal-pepam
 Madagascar Rural Access To New Opportunities For Health And Prosperity (RANO-HP) Ex-Post Evaluation. (2017, June 1). USAID. Retrieved from https://www.globalwaters.org/resources/assets/madagascar-rural-access-new-opportunities-health-and-prosperity-rano-hp-ex-post-0
 Kobayashi, N. (2017). Kingdom of Cambodia: FY2017 Ex-Post Evaluation of Technical Cooperation Project: “Technical Service Center for Irrigation System Project – Phase 2 / The Improvement of Agricultural River Basin Management and Development Project (TSC3)”. Retrieved from https://www2.jica.go.jp/en/evaluation/pdf/2017_0900388_4.pdf
Learning from a river of ex-post project evaluations and tools… Thanks USAID!
Dear ex-post aficionados. It’s raining ex-post project evaluations. Here’s hoping learning from such evaluations in water/ sanitation, maternal/child health and even capacity building/ peacekeeping, and their number increases!
1. WATER/ SANITATION & HYGIENE:
USAID has a series of six ex-post evaluations of the water/ sanitation and hygiene sectors since 2017! What is exciting is that they are also looking to the future. These evaluations will “provide insight into what happens after an activity ends, and how to mitigate challenges in future programming, potentially. The series will inform USAID’s WASH activity design and implementation and contribute to a larger sector discussion on achieving sustainability.”
The E3 water division (Water CKM ) took sustainability on as their strategy and have made great strides these last two years. They have done five ex-post project evaluations, cited below, and MSI has completed one more wat/san/ hygiene ex-post evaluations, specifically:
Madagascar Rural Access to New Opportunities for Health and Prosperity (RANO-HP) – Published June 2017
The first evaluation in the series explores the sustainability of the sanitation and hygiene components of the RANO-HP activity, implemented in 26 communes from 2009–2013.
Indonesia Environmental Services Program (ESP) – Published August 2017
The second evaluation in the series examines the sustainability of water utility capacity building, microcredit, and financial outcomes associated with the ESP activity, which was implemented from 2004–2010.
Ethiopia Millennium Water Alliance (MWA-EP) – Published May 2018
The third evaluation in the series examines the long-term sustainability of outcomes related to rural water point construction, rehabilitation, and management, as well as participatory sanitation and hygiene education and construction related to the MWA-EP activity, implemented in 24 rural districts between 2004–2009.
Financial Institutions Reform and Expansion–Debt and Infrastructure (FIRE-D) – Published September 2018
This evaluation is the fourth in the series. It examines how urban water and sanitation services in India have changed since FIRE-D closed and to what extent policies, practices, and financing mechanisms introduced through FIRE-D have been sustained.
Millennium Water and Sanitation Program in Senegal (PEPAM/USAID) – Published July 2019
The fifth ex-post evaluation in the series looks at the PEPAM project (Programme d’Eau Potable et d’Assainissement du Millénaire au Sénégal), implemented from 2009–2014 to improve sustainable access to WASH in four regions of Senegal.
USAID-funded by MSI: USAID/Ghana’s Water Access, Sanitation, and Hygiene for Urban Poor (WASH-UP)– published Nov 2018
Also USAID and Rotary International developed a WASH Sustainability Index Tool, “to assess a WASH activity’s likelihood to be sustainable according to the following factors: availability of finance for sanitation; local capacity for construction and maintenance of latrines; the influence of social norms; and governance.” This is similar to what we learned from USAID/ FFP/ Tufts/ FHI360 12 ex-posts that resources, capacities, motivation and linkages (aka partnerships, including governance) are vital to sustaining outcomes and impacts.
It will be interesting to see whether they examine the other ex-posts for excellent lessons, as they have the Senegalese evaluation:
- “Whether or not to subsidize sanitation access …Based on this evaluation’s findings and exploration of the literature, subsidies can help improve the quality of household latrines, but increasing use of those latrines remains a challenge.
- In contrast, CLTS (a nonsubsidized approach) is often credited with increasing use of unimproved latrines, but serious questions linger about quality and long-term sustainability of the latrines built after CLTS triggering, particularly as it relates to moving up the sanitation ladder. This evaluation… provides the opportunity to examine the potential value of a hybrid approach….
- The handwashing results suggest that low-cost, low-quality handwashing stations such as tippy taps do not lead to sustained behavior change. It may be worth considering hygiene investments that reduce the behavior change burden on targeted beneficiaries.
2. MATERNAL/ CHILD HEALTH & NUTRITION:
“Sustainability of a Community-Based CHOICE Program to Improve the Health and Nutrition Status of Mothers and Infants in Indonesia,” The report focused on whether the USAID-funded CHOICE program had left sustainable impacts: improving the health and nutrition status of children under the age of five, as well as the health status of pregnant and lactating women and mothers or caretakers of young children in the Pandeglang District of Indonesia. “After examining the data collected from the PSS, the researchers found that there were significant improvements in many indicators—such as births attended by skilled personnel, the treatment of diarrhea, and the nutritional quality of food fed to infants—in the six years after the CHOICE program ended. However, despite these improvements, the researchers found no significant statistical differences between villages that received the CHOICE program interventions and comparison villages, which did not. This speaks to using such a comparison methodology to focus on actual contribution and rule out the “rising tide lifts all boats” phenomenon.
3. CAPACITY DEVELOPMENT & PEACEBUILDING:
USAID’s Regional Office in Thailand evaluated its capacity building and peacebuilding program 1.5 years ex-post. While civil society was strengthened and there were inroads made on peacebuilding, “many interventions initiated during Sapan did not continue post-Sapan, although some did remain. For example, “stakeholders cite evidence of continuing to use some governance tools in local governance related to service delivery [although] because of limited financial resources after Sapan ended, they had to change some of their interventions and reduce the range of people they could include. There are lessons for whose capacities are built, two-way feedback loops with local partners, using local organizations such as universities to sustain training, planning sufficient time for partners to internalize training lessons, etc.
4. USAID FUNDED GUIDANCE:
‘Impact Evaluations’ have a new focus on long-term impact, rather than effectiveness during implementation (which was at least the original intent of impact evaluation in the 1980s)! In September 2018, USAID and Notre Dame issued a Guide for Planning Long-term Impact Evaluations as part of the Utilizing the Expertise of the ERIE Program Consortium. The guide covers the difference between traditional impact evaluation designs and data collection methods and how to apply them to long-term impact evaluations (LTIE). It also shares examples across a range of sectors, including later evaluating past impact evaluations, which ended before final evaluation.
Finally, in new 2018 USAID guidance, ex-post evaluation is clarified as the source of the sustainability of services and benefits. USAID clarifies that “questions about the sustainability of project services and benefits can be asked at any stage, but must usually be adjusted to take evaluation timing into account. Thus, for example, in a mid-term evaluation, a question about the existence of a sustainability plan and early action on that plan might be appropriate. An end-of-project evaluation could address questions about how effective a sustainability plan seems to be, and early evidence concerning the likely continuation of project services and benefits after project funding ends. Only an ex-post evaluation, however, can provide empirical data about whether a project’s services and benefits were sustained.”
Such richness that we can learn from. Keep the momentum going on the 99% of all global projects yet unevaluated ex-post, and change how we fund, design, implement, monitor and evaluate global development projects!
Assuming Sustainability and Impact is Dangerous to Development
(+ OECD/ DAC evaluation criteria)
We all do it; well, I used to do it too. I used to assume that if I helped my field staff and partners target and design funded projects well enough, and try to ensure a high quality of implementation and M&E, then it would result in sustainable programming. I assumed we would have moved our participants and partners toward projected long-term, top-of-logical-framework’s aspirational impact such as “vibrant agriculture leading to no hunger”, “locally sustained maternal child health and nutrition”, “self-sustained ecosystems”.
INTRAC nicely differentiates between what is typically measured (“outputs can only ever be the deliverables of a project or programme…that are largely within the control of an agency”) and what is not: “impact as the lasting or significant changes in people’s lives brought about by an intervention or interventions” . They continue: “as few organisations are really judged on their impact, the OECD DAC impact definition (“positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended“) allows for long-term changes in institutional capacity or policy change to be classed as impact” . Do we do this? Virtually never. 99% of the time we only evaluate what happened while the project and its results is under the control of the aid implementer. Yet the five OECD/DAC evaluation criteria asks us to evaluate relevance, effectiveness, efficiency (fair enough, this is important to know if a project was good) and also impact and sustainability. So in addition to the prescription to evaluate ‘long-term effects’ (impact), evaluators are to measure “whether the benefits of an activity are likely to continue after donor funding has been withdrawn… [including being] environmentally as well as financially sustainable” .
How do we know we are getting to sustained outcomes and impacts? We ask people on the receiving end ideally after projects end. It is dangerous to assume sustainability and impact, and assume positive development trajectories (Sridharan) unless we consistently do “ex-post” project evaluations such as these from our research or catalytic organizations that have done at least one ex-post. At very minimum we should evaluate projected sustainability at end of project with those tasked to sustain it before the same project is repeated. Unfortunately we rarely do so and the assumed sustainability is so often not borne out, as I presented at the European Evaluation Society conference Sustainability panel two weeks ago along with AusAid’s DFAT, the World Bank, University College London and UNFEM.
Will we ever know if we have gotten to sustained impacts? Not unless the OECD/DAC criteria are drastically updated and organizations evaluate most projects ex-post (not just good ones :)), learn from the results and fund and implement for country-led sustainability with the country nationals. We must, as Sanjeev Sridharan tells us in a forthcoming paper embed sustainability into our Theories of Change from the onset (“Till time (and poor planning) do us part: Programs as dynamic systems — Incorporating planning of sustainability into theories of change” (Canadian Journal of Program Evaluation, 2018).*
There are remarkable assumptions routinely made. Many projects put sustainability into the proposal, yet most close out projects in the last 6 months. Rarely do projects take the time to properly phase down or phase over (unlike CRS Niger); many exit ceremonially ‘handing over’ projects to country-nationals, disposing project assets, and leaving only a final report behind. Alternatively, this USAID Uganda CDCS Country Transition Plan which looks over 20 years in the future by when it assumes to have accomplished sustained impact for exit . Maybe they will measure progress towards that goal and orient programs toward handover, as in the new USAID “Journey to Self-Reliance” – we hope! Truly, we can plan to exit, but only when data bears out our sustained impact, not when the money or political will runs out.
As OXFAM’s blog today on the evaluation criteria says, “Sustainability is often treated as an assessment of whether an output is likely to be sustained after the end of the project. No one, well, hardly anyone, ever measures sustainability in terms of understanding whether we are meeting the needs of the present, without compromising the ability of future generations to meet their own need” and “too often in development we evaluate a project or programme and claim impact in a very narrow sense rather than the broader ecology beyond project or programme parameters” . In fact, most ‘impact evaluations’ actually test effectiveness rather than long-term impact. Too rarely do we test impact assumptions by returning 2-10 years later and gather proof of what impacted locals’ lives sustainably, much less – importantly – what emerged from their own efforts once we left (SEIEs)! Oh, our hubris.
if you’re interested in the European Evaluation Society’s DAC criteria update discussion, see flagship discussion and Zenda Offir’s blog which stresses the need for better design that include ownership, inclusivity, empowerment . These new evaluation criteria need to be updated, including Florence Etta’s and AGDEN‘s additional criteria participation, non-discrimination and accountability!
We can no longer afford to spend resources without listening to our true clients – those tasked with sustaining the impacts after we pack up – our partners and participants. We can no longer fund what cannot be proven to be sustained that is impactful. We talk about effectiveness and country ownership (which is paramount for sustainability and long-term impact), with an OECD report (2018) found “increases [in[ aid effectiveness by reducing transaction costs and improving recipient countries ownership” . Yet donor governments who ‘tie’ aid to their own country national’s contracts benefit a staggering amount from ‘aid’ given. “Australia and the United Kingdom both reported … 93 percent and 90 percent of the value of their contracts respectively went to their own firms” . It is not so different in the USA where aid is becoming bureaucratically centralized in the hands of a few for-profit contractors and centralized hundreds of millions in a handful of contracts. We must Do Development Differently. We can’t be the prime beneficiaries of our own aid; accountability must be to our participants; is it their countries, not our projects, and we cannot keep dangerously assuming sustained impact. Please let us know what you think…
[*] This paper is now available at https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/53055
 Simister, N. (2015). Monitoring and Evaluation Series: Outcomes Outputs and Impact. Retrieved from https://www.intrac.org/wpcms/wp-content/uploads/2016/06/Monitoring-and-Evaluation-Series-Outcomes-Outputs-and-Impact-7.pdf
 OECD. (n.d.). DAC Criteria for Evaluating Development Assistance. Retrieved September, 2018, from https://web.archive.org/web/20180919035910/http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm
 USAID. (2016, December 6). USAID Uganda Country Development Cooperation Strategy 2016-2021. Retrieved October, 2018, from https://www.usaid.gov/uganda/cdcs
 Porter, S. (2018, October 18). DAC Criteria: The Hand That Rocks the Cradle. Retrieved from https://views-voices.oxfam.org.uk/2018/10/dac-criteria-the-hand-that-rocks-the-cradle/
 European Evaluation Society Biennial Conference: Flagship Symposia. (2018). Retrieved from http://www.ees2018.eu/1539782596-flagship-symposia.htm
 Ofir, Z. (2018, October 13). Updating the DAC Criteria, Part 11 (FINAL). From Evaluation Criteria to Design Principles. Retrieved from https://zendaofir.com/dac-criteria-part-11/
 OECD. (2018, June 11). 2018 Report On The DAC Untying Recommendation. Retrieved from http://www.oecd.org/dac/financing-sustainable-development/development-finance-standards/DCD-DAC(2018)12-REV2.en.pdf
Public and Private paths to Sustained Global
(Reposted from: https://medium.com/@jindracekan/public-and-private-paths-to-sustained-global-development-impacts-9b7523891fce)
Six years. That’s how long ago I began researching proof of sustained impact(s) through its ex-post project evaluation. Until now Valuing Voices has focused on aid donors. We are expanding to the private sector.
In my PhD I was sure it was a lack of researched and shared proof of successful prevention of famine that led to inaction. In Valuing Voices’ research on ex-post project evaluation, I again felt “if only they knew, they would act”. I pulled together a variety of researchers and consultants who (often pro-bono, or for limited fees) researched the shockingly rare field evaluations of what was sustained after projects closed, why, and what participants and partners did themselves to sustain impacts.
Sustaining the outcomes and achieving impacts, are, after all, what global development projects promise. These ‘sustainable development’ results are at the top (or far-right, below) of our ‘logical frameworks’. We promise the country-level partners, our taxpayers and donors, that we will achieve them, yet…
We have done six post-project Sustained and Emerging Impacts Evaluations. We have created checklists on ex-post project evaluability thanks to a Faster Forward Fund grant by esteemed evaluator Michael Scriven. We have created preliminary guidance on Sustained and Emerging Impacts Evaluations (SEIE) and shared 25 such ex-post closure evaluations that we found returned to ask participants 2-15 years after close-out in (one of?) the only database on such evaluations in the world. We have drawn valuable lessons from the evaluations throughout nearly 60 blogs and presented at 10 conferences. We have found that results at the end of project are dynamic, that there can be greater failure – or sometimes greater success – than we would ever expect in our project assumptions. We have found that communities can create ‘emerging’ outcomes, adapting the activities to succeed onward with no further donor funding, and that when we design for long-term sustainability with our partners, then remarkable success can ensue. So many lessons for programming that we need to learn from, including partnering with country-nationals, focusing on youth, questioning assumptions at exit, etc.
We have applied to many grants for support, unsuccessfully, and have applied to evaluate a handful more ex-post sustainability evaluations which other consultants have won – while we were disappointed, in equal measure we are happy others are learning to do this, as we share our resources freely to promote exactly such practices across hundreds of thousands unevaluated projects! We are currently doing an ex-post project evaluation of an agriculture value chain in Tanzania, yet there are a handful done per year. At one conference, our discussant Michael Bamberger joked we were lucky not to be found dead under a bridge for taking on such a dangerous topic. We remain undeterred, and delight in colleagues we promote such work and thanking us for ours.
At the same time, several things have become apparent:
- Vital lessons for how aid can do better remain unexplored, and true accountability to our country-national participants and partners ends when fixed-time, fixed deliverable project resources are spent and proof of accountability for money and results that donors want are filed away. Sadly, while capacity building is done throughout implementation, knowledge management about results is abysmal as ‘our projects’ data almost always dies quietly in donor and implementer computer hard drives after close-out, rather than being accessible in-country for further learning. Go partner!
- We hardly ever return after all our evaluations to share with communities which speaks to ‘partnerships’ not being with the participants, and we often ‘exit’ without giving ample time to handover so that things can be sustained, e.g. local partners found, local and other international funding harnessed, etc. Learn together!
- There is a real need to fund systematized methods for such evaluations, mandate access to quality baseline, midterm and final evaluations, and mandate that all projects above a certain funding level (e.g. $1mil) include funding for such evaluation and learning 2-10 years later. Many so-called ex-post evaluations are in fact either delayed final evaluations, desk studies without any fieldwork, rather methodologically flawed comparisons or with fieldwork which doesn’t talk to the intended ‘beneficiaries’ for such pivotal ground-level feedback. Innovate by listening!
- It is unclear to us how any organization that has done an ex-post sustainability evaluation has learned from it and changed their systems, although we have been told some are ‘looking for a successful project to evaluate’, and that after a failed one, they are discontinued. We know of some (I)NGOs who are putting ex-posts into their new strategies, and two INGOs who are researching exits more – good. Be brave!
- Recently, we are delighted some new NGOs are dipping their feet into their first evaluations of sustainability, they do so bravely. The tension between accountability and learning is heightened at the prospect that implementers and donors have failed to create sustained impact. But why judge them when all the design and systems in place are to reward success while projects are running (and even those don’t always show much) so that they all get congratulations and more funding for very similar projects? Who knows who is focused on sustaining impacts with funding capacities, partnerships and country-led design, implementing with feedback loops and adjusting for the long-term, helping communities evaluate us rather than how well they are fulfilling our targets, etc. Sustaining impacts will win you funding!
- Logically, here are many indications among ex-post sustainability evaluations that profitable, but low-risk and diversified agriculture, microenterprise/ business projects are better sustained (Niger, Ethiopia, Tanzania, Nepal, etc.). This does not mean that all projects need to be profitable, but cost-covering projects even in the health and education/ vocational training/ sectors is important as many of us know. Self-funding!
So rather than giving up on sustained impacts, we are adding another branch to the Valuing Voices tree.
My partners and I have extensively researched the need for and co-founded Impact Guild. We will work alongside NGOs and impact investors to foster:
- FUNDING: The money available from development aid donors is shrinking in volume + value, while development financing is scaling up exponentially.
The SDGs and the Paris Agreement are prompting a massive scale-up of development financing from billions to trillions of dollars into ‘sustainable development’, yet with rare Scandinavian and Foundation exceptions, donors appear to be switching from longer-term development to humanitarian aid. Further, despite decades of experience, international and national nonprofit development implementers are mostly absent in the conversation around scaling-up the flow of capital to achieve and sustain development goals. Exceptions are some in the International NGO Impact Investing Network (AMPLIFY)
2. RESULTS: Funding for projects that can show great results (e.g. Social Impact Bonds/ Development Impact Bonds, which are in fact pay-for-performance instruments), even sustained impacts from partnering with local small and medium enterprises, national level ministries, and local NGOs. Far too long, implementers have been able to get funding for projects with mediocre results; impact investors are raising the bar and even donors are helping hedge risk. This includes M&E ‘impact’ value that rigorously tracks results (savvy private-sector donors require counterfactual/ control group data, isolating results from that intervention).
3. LEARNING: Impact Investors have a lot to learn from non-profits and aid donors as well.
- They talk about impact but too often that is synonymous with generic results, while International and National nonprofits (NGOs) have detailed, grassroots systems in place;
- Most seem to be content – for now – to invest in the 17 Sustainable Development Goal areas (e.g. vetting investable projects by screening criteria of not only getting a financial return, but also by broad sectoral investments, e.g. poverty, hunger, climate etc.). Many claim they have affected change, without data to prove it. The SDGs are slowly creating indicators to address this, and investors also need to be brought along to differentiate between corporate efficiency activities for their operations and those that affect change at the output, outcome and impact levels in communities;
- There are still large leaps of logic and claims among investors and some know that data is lacking to claim good grassroots targeting and actual results that prove they are changing hunger, poverty and other sectors in Africa, Asia, Latin Americ. Good development professionals would see that the very design would make results accessible only to the elite of that country (e.g. $1 nutrition bars are inaccessible to most of a country’s population living on income of $2.00 a day)
We will bring with us all we know about great potential sustained impacts programming, such as Theory of Sustainability, looking for emerging results alongside planned early on, learning from failure for success, partnering successfully for country-led development, etc.
So keep watching these ‘spaces’: www.ValuingVoices.com and www.ImpactGuild.org for updates on bridging these worlds, hopefully for ever-greater sustained impacts. Let us know if you would like to partner!