Fostering Values-Driven Sustainability Through an Ex-Post Capacities Lens (reposting a book chapter)

We all want our project results to be sustained, but without doing ex-post sustainability evaluations, we don’t know if they are. However, ex-post evaluations can also teach us how to fund, design, monitor, and evaluate projects before they close. They also require some evaluator competencies, and the checklists below are designed to help build capacities to make implemented projects more sustained, This research was also informed by excellent research by INTRAC and CDA. Enjoy! Also, you can download it from this great array of evaluator competencies via the Journal of Multidisciplinary Evaluation.

Fostering Values-Driven Sustainability Through an Ex-Post Evaluation Capacities Lens

 

Jindra Cekan/ova

Founder of Valuing Voices at Cekan Consulting LLC

Background: Ex-post evaluation of sustainability has been done for 40 years in global development. However, it has been done far less than 1% of all global development projects, for there is little proof that “sustainable” development is or is not. Similarly, foreign aid projects are implemented to foster sustainability, but without the benefit of evidence from ex-post evaluations of what drove it and limited research on the benefits of robust exit strategies..

 

Purpose: Transparency in values we hold, and evaluative capacities’ best practices that we bring to our evaluations inform how they are done, with whom, and for what. Using the evidence base from ex-post evaluations and exit strategies led to these nine checklists. Professionals in monitoring and evaluation should use them to foster long-term sustainability and learning.

 

Setting: Drawing on primary and secondary research across 91 ex-post evaluations of foreign aid sustainability plus two major studies of exit strategies globally..

 

Intervention: Not applicable.

 

Research Design: The checklists were drafted based on sustainability and exit studies and then vetted with lead researchers of the two exit studies. They were revised, and additional research was done on both values-driven evaluation and evaluation competencies.

Data Collection and Analysis: Some primary data was collected during ex-post evaluations by the author, complemented by secondary research.

 

Findings: Sustained exit commitments and conditions checklists can build evaluator capacities in evaluating sustainability. Several have been used by Tufts, USAID, the GEF, and the Adaptation Fund and verified actual sustainability and its prospects. Also, evaluator capacities can be built.

 

Keywords: ex-post evaluation; sustainability; monitoring and evaluation; values; competencies; M&E checklists

Abstract

 

Monitoring and evaluation (M&E) work is guided by an array of values held by funders, implementers, M&E experts, and project participants and partners. Some values are explicit, while others are assumed, such as the truth of “values-neutral” evaluation or that projects are sustainable in the long term. I espouse Patton’s (2022) “activist interventionist change-committed evaluation” by both advocating for ex-post evaluation of many development aid projects’ untested hypotheses about durability, and suggesting ex-post lessons can shape development aid projects from design to closure. Ex-post lessons are valuable for current project planning, design, implementation, and M&E. Using them can make development results more sustainable. Checklists created to ease monitoring and evaluation of prospects for sustainability should be used with country nationals. Six evaluator competencies support sustainability practice, namely systems thinking competency, collaboration competency, anticipatory competency, and reflective, technical, and situational practice competencies. Drawing on several studies that validate this approach, this paper shows how infrequently ex-post evaluations of sustainability are conducted. This seems to indicate that the lessons learned from ex-post evaluation are not valued. Bringing lessons from rare ex-post evaluations to benefit current implementation and exit is the core of the checklists described in this article. Learning from ex-post evaluations and exit studies is very beneficial to inform current aid projects and helps results last. Evaluator competencies are built through this paper. Evaluating both the results expected by donors and new, locally emerging outcomes from local efforts to sustain results also adds value to the canon. Ongoing learning and sharing lessons from progress around the project cycle, from participants to donors, and among M&E experts is vital, especially bringing those lessons back to new projects. The six competencies, the technical checklists, and evaluative thinking about sustainability can help shift programming toward locally led and sustainable development.

Introduction

 

This paper explores a range of values and capacities needed to support the sustainability of foreign aid development projects. It draws on 12 years of Valuing Voices research.[1] This initiative, aimed to increase sustainable solutions for excellent impact through learning from ex-post project sustainability evaluations, also focuses on how evaluators can promote the design, monitoring, and evaluation of sustainability pre-closure and draw on germane evaluator competencies. This paper explores a range of evaluators’ views on the values we bring as monitoring and evaluation experts, as well as the competencies needed to design, implement, monitor, and evaluate for long-term sustainability.

 

Both implicit and explicit values that donors, implementers, and M&E commissioners bring to global development work influence how that work is done. Evaluators need to be aware of and promote the explicit and implicit values that drive M&E work to build evaluation capacity that manifests evaluation values to ascertain which project results are sustainable, by whom, for how long, and why.

 

Sustainability, i.e., the long-term durability of project results, does not happen by itself; it needs to be fostered during the project, but more needs to be known about the conditions required for sustainability to take root after project closure and exit. Valuing Voices’ founder, consultants, and clients believe that evaluating sustainability cannot be limited to desk studies; that eliciting the views of country-based former project participants and partners is key. Based on the lessons from 10 such ex-post sustainability and exit evaluations done by Valuing Voices and over 90 other studies that include participant responses from a variety of donors and implementers,[2] plus seminal studies of exit strategies from Lewis (2016) and CDA (2020), we found nine elements need to be monitored and evaluated from project design to the ex-post years after closure. Development practitioners, including evaluators, need to build their knowledge about what has been sustained in ex-post evaluations and have this inform how they advocate to include these nine elements in project design, implementation, monitoring, and evaluation. This will need equal participation by national partners and participants to be built in throughout to foster long-term results and for new emerging pathways to emerge.

 

The nine elements are presented below in the form of checklists, which function as evaluator capacities tools. For by identifying what elements are needed to foster sustainability in programming, evaluators can inform clients and employers of what needs to be designed, implemented, monitored, and evaluated. The checklists cover two kinds of sustainability drivers: (a) commitments to sustainability, which includes designing beyond the project lifetime through a theory of sustainability, thinking about how to foster sustainability through the process of exit/handover, and considering risks and resilience; and (b) building conditions within the very project to foster lasting sustainability. This involves looking beyond resources as the only driver of durability, to seeing what makes local ownership of results robust. This includes considering several questions: How should equitable partnerships be fostered for long-term results? What capacities to keep disseminating behavior change exist? How adaptive are the timeframe and exit to foster sustainability? How accountable are projects in their communications to partners as they exit?

 

One of the greatest shocks that threatens the sustainability of most global development aid investments is climate change, which is why the natural world and access to viable nature is part of both risks and resilience to shocks. It is discussed separately, given the urgency with which we need to monitor and evaluate its progression and effect on sustainability. Some evaluator competency-building resources that help to evaluate the natural world have been added (e.g., Brouselle, 2022; Rowe, 2019). This is because nature is assumed and often overlooked in much global development programming design and evaluation, as seen in the review of several hundred ex-posts, exit reports, webinars, and evaluations, including blog posts about sustainable development by Cekan (2020a; 2020b), and underscored by Rowe (2019). The natural world and its environmental sustainability are a missing link, while the oft-stated but rarely evaluated “resilience” is often unproven (except for new ex-post research by the Adaptation Fund (2022). A viable natural world continuing to support lives and livelihoods underpins sustainability across so much of global foreign aid and urgently needs inclusion in all evaluations.

 

Defining Evaluation, Its Values, and Sustainability

 

Michael Scriven defined evaluation this way: “Evaluation determines the merit, worth, or value of things” (Scriven, 1991, as cited in Coffman, 2004, p. 1). “Valuation” (measurement, estimation of worth) is embedded in our work as evaluators. Increasingly, the field of evaluation is discussing the values that underpin the work of evaluators. Thomas Archibald notes in a book review, “Schwandt, House, and Scriven—call into question the dubious ‘value-free doctrine’ of the social sciences… [and] emphasize[s] the obvious yet frequently ignored primacy of values and valuing in evaluation” (2016, p. 448). Evaluation, from the perspective of Michael Scriven, is filled with values:

 

If evaluators cling to a values-free philosophy, then the inevitable and necessary application of values in evaluation research can only be done indirectly, by incorporating the values of other persons who might be connected with the programs, such as program administrators, program users, or other stakeholders. (Encyclopedia.com, 2018, para. 26)

 

This opens a door for participatory input from those most closely connected to projects¾the partners and the participants.

 

Michael Quinn Patton highlights tensions between evaluations that seek independent definitive judgments versus those that honor diverse perspectives. He values work done via participatory co-creation by activist, interventionist, change-committed evaluators, where the evaluation itself engages in change. This paper explicitly encourages those involved in monitoring and evaluation to work through participatory co-creation, because sustainability can only be maintained if it is locally driven. Evaluation also needs change-committed evaluators who embrace long-term sustainability.

 

The Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD/DAC) defines sustainability as the basis for ex-post project evaluation. Their definition includes that same reference to long-term sustainability, and its evaluation is part of the change needed in our field¾namely, a focus on longitudinal results: “the continuation of benefits from a development intervention after major development assistance has been completed…. [and] [t]he probability of continued long-term benefits. The resilience to risk of the net benefit flows over time” (2002, p. 37). In OECD/DAC’s updated and detailed definition, evaluators are directed to consider sustainability

 

at each point of the results chain and the project cycle of an intervention. Evaluators should also reflect on sustainability in relation to resilience and adaptation in dynamic and complex environments. This includes the sustainability of inputs (financial or otherwise) after the end of the intervention and the sustainability of impacts in the broader context of the intervention. For example, an evaluation could assess whether an intervention considered partner capacities and built ownership at the beginning of the implementation period as well as whether there was willingness and capacity to sustain financing at the end of the intervention. In general, evaluators can examine the conditions for sustainability that were or were not created in the design of the intervention and by the intervention activities and whether there was adaptation where required…. If the evaluation is taking place ex post, the evaluator can also examine whether the planned exit strategy was properly implemented to ensure the continuation of positive effects as intended. (2019 Sustainability, para. 3, 6).

 

These key elements, especially the “conditions for sustainability,” inform the checklists in this paper.

 

The OECD also differentiates between durability and ecological sustainability. With the latter being relegated to:

 

Confusion can arise between sustainability in the sense of the continuation of results, and environmental sustainability or the use of resources for future generations…. environmental sustainability is a concern (and may be examined under several criteria, including relevance, coherence, impact, and sustainability). (2019, Sustainability, para. 2)

 

Yet sustainability rests on our valuing the environment and planning for risks and resilience to sustainability (see Figure 8). As evaluators, we need to push donors and implementers to examine the natural system’s resilience, which supposedly unrelated sectors rely on. For instance, the environment affects sectors such as income generation (e.g., natural products being processed by people generating income) and education (e.g., the gardens that subsidize teacher salaries, or the farming, relying on rain, that supports parents to afford school fees). In “Planting Seeds for Change,” evaluator Brouselle (2022) reminds us of the primacy of climate values in Evaluation’s COP26 compendium:

 

We must challenge the ways that evaluations are commissioned; how policies and programmes are framed¾to take risks, going beyond existing evaluation mandates, to improve equity, health and prosperity; reduce pollution; take care of our air, waters and lands; and protect biodiversity… we should use our facilitating skills to foster democracy and engagement. Evaluators can contribute to creating spaces for dialogue and debate with commissioners, participants, and stakeholders, on the socio-ecological impacts of projects, programmes and policies. (para. 4)

 

Linking Competencies and Capacities to Sustainability via Valuing Voices Sustained Exit Checklists

There are six types of evaluator competencies that are relevant to focus work planning for sustainability during design/implementation or conducting an ex-post sustainability evaluation.

 

Evaluation as a field needs to embrace a variety of such competencies as we seek to address a range of complex problems. The first three competencies come from the United Nations Educational, Scientific and Cultural Organization (UNESCO), from a 2017 report called “Education for Sustainable Development Goals: Learning Objectives,” which informs the macro view for sustainability and locally led development.

 

Systems Thinking Competency

UNESCO (2017) defines this competency as “the abilities to recognize and understand relationships; to analyse complex systems; to think of how systems are embedded within different domains and different scales; and to deal with uncertainty” (p. 10). This is key as interventions interact with complicated societies, often with wider aims than what just one project wants to achieve. Uncertainty affects projects in implementation (which is why adaptive management is a checklist item (see Figure 7). Further, because ex-posts are not about direct attribution, given the complexity of communities, but contribution, it is vital to look at a range of outside influences post–project closure that could explain the results (not) seen.

 

Collaboration Competency

 

This competency is pivotal in designing, implementing, monitoring, and evaluating sustainability, which lies in both “the abilities to learn from others; to understand and respect the needs, perspectives and actions of others… and to facilitate collaborative and participatory problem solving” (UNESCO, 2017, p. 10). Listening to those who will be tasked with sustaining results or innovating emerging outcomes involves a close collaboration, as does using participatory methods to both design for and troubleshoot/problem-solve with.

 

Anticipatory Competency

 

Anticipatory competency is “the ability to understand and evaluate multiple futures¾possible, probable and desirable¾and to create one’s own visions for the future, to apply the precautionary principle, to assess the consequences of actions, and to deal with risks and changes” (UNESCO, 2017, p. 10). This competency is key to the whole field of sustainability as a field of study. Often projects assume sustainability will be the long-term result of development efforts. But, as Rogers and Coates (2015) note,

 

Hope is not a strategy. Sustainability plans that depend on the expectation, or hope, that individuals and organizations will continue to function without the key factors previously identified are not likely to achieve this goal. Such plans should take account of what is feasible within the economic, political, and social/cultural context of the areas in which they work. (p. 44)

 

This also relates to two other competencies, systems thinking (discussed above) and situational practice (discussed below).

 

The Canadian Evaluation Society (CES; 2018) provides us with the second three domains relevant to sustainability that evaluators need to consider in terms of how the M&E is done.

 

Reflective Practice Competencies

CES’s Reflective Practice domain includes competencies that “focus on the evaluator’s knowledge of evaluation theory and practice; application of evaluation standards, guidelines, and ethics; and awareness of self, including reflection on one’s practice and the need for continuous learning and professional growth” (2018, p. 5). This competency applies to the content of the sustainability methods presented below, as well as the knowledge evaluators will gain from evaluating prospects for sustainability and emerging outcomes (Figure 1) in projects. Additionally, this competency domain includes both considering “the well-being of human and natural systems in evaluation practice” and being “committed to transparency” (p. 6), which is the aim of using the checklists as a whole sustainability learning process. It is important in such reflection to clarify one’s values.

 

Technical Practice Competencies

These competencies focus on the “strategic, methodological, and interpretive decisions required to conduct an evaluation” (CES, 2018, p. 5), which directly applies to the five sustained exit commitments and conditions (see Figure 3). One competency, “assesses program evaluability,” is germane to ex-post evaluation and prospects for long-term sustainability. Cekan and Legro (2021) have applied the elements in the nine checklists which comprise the Embedding Sustainability in the Project Cycle framework to a World Bank sustainability study, and Cekan has used it in ex-post evaluations, such as a recent one for youth employment (USAID Mali, 2022). It has informed the training materials created for the Adaptation Fund (2023) on how to evaluate sustainability and resilience ex-post.

 

Situational Practice Competencies

As so few projects are “cookie-cutter” versions of each other, it is always vital to contextualize each project and its prospects for sustainability in its unique context, applying CES’s third competency domain, Situational Practice: “Focus on understanding, analyzing, and attending to the many circumstances that make every evaluation unique, including culture, stakeholders, and context” (CES, 2018, p. 6), identifying how specifically the project has moved around the project cycle (see Figure 2), particularly monitoring “organizational changes and changes in the program environment during the course of the evaluation” (p. 7) as well as tracing changes that lead to likely sustainability post-project, and building evaluation capacity by “engag[ing] in reciprocal processes in which evaluation knowledge and expertise are shared between the evaluator and stakeholders” (p. 7) throughout both the analysis and the sharing of the learning results.

 

Competencies that M&E professionals need can be used when monitoring and evaluating prospects for sustainability during project implementation as well as during ex-post evaluations. Sustainability prospects increase when they are designed and planned for, as Zivetz et al. (2017) found in researching ex-posts. There are clear advantages of planning for sustainability measurement from the outset of the project as well as measuring sustainability through the entire project cycle. Donors, implementers, and experts in monitoring and evaluation, as well as national partners, need to be trained in these competencies.

Evaluating Sustainability in Practice

Aid experts including evaluators embed values in their work in a myriad of ways, starting with how projects are funded and designed and by whom; for this reason, much M&E emphasis is on final rather than ex-post evaluations and learning from them. Over $3.5 trillion has been spent on public foreign aid projects in the past 70 years (OECD, 2019). Yet, the aid industry has evaluated fewer than 1% of these projects for sustainability (Cekan, 2015). Valuing Voices’ ex-post research on 39 organizations’ ex-post evaluations of sustainability shows that most project results decrease (10–90%) as early as 2 years ex-post (Valuing Voices, 2012).

 

Except for the Japan International Cooperation Agency (JICA), which has done over 2,500 ex-post evaluations on their grants, loans, and technical assistance, learning from what lasts is rare among international aid donors and implementers. An Asian Development Bank study (2010) of post-completion sustainability found that “some early evidence suggests that as many as 40% of all new activities are not sustained beyond the first few years after disbursement of external funding” (p. 1). The World Bank and Inter-American Development Bank, both multilateral banks, show less stellar investments in ex-post learning (Lopez, 2015; Cekan, 2022). Ex-post evaluations are rare, as is illustrated by a Sustainable Governance Indicators overview of EU member state policy evaluations, with most countries using them rarely or not at all (Sustainable Governance Indicators, n.d.).

 

Often in the ex-post evaluation of sustained impact, we see some results fade as early as 2 years ex-post. It is key to prioritize learning from what was sustained by asking our project participants and local/national partners directly during implementation about sustainability prospects. Field inquiry gives no time to test assumptions about drivers/barriers that the project is being implemented under and test whether optimistic trajectories will hold post-closure, as is widely assumed in the global development industry. For as Sridharan and Nakaima (2010) write:

 

There is no reason for the trajectory of performance outcomes to be linear or monotonic over time¾this has important implications for an evaluation system… [and] should programs that do not have a ‘successful’ trajectory of ‘performance measures’ be terminated? (p. 144)

 

To make sustainability more likely, designing, implementing, monitoring, and evaluating for sustainability is key, and makes successful trajectories more likely. While widespread ex-post learning would be the most effective, lessons can be learned to manifest our values of pro-sustainable development by extracting learning from the ex-post evaluations and exit studies that have been done. This is the aim of the rest of this article.

 

Most ex-posts have found mixed results of some activities being sustained, and others not. Often, what was relevant and locally owned, was sustained, whereas activities that relied on donor incentives such as food aid failed to continue (Catholic Relief Services [CRS], 2016). A 2020 Jones and Jordan ex-post study of USAID Global Waters projects found that while 25 million have gained access to water and sanitation,

 

despite tremendous achievements within the life of our programs, they have largely not endured… Rural water systems that, at activity close, delivered safe water to households have fallen into disrepair. Basic latrine ownership and use have dwindled. Communities certified as open-defecation free are backsliding, and gains in handwashing have not been sustained. [Nonetheless,] where USAID invested in providing technical assistance to committed government partners and utilities, gains in service provision and local capacity were sustained, with local actors taking up and expanding upon best practices introduced during activity implementation. (para. 3, 4)

 

This again supports designing and implementing for sustainability during the project, which is the aim of this paper. But such reviews are rare among donors.

 

The dearth of ex-post evaluations suggests that most global development evaluations currently being conducted are not value neutral. Commissioners seem to value short-term results rather than showing and learning from sustained impacts. Further, donors and implementers design and fund aid projects and their evaluations. Country nationals need to be engaged throughout the project cycle (Figure 2), for they will be left to sustain results. As Scriven stated in discussions with Donaldson, Patton and Fetterman (2010),

 

I want to hear, not just about intended use or users of the evaluation. I want to find out about impact on intended and actual impactees—the targeted and accidental recipients of the program, not just the people that get the evaluation. So I consider my task as an evaluator to find out who it is that this program is aimed at reaching and helping. (p. 23)

 

Emerging Outcomes

 

Typical ex-post evaluations focus on what lasted from what donors funded. Few evaluations return ex-post to also ask the front-line users, project participants, and partners what lasted of the prior project, and what emerged from their local efforts to sustain results with fewer or different resources, partnerships, etc. This glaring omission speaks to a lack of valuing sustained results, much less learning from local capacities to sustain results differently. Thus an innovation by Valuing Voices in evaluating sustainability, either ex-post or for monitoring sustainability, is the search for emerging outcomes, namely what emerges from local efforts to sustain results, rather than focusing only on expected donor-designed pathways to still exist.

 

The example in Figure 1 comes from 2023 Adaptation Fund training materials on ex-post; it draws on a three-year World Food Program Ecuadorian ex-post evaluation of sustainability and resilience. The expected change was that improving the water supply for crops would lead to improved food security. While that was happening to some degree, other outcome pathways were happening as well. In some areas, more water was used to improve cultivation methods, which led to an emerging outcome of children returning home to their rural villages to help their parents and continued to sustain food security, which decreased family vulnerability. Elsewhere, maladaptive pathways also emerged, in which a landslide eliminated the stable water reservoir source in one site, leading farmers to revert to drawing water from a river via pump systems, which likely led to decreased water for the community.

Figure 1. Expected, Emerging, and Unexpected Outcomes Ex-Post

 

 

 

 

 

 

 

 

Note. From Training Material for Ex Post Pilots, by Adaptation Fund, 2023 (https://www.adaptation-fund.org/document/training-material-for-ex-post-pilots/).

 

The picture is incomplete without looking at what was expected to be sustained and what local communities had to innovate to maintain results. Unless we look at both what was expected to be sustained and what local communities had to innovate to sustain results, the picture would be incomplete. Both can be traced during implementation and at ex-post evaluation.

 

Sustainability Around the Project Cycle

We need to build sustainability in from the onset, from funding and design to implementation, while looking out for alternative paths that locals create (see the orange slices in Figure 2). Once local stakeholders are involved throughout the project cycle (green slices in Figure 2), results are more likely to be sustained, for the programming is done with country nationals who will sustain results after donors leave. Assumptions need to be checked, adaptation to foster durability needs to be monitored and evaluated, and exit needs to include consultations on ownership, resources, partnerships, adaptation, resilience, and communications, much of which can be traced in a theory of sustainability.

 

Figure 2. Embedding Sustainability in the Project Cycle

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Note. From “What Happens After the Project Ends?”, by J. Cekan, 2016 (https://valuingvoices.com/what-happens-after-the-project-ends-country-national-ownership-lessons-from-post-project-sustained-impact-evaluations-part-2/ ).

 

As ex-post evaluation of projects is an important link missing before exiting with participants and partners leading sustainability; this paper focuses on lessons learned from the 90+ ex-posts reviewed. Lessons come from projects such as those below. Roughly 80% of the CRS Niger PROSAN food security project was sustained 3 years ex-post. It was implemented for sustainability by taking the final 18 months to exit, rather than 3 to 6 months. National partners were co-implementers pre–project closure. The UK charity EveryChild similarly worked with INTRAC (Lewis, 2016; Morris et al., 2021) to evaluate sustainability during exit. They did so in four countries 5 years ex-post, learning similar lessons about phasing down and over before exiting sustainably.

Were national stakeholders to partner equally, these local “targeted recipients” as Scriven tells us, could require projects not to close until further funding was secured, as EveryChild UK did. Donors, implementers, and evaluators need to listen to what locals want and can sustain. All of us who value sustainable development need to design M&E to incorporate sustainability. Exemplary studies are an ex-post tracing national primary teacher training (USAID Uganda, 2017) and final evaluation projecting sustainably prospects pre-exit from migrants and NGOs in Bangladesh (Hasan, 2021).

Thus, the checklists below help foster sustainability through M&E that involves questioning assumptions that donors and implementers, partners, and participants hold about the sustainability of results. It means building capacities to monitor and evaluate conditions for sustainable impact that are embedded in a traceable, relevant way as projects are implemented. It means documenting and learning from data throughout implementation, planning sustained exit beyond the final evaluation, and retaining data to be evaluated ex-post. This involves building understanding and capacities for ex-post evaluation and project planning (funding, design, implementation, and M&E) to foster it. This includes national stakeholders and evaluators who have a greater stake in their countries who can help foreign national stakeholders focus on learning what excelled or failed and how to use it for future projects in-country.

 

Validation

 

Several sources of expertise inform and validate the checklists (see Figures 4 to 8). In their 2015 analysis of exit strategies and sustainability for four USAID / Food and Peace countries, Rogers and Coates highlighted monitoring and evaluating the presence of four “drivers” of sustainability. These drivers create conditions that both are used to evaluate sustainability ex-post and are likely indicators for how likely sustainability is (if such drivers were put in place during implementation pre-exit). Rogers and Coates’ drivers are (a) sustained motivation/ ownership by national stakeholders to sustain a project’s activities; if activities are yielding relevant results, they are far more likely to be sustained; (b) a sustained flow of resources from, national or international sources; (c) sustained technical and managerial capacities passed on to new participants; and (d) linkages/partnerships with governmental/private or other organizations, for an array of support. Negi and Sohn (2022) confirmed the presence of these drivers across Global Environment Facility (GEF) projects created by Rogers and Coates and applied by Cekan and Legro (2022). Negi and Sohn’s review of 62 projects also confirmed that project design, a key sustainability driver, feeds into OECD’s (2019) Relevance criterion, as well as Figure 4. Similarly, USAID Uganda (2017) found the same four drivers were operational in sustainability.

These elements of sustainability draw on ex-post research by Cekan and key studies about participatory implementation and exit. One is Anderson, Brown, and Jean’s (2012) report Time to Listen. They interviewed 6,000 recipients and implementers of international aid across 20 countries from inside and outside the aid system. Their study focuses on unearthing stories “on the ways that people on the receiving side of aid suggest it can become more effective and accountable” (p. i). A second source was CDA (2020) case studies research led by Jean and a consortium of non-governmental organizations (NGOs), focused on improving exit. This work, Stopping as Success, highlighted that a gradual exit process contributes to sustainability. This research informs one of the commitments mentioned in Figure 3, namely phasing down over time during implementation and to national partners before exiting. These studies underscore that global development should be informed by local conditions and country nationals. Local participation is important while checking on sustainability prospects, as is getting local feedback on how well exit is going pre-closure. These checklists below also draw on seminal research by Lewis for INTRAC (2016), from extensive work on exit among NGOs.

Sustained Exit Commitments and Conditions Checklists

 

Figure 3. Valuing Voices Sustained Exit Commitments and Conditions Checklists

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Note. From “Exit for Sustainability Checklists,” by Valuing Voices, 2020 (https://valuingvoices.com/wp-content/uploads/2021/03/Exit-For-Sustainability-Checklists-Dec2020-2.pdf).

 

Now, let’s return to reflect on how the evaluator competencies articulated by UNESCO and CES fit into these Figure 3 commitments and conditions. Systems thinking competency leads us to consider what a theory of sustainability could consist of, and how to plan for it, given the complex ecosystems any project is embedded in. Collaborative and anticipatory competencies are brought into play when handing over projects during implementation, pre-exit. This is especially relevant to partnerships seeking to best face unknown future risks to sustainability and foster resilience to shocks pre-closure. Taking these commitments to heart predisposes projects to continuation. Another competency, reflective practice, needs to be used to discern which conditions of sustainability are driving change. Further, technical and situational practice are used in the field, examining if and to what degree sustainability is driven by these six conditions. While four of the six conditions (ownership, resources, capacities, and partnerships) driving sustainability come from the Rogers and Coates study, two additional conditions have been found to be important in the exit literature. Namely, how well timeframes pre-exit can be shifted to enable sustainability, and how clear and accountable the communication is between those closing out and those being left before closure. Consider using the nine checklists listed in Figures 4 through 8 along a scale of high–medium–low and revisiting them periodically to gauge change.

Revising a theory of change into a theory of sustainability (Figure 4) is helpful to chart stakeholders, assumptions, trajectories, key questions, and whom to ask.

 

Figure 4. Sustainability Ex-Post Project: Theory of Sustainability

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Ask all stakeholders involved long before exit about how much they feel they “own” the project’s continuation and the resources needed. There is a wide range of resources to be explored and questions to ask about how much the interventions are generating local results that are valued (see Figure 5).

 

Figure 5. Designing for Exit: Ownership/Motivation and Resources

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The questions in Figure 6 can be used during baseline and midterm evaluations. Some questions can also be selected, as part of ongoing monitoring, from the lists of resources and ownership (above) and capacity strengthening and partnerships. With such data, evaluating sustainability during ex-post evaluations is much easier.

 

Figure 6. Checking Assumptions: Capacity Strengthening and Partnerships

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Two of the elements that tell the most about the extent to which project implementation fosters sustainability are the amount of planning that has gone into project exit and handover, as well as adapting timeframes to readiness for exit (see Figure 7).

 

Figure 7. Monitoring and Adaptation: Exit/Handover, Timeframe, and Adaptation of Implementation

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Finally, long-term sustained and responsible exit fostering local ownership is based on planning for the immediate term (communications about who leaves and who knows why the project is closing, how respectfully this is this done and with how much involvement by local partners). As shown in the two checklists in Figure 8, it is vital to examine how well consideration of present and future risks and resilience to shocks have been embedded in programming.

 

Figure 8. Exit Consultations and Close: Risks/Resilience and Accountable Communications

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Conclusions

 

In addition to infusing sustainability into the project cycle during implementation, it is important to live one’s values and use evaluator capacities as guiding lights for one’s work. What also matters is monitoring and evaluating sustained ownership and the other hallmarks of sustainability within the checklists during programming and at ex-post evaluation. Further, it’s important to look for the capacities that remain behind after projects close (emerging outcomes) and learn from ex-post evaluations to inform current programming to facilitate sustainability while there are sufficient resources, partnerships, capacities and other conditions. Also important is fostering what national and local stakeholders want to sustain through their commitments and conditions. Six competencies equipping monitoring and evaluation experts to do this well have been outlined above, namely systems thinking competency, collaboration competency, anticipatory competency, and reflective, technical, and situational practice competencies. These types of “evaluative thinking” lenses can and should be used, as Archibald (2021) calls for an “ethical accountability” in locally led development. Values-driven sustainability can be a powerful driving force to improve public accountability and good governance. Equipped with such skills, evaluators simultaneously bolster evaluation systems and capacities among national evaluators and program implementers alike. For equitable, values-driven accountability for sustainability to happen, power needs to shift to people at national and local levels to determine what resources, partnerships, and capacities are needed and what is a priority for them to take ownership of. We can begin as soon as possible by building the most likely conditions for sustainability and commitments to foster sustainable exit into the project cycle. We have no time to lose; embracing such values-driven sustainability would be of great benefit.

 

 

References

 

Adaptation Fund (2022). Training material for ex post evaluations. https://www.adaptation-fund.org/about/evaluation/publications/evaluations-and-studies/ex-post-evaluations/

Adaptation Fund. (2023). Training materials for ex post pilots. https://www.adaptation-fund.org/document/training-material-for-ex-post-pilots/

Anderson, M. B., Brown, D., & Jean, I. (2012, December 1). Time to listen: Hearing people on the receiving end of international aid. CDA Collaborative. https://www.cdacollaborative.org/publication/time-to-listen-hearing-people-on-the-receiving-end-of-international-aid/

Archibald, T. (2016). Evaluation foundations revisited: Cultivating a life of the mind for practice [Review of the book Evaluation foundations revisited: Cultivating a life of the mind for practice, by T. A. Schwandt]. American Journal of Evaluation, 37(3), 448–452. https://doi.org/10.1177/1098214016648794

Archibald, T. (2021, February 18). Critical and evaluative thinking skills for transformative evaluation. Eval4Action. https://www.eval4action.org/post/critical-and-evaluative-thinking-skills-for-transformative-evaluation

Asian Development Bank. (2010, October). Special evaluation study on post-completion sustainability of Asian Development Bank-assisted projects. Organisation for Economic Co-operation and Development. https://www.oecd.org/derec/adb/47186868.pdf

Brouselle, A. (2022). Planting seeds for change. Evaluation28(1), 7–35. https://journals.sagepub.com/doi/full/10.1177/13563890221074173

Canadian Evaluation Society. (2018, November). Competencies for Canadian evaluation practice. Evaluation Canada. https://evaluationcanada.ca/files/pdf/2_competencies_cdn_evaluation_practice_2018.pdf

Catholic Relief Services. (2016, October 7). Participation by all: The keys to sustainability of a CRS food security project in Niger. https://www.crs.org/our-work-overseas/research-publications/participation-all

CDA. (2020). Stopping as success: Research findings case studies. CDA, Peace Direct, Search for Common Ground. https://www.stoppingassuccess.org/resources/

Cekan, J. (2015). When funders move on. Stanford Social Innovation Review. https://ssir.org/articles/entry/when_funders_move_on

Cekan, J. (2016, February 19). What happens after the project ends? Country-national ownership lessons from post-project sustained impacts evaluations (Part 2). Valuing Voices. https://valuingvoices.com/what-happens-after-the-project-ends-country-national-ownership-lessons-from-post-project-sustained-impact-evaluations-part-2/

Cekan, J. (2020a, April 20). Sustaining sustainable development. Valuing Voices. https://valuingvoices.com/sustaining-sustainable-development/

Cekan, J. (2020b, October 28). Sustained exit? Prove it or improve it! [Webinar]. Valuing Voices. https://valuingvoices.com/interactive-webinar-sustained-exit-prove-it-or-improve-it-nov-6-2020/

Cekan, J. (2022, November 26). Inter-American Development Bank (IDB) – Where have your ex-post evaluations, and learning from them, gone? Valuing Voices. https://valuingvoices.com/inter-american-development-bank-idb-where-have-your-ex-post-evaluations-and-learning-from-them-gone/

Cekan, J., & Legro, S. (2022). Can we assume sustained impact? Verifying the sustainability of climate change mitigation results. In J. I. Uitto & G. Batra (Eds.), Transformational change for people and the planet: Evaluating environment and development. Springer. https://link.springer.com/book/10.1007/978-3-030-78853-7

Coffman, J. (2004). Michael Scriven on the differences between evaluation and social science research. Evaluation Exchange, 9(4). https://archive.globalfrp.org/evaluation/the-evaluation-exchange/issue-archive/reflecting-on-the-past-and-future-of-evaluation/michael-scriven-on-the-differences-between-evaluation-and-social-science-research

Donaldson, S., Patton, M., Fetterman D., & Scriven, M. (2010). The 2009 Claremont debates: The promise and pitfalls of utilization-focused and empowerment evaluation. Journal of MultiDisciplinary Evaluation, 6(13). https://www.researchgate.net/publication/41391464_The_2009_Claremont_Debates_The_Promise_and_Pitfalls_of_Utilization-Focused_and_Empowerment_Evaluation

Encyclopedia.com. (2018, May 17). Evaluation research: Brief history. https://www.encyclopedia.com/social-sciences-and-law/sociology-and-social-reform/sociology-general-terms-and-concepts/evaluation-research

Hasan, A. A. (2021, January 17). Ex-post eval week: Are we serious about project sustainability and exit? American Evaluation Association AEA365. https://aea365.org/blog/ex-post-eval-week-are-we-serious-about-project-sustainability-and-exit-by-abu-ala-hasan/

Japan International Cooperation Agency. (n.d.). Ex-post evaluation (technical cooperation). https://www.jica.go.jp/english/our_work/evaluation/tech_and_grant/project/ex_post/index.html

Jones, A., & Jordan, E. (2020, October 19). Unpacking the drivers of WASH sustainability. USAID Global Waters. https://www.globalwaters.org/resources/blogs/unpacking-drivers-wash-sustainability

Lewis, S. (2016, January). Developing a timeline for exit strategies: Experiences from an Action Learning Set with the British Red Cross, EveryChild, Oxfam GB, Sightsavers and WWF-UK. INTRAC. https://www.intrac.org/wpcms/wp-content/uploads/2016/09/INTRAC-Praxis-Paper-31_Developing-a-timeline-for-exit-strategies.-Sarah-Lewis.pdf

Lopez, K. (2015, April 7). IEG blog series part II: Theory vs. practice at the World Bank. Valuing Voices. https://valuingvoices.com/ieg-blog-series-part-ii-theory-vs-practice-at-the-world-bank/

Morris, L., George, B., Gondwe, C., James, R., Mauney, R., & Tamang, D. D. (2021, June). Is there lasting change, five years after EveryChild’s exit? Lessons in designing programmes for lasting impact. INTRAC. https://www.intrac.org/wpcms/wp-content/uploads/2021/07/Praxis-Paper-13_EveryChild-exit.pdf

Negi, N. K., & Sohn, M. W. (2022). Sustainability after project completion: Evidence from the GEF. In J. I. Uitto & G. Batra (Eds.), Transformational change for people and the planet: Evaluating environment and development. Springer. https://doi.org/10.1007/978-3-030-78853-7_4

OECD. (2019). Applying evaluation criteria thoughtfully. [Chapter: Understanding the six criteria: Definitions, elements for analysis and key challenges]. OECD Publishing. https://www.oecd-ilibrary.org/sites/543e84ed-en/1/3/4/index.html?itemId=/content/publication/543e84ed-en&_csp_=535d2f2a848b7727d35502d7f36e4885&itemIGO=oecd&itemContentType=book#section-d1e4964

OECD/DAC. (2002). Evaluation and Aid Effectiveness No. 6 – Glossary of Key Terms in Evaluation and Results Based Management (in English, French and Spanish). https://read.oecd-ilibrary.org/development/evaluation-and-aid-effectiveness-no-6-glossary-of-key-terms-in-evaluation-and-results-based-management-in-english-french-and-spanish_9789264034921-en-fr#page37

Patton, M. Q. (2022, June 13). Why so many evaluation approaches: The short story version [Video]. YouTube. https://www.youtube.com/watch?v=6xY9jMUUorM.

Rogers, B. L. & Coates, J. (2015, December). Sustaining development: A synthesis of results from a four-country study of sustainability and exit strategies among development food assistance projects. Food and Nutrition Technical Assistance III Project (FANTA III) for USAID. https://pdf.usaid.gov/pdf_docs/PA00M1SX.pdf

Rowe, A. (2019). Sustainability-ready evaluation: A call to action. New Directions for Evaluation, 162. https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20365

Sustainable Governance Indicators. (n.d.). Evidence-based instruments. https://www.sgi-network.org/2020/Good_Governance/Executive_Capacity/Evidence-based_Instruments/Quality_of_Ex_Post_Evaluation

Sridharan, S., & Nakaima, A. (2010). Ten steps to making evaluation matter. Evaluation and Program Planning, 34(2), 135–146. https://doi.org/10.1016/j.evalprogplan.2010.09.003

UNESCO. (2017). Education for sustainable development goals: Learning objectives. https://unesdoc.unesco.org/ark:/48223/pf0000247444

USAID Mali. (2022, December). Ex-post evaluation of the USAID/Mali Out Of School Youth Project (PAJE-NIETA): Final evaluation report. https://pdf.usaid.gov/pdf_docs/PA00ZTBJ.pdf

USAID Uganda. (2017, October 11). Uganda case study summary report: Evaluation of sustained outcomes. https://pdf.usaid.gov/pdf_docs/PBAAJ314.pdf

Valuing Voices. (2020, December). Exit for sustainability checklists. https://valuingvoices.com/wp-content/uploads/2021/03/Exit-For-Sustainability-Checklists-Dec2020-2.pdf

Valuing Voices. (2012). Catalysts for ex-post learning. https://valuingvoices.com/catalysts-2/

Zivetz, L., Cekan J., & Robbins, K. (2017, May). Building the evidence base for post-project evaluation: Case study review and evaluability checklists. Valuing Voices. https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf

[1] https://valuingvoices.com/

[2] https://valuingvoices.com/catalysts-2/

This chapter in published form can be accessed at: https://www.researchgate.net/publication/376497201_Fostering_Values-Driven_Sustainability_Through_an_Ex-Post_Capacities_Lens

The EU and Sustainability

 

The EU and Sustainability

 

At the Prague European Summit I was invited to last week, sustainability was on the program three times. The most relevant session was Towards a More Sustainable and Prosperous European and Global Economy through Trade and Investments, pictured below.

 

 

 

 

 

 

 

I asked Panelists like Sabine Weyand, Director General for Trade at the European Commission, to define Sustainability. She defined it as actions the EU is taking, including mutually reinforcing Twin Transitions (Green and Digital) to foster a carbon-neutral EU by 2050 and the three pillars of sustainability, namely:

  • 1st pillar is ‘climate and environment,
  • 2nd, a zero carbon economy (including biodiversity), and
  • 3rd, social sustainability (equity by member states and internationally).

These were followed by several panelists mentioning the issue of member countries needing to support and fund these. One would think that funding would be gratefully accepted and reciprocated given the astonishing €338bil the EU already spent across Europe, eight times the size of the Marshall Plan.

The panelists of these sessions sketched out ambitious plans. I researched this oft-promised “EU Green Deal” to understand its scale, if the sustainability of results (which I work on ex-post), the sustainability of our global ecology, and/or sustainability in terms of business functions resides, if at all. The results of this cursory research did not make these clear. But ambitions are massive. 

EU Green Deal

The European Green Deal began when the EU Parliament adopted the EU Climate Law in 2021, which makes “legally binding a target of reducing emissions 55% by 2030 and climate neutrality by 2050. This moves the EU closer to its post-2050 objective of negative emissions and confirms its leadership in the global fight against climate change.”

 

SIDENOTE1: Given that our lawsuit against the Czech Republic government for not meeting its Paris Agreement commitments just failed and the Ministry of the Environment successfully claimed that reducing emissions by only 26% was enough (rather than the EU’s target of 55%), I wonder how much of these ‘legally binding’ targets will actually be met…

 

Turning to what I could find of the 1st and 2nd pillars (as the 3rd was missing from Green Deal online documents which I found other than mentions of a ‘Just Transition’):

 

  1. AID FUNDING (SDGs)

The EU as a leading global partner for the funding the SDGs & Paris Agreement

The EU seems to be shifting from the SDGs which appear to be bilaterally funded, although ‘collectively’ claimed. “The EU and its Member States are the leading donors of official development assistance (ODA) globally. In 2022, they collectively provided €92.8 billion (based on preliminary OECD figures), which accounts for 43% of global assistance.” So what progress is that generating? Not much:

 

 

 

 

 

 

 

 

 

SOURCE:https://unstats.un.org/sdgs/report/2023/progress-chart/Progress-Chart-2023.pdf

 

SIDENOTE2: And since we’re talking about climate, can the EU or other donors aid-fund functioning weather stations? They have 5% of the number the US and EU do which massively limits mitigation: https://africanarguments.org/2023/11/without-warning-africa-lack-of-weather-stations-is-costing-lives/

 

  1. SUSTAINABLE FINANCE FOR CLIMATE, ENVIRONMENT, AND ECONOMY (the 1st 2 pillars of the mentioned Sustainability)

So, for the EU, where I now live most of the time, sustainable finance plans seem to be off to a good start. Nearly 2/3 of the promised €300 billion has been secured in only two years. Notably, this is 1/3 of the EU’s planned €1 Trillion, including private investments. Two German policy researchers, Findeisen and Mack, question their achievability. “Europe needs to spend an additional €350 billion on climate action every year until the end of this decade…to reach its 55% greenhouse gas reduction target by 2030.” Furthermore,shortcomings of [Investment Plan] InvestEU in combatting climate change can be addressed and why it is no substitute for fresh public spending at EU level.” (See the excellent brief including concerns about the Sustainable Europe Investment Plan, outlined below. This overview of the EU’s Investment plan has broadly planned outputs (e.g., ambitions, financing, research, mobilizing), outcomes (e.g., energy generated, building new energy sources, restoring ecosystems, and building food systems and mobility), and impacts (e.g., a sustainable future):

 

 

 

 

 

 

 

 

 

Source: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52020DC0021&rid=7

 

Much of the plan is, understandably, European-focused, yet we all know that the Global South is where the deepest needs and opportunities lie. Drilling down further to see investments, activities, and even results on the ground was both interesting and harder:

The inaugural milestone of the Global Gateway was the Africa-Europe Investment Package with approximately €150 billion of investment dedicated to bolstering cooperation with African partners. We have also started implementing Global Gateway in Asia and the Pacific and in Latin America and the Caribbean, where President von der Leyen announced a global investment by the EU and its Member States of over €45 billion. In 2023, ninety key projects were launched worldwide across the digital, energy, and transport sectors through Global Gateway to strengthen health, education, and research systems globally.”

So, as an Africanist and evaluative researcher, I looked into the EU’s reported figures (and results). Under Global Gateway, the EU has funded 33 projects in Sub-Saharan Africa, 11 of which are climate-related and another 7 that are transport-related. There are only short descriptions of projects without substantive partners, activities, or data, such as a €1billion “Climate Change Adaptation and Resilience in Africa“. Details would be instructive, to say the least. Research on African Voices regarding their projects unearthed one that should already be of concern: Namibia’s $10 billion Green Hydrogen project, which is to help 2.5 million people decarbonize. However, the tendering is not transparent, locals are in the dark, there are biodiversity concerns, and initial funding is missing. Much more transparent monitoring & evaluation data is needed to know it is on track. 

 

I also consulted African Voices regarding COP28 that are relevant to the EU’s Africa investments.

  • Funds are needed. Lorraine Chiponda. a Coordinator of the Africa Movement Building Space notes that “Africa receives a meager 3% of the total global climate finance and African countries still play a marginal role in the global finance system, making it a mammoth task to obtain funds for renewable energy investment.”
  • Africa has much to offer. Joseph Nganga is Interim Managing Director at the Global Energy Alliance for People and Planet (GEAPP) reminds us that “Africa, a region with a developing economy endowed with abundant natural resources, stands at a crossroads where strategic financial investments can steer its trajectory towards a prosperous climate-resilient future.”
  • Moreover, financing needs are clearly expressed, which the EU, for one, should listen to. “This year at the African Climate Summit, countries were united in their call for a series of finance reforms that would have a meaningful impact on their fiscal and policy space to address climate change. These include a reduction in borrowing costs and risk premiums; debt management, restructuring and relief; scaling concessional climate finance from multilateral developments banks (MDBs); and reforms to the global tax regime.Olivia Rumble is a climate change legal and policy expert and a director of Climate Legal

 

Much more is needed for equitable partnerships to address our increasingly desperate climate needs. While I was at the conference, the Copernicus Climate Change Service issued a warning that “For the first time ever, the planet globally exceeded a key warming threshold on Friday for the first time since at least the beginning of instrument records, new data shows. Simply put, a 2-degree rise in global temperatures was considered as a target for the end of the century and is considered a critical threshold above which dangerous and cascading effects will occur.” COP28 has a lot to accomplish.

Helpful? Let me know in the comments…

 

 

5 Ways to Foster Sustainability and Resilience to Climate Change 

5 Ways to Foster Sustainability and Resilience to Climate Change 

by Omar Abdou, M.A. and Jindra Cekan/ova, Ph.D

In the global climate change graphic, the blue are countries that contribute very few emissions to #climatechange. They are also those who suffer the most from climate change effects and who have most #globalaid projects which are most at risk of no longer being #sustained:

 

 

 

 

 

 

 

https://globalgovernanceforum.org/global-issues/climate-change/ 

Omar Abdou, a food security and M&E expert from Niger, lives a vulnerability to climate change there is indisputable. Sahelien production systems suffer the full brunt of the negative effects. Indeed, the essential rural activities, namely agriculture, and livestock which occupy at least 80% of the population, are becoming more and more uncertain as they rely on two pillars: rainfall and the exploitation of natural resources (soil, vegetation, other water sources). But, in recent decades, we have witnessed a drastic reduction in rainfall. In pastoral zones, this decrease is reflected in the degradation of the vegetation cover and the deficits in livestock-vital fodder. For instance, the fodder assessment from 2000 to 2020 for the Tahoua Region has been in deficit for 15 years. More and more recurrent and frequent crises seriously affect the main productive capital of households, which is the herd.

 

 

 

 

https://intraacpgccaplus.org/story/niger-centre-studies-climate-smart-agriculture-and-climate-finance/

Emergency sales of increasingly emaciated animals at a low price are growing to pay for consumer goods and to buy fodder to feed a few animals. These are kept for breeding after the crisis, but what used to be seasonal crisis (dry-season) has become chronic.

In agro-pastoral zones, agricultural production in the rainy season, which has changed its duration and strength, becomes uncertain to feed the household members. The harvest is often exhausted at most by the 6th month of the year for many households. Thus, there is no more food surplus to supply the area’s food needs where cereals are not grown. Suddenly, cereals become expensive and inaccessible for many families. Their coping strategies are diverse but not always effective. In pastoral settings, parents are forced to take their children out of school to go further south in search of food and pasture. Sometimes more sedentary agro-pastoralist farmers pledge or sell their land to pay for food. Those left behind who have nothing to sell, are forced to cut trees and sell the wood for charcoal, further degrading the environment. Unfortunately, Niger’s neighbors are now mostly at war, so these young men run the risk of being recruited by armed groups or are killed. In these conditions, there is serious instability where the means of production (natural and human) are sold for survival and overexploited natural resources decrease. Increasingly, it is practically impossible to talk about sustainability. It is, therefore, necessary to think of consistent and innovative projects to enable households to produce sustainably while preserving the ecosystem for future generations. Outlets such as migration of the able-bodied workers to neighboring countries to provide labor in the lean season work now but are unsustainable. 

 

Jindra notes that 31 Oct the UN Climate Change Conference (COP26) begins. Climate science documents life-threatening effects such as Omar and billions of others are already experiencing around the ‘developing’ world. The IPCC Working Group report shows that “the world will probably reach or exceed 1.5 degrees C (2.7 degrees F) of warming within just the next two decades. Whether we limit warming to this level and prevent the most severe climate impacts depends on actions taken this decade. Only with ambitious emissions cuts can the world keep global temperature rise to 1.5 degrees C, the limit scientists say is necessary for preventing the worst climate impacts. Under a high-emissions scenario, the IPCC finds the world may warm by 4.4 degrees C by 2100 — with catastrophic results.” Erratic rainfall, greater heatwaves will affect 70% of all farmers and herders who rely on rain for their production and a new report on Adaptation to climate across West African shows aridity increasing, which must inform global development.

 

What can we do: 

1- LISTEN AND ACT GLOBALLY: There are excellent resources to track demands that the Global North divest and stop subsidizing fossil fuels. For instance, “In 2017/18, the G7 collectively spent more on fossil fuel finance than on climate finance, according to analysis by Oil Change International. They allocated nearly $40bn that year.” Hypocrisy abounds as this Climate Change News article goes on to note that “Since the Paris Agreement was signed, G20 countries’ export credit agencies have provided 14 times as much support for fossil fuels as clean energy “to $64 billion a year, and investors still prioritize profit over planet & people.  

 

2- LISTEN AND PRIORITIZE GLOBAL SOUTH VOICES: While funding demands for over $100billion to LDCs is a demand of COP26, the need to support local research and activist organizations who show measured results is mostly missing (more from Sweden here). Other resources include WRI’s Allied for Climate Transformation by 2025 podcast series among the most Climate Change vulnerable nations. “ACT2025 consortium to ensure voices from countries most exposed to climate change are heard, empowered, mobilized, and adequately supported in international climate negotiations.“ The series includes how countries such as Niger adapt to climate changeIf nothing else, consider the tragedy of the loss of $1.5 billion in aid invested in Niger, not to mention the ensuing suffering global Northern consumption is unleashing on the global South.

 

3- DESIGN CLIMATE SMARTLY: Global Development needs to do much more climate-smart programming as 50 years of investments in foreign aid to ‘developing’ countries has already started to be threatened. What can be done? From sequestering the carbon content of soil via farming to fostering climate-smart agriculture, which, according to the UN’s FAO means “agriculture that sustainably increases productivity, enhances resilience (adaptation), reduces/removes GHGs (mitigation) where possible, and enhances achievement of national food security and development goals”. Importantly this involves productivity plus adaptation and mitigation against changes wrought by climate change.

 

4- MEASURE SUSTAINABILITY: Exciting measurements are beginning! Scotland is valuing its nature, and there are calls for The Rights Of Nature In Evaluation Of Environmental Sustainability.  Jindra is consulting to a wonderful team at the Adaptation Fund. We are evaluating what could be sustained, and what withstood climatic changes is ex-post project sustainability and resilience evaluation. The Adaptation Fund on evaluating projects’ longer-term (3-5 year ex-post project closure) sustainability and, importantly their resilience to climate change. We are piloting these rare kinds of evaluations in six countries, starting with Samoa and Ecuador in 2021. As the Fund has funded over 100 projects in 100 countries, with almost 30 million participants, we have a rich array to choose from. Such a learning opportunity about what lasts and has withstood disturbances is vital. As the Fund notes: “Climate change is predicted to greatly affect the poorest people in the world, who are often hardest hit by weather catastrophes, desertification, and rising sea levels, but who have contributed the least to the problem of global warming. In some parts of the world, climate change has already contributed to worsening food security, reduced the predictable availability of freshwater, and exacerbated the spread of disease and other threats to human health. Helping the most vulnerable countries and communities is an increasing challenge and imperative for the international community, especially because climate adaptation requires significant resources beyond what is already needed to achieve international development objectives.”

 

5- TAKE PERSONAL RESPONSIBILITY: For 8 years, Valuing Voices has pushed back to evaluate the ‘Sustainable Development” we have promised for 50 years. We must be accountable primarily to our participants and partners, rather than donors, and only by listening to these country nationals, we can learn what was sustained not, why or why not, and what emerged from their efforts. In 2019, we wrote all of us are accountable to improve the SDGs. It is wonderful to have climate change leading organizations invest in ex-post evaluation that adds resilience, prioritizing organizational learning and accountability. We all must increase our awareness that curb our consumption, for the sake of Niger, small island states, and many other parts of our planet can no longer stand our business-as-usual. Limiting climate change begins with us, from changing our food consumption to consumer choices to political advocacy to seeing the Earth as indivisible from us and us from the Earth. Let us start to invest in global sustainability today.

Do you see any bright spots on the horizon? Do share!

“What IS Sustainability?” It depends on whom you ask: OECD, the UN, or Harvard Business School

“What IS Sustainability?” It depends on whom you ask: OECD, the UN, or Harvard Business School

Recently I’ve had conversations where I had to define which sustainability we were talking about. Was it:

  1. ex-post-project sustainability of outcomes and impacts,
  2. environmental sustainability, or
  3. business sustainability?

 

Since I spend most of my time evaluating the ex-post sustained and emerging impacts of foreign aid projects years after projects close, or at least advocate for it, let’s start there.

  1. The Organisation for Economic Co-operation and Development (OECD) is a “forum and knowledge hub for data and analysis, exchange of experiences, best-practice sharing, and advice on public policies and international standard-setting.” Regarding evaluation specifically, the OECD has “established common definitions for six evaluation criteria – relevance, coherence, effectiveness, efficiency, impact, and sustainability – to support consistent, high-quality evaluation”. Focusing on long-term sustainability, their evaluation guidance is:

 

 

Source: OECD, Better Criteria for Better Evaluation, 2019

 

The good news is that in this recent publication on Applying Evaluation Criteria Thoughtfully (2021), OECD keeps the updated definition but inches towards recommending actual ex-post project sustainability evaluation, rather than just projected (and assumed “likely to continue” sustainability). For this, “likely” is the most significant reason evaluators for donors and implementers have assumed, rather than evaluated, sustainability for decades. Further, positive, ‘sustained’ trajectories are also assumed at close-out/ exit, but rarely tested ex-post.

The OECD criteria give not evaluating it as an option. I far prefer “net benefits of the intervention continue” as it is a marching order: Prove that results were sustained. In this evolution, this 2021 report states, “After the completion of the intervention, and evaluation of sustainability would look at whether or not the benefits did continue, this time drawing on data and evidence from the intervention’s actual achieved benefits.”

 

OECD even goes on to recommend implementing and monitoring for sustainability. The new piece de resistance is: “Sustainability should be considered at each point of the results chain and the project cycle of an intervention”:

  1. “The sustainability of inputs (financial or otherwise) after the end of the intervention and the sustainability of impacts in the broader context of the intervention…. as well as whether there was willingness and capacity to sustain financing (resources) at the end of the intervention
  2. For example, an evaluation could assess whether an intervention considered partner capacities
  3. Built ownership at the beginning of the implementation period…. And
  4. In general, evaluators can examine the conditions for sustainability that were or were not created in the design of the intervention and by the intervention activities and whether there was adaptation where required.”

Yes, Valuing Voices highlighted this at the American Evaluation Association presentation “Barking up a Better Tree: Lessons about SEIE Sustained and Emerging Impact Evaluation” in 2016, and we have developed this into 2020’s Exiting for Sustainability trainings and checklists. Wonderful to see implementing for sustainability in guidance by the OECD! Many of the elements his research cites overlap with our Exiting for Sustainability checklislts which includes ownership, resources, capacities, partnerships, and implementation/ M&E process.

 

Moreover, while the 2019 OECD report mentioned resilience in passing, related to sustainability, “encourages analysis of potential trade-offs, and of the resilience of capacities/ systems underlying the continuation of benefits”. Such resilience and continuation of benefits evaluation involve examining huge systems (the financial, economic, social, environmental, and institutional capacities) that projects and programs are implemented within, whose stability is needed to sustain net benefits over time. Yes, for ex-post sustainability questions for evaluators to consider should include: “To what extent did the intervention contribute to strengthening the resilience of particularly disadvantaged or vulnerable groups” on which the sustained impacts of so much of our “Leave No One Behind” myth of Sustainable Development rely.

 

However, OECD makes suggestions to evaluate even broader, overwhelming what is feasible: “…this involves analyses of resilience, risks, and potential trade-offs.” Whose? All stakeholders, from participants to local partners and national and international implementers, and international donors? How far back and how far forward? What a huge undertaking. Further, the OECD points evaluators to define resilience, but as I learned in my Famine Early Warning System research and a current ex-post evaluation process for the Adaptation Fund, that involves creating evaluable boundaries by determining resilient to what kinds of shocks? Vital questions current industry monitoring and evaluation budgets for all evaluations, much less (too-rare) ex-post project evaluations, are insufficient for as they hover around 3-5% of total costs.

  1. Slight progress at OECD is being made by acknowledging environmental sustainability first brought up by the Brundtland Report, “Our Common Future” back in 1987. This linchpin report highlighted that “critical global environmental problems were primarily the result of the enormous poverty of the South and the non-sustainable patterns of consumption and production in the North. It called for a strategy that united development and the environment – described by the now-common term “sustainable development”… that meets the needs of the present without compromising the ability of future generations to meet their own needs.”

 

While an OECD brief in 2008 considers the environmental aspects of our thinking about sustainability, it argues that sustainability primarily about “using economic development to foster a fairer society while respecting ecosystems and natural resources.” The 2021 Applying Evaluation Criteria Thoughtfully rather unhelpfully mostly ignores the environment’s role in sustainability: “Confusion can arise between sustainability in the sense of the continuation of results, and environmental sustainability or the use of resources for future generations. While environmental sustainability is a concern (and maybe examined under several criteria, including relevance, coherence, impact, and sustainability), the primary meaning of the criteria is not about environmental sustainability as such; when describing sustainability, evaluators should be clear on how they are interpreting the criterion.” Given rapid climate change, I would argue that any sustained and emerging outcomes and impacts of projects that does not include an evaluation of the environmental context will fail to foster sustained resilience. Yet donors’ fixed funding timeframes that set completion to disbursement without evaluating sustainability or resilience continue to be huge barriers.

 

In 2022 a new resource on Sustainability from the media perspective questions the OECD. Sustainability: Going Beyond the Buzzword groups sustainability described in various media as: 1) Economic viability, 2) Social Sustainability, 3) Institutional Sustainability and 4) technical sustainability. But we’ve learned in 2022 from consulting work that the technology piece mentioned in this Buzzwords study regarding ‘baking in’ sustainability via the infrastructure/ assets created (The Adaptation Fund’s ex-post and resilience evaluations) and incorporating INGOs’ considerations of buying-local, seeing what local spare parts and technicians will be available in the future is a great added consideration. So too is promoting locally-led development through the very participatory way projects are implemented, with locals leading. Among the best resources is CDA/ Peace Direct/ Search for Common Ground’s Stopping as Success studies for USAID.

 

  1. Finally, business sustainability brings together these impacts on communities and society along with impacts on the environment. These are called ESG (Environmental, Social, and Governance) criteria. A Harvard Business School brief defines sustainability as “doing business without negatively impacting the environment, community, or society as a whole. “Where applied well, the aspiration is that “beyond helping curb global challenges, sustainability can drive business success.” While Harvard Business Review highlights “What Works’ in Calculating the Value of Impact Investing, they are, like almost all of global development ‘while- we-are-there’-measures. There is one mention of ‘terminal value’, 5 years after close of ownership, and they estimate social return on investments. This is a good, step, but as insufficient as foreign aid – for these are projected, not actual results.

At Valuing Voices, we have found hopeful examples such as IKEA as well as where ‘impact investing’ hype does not match the claims. Nonetheless, increasingly businesses are trying to consider circular economy systemic principles of “economic development designed to benefit businesses, society, and the environment.” This is regenerative, aims to decouple growth from the consumption of finite resources, not generate excess waste that cannot be reused and actuals seem to be measured at least during investments. As Harvard notes, “this leads investors to look at factors such as a company’s carbon footprint, water usage (both Environment), community development efforts (Social), and board diversity (Governance).” We encourage them to measure long-term/ longitudinally. A current Harvard Business Review sobering article on the ineffectiveness so far of measuring environmental sustainability and ESG. “…reporting is not a proxy for progress. Measurement is often nonstandard, incomplete, imprecise, and misleading. And headlines touting new milestones in disclosure and socially responsible investment are often just fanciful ‘greenwishing’”.

Australia’s RMIT defines business sustainability as comprising 4 pillars: Human, Social, Economic, and Environmental which combines a) “Human sustainability focuses on the importance of anyone directly or indirectly involved in the making of products, or provision of services or broader stakeholders;… b) Social sustainability focuses on maintaining and improving social quality with concepts such as cohesion, reciprocity and honesty and the importance of relationships amongst people;… c) Economic sustainability aims to improve the standard of living [and] the efficient use of assets to maintain company profitability over time;… d) Environmental sustainability places emphasis on how business can achieve positive economic outcomes without doing any harm, in the short- or long-term, to the environment.” But how well measured?!

Would ESG success be sustained over the long-term rather than short-term shareholder profit cycles? Will the OECD start to recommend extensive ex-post evaluation? Will they develop guidance to incorporate environmental concerns in evaluation for our common good? I do not yet know, but I implore these silos to start talking. No time to waste!

As my colleague and  collaborator Susan Legro commented, we need to:

1) Continue to seek clarity and specificity in the terminology that we use, ensuring that it is clear to all stakeholders and beneficiaries; and

2) Find ways to study projects and initiatives over the longer term, which is the only way to study the designation of “sustainable” for any initiatives seeking that label.

3) I’ll add spread the word, as I just found out Harvard’s Extension School course on Sustainability features this blog in its curriculum 🙂

What are your thoughts?

 

Implementing, Scaling and Planning for Aid Exit and Sustainability

Reposted from: https://medium.com/@jindracekan/implementing-scaling-and-planning-for-aid-exit-and-sustainability-b1b92e70fb36?postPublishedType=initial

Rarely do funders return to evaluate (ex-post) what lasts after aid projects end, but when they do, we can find myriad pleasures:

1) sustainability of activities we launched and nurtured together and hoped would last as is, even 15 years later, or

2) new ways local participants or partners made old activities last which we would have never imagined, or

3) wholly new activities or collaborations that emerged which we could not have foreseen but which meet the evolving needs of participating country nationals.

All three elicit two questions: a) ‘what did we (funders, designers, implementers, evaluators) do right during design, implementation, and exit?’ and b) ‘what did they do so well after we left?’ There are rich answers for this, which involves how we co-funded, co-designed, co-implemented, and co-evaluated all along the program cycle, and how we exited.

Did we make enough time and measurement to foster sustainability, as we phased-down and phased-over an array of activities, alongside those remaining (white paper forthcoming)? Did we abruptly phase-out leaving partners and participants at a loss? Sharing power over all these decisions will influence what lasts.


There is an amazing breadth of local, ongoing resources, skills & capacities, linkages, motivation (thanks to Tufts FHI360’s work for USAID’s Food For Peace) that we can explore and learn from. There are local innovations and an array of unplanned collaborations (e.g., funding for health staff (Niger), training in small enterprise from the national government (Bangladesh), or private sector markets (Ethiopia) that can be accessed when partnerships are transparent and created one or more years pre-exit to collaborate on post-exit.

Ideally, we design and implement for exit from the onset. When we jointly set the timeframe and jointly assess risks to sustainability and adaptively manage exit, rather than exit based on pre-set timeframes, all sides win, with partners and participants able to foster sustainability. As USAID/ GIIN wrote about Responsible Exits for Impact Investors (2018), “the foundations for a responsible exit are laid even before an investment is made. To increase the likelihood of continued impact after exit, investors often select investees based on whether impact is embedded in their business model or inextricably linked to financial success. They also seek to understand the likely growth trajectory of the business, which has implications for which exit paths and options will become available.” They also note that a “growth strategy’ is needed throughout and at (investment) exit is “a company’s continued access to the right resources, networks, and knowledge” for sustained impact.

The need for a thoughtful approach to sustainability is shown by Hiller, Guthrie, and Jones in “Overcoming Ex-Post Development Stagnation“ (2016). The authors cite “limited evidence of program efficacies coupled with government and agency preference for planning, approval, and implementation processes rather than sustainment of outputs, outcomes, and impacts means that ex-post performance, scaling, and sustainability is not well understood or well pursued…. [There is a] lack of willingness to commit time and resources to rigorous evaluation of post-project effectiveness“. This affects a vast number of projects. For instance, they found 63,000 projects in 2003 alone, and “relative to the number of development projects undertaken, ex-post project [evaluations] are not commonly carried out, meaning that rates of success are often unknown and the complexity of causalities and ex-post dynamics of interactions and processes are not well understood.“ This limits our learning from what has (not) worked and what to do more (or less) of, including those that could not be sustained with only local resources.

We make sustainability assumptions are participants long for them, as Valuing Voices also found. Hiller et al. state that “whilst project documentation commonly conveys an expectation that some process of spread will occur ex-post, it rarely does, despite strong ex-post case-study evidence of stakeholder requests for further development opportunities.” This cautionary feedback could mean some project activities could be so resource-intensive that they could not feasibly be sustained or spread without long-term support, and retaining results may be limited to less costly activities. Valuing Voices found that other activities could be remunerative enough (financially, in health or education outcomes, for instance), as to be locally demanded and continued to be pursued. We have found in our Valuing Voices research and the Tufts research that activities where incentives did not continue, tended to die ex-post, while those which continued to bring benefits, such as cash crops and credit, water supply, and health, were prioritized ex-post, even in the absence of external funding continuing.

Hiller et al. outline that there are multiple ways in which scaling-up environmental sustainability over time, over area and, interestingly, scaling-within projects. The Ex-post Development Stagnation authors are clear that “creating conditions to support longer-term sustainability beyond project completion represents a recurring challenge, and it is not uncommon for activities and institutions to become inactive ex-post or for stakeholders to revert to previously unsustainable practices.” They note that some watershed studies have even found that participants actively destroy project measures in some cases. Certainly, the inability of locals to sustain often expensive activities without a project or larger organization’s support is common. “If it is assumed that development needs remain outstanding, then there may be merit in ensuring that development projects do not just remain “isolated, one-time interventions, like unconnected dots on a white page” or “islands of salvation.” The authors concur, “based on project subsidiarity and participatory principles, scaling-within management should be devolved to the local level (local authorities and local communities) to allow communities and individuals to filter out irrelevant practices and encourage adaptation and evolution of activities which are of greatest perceived livelihood benefit”.

As Valuing Voices research on exit has shown, it is a process, not an event(forthcoming, with thanks to I. Davies). The Hiller article notes that sustainability is best enhanced by capacity-building during implementation and with time for handover where “organizations adopt modes of functioning that allow local communities and organizations to build conceptual, operational, and institutional capacities. While scaling-down does not mean that governments disengage from processes such as community-driven development — it does, however, require it to be more flexible and responsive to locally generated demand to ensure the terrain is fertile for community organizations to emerge, learn, and grow.“

Let’s work together to extend the sustainability of impacts. Would love to hear your thoughts…

Assuming Sustainability and Impact is Dangerous to Development (+ OECD/DAC evaluation criteria)

 

Assuming Sustainability and Impact is Dangerous to Development
(+ OECD/ DAC evaluation criteria)

 

We all do it; well, I used to do it too. I used to assume that if I helped my field staff and partners target and design funded projects well enough, and try to ensure a high quality of implementation and M&E, then it would result in sustainable programming. I assumed we would have moved our participants and partners toward projected long-term, top-of-logical-framework’s aspirational impact such as “vibrant agriculture leading to no hunger”, “locally sustained maternal child health and nutrition”, “self-sustained ecosystems”.

INTRAC nicely differentiates between what is typically measured (“outputs can only ever be the deliverables of a project or programme…that are largely within the control of an agency”) and what is not: “impact as the lasting or significant changes in people’s lives brought about by an intervention or interventions” [1]. They continue: “as few organisations are really judged on their impact, the OECD DAC impact definition (“positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended“) allows for long-term changes in institutional capacity or policy change to be classed as impact” [1]. Do we do this? Virtually never. 99% of the time we only evaluate what happened while the project and its results is under the control of the aid implementer. Yet the five OECD/DAC evaluation criteria asks us to evaluate relevance, effectiveness, efficiency (fair enough, this is important to know if a project was good) and also impact and sustainability. So in addition to the prescription to evaluate ‘long-term effects’ (impact), evaluators are to measure “whether the benefits of an activity are likely to continue after donor funding has been withdrawn… [including being] environmentally as well as financially sustainable” [2]. 

How do we know we are getting to sustained outcomes and impacts? We ask people on the receiving end ideally after projects end. It is dangerous to assume sustainability and impact, and assume positive development trajectories (Sridharan) unless we consistently do “ex-post” project evaluations such as these from our research or catalytic organizations that have done at least one ex-post. At very minimum we should evaluate projected sustainability at end of project with those tasked to sustain it before the same project is repeated. Unfortunately we rarely do so and the assumed sustainability is so often not borne out, as I presented at the European Evaluation Society conference Sustainability panel two weeks ago along with AusAid’s DFAT, the World Bank, University College London and UNFEM.

 

 

Will we ever know if we have gotten to sustained impacts? Not unless the OECD/DAC criteria are drastically updated and organizations evaluate most projects ex-post (not just good ones :)), learn from the results and fund and implement for country-led sustainability with the country nationals. We must, as Sanjeev Sridharan tells us in a forthcoming paper embed sustainability into our Theories of Change from the onset (“Till time (and poor planning) do us part: Programs as dynamic systems — Incorporating planning of sustainability into theories of change” (Canadian Journal of Program Evaluation, 2018).*

There are remarkable assumptions routinely made. Many projects put sustainability into the proposal, yet most close out projects in the last 6 months. Rarely do projects take the time to properly phase down or phase over (unlike CRS Niger); many exit ceremonially ‘handing over’ projects to country-nationals, disposing project assets, and leaving only a final report behind. Alternatively, this USAID Uganda CDCS Country Transition Plan which looks over 20 years in the future by when it assumes to have accomplished sustained impact for exit [3]. Maybe they will measure progress towards that goal and orient programs toward handover, as in the new USAID “Journey to Self-Reliance” – we hope! Truly, we can plan to exit, but only when data bears out our sustained impact, not when the money or political will runs out.

As OXFAM’s blog today on the evaluation criteria says, “Sustainability is often treated as an assessment of whether an output is likely to be sustained after the end of the project. No one, well, hardly anyone, ever measures sustainability in terms of understanding whether we are meeting the needs of the present, without compromising the ability of future generations to meet their own need” and “too often in development we evaluate a project or programme and claim impact in a very narrow sense rather than the broader ecology beyond project or programme parameters” [4]. In fact, most ‘impact evaluations’ actually test effectiveness rather than long-term impact. Too rarely do we test impact assumptions by returning 2-10 years later and gather proof of what impacted locals’ lives sustainably, much less – importantly – what emerged from their own efforts once we left (SEIEs)! Oh, our hubris.

if you’re interested in the European Evaluation Society’s DAC criteria update discussion, see flagship discussion and Zenda Offir’s blog which stresses the need for better design that include ownership, inclusivity, empowerment [5][6]. These new evaluation criteria need to be updated, including Florence Etta’s and AGDEN‘s additional criteria participation, non-discrimination and accountability!

 

 

We can no longer afford to spend resources without listening to our true clients – those tasked with sustaining the impacts after we pack up – our partners and participants. We can no longer fund what cannot be proven to be sustained that is impactful. We talk about effectiveness and country ownership (which is paramount for sustainability and long-term impact), with an OECD report (2018) found “increases [in[ aid effectiveness by reducing transaction costs and improving recipient countries ownership” [7]. Yet donor governments who ‘tie’ aid to their own country national’s contracts benefit a staggering amount from ‘aid’ given. “Australia and the United Kingdom both reported … 93 percent and 90 percent of the value of their contracts respectively went to their own firms” [7]. It is not so different in the USA where aid is becoming bureaucratically centralized in the hands of a few for-profit contractors and centralized hundreds of millions in a handful of contracts. We must Do Development Differently. We can’t be the prime beneficiaries of our own aid; accountability must be to our participants; is it their countries, not our projects, and we cannot keep dangerously assuming sustained impact. Please let us know what you think…

 

 

Footnotes:

[*] This paper is now available at https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/53055

 

Sources:

[1] Simister, N. (2015). Monitoring and Evaluation Series: Outcomes Outputs and Impact. Retrieved from https://www.intrac.org/wpcms/wp-content/uploads/2016/06/Monitoring-and-Evaluation-Series-Outcomes-Outputs-and-Impact-7.pdf

[2] OECD. (n.d.). DAC Criteria for Evaluating Development Assistance. Retrieved September, 2018, from https://web.archive.org/web/20180919035910/http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm

[3] USAID. (2016, December 6). USAID Uganda Country Development Cooperation Strategy 2016-2021. Retrieved October, 2018, from https://www.usaid.gov/uganda/cdcs

[4] Porter, S. (2018, October 18). DAC Criteria: The Hand That Rocks the Cradle. Retrieved from https://views-voices.oxfam.org.uk/2018/10/dac-criteria-the-hand-that-rocks-the-cradle/

[5] European Evaluation Society Biennial Conference: Flagship Symposia. (2018). Retrieved from http://www.ees2018.eu/1539782596-flagship-symposia.htm

[6] Ofir, Z. (2018, October 13). Updating the DAC Criteria, Part 11 (FINAL). From Evaluation Criteria to Design Principles. Retrieved from https://zendaofir.com/dac-criteria-part-11/

[7] OECD. (2018, June 11). 2018 Report On The DAC Untying Recommendation. Retrieved from http://www.oecd.org/dac/financing-sustainable-development/development-finance-standards/DCD-DAC(2018)12-REV2.en.pdf