by Jindra Cekan | Mar 22, 2021 | ex-post evaluation, USAID, water/ sanitation
Reblog Unpacking the Drivers of WASH Sustainability
In Nov 2019 I highlighted several water/ sanitation/ hygiene ex-posts happening at USAID’s E3 Water Bureau. They’ve just come up with a synthesis, which is important, both in learning and transparency for accountability. Kudos! My only quibble is that we don’t know the full scale after only 6 very different ex-posts.
The authors Jones & Jordan’s overview: In its evaluation series, USAID looked back at the results of six water, sanitation, and hygiene (WASH) activities to inform future USAID investments in the sector and to better understand the long-term impact and sustainability of its interventions several years after projects close. Here: https://www.globalwaters.org/resources/blogs/unpacking-drivers-wash-sustainability
The water, sanitation, and hygiene (WASH) Ex-Post Evaluation Series represents a key milestone in the Agency’s ongoing quest to unpack the drivers of sustainability within our WASH programs. The commitment, made in the first USAID Water & Development Strategy (2013–2018), came in the wake of headlines about high rates of nonfunctional water systems in partner countries where USAID has long invested. At that time, many governments and development partners were grasping for similar answers, and a number of sustainability assessment tools emerged. For its part, the Agency pursued a multi-pronged approach that focused on: 1) measuring progress towards sustainable outcomes (such as through the development and application of the Sustainability Index Tool with Rotary International); 2) gaining a better understanding of the drivers of long-term outcomes through this WASH Ex-Post Evaluation Series; and 3) testing new ideas, approaches, and tools to strengthen the local systems that can deliver WASH service sustainability through the Sustainable WASH Systems (SWS) Learning Partnership.
The Agency, together with its development partners, has achieved staggering results in terms of delivering first-time water and sanitation access to people the world over. Since 2013, USAID has helped roughly 25 million people gain access to at least basic drinking water services and 18 million people gain access to at least basic sanitation. Together with partners, USAID has mobilized almost $100 million in new funding for the sector and supported nearly 17,000 communities become open defecation free.
Yet the results of this ex-post series are sobering. Despite tremendous achievements within the life of our programs, they have largely not endured. This is especially the case in countries and communities with the highest levels of poverty at baseline, where the Agency’s resources are needed the most. Rural water systems that, at activity close, delivered safe water to households have fallen into disrepair. Basic latrine ownership and use have dwindled. Communities certified as open-defecation free are backsliding, and gains in handwashing have not been sustained.
The series did reveal some programming bright spots. Where USAID invested in providing technical assistance to committed government partners and utilities, gains in service provision and local capacity were sustained, with local actors taking up and expanding upon best practices introduced during activity implementation. Often these successes endured in countries and communities that had higher levels of capacity at the outset. However, the successes in these contexts demonstrate important lessons about investing time and resources into partnering with local institutions and focusing on plans for management of services, not just first time access.
In the course of the roughly 15 intervening years since most of the activities evaluated in this series were designed, the sector has evolved. For instance, in resource-constrained environments, the sector is now coalescing around facilitating the development of professionalized support to community-managed rural water schemes in various forms, rather than expecting voluntary committees to manage essential services alone. Additional approaches beyond community-led total sanitation, including smart subsidies and market-based sanitation, are seen as necessary to move households up the sanitation ladder, and are being applied through USAID programming. And the Agency is shifting its WASH social and behavior change programming to more holistic approaches that address emotional drivers, convenience, and social norms to modify intractable behaviors rather than communication or health promotion alone. The WASH Ex-Post Evaluation Series validates why those shifts were essential.
All those with a stake in promoting lasting development gains in the sector need to internalize these findings and take a long look in the mirror. USAID has and is seeking to do better. First, under the USAID Water and Development Plan (2018–2022) in support of the Global Water Strategy, USAID codified its commitment to sustainability with the goal of increasing the availability and sustainable management of safe water and sanitation, and an emphasis on improving the underlying governance, finance, and management of water resources that underpin sustainability. Second, the Agency has issued a set of technical briefs that provide new guidance on important topics for developing and implementing WASH activities, as well as recommendations for activity design, implementation, and monitoring. Third, USAID has launched the Water Security, Sanitation and Hygiene Implementation Research Agenda that identifies and prioritizes sector-specific research questions to close lingering evidence gaps directly related to accomplishing USAID’s goal of increasing access to sustainable water and sanitation services. Finally, the Agency is rethinking its approach to sector targets and key performance indicators in its solicitations, recognizing that targets can cause perverse incentives to undermine sustainability from the outset. Doing so underscores the Agency’s commitment to sustainability and its willingness to be held accountable to deliver against those results.
For more information, take a look at the full evaluation or access other ex-post evaluations on Globalwaters.org.
By Abigail Jones and Elizabeth Jordan, Water and Sanitation Specialists, USAID Center for Water Security, Sanitation, and Hygiene
by Rachel Sadoff | Jun 28, 2020 | EU, Exit strategies, USAID
Sustainable Exit Strategies: USAID vs. EU
Once malnutrition has decreased, students’ attendance has risen, or the number of small businesses has doubled, program implementers may be quick to pack their bags and leave. But the impact of their work may be undermined if not undone if they leave before implementing context-specific, comprehensive plans for post-project sustainability. It is for this reason that exit strategies – and particularly institution-wide mandates by donors – are crucial components of effective development programming. The United States Agency for International Development (USAID and the European Union (EU) Commission both have close-out protocols in place, but much to learn from one another in ensuring long-term progress in the wake of their close-outs.
Recommendations for Project Close-Out
Although every program should develop and follow context-specific exit measures, non-profits and ex-post evaluators have found that some exit strategies consistently lead to long-term, sustainable impact. The International NGO Training and Research Centre (INTRAC), for example, states that donors and implementers should actively plan for close-out throughout the project – from program design to the post-exit stage [1]. It recommends the following:
- Implementers should “build exit thinking into the design of the project,” and during the program, representatives should be monitoring pre-determined exit indicators [1]
- “Senior staff and management need to prioritize resources for exits in order to do them well, embed learning within organizations and across teams, and ensure mistakes are not repeated” [1]
- “Practitioners should also consider going the extra mile to look after staff at headquarters and within partner organizations” [1]
- “Document experiences and share them externally” [1]
- “Consider whether a timeline approach would work for you” [1]
The USAID Approach to Project Close-Out
According to USAID’s 2020 Program Cycle Operational Policy, the formal close-out process for specific projects are carried out by the USAID Mission team in the recipient country and Operating Units based in Washington, DC [2]. However, the Program Cycle does not reference the close-out process, let alone offer guidance [3]. Unlike country exits, which require thorough evaluation and international cooperation [4],ᴬ the sustainability measures of close-out procedures for activities – components of country-wide programs – seem to be delegated to implementing agencies.
Tellingly, the USAID formally defines “sustainability” as “the ability of a local system to produce desired outcomes” after its projects end, which the Agency may “contribute to” by strengthening capacity in their respective realms [5]. After close-out, USAID currently considers the sustainability of its projects to be a matter of “the country and/or targeted community’s commitment and capacity to achieve development solutions” [5]. Commitment and capacity, in turn, are gauged by 17 country-level “self-reliance metrics” ranging from “liberal democracy” to “export sophistication” [6,7].ᴮ Even post-project monitoring and evaluation are explicitly meant to “reveal whether implementation is on track and results are being achieved” with no mention of sustainability or long-term follow-up [3].
In effect, this framework not only puts the onus of project sustainability on recipients, but evaluates it based on large-scale and potentially tangential metrics. The stakes of this practice are raised by the fact that the resultant “Country Roadmaps” inform when USAID ends its programming in a given country, endangering a range of populations that these metrics do not account for, like seniors and prisoners [6].
USAID’s official Program Cycle [3]
During close-out, USAID representatives seem to focus on administrative tasks. When a Mission is preparing for activity close-out, for example, Mission Directors are responsible for developing plans that include the following:
- “The retention of sufficient and suitable staff members and the delegation of authority and assignment of specific responsibility to each to carry to completion the required close-out action” [8]
- “Estimates of the personnel, space, and funds required to complete the close-out” [8]
- “The timetable for phasing out, transferring, or terminating U.S. direct-hire, participating agency, and contract personnel and the replacement of assistance from other resources” [8]
- “Actions for terminating the services of cooperating country and third country national staff” [8]
- In its 2003 version, but since removed:
- “Missions must think in terms of leaving sustainable and useful units of assistance” [9]
- “A monthly timeframe/activity projection for the entire close-out period [approximately six months] of major activities that must occur” [9]
Similarly, in the case of Interagency Agreements, USAID Agreement Officers’ (AO) personal and delegated close-out responsibilities are limited to the collection of financial reports and compliance-related documents [10]. AOs are expected to work in this same, limited capacity as they terminate Assistance and Acquisition Awards [11].
By contrast, when USAID gives grants to NGOs and has “little direct involvement,” recipients are strongly encouraged to adhere to six substantial, well-rounded “tips,” which include [12,13]:
*Staff refers to each NGO’s own staff; based on [13]
Among all of the terms and conditions reviewed, USAID’s exit strategies for foreign Missions, Agreements, and Awards have been almost exclusively administrative in nature. Although the Agency should not be solely responsible for ensuring the sustainability of its programming, its lack of regulation in this realm risks both undue pressure on implementing partners to do so and short-lived progress in the wake of its exit [14].
Fortunately, USAID has demonstrated its interest in sustainable close-out by commissioning studies and reports. In March 2020, USAID’s three-year “collaborative learning project” called Stopping as Success published a checklist for sustainable project transition [15,16]. The publication offers 16 concrete suggestions ranging from “build[ing] trust” with local communities to requiring implementers to report their progress in carrying out a pre-determined sustainability plan [16]. Similarly, the Food and Nutrition Technical Assistance III Project – a partnership between USAID and FHI 360 – studied the consequences of four food assistance programs and made thorough recommendations for ensuring the long-term impact of future projects [14]. These recommendations include accounting for “predictable external shocks and stressors” like natural disasters, and building “vertical linkages […] between community-based organizations or individuals and existing public or private sector institutions” [14]. Armed with this impressive base, USAID is well-equipped to account for new, evidence-based close-out practices.
UPDATE: A NEW DISCUSSION NOTE BY USAID ON EX-POST EVALUATION WHERE EXIT IS MENTIONED.
The EU Approach to Project Close-Out
Given the sheer scale of EU development programming – 28 countries providing 47% of the world’s development aid – no specific, all-encompassing mandates for the close-out process was found [17].ᶜ Perhaps as a result, there is notoriously little development policy coordination among Member States:
- Due to inconsistencies among the eight documents outlining the EU’s development objectives and indicators, an internal audit characterized its collective approach as “mixed – at best” [18]
- High-level meetings have explicitly prioritized “consensus on development” and “working better together,” producing appropriately vague guidelines that do not specifically address closure [19,20]
- e.g. EU staff should “help to promote more inclusive, sustainable growth that does not compromise the ability of future generations to meet their needs” [19]
- Its main development finance body, the Development Cooperation Instrument, has been critiqued for its “ambitio[us] desig[n] to address a very broad range of issues, from policy priorities in the agenda for change to EU policy concerns” [21]
For insight into the EU’s exit strategies, one must look to guidelines for specific types of development programming.
[33]
In the case of joint programming, when Member States work together to design and implement a country-wide, multi-sectorial project, donors must “reach agreement on a responsible exit strategy” for “phasing out sectors outside focal/centration areas, without creating a financing gap and with minimum disruption to the partner country” [20]. The policy offers two examples of such a strategy: “including scenarios in which other participating partners ‘plug the gap’ and/or other forms of cooperation beyond ODA [official development assistance] (e.g. the private sector)” [20]. The strategy must be drafted with input from the recipient country “as much as possible,” and may include either the intervention of other stakeholders or the private sector [20]. These discussions make take place at any of the following four “entry points” [20]:
- “At the start of a new national development plan cycle, or a major change in the country” [20]
- “At the start of a new strategic programming period or a review process for several important development partners” [20]
- “During high-level country meetings […] where partner governments present their plans and solicit development partners support” [20]
- “In situations of fragility or transition from humanitarian aid to more structured programmatic aid and more development partners” [20]
The development of context-specific exit strategies by Member States and recipients allows for practicable, fairly delegated, and concrete steps to exiting. However, this provision omits mention of enforcement or evaluation, may take place at the end of the implementation period, and does not encompass best practices as defined by INTRAC [1].
Outside of joint programming, the EU oversees projects funded by its executive implementing agencies, such as the Consumers, Health, Agriculture and Food Executive Agency and the Executive Agency for Small and Medium-sized Enterprises (EASME). In those cases, agency-wide final report guidelines could shed light on collective guidance but these are publicly inaccessible [22].ᴰ However, an early iteration of the Horizon 2020 Programme’s final report guide is still available online. The EU had required that implementers submit 20-30 lines of text addressing the following:
“Inform about activities undertaken during the project to ensure that your action
continues and reaches out further. Inform also about plans and intentions to continue
and sustain the project activities after the end of the project. Indicate the state
of advancement (e.g. agreed action, concrete proposal, idea)” [23].
Although field staff may not have been required to tangibly ensure the sustainability of their projects – and may not even make these assessments anymore – it is clear that at one time, the EU mandated assessments of expected sustainability at the end of program-specific project implementation.
With no overarching guidance for EU project exits, countries and individual NGOs have resorted to choosing or developing their own. The think tank E3G – funded in part by EASME [24] – has issued both internal and guidance for sustainable programming [25,26]. Its key points include “consultative engagement[s with] a broad range of stakeholders,” “integrat[ing programs] with long term national development plans,” and “strengthening city and local-level planning and institutions” [25]. These recommendations, however, are scattered among legal studies, commissioned reports, and briefing papers. By contrast, another EASME-funded program called Carbon Market Watch adheres to the project cycle guidelines of UN committees [27,28,29].ᴱ Although their Project Design Document and methodology forms request specific information about monitoring and evaluation, they do not ask for implementers’ plans for program close-out or long-term impact [30].
[32]
Other EU member-country NGOs end up developing sustainable exit measures mid-project. One example is a 2011-2013 agricultural project in Ethiopia implemented by People in Need, an international NGO based in the Czech Republic and funded in part by the EU’s Commission for Civil Protection and Humanitarian Aid [31]. Two months before its end, an evaluation of the program’s sustainability was conducted, receiving such recommendations as: 1) “increasing [recipients’] financial sustainability;” 2) “ensuring that FTCs’ [Farmer Training Centers] services benefit all, not only model farmers;” and 3) “providing books in Amharic language, not English” [32]. In the remaining time, 75% of these recommendations were met, leading to the internal assessment that the “sustainability of the project’s outcomes is likely to be largely positive” [32].
Learning from One Another
USAID and the EU operate at starkly different scales – one a country-specific bilateral donor, and the other an economic union of dozens of donor countries – but each of their exit strategies could be strengthened by incorporating elements from the other’s.
The EU could learn from the USAID’s multifaceted guidance for NGOs; the considerations listed in its final report guide are limited to the post-project involvement of EU field staff. It should explicitly promote local participation for sustainability during project close-out and beyond, such as transferring operations to local organizations or businesses.
In turn, USAID’s administration-centric strategy could benefit from the large-scale considerations of EU policies. Discussions with recipient governments, particularly in turbulent circumstances, could strengthen diplomatic relations with the US in addition to supporting reasonable transition plans. Formally soliciting exit plans from project implementers could also produce more sustainable project results.
NOTE: Valuing Voices thanks our excellent Harvard intern Rachel Sadoff for this research! We have a question: we found no EU/EC sites discussing exit strategies recommendations or guidance for global development projects. Do you have some? Please share in the comments below. Thank you.
Footnotes:
A) Projects and activities need to explain and demonstrate how their programming advances the objectives of USAID’s Country Development Cooperation Strategies for the given state (see ADS Chapter 201).
B) Current totals are publicly available on the Agency’s Country Roadmap website.
C) VV’s search spanned over five hours.
D) As of writing, final report guidelines are only available through an exclusive portal.
E) These are the Clean Development Mechanism and Joint Implementation Committee of the United Nations Framework Convention on Climate Change (UNFCCC).
Sources:
[1] Lewis, S. (2016, January). Developing a Timeline for Exit Strategies. Retrieved from https://www.intrac.org/resources/praxis-paper-31-developing-timeline-exit-strategies/
[2] USAID. (2020, June 16). ADS Chapter 201: Program Cycle Operational Policy. Retrieved from https://www.usaid.gov/sites/default/files/documents/1870/201.pdf
[3] USAID. (2016, December 8). ADS 201: Program Cycle Operational Policy – Fact Sheet. Retrieved from https://usaidlearninglab.org/library/ads-201-program-cycle-operational-policy-fact-sheet
[4] USAID. (2020, February 26). Strategic Transitions Fact Sheet. Retrieved from https://www.usaid.gov/documents/1870/strategic-transitions-fact-sheet
[5] USAID. (2020, May). Discussion Note: Ex-Post Evaluations. Retrieved from https://usaidlearninglab.org/library/discussion-note-ex-post-evaluations
[6] USAID. (2020, June 3). USAID’S Country Roadmaps Fact Sheet. Retrieved from https://www.usaid.gov/documents/1870/usaids-country-roadmaps-fact-sheet
[7] USAID. (n.d.). The Journey To Self-Reliance: Country Roadmaps. Retrieved June 2020 from https://selfreliance.usaid.gov/
[8] USAID. (2018, July 17). Mandatory Reference for ADS Chapter 527: Administrative Guidance on How to Close a USAID Operating Unit – Checklists. Retrieved from https://www.usaid.gov/sites/default/files/documents/1868/527mab.pdf
[9] USAID. (2003, October 1). ADS 527 Mandatory Reference: Guidance on How to Close a USAID Mission – Checklist. Retrieved from https://2012-2017.usaid.gov/sites/default/files/documents/1868/527mab.pdf
[10] USAID. (2015, November 30). ADS Chapter 306: Interagency Agreements. Retrieved from https://2012-2017.usaid.gov/sites/default/files/documents/1868/306.pdf
[11] USAID. (2010, October 6). Guidance on Closeout Procedures for A&A Awards. Retrieved from https://2012-2017.usaid.gov/sites/default/files/documents/1868/302sat.pdf
[12] USAID. (2020, February 13). Doing Business with USAID. Retrieved from https://www.usaid.gov/work-usaid/how-to-work-with-usaid
[13] USAID, & FHI 360. (2012). The Essential NGO Guide to Managing Your USAID Award. Retrieved from https://www.fhi360.org/resource/essential-ngo-guide-managing-your-usaid-award
[14] Food and Nutrition Technical Assistance. (2015, December). Effective Sustainability and Exit Strategies for USAID FFP Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
[15] Stopping as Success. (n.d.). About Us. Retrieved June 2020 from https://www.stoppingassuccess.org/about-us
[16] Stopping as Success. (2020, March 17). USAID Mission Checklist for Sustainable Transitions. Retrieved from https://www.stoppingassuccess.org/resources/usaid-mission-checklist/
[17] European Commission. (2015, December). EU Site Explorer: Overview. Retrieved June 2020 from https://euaidexplorer.ec.europa.eu/content/overview_en
[18] Froidure, P., & Devillé, H. (2015). Review of the risks related to a results oriented approach for EU development and cooperation action. Retrieved from https://op.europa.eu/en/publication-detail/-/publication/b2921ec1-c589-11e5-a4b5-01aa75ed71a1
[19] European Commission Directorate-General for International Cooperation and Development. (2018, November 19). The New European Consensus on Development: ‘Our World, Our Dignity, Our Future’. Retrieved from https://op.europa.eu/en/publication-detail/-/publication/5a95e892-ec76-11e8-b690-01aa75ed71a1
[20] European Commission. (2018, June). Joint Programming Guidance: Supporting EU Delegations to Work Better Together with Member States, Like-Minded Partners and Country Stakeholders. Retrieved from https://knowledge.effectivecooperation.org/node/77
[21] Parry, M., & Segantini, E. (2017, October). How the EU Budget is Spent: Development Cooperation Instrument. Retrieved from https://www.europarl.europa.eu/RegData/etudes/BRIE/2017/608764/EPRS_BRI(2017)608764_EN.pdf
[22] Linköping University. (2020, May 25). Reporting to the European Commission. Retrieved June 2020 from https://insidan.liu.se/forskningsfinansiering/horizon-2020/rapportera-till-eu-kommissionen?l=en
[23] European Commission: Intelligent Energy Europe. (2012). Guidelines for Drafting the Final Technical Implementation Report. Retrieved June 2020 from https://ec.europa.eu/easme/en/technical-reporting
[24] LIFE Program. (2019, June 3). 2019 Operating Grants for NGOs. European Commission. Retrieved from https://ec.europa.eu/easme/en/section/life/2019-life-call-proposals-ngos
[25] Wright, H., Dimsdale, T., & Healy, C., et al. (2018, June 26). Sustainable Infrastructure and the Multilateral Development Banks: Changing the Narrative. E3G. https://www.e3g.org/library/sustainable-infrastructure-and-the-multilateral-development-banks-changing
[26] Naidoo, C., Amin, A., & Dimsdale, T., et al. (2014, April 9). Strategic national approaches to climate finance: Report on scoping work in Peru, Chile and Colombia on national climate finance pathways and strategies. Retrieved from https://www.e3g.org/news/media-room/strategic-national-approaches-to-climate-finance
[27] LIFE Program. (2019, June 3). 2019 Operating Grants for NGOs. European Commission. https://ec.europa.eu/easme/en/section/life/2019-life-call-proposals-ngos
[28] Carbon Market Watch. (2014, June 25). The Joint Implementation offsetting mechanism is deeply flawed – time to consign it to history? Retrieved from https://carbonmarketwatch.org/2014/06/25/the-joint-implementation-offsetting-mechanism-deeply-flawed-time-to-consign-it-to-history/
[29] Carbon Market Watch. (n.d.) CDM Project Cycle. Retrieved 2019, May 10 from
https://web.archive.org/web/20190510064748/http:/archive.carbonmarketwatch.org/learn-about-carbon-markets/cdm-project-cycle/
[30] United Nations Framework Convention on Climate Change. (n.d.). Guidelines for Completing the Project Design Document (CDM-PDD), the Proposed New Methodology: Baseline (CDM-NMB) and the Proposed New Methodology: Monitoring (CDM-NMM). Retrieved June 2020 from https://cdm.unfccc.int/Reference/Documents/Guidel_Pdd/English/Guidelines_CDMPDD_NMB_NMM.pdf
[31] Directorate-General for European Civil Protection and Humanitarian Aid Operations. (n.d.). Agreements for Humanitarian Aid and Emergency Support Within the EU Awarded in 2017 by DG ECHO. Retrieved June 2020 from https://ec.europa.eu/echo/files/funding/agreements/agreements_2017.pdf
[32] People in Need Ethiopia. (n.d.). Final Evaluation: PIN’s Support to Strengthening Agricultural Extension Services in SNNPR, Ethiopia. Retrieved from https://www.clovekvtisni.cz/en/learning-series-on-agricultural-extension-738pub
by Jindra Cekan | Apr 20, 2020 | Aid effectiveness, climate change, environment, ex-post evaluation, Impact, international development, JICA, OECD, Participants, post project evaluation, Sustainability, Sustainable development, USAID, Vietnam, water/ sanitation
Sustaining “Sustainable Development”?
As a global development industry, we have almost no evidence of how (un)sustained the outcomes or impacts of 99% of our projects because we have never returned to evaluate them. But from early indications based on the ex-posts, we have evaluated 2-20 years after donor departure it is, learning from what was and was not sustained is vital before replication and assuming sustainability. Most results taper off quite quickly, showing 20-80% decreases as early as two years post-closure and donor exit. A few cases of good news also appear, but more trajectories falter and fail than rise or remain. Sustainability, then, is not a yes-no answer, but a how much, yet too few ask… hence if they were, resilient, they are less so, or even not at all, now.
At Valuing Voices we focus on the sustainability of projects after external support ends. Still, those projects are also dependent on the viability of the environment in which they are based. As Andy Rowe, an evaluator on the GEF’s Adaptation Fund board, noted at IDEAS’ Conference in Prague late 2019 [1], a need for sustainability-ready evaluation to help us know how viable the resources are on which so many of our projects rest [2]. He states, “the evaluation we have today treats human and natural systems as unconnected and rarely considers the natural system”. He goes on to differentiate between biotic natural capital (air, water, plants, and trees) and abiotic natural capital sources (fossil fuels, minerals, and metals, wind, and solar).
How much are projects designed assuming those resources are and will remain plentiful? How often do we evaluate how much our projects drain or rely on these environmental elements? Many projects are required to do environmental compliance and safeguarding against damage at project onset [3]. Others, such as agriculture and natural resource management or water/ sanitation, often focus on improving the environment on which those activities rely, e.g., improving soil or terrain (e.g., terraces, zais), planting seedlings, and improving access to potable water for humans and animals. Still, many projects ‘assume’ inputs like rainfall, tree cover, solar power, or do not consider the sustainability of natural resources for the communities in which they intervene. Examples are both those that rely on natural systems as well as those supposedly beyond them, e.g., enterprise development, education, safety nets, etc. Yet many enterprises, schools, safety nets do rely on a. viable environment in which their participants trade, learn, and live, and all are subject to the growing climate change disruptions.
Why is this urgent? The OECD/DAC reminds us that “Natural assets represent, on average 26% of the wealth of developing countries compared to 2% in OECD economies” [4]. Unless we protect them and address the demand for natural resources, demand will far outstrip supply. “By 2030, an additional 1 billion people are expected to live in severely water-stressed areas, and global terrestrial biodiversity is expected to decline an additional 10%, leading to a loss of essential ecosystem services. By 2050, growing levels of dangerous air emissions from transport and industry will increase the global number of premature deaths linked to airborne particulate matter to 3.6 million people a year, more than doubling today’s levels. Failure to act could also lead to a 50% increase in global greenhouse gas emissions by 2050, and global mean temperature increases of 3-6°C by the end of the century, in turn contributing to more severe and sometimes more frequent natural disasters… [so] reconciling development with environmental protection and sustainable resource management is broadly agreed as a central concern for the post-2015 development agenda.”
When we return to projects that are a mix of behavior change and environment, we find a wide range of results:
- Some projects, such as JICA Vietnam’s water supply and irrigation infrastructure reached 80% of the final results two years later [5]. And while the pilot projects were worse off (as low as 28% of irrigated hectares), longer-standing projects sustained as much as 72% of final results. While such agricultural development assumes continued water supply and access, does it evaluate it? No.
- Some can define what ex-post lessons are more narrowly as functioning mechanisms: New ex-posts of water/ sanitation showed better – but still mixed results, such as USAID Senegal’s [6]. “While a majority (63 percent) of the water points remained functional, the performance varied significantly based on the technology used. Of the different technologies, the Erobon rope pumps performed poorly (27 percent functional), while the India Mark (74 percent functional) and mechanized pumps (70 percent functional) performed the best.”
- Some projects that include environmental considerations illustrate our point by only focusing on behavior change as this sanitation/ hygiene ex-post from Madagascar did, where results fell off precipitously three years ex-post but without considering water supply or quality much [7].
[7]
- There can be useful learning when one combines an evaluation of both types of sustainability (ex-post and environmental). A JICA irrigation project in Cambodia shows that when irrigation canals were mostly sustained over the five-years ex-post, they could serve increasing needs for land coverage and rice production [7]. The area of irrigated fields at the national level in 2010 reached the target, and the irrigated field area has since continued to increase in most areas. Even the largest drop [in area irrigated] post-closure was only 11%. They reported that the unit yield of rice at the end-line survey in 2012 at 11 sites was 3.24t/ha (average) versus 3.11t/ha of unit yield of rice at the ex-post evaluation in 2017, which [almost] maintains the 2012 level. The ex-post showed that “continuous irrigation development in the said site can be considered as the main reason for the increase in land area. Securing an adequate amount of water is an important factor in continuously improving rice productivity.” The research also found that 81% of agricultural incomes as a result of the irrigation had increased, 11% stayed the same, and 8% had decreased. Again, this looks to be among the most resilient projects that, based on ex-post research, included environment which was also found to be as resilient as the livelihoods it was fostering.
- Sometimes more bad than good news is important when tracking environment and ex-post sustainability: Food for the Hungry, ADRA, and CARE Kenya found that unreliable water supply reduced the motivation to pay for water, threatening the resources to maintain the system [8]. What improved prospects of sustainability understand why communities could not sustain water and sanitation results based on willingness-to-pay models, as well as water being unavailable. Further, a lesson the organizations ideally learned was that “gradual exit, with the opportunity for project participants to operate independently prior to project closure, made it more likely that activities would be continued without project support.” So the question remains, what was learned by these organizations to avoid similar bad results and improve good, resilient results in similar circumstances?
[6]
Neither sustainability nor environmental quality can be assumed to continue nor to have positive results. Both are extensively under-evaluated, and given climate change disruptions, and this must change. Rowe concludes: “Climate change is a major threat to the long-term sustainability both attacking the natural systems (e.g. lower rainfall or higher floods, worse soil quality, increasing pests attacking crops, disappearing fish stocks, microplastics in our air and water, increasing sea levels from melting glaciers, worsening public health etc.) and destabilizining our Earth’s regenerative capacity. Fortunately, technical barriers do not prevent us from starting to infuse sustainability into evaluation; the barriers are social and associated with the worldview and vision of evaluation.”
Sources:
[1] IDEAS 2019 Global Assembly. (n.d.). Retrieved from https://2019.global-assembly.org/
[2] Rowe, A. (2019). Sustainability‐Ready Evaluation: A Call to Action. New Directions for Evaluation, 162, 29-48. Retrieved from https://www.researchgate.net/publication/333616139_Sustainability-Ready_Evaluation_A_Call_to_Action
[3] USAID. (2013, October 31). Environmental Compliance Procedures. Retrieved from https://www.usaid.gov/our_work/environment/compliance/pdf/216
[4] OECD. (2015). Element 4, Paper 1: Global and local environmental sustainability, development and growth. Retrieved from https://www.oecd.org/dac/environment-development/FINAL%20POST-2015%20global%20and%20local%20environmental%20sustainability.pdf
[5] Haraguchi, T. (2017). Socialist Republic of Viet Nam: FY 2017 Ex-Post Evaluation of Japanese ODA Loan Project “Small-Scale Pro Poor Infrastructure Development Project (III)”. Retrieved from https://www2.jica.go.jp/en/evaluation/pdf/2017_VNXVII-5_4.pdf
[6] Coates, J., Kegode, E., Galante, T., & Blau, A. (2016, February). Sustaining Development: Results from a Study of Sustainability and Exit Strategies among Development Food Assistance Projects: Kenya Country Study. USAID. Retrieved from https://www.globalwaters.org/resources/assets/ex-post-evaluation-senegal-pepam
[7] Madagascar Rural Access To New Opportunities For Health And Prosperity (RANO-HP) Ex-Post Evaluation. (2017, June 1). USAID. Retrieved from https://www.globalwaters.org/resources/assets/madagascar-rural-access-new-opportunities-health-and-prosperity-rano-hp-ex-post-0
[8] Kobayashi, N. (2017). Kingdom of Cambodia: FY2017 Ex-Post Evaluation of Technical Cooperation Project: “Technical Service Center for Irrigation System Project – Phase 2 / The Improvement of Agricultural River Basin Management and Development Project (TSC3)”. Retrieved from https://www2.jica.go.jp/en/evaluation/pdf/2017_0900388_4.pdf
by Jindra Cekan | Nov 10, 2019 | Aid effectiveness, civil society, ex-post evaluation, Ghana, Impact, impact evaluation, Indonesia, Maternal Child Health, MCHN, NGOs, post-project evaluation, sanitation & hygiene, Senegal, Sustainable development, USAID, water/ sanitation
Learning from a river of ex-post project evaluations and tools… Thanks USAID!
Dear ex-post aficionados. It’s raining ex-post project evaluations. Here’s hoping learning from such evaluations in water/ sanitation, maternal/child health and even capacity building/ peacekeeping, and their number increases!
1. WATER/ SANITATION & HYGIENE:
USAID has a series of six ex-post evaluations of the water/ sanitation and hygiene sectors since 2017! What is exciting is that they are also looking to the future. These evaluations will “provide insight into what happens after an activity ends, and how to mitigate challenges in future programming, potentially. The series will inform USAID’s WASH activity design and implementation and contribute to a larger sector discussion on achieving sustainability.”
The E3 water division (Water CKM ) took sustainability on as their strategy and have made great strides these last two years. They have done five ex-post project evaluations, cited below, and MSI has completed one more wat/san/ hygiene ex-post evaluations, specifically:
Madagascar Rural Access to New Opportunities for Health and Prosperity (RANO-HP) – Published June 2017
The first evaluation in the series explores the sustainability of the sanitation and hygiene components of the RANO-HP activity, implemented in 26 communes from 2009–2013.
Indonesia Environmental Services Program (ESP) – Published August 2017
The second evaluation in the series examines the sustainability of water utility capacity building, microcredit, and financial outcomes associated with the ESP activity, which was implemented from 2004–2010.
Ethiopia Millennium Water Alliance (MWA-EP) – Published May 2018
The third evaluation in the series examines the long-term sustainability of outcomes related to rural water point construction, rehabilitation, and management, as well as participatory sanitation and hygiene education and construction related to the MWA-EP activity, implemented in 24 rural districts between 2004–2009.
Financial Institutions Reform and Expansion–Debt and Infrastructure (FIRE-D) – Published September 2018
This evaluation is the fourth in the series. It examines how urban water and sanitation services in India have changed since FIRE-D closed and to what extent policies, practices, and financing mechanisms introduced through FIRE-D have been sustained.
Millennium Water and Sanitation Program in Senegal (PEPAM/USAID) – Published July 2019
The fifth ex-post evaluation in the series looks at the PEPAM project (Programme d’Eau Potable et d’Assainissement du Millénaire au Sénégal), implemented from 2009–2014 to improve sustainable access to WASH in four regions of Senegal.
USAID-funded by MSI: USAID/Ghana’s Water Access, Sanitation, and Hygiene for Urban Poor (WASH-UP)– published Nov 2018
Also USAID and Rotary International developed a WASH Sustainability Index Tool, “to assess a WASH activity’s likelihood to be sustainable according to the following factors: availability of finance for sanitation; local capacity for construction and maintenance of latrines; the influence of social norms; and governance.” This is similar to what we learned from USAID/ FFP/ Tufts/ FHI360 12 ex-posts that resources, capacities, motivation and linkages (aka partnerships, including governance) are vital to sustaining outcomes and impacts.
It will be interesting to see whether they examine the other ex-posts for excellent lessons, as they have the Senegalese evaluation:
- “Whether or not to subsidize sanitation access …Based on this evaluation’s findings and exploration of the literature, subsidies can help improve the quality of household latrines, but increasing use of those latrines remains a challenge.
- In contrast, CLTS (a nonsubsidized approach) is often credited with increasing use of unimproved latrines, but serious questions linger about quality and long-term sustainability of the latrines built after CLTS triggering, particularly as it relates to moving up the sanitation ladder. This evaluation… provides the opportunity to examine the potential value of a hybrid approach….
- The handwashing results suggest that low-cost, low-quality handwashing stations such as tippy taps do not lead to sustained behavior change. It may be worth considering hygiene investments that reduce the behavior change burden on targeted beneficiaries.
2. MATERNAL/ CHILD HEALTH & NUTRITION:
“Sustainability of a Community-Based CHOICE Program to Improve the Health and Nutrition Status of Mothers and Infants in Indonesia,” The report focused on whether the USAID-funded CHOICE program had left sustainable impacts: improving the health and nutrition status of children under the age of five, as well as the health status of pregnant and lactating women and mothers or caretakers of young children in the Pandeglang District of Indonesia. “After examining the data collected from the PSS, the researchers found that there were significant improvements in many indicators—such as births attended by skilled personnel, the treatment of diarrhea, and the nutritional quality of food fed to infants—in the six years after the CHOICE program ended. However, despite these improvements, the researchers found no significant statistical differences between villages that received the CHOICE program interventions and comparison villages, which did not. This speaks to using such a comparison methodology to focus on actual contribution and rule out the “rising tide lifts all boats” phenomenon.
3. CAPACITY DEVELOPMENT & PEACEBUILDING:
USAID’s Regional Office in Thailand evaluated its capacity building and peacebuilding program 1.5 years ex-post. While civil society was strengthened and there were inroads made on peacebuilding, “many interventions initiated during Sapan did not continue post-Sapan, although some did remain. For example, “stakeholders cite evidence of continuing to use some governance tools in local governance related to service delivery [although] because of limited financial resources after Sapan ended, they had to change some of their interventions and reduce the range of people they could include. There are lessons for whose capacities are built, two-way feedback loops with local partners, using local organizations such as universities to sustain training, planning sufficient time for partners to internalize training lessons, etc.
4. USAID FUNDED GUIDANCE:
‘Impact Evaluations’ have a new focus on long-term impact, rather than effectiveness during implementation (which was at least the original intent of impact evaluation in the 1980s)! In September 2018, USAID and Notre Dame issued a Guide for Planning Long-term Impact Evaluations as part of the Utilizing the Expertise of the ERIE Program Consortium. The guide covers the difference between traditional impact evaluation designs and data collection methods and how to apply them to long-term impact evaluations (LTIE). It also shares examples across a range of sectors, including later evaluating past impact evaluations, which ended before final evaluation.
Finally, in new 2018 USAID guidance, ex-post evaluation is clarified as the source of the sustainability of services and benefits. USAID clarifies that “questions about the sustainability of project services and benefits can be asked at any stage, but must usually be adjusted to take evaluation timing into account. Thus, for example, in a mid-term evaluation, a question about the existence of a sustainability plan and early action on that plan might be appropriate. An end-of-project evaluation could address questions about how effective a sustainability plan seems to be, and early evidence concerning the likely continuation of project services and benefits after project funding ends. Only an ex-post evaluation, however, can provide empirical data about whether a project’s services and benefits were sustained.”
Such richness that we can learn from. Keep the momentum going on the 99% of all global projects yet unevaluated ex-post, and change how we fund, design, implement, monitor and evaluate global development projects!
by Jindra Cekan | May 16, 2018 | Accountability, Aid effectiveness, CGDev, Evaluation, ex-post evaluation, Feed the Future, Feedback Labs, foreign aid, impact investing, Impact Investors, International aid, international development, Local Participants, Participants, Participation, post project evaluation, post-project evaluation, Return on Investment (ROI), SALT, Sidekick Manifesto, SUSRoi, Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation, Sustained and Emerging Impacts Evaluations (SEIE), USAID, Valuing Voices
Setting a higher bar: Sustained Impacts are about All of us
Global development aid has a problem which may already affect impact investing as well.
It is that we think it’s really all about us (individuals, wealthy donors and INGO implementers) not all of us (you, me, and project participants, their partners and governments). It’s also about us for a short time.
All too often, the measurable results we in global development aid and Corporate Social Responsibility (CSR) funded projects that last 1-5 years track and report data for two reasons:
1) Donors have Compliance for grantees to meet (money spent, not lost, and results met by fixed deadlines of 1-5 years – look at some of the European Commission Contracting rules) and
2) Fund recipients and the participants they serve are accountable to ‘our’ donors and implementers who take what happened through their philanthropic grants as ‘their’ results.
Both can skew how sustainably we get to create impacts. An example of such strictures on sustainability from USAID. As respected CGDev Elliot and Dunning researchers found in 2016 when assessing the ‘US Feed the Future Initiative: A New Approach to Food Security?‘ the $10.15 billion leveraged $20 billion from other funders for disbursement over three years (2013-16). “We are concerned that pressure to demonstrate results in the short term may undermine efforts to ensure any impact is sustainable…. Unfortunately, the pressure to show immediate results can encourage pursuit of agricultural investments unlikely to be sustained. For example, a common response to low productivity is to subsidize or facilitate access to improved inputs… it can deliver a quick payoff… however, if the subsidies become too expensive and are eliminated or reduced, fertilizer use and yields often fall…..
With so much focus on reporting early and often about the progress in implementing the initiative, there is a risk that it increases the pressure to disburse quickly and in ways that may not produce sustainable results. For example, for 2014, Feed the Future reports that nearly 7 million farmers applied “improved technologies or management practices as a result of U.S. Government assistance,” but only 1,300 received “long-term agricultural sector productivity.” Are the millions of others that are using improved inputs or management practices because of subsidies likely to have these practices sustained? And how likely are they to continue using improved practices once the project ends?”
3) Impact investors stick to the same two paths-to-results and add a new objective: market-competitive financial returns. They also need to show short-term results to their investors, albeit with social, environmental and governance results like non-profits (future blog).
4) Altruists create things we want ‘beneficiaries’ (our participants) to have. For instance a plethora of apps for refugees cropped up in recent years, over 5,000 it is estimated, which can be appropriate, nor not so helpful. Much like #2 above, ‘we’re’ helping ‘them’ but again, it seems to be a ‘give a man a fish’… and my fish is cool sort of solution… but do our participants want/ need this?
How often is our work-for-change mostly about us/by us/ for us... when ideally it is mostly about ‘them’ (OK, given human self-interest, shouldn’t changes we want at least be about all of us?).
All too often we want to be the solution but really, our ‘grassroots’ clients who are our true customers need to generate their own solution. Best if we listen and we design for long-term sustainability together?
As the Brilliant Sidekick Manifesto stated in two of its ten steps:
a) “I will step out of the spotlight: Sustainable solutions to poverty come from within are bottom-up, and flow from local leaders who are taking the risks of holding their politicians accountable and challenging the status quo.”
b) “I will read “To Hell with Good Intentions” again and again: Politicians, celebrities and billionaire philanthropists will tell me that I can be a hero. I cannot. The poor are not powerless or waiting to be saved. Illich will check my delusions of grandeur.”
We have examples of where we have stepped away and participants had to fend for themselves. At Valuing Voices, we’ve done post project-exit evaluations 2-15 years afterward. What did participants value so much that they sustained it themselves (all about them, literally)? These Sustained and Emerging Impacts Evaluations (SEIE) also give us indications of Sustained ROI (Sustained Return on Investment (SusROI) is a key missing metric. As respected evaluator Ricardo Wilson-Grau said in an email, “I think calculating cost-effectiveness of an intervention’s outcomes would be a wonderful challenge for a financial officer searching for new challenges — if not a Nobel prize in economics!”)
Most of these evaluations are pretty bad news mixed with some good news about what folks could sustain after we left, couldn’t and why not. (These are the ones folks expect to have great results, otherwise they wouldn’t share them!) While most clients are understandably interested in what of ‘theirs’ was still standing, and it was interesting disentangling where the results were attributable by implementation or design or partnership flaws or something else, what was mesmerizing was what came from ‘them’.
The key is looking beyond ‘unexpected’ results to look at emerging impacts that are about ‘them’ (aka what we didn’t expect that was a direct result of our project, e.g. spare parts were no longer available to fix the water well pump once we left or a drought rehabilitation water project that decreased violence against women), to what emerging results are attributable not to use but only to our participants and partners who took over after our projects closed. One example is a Nepalese project ended yet the credit groups of empowered women spawned groups of support groups for battered women. Another is a child maternal health project changed how it worked as women reverted to birthing at home after NGOs left; community leaders punished both parents with incarceration in the health clinic for a week if they didn’t given birth there (wow did that work to sustain behavior change of both parents!).
Many of us at Valuing Voices are shocked that funders don’t seem that interested in this, as this is where they not only take over (viz picture, sustaining the project themselves), but they are making it theirs, not ours. Imagine assuming the point of development is to BE SUSTAINABLE.
Source: Community Life Competence
Our participants and national stakeholder partners are our true clients, yet… Feedback Labs tell us Americans alone gave $358 billion to charities (equivalent to the 2014 GDP of 20 countries) – in 2014 but how much of this was determined by what ‘beneficiaries’ want? Josh Woodard, a development expert, suggests a vouchers approach where our true clients, our participants, who would “purchase services from those competing organizations… [such an] approach to development would enable us all to see what services people actually value and want. And when we asked ourselves what our clients want, we would really mean the individuals in the communities we are in the business of working with and serving. Otherwise we’d be out of business pretty quickly.”
This opens the door to client feedback – imagine if participants could use social media to rate the sustained impacts on them of the projects they benefited from? A customer support expert wrote in Forbes, “Today, every customer has, or feels she has, a vote in how companies do business and treat customers. This is part of a new set of expectations among customers today that will only grow ... you can’t control product ratings, product discussions or much else in the way of reviews, except by providing the best customer experience possible and by being proactive in responding to negative trends that come to the surface in your reviews and ratings stronger.”
So how well are we working with our participants for ‘development’ to be about them?
What do you think?
by Jindra Cekan | Nov 29, 2017 | Aid effectiveness, Better Evaluation, ex-post evaluation, Honduras, Madagascar, Niger, Nigeria, post-project evaluation, Sustainable development, Sustained and Emerging Impacts Evaluation, USAID
Can’t wait to learn from post-project sustainability evaluation? If not why?
A colleague who has been promoting ex-post sustainability evaluation in her organization questioned my claim that doing them had “benefits” for future programming. It was an “untested assumption that there will be sufficient, strong enough evidence to apply to future programming… [and] the need to have evidence to cite for future work is not pressing enough.”
If you are on aid’s receiving end, what you care about is that good results are sustained, and you are able to live better, longer. You might want to show others evidence of what was sustained, rather than only what worked while external investments were there but stopped since. Absolutely, aid donors need to have evidence that something designed, funded, implemented, and monitored & evaluated showed good results, but we assume our results will be sustained after we have closed out and moved on. How well have we done? Let’s see.
At the American Evaluation Association meetings this month, several post-project evaluations were presented. Some came from Valuing Voices research, some from Social Impact, PLAN and World Vision and some others [1].
Results 3-5 years post close-out were, shall we say unexpected, from CRS Madagascar:
[2]
Nigeria,
[3]
and Honduras.
[4]
While there were some successes, including Niger
[5]
and Burkina Faso, where MCC/ PLAN found that three years post project “BRIGHT still had a significant positive impact—6.0 percentage points for children between ages 6 and 22—on self-reported enrollment. The impacts are smaller than estimated impacts on enrollment at 7 and 3 years after the start of the program,” they were rare [6].
If we don’t wonder why things didn’t work or why they did, and don’t return to find out if it happened again and what to do/ not to do again?, Often we continue to do very similar programming elsewhere, again assuming great results. How can we close our eyes and not do post project, Sustained and Emerging Impacts Evaluations (SEIE) and see, learn, do better? How can we continue to do very similar water/ sanitation, health, food security, and education programs and projects (with potentially similar results), and call ourselves sustainable development professionals? Shouldn’t we always ask not how effective is our aid when it’s there, but after its gone?
If you want more data, see a presentation we did at USAID. What do you think?
Sources:
[1] Cekan, J., PhD. (2014, April 7). Evaluation of ERCS/Tigray’s “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity”. Retrieved from https://valuingvoices.com/wp-content/uploads/2017/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf
[2] Madagascar Rural Access To New Opportunities For Health And Prosperity (RANO-HP) Ex-Post Evaluation. (2017, June 1). USAID. Retrieved from https://www.globalwaters.org/resources/assets/madagascar-rural-access-new-opportunities-health-and-prosperity-rano-hp-ex-post-0
[3] The World Bank. (2014, June 26). Project Performance Assessment Report, Nigeria: Second National Fadama Development Project. Retrieved from http://ieg.worldbankgroup.org/sites/default/files/Data/reports/Nigeria_Fadama2_PPAR_889580PPAR0P060IC0disclosed07070140_0.pdf
[4] Rogers, B. L., Sanchez, L., & Fierstein, J. (n.d.). Exit Strategies Study: Honduras. Retrieved from https://www.fsnnetwork.org/sites/default/files/fanta_exit_strategies_presentation_honduras_-_final_-_feb_5_0.pdf
[5] Cekan, J., PhD, Kagendo, R., & Towns, A. (2016). Participation by All: The Keys to Sustainability of a CRS Food Security Project in Niger. Retrieved from https://www.crs.org/our-work-overseas/research-publications/participation-all
[6] Davis, M., Ingwersen, N., & Kazianga, H. (2016, August 29). Ten-Year Impacts of Burkina Faso’s BRIGHT Program. Retrieved from https://www.mathematica.org/our-publications-and-findings/publications/ten-year-impacts-of-burkina-fasos-bright-program
