Who is accountable for the ‘sustained development’ of those who suffer, and for how long?
Some of the rarely discussed myths of ‘sustainable development is that the aid we donors and implementers bring will help everyone, and that recipient governments can take over once donors leave and that what we left when the project ended was sustainable. The fact that our ‘global development aid’ helps a small fraction of all those who have equally ‘worthy’ needs in the countries we target is unspoken, as is the fact that these projects aren’t often built to sustainably withstand shocks such as climate changes bearing down now. As evaluator Michael Quinn Patton writes, “Effective programs… create islands of protected effectiveness in a sea of need and suffering… [we must] assess sustainability over time for Adaptive Resilience.”[1]
More often, countries need to be: a) so poor, b) so ‘fragile’, or c) so geopolitically important to warrant our aid. The fact that most USAID and EU bilateral funding (not necessarily multilateral funding to IMF, World Bank, African or Asian or other Development Banks) is focused on fragile/ war-torn or the strategically important countries edges out what is left for the impoverished people of the world. Most of our taxpayers have little idea of this and believe it is needs-based. So, many of us assume that: d) our mostly short-term aid is either only the brief support that they need (the ‘shot in the arm school’) as might be true of emergency or humanitarian aid post-emergencies or e) the results will be so excellent as to spontaneously spread (scale) ‘forever’ that no more aid will be needed or f) that national governments – not donors- need to carry the accountability for sustained improvements forward. Yet those stakeholders in poor or fragile countries (governments, non-profit NGOs) often have the weakest capacities to sustain results. Rutere Kagendo, a fellow Kenyan on the Valuing Voices team, wrote a moving blog about who ends up tasked with sustaining project activities and results: communities, especially women.
Jindra Čekan/ova of the USA/ Czech Republic and Peter Kimeu of Kenya offer our perspectives about who is accountable for aid, and for how long.
Jindra
I have worked for international NGOs (e.g. Catholic Relief Services, the international division of the American Red Cross, large INGOs such as CARE, World Vision, Lutheran World Relief, and others), including bilaterals (USAID) and foundations (the Bill & Melinda Gates Foundation, Aga Khan Foundation) for over 30 years. Most of our projects strove to fulfill objectives of the grants and were successful. Yet rarely did we ask ‘for how long?’ At one point, I worked for a huge International NGO with a program that had been feeding 50,000 West African children breakfast for 30 years. This was part of a bilateral aid education support + ‘safety net’ program. Many studies, including this terrific one from the UN’s World Food Program, show that such school feeding improves “school participation (enrolment, attendance, completion) and learning (scores on cognitive, language and mathematics tests)… [and] decrease child marriages, etc.“
Clearly, they do good. Yet the donor suddenly wanted proof that the effects of this long-term aid had improved national GDP rather than just the nutritional and learning outcomes of assisted individuals, otherwise, the project would be canceled. Letters from the impoverished country– including the country’s president and cabinet -stating they themselves had benefitted from the program which in turn led them to great educational outcomes and leadership had some effect. Yet without lobbying from the donor country’s agriculture and food aid industry that cutting food aid exports would harm them, it would have been canceled. Did they care about the schoolchildren’s learning or only providing outlets to US agricultural surplus producers? Was 30 years too long to keep helping? How do we have these discussions as equal partners? As evaluator Zenda Offir notes regarding the SDG’s No One Left Behind, “the burden of supporting and sustaining a majority of ‘leaving no-one behind’ efforts fall inevitably on many of the poorest (low-income) countries in the Global South. The problem is that they cannot afford it, nor can they sustain it. It will therefore be unfair to hold such countries accountable for ‘leaving no-one behind’ strategies. “
This brings up the questions of sustainability and accountability ‘to whom’ and ‘for how long’? You may have other questions, including ‘why’ and ‘how we know, which I look forward to addressing, but for now, these two are the focus of this blog. While 65% of Americans favor foreign aid, believing we spend up to 25% of our GDP abroad, the US spends just over 1%. Most US aid goes to the fragile and geopolitically strategic: “More than two hundred countries receive U.S. aid. It disproportionately goes to a few, however, with the top five all receiving over $1 billion per year as of 2016: Iraq ($5.3 billion), Afghanistan ($5.1 billion), Israel ($3.1 billion), Egypt ($1.2 billion), and Jordan ($1.2 billion).” In Europe, only 3 countries met the OECD goal of giving 0.7% of GNI: Norway, Sweden, and Denmark while the Czech Republic was at 0.13%, just below the USA 0.16%(2019). A fascinating measure of ‘commitment to development” (CDI) looking at the ‘quality of aid’ found that in the Czech Republic aid performance was very poor: “Adding up both quality-adjusted aid (95.7 million USD) and quality-adjusted charity induced by public policy (1.1 million USD), we arrive at 96.8 million USD for 2009 which amounts to 0.054% of GNI. Translating the percentage onto the standardised CDI scale the Czech Republic… [has] the third least favorable aid policy towards developing countries among DAC countries… [and] Aid allocation is not primarily focused on low-income countries” which is in part explained by the recent shift from aid recipient to aid donor (Syrovatka and Krylova 2012). Even, worse now, donors and international NGOs distribute aid (especially what is left for the poor countries of the world) that is annually allocated, but as the pandemic has led more spending to be domestic, aid to the poorest has decreased among almost all donor countries, bad news as Covid-economic downturns continue and climate change ramps up.
So who is accountable to the poor whom we help? Peter comments on that from the perspective of the CEO of a local Kenyan NGO targeting 15,000 farmers.
Peter
I have over forty years of experience in development; 8 years lead in Community Initiated (Harambee) High schools, 35 years with Catholic Relief Services in Emergency Relief, Sustainable Development and Justice and Peace Programs, and currently 6 years with the (Kenyan) county-devolved sustainable development. I am the Founder and CEO of Decent Living Institute of Organic Farming promoting avocado farming, aquaculture, and apiculture for improved decent livelihoods. My early life as a young boy makes me a living witness of a life in deep poverty, which the New York Times featured.
The question ‘who is accountable for sustained development’ and ‘for how long’ has an assumption that it is possible to attain sustainable development without the continued involvement of those who suffer. I don’t think so. Sustained development occurs as a process to a transformed situation from abject poverty, a condition of want without the capacity to satisfy even the most basic needs, a position of lacking continuously leading to untold suffering and living in dehumanized conditions for the sufferer and the generations to come to the desired decency of fulfilled living. Living as a pauper in my first 30 years of life, having been born in a paupers’ family, I accepted the conditions of poverty and hunger as a way of life. After all, you know nothing better and when you see wealth around you, it is meant for the lucky few, and not for you. The situation limits the poor to survival conditions, eating from hand to mouth and everything is left to luck.
Aid to the poor would make sense if it is used as a catalyst to motivate and enable the poor depart from the circle of poverty (the poor giving birth to more poor) and is able to sustain the conditions of being above the poverty line of US$2 a day. Such aid would enable them to have enough to take care of their daily basic needs and create wealth without falling back below the poverty line repeatedly, for generations to come. The impact measure for aid should therefore be participatory learning from and measuring the extent to which success is sustained documenting representative success stories by participants who have left the circle of poverty sustainably. Such would include ‘in the past I couldn’t to find enough to eat occasionally slept without food and now my family has no idea of how it feels like to be hungry.
Unfortunately, the manner of delivering aid is seen as pure luck by the targeted poor for it comes without involving the poor as to strategically plan long-term impact that they can sustain. The aid donors and implementing agencies will target a given county, while the identification of targeted community cluster location for aid will be influenced by either by powerful persons from the locality or larger numbers in the public participation, so those with greater authority or louder voice will take the day. The decision on who will participate in the project finally will be determined by the same criteria and not the poverty levels. One example is the aid fund for COVID-19 response in Kenya which was distributed to the well connected to persons of authority and not to those who championed the control of the coronavirus. A decision was made at the county governments to disperse one million Kenya shillings to every cluster of villages to pay the youth for engaging in communal work such as community road works, terracing a degraded land, or even constructing an earth dam and paid per piece work completed – termed employment -to cushion the youth who have lost job opportunities due to the COVID -19 effects. A million shillings in a cluster of ten villages would perhaps engage 100 young people for a week earning Kenya shillings 500 (US$5) a day or $30 a week. The rest of the money – 50% of the total or more – does not go for wages as planned but is used to cover the management of the program by the county officials. The youth will spend this money like it is good luck for it is too little to ever think of the future investments.
However, the same amount is what it costs to support a member of the self-help group and collective community-led development in our Decent Living NGO per family of a vulnerable child to grow a vegetable garden, keep six chicks and grow three Hass avocado seedlings. Further, the participants commit to support another poor family with six chicks in a years’ time. From the onset, the poor are involved in ‘planning in advance’ to help others. Their developing vision is guided by the long-term impact they hope for, such as the family economic boost that will cover the full cost of schooling, medical expenses, and family meals, clothing, and shelter for all the children including the most vulnerable. Other long-term indicators will be the percentage of poor families that are above the poverty line meeting the family basic needs sustainably.
I see the role of the aid donor as to holding the aid receiver (local government and recipient communities) to their goals of sustainable development and to account for the funds given by reaching their goals and targets that must be time-bound.The aid receivers are also responsible to account for their aid distribution to their intermediary implementing partners (often local non-profits/ NGOs) to meet their targets, goals and should track expected and measurable long-term outcomes within three years after the closure of the project. This means to deliver not only the aid funds but also through the funding the systems established or improved at the conception of the project should be accounted for during the project implementation period and will be impacting long term results transforming the community to the desired state long after the project activities.
It could be building sustainable infrastructure for long-term support to the poor. The sustained impact would then be numbers of poor that have transformed their poverty and created wealth through the developed infrastructure in an intergenerational, long-lasting way that could be measured in later years. Sadly, most of the aid givers do not see their role beyond the performance short-term outputs such as trainings given or outcomes leading to a change in farming practices, like the deliverables for the specific objectives in an agriculture project. Hence the success of the project is defined by these short-term indicators that measure outputs such as the target number reached with food aid, or even some changes in practices leading to improved yields, but once the project ends, all tracking of results end. The national stakeholders are – or should be- responsible to demonstrate how the results of the project will be assumed by the community’s self-help groups so that the impacts become intergenerational. For it is vital to see that the project does not end with the implementing agencies. It is not only short-sighted by aid donors to believe that it ends, but national stakeholders are absconding their responsibility of accountability to the long-term impacts that are related to relationships and behavior change sustainably when they do not sustain them.
The UN’s ‘Sustainable Development Goals’ are merely a dream for most poor until the individual struggling with conditions of want is able to take steps towards permanent solutions for themselves and their future generations. It takes an oppressed dreamer (the poor with empty stomachs) who believes a change is possible to demand accountability. It also takes a progressive facilitator (donors and national stakeholders) who believes in creating enabling conditions for the oppressed to succeed. Both the oppressed dreamers (project participants, local implementing NGO agencies and the progressive facilitator (donor) are accountable to the transformed conditions. For the ‘sustained development’ to occur it must be intentionally dreamed of by all parties engaged in the process of development.
I dream of “a just world where everyone is fully a participant and celebrates sustainable development for all” wrote Pope Francis in his 2015 Laudato Si encyclical. He calls for all humanity to take care of our Mother Earth and in return, she will provide for all, including addressing issues of global warming. In my world dream, I see a time when a transformation of the sufferer from distressful and oppressive conditions of living is eased by putting future dreams into action, for those who suffer with deprivation today and are thirsty for change. I wish to make reference to a story told in the bible Jesus meeting a blind beggar (Mark 10:51) shouting to Jesus for help. Jesus asked the beggar to identify what type of aid he needed. And the blind beggar’s request for the power to see was heard and his sight was fully restored, emancipating him from the bondage of begging. We are told he transformed from a beggar into a disciple of Jesus. The transformation of conditions to sustainable options starts with the bilateral donor engaging a poor government to undertake a particular development agenda that in return facilitates its citizens to enjoy sustained development.The donor government should hold the recipient government responsible and accountable of delivering sustainable options for its citizen as per the grant agreements with evaluation two to three years after the project closure.
The poor who may be targeted with the aid may seem passive, not having been involved right from the beginning of design, and may have limitations of identifying what to ask for, perhaps those with intermediary implementing INGOs may be aware of how well what is being offered can meet their needs. Setting up the appropriate structures, they may dictate and demand sustained development options for themselves and those who are suppressed in poverty. The major issue is that most often the victims of poverty are never engaged in aid’s design and only implement what is offered. The situation creates room for corrupt national governments, INGOs, and NGOs to make quick money. The donors should hold recipient countries and INGOs accountable for tangible results toward the Sustainable Development Goals indicators for every grant in aid for as long as it takes, not just reporting at the national level.
I see a world where what matters most, is how engaged those who have empty stomachs are in the development aid agenda, and how the aid is administered and accounted for themselves and the neighboring suffering households. That development is all about a sustained transformation for empty stomachs of our project participants, their immediate neighbors, their children, their husbands, and their fathers/in law and mothers /in law, their brothers/ in law and sisters/ in law. It is about a better living standard of their neighbors who lends salt and water, the generosity of their firewood friends, those neighbors who will never turn down an opportunity to offer help no matter what. If these impacts and long term outcomes are not evaluated and accounted for, those who suffer poverty will always consider projects as myths of ‘sustainable development’ and the aid provided by bilateral, multilateral donors or through INGOs/NGOs as beneficial to the lucky few, while recipient governments and participating communities and their future generations have no sustainable impact results.
Your thoughts?
[1] MQ Patton in New Directions for Evaluation “Transformation to Global Sustainability: Implications for Evaluation and Evaluators, 2019 (link inaccessible)
Ex-post Eval Week: Are we serious about project sustainability and exit? By Abu Ala Hasan
Published on the American Evaluation Association AEA365blog series: https://aea365.org/blog/ex-post-eval-week-are-we-serious-about-project-sustainability-and-exit-by-abu-ala-hasan/ January 18, 2021
I am Abu Ala Hasan, an independent consultant. I am an anthropologist, have been working in the NGO sector for 17 years. A large portion of my work are evaluations and research.
In 2018 we (Jindra Cekan and I) were selected to design exit strategy and learning documentation for a client. When we started, we found, despite having a plan for devising exit strategy, the implementing organisations involved only paid attention to that at the end of the project, having only three months.
Most startling was the fact that the informants, ranging from partner NGO officials, to community members involved gave us blank looks when asked about readiness for exit. Few NGO executives said that they would be able to continue some of the activities themselves. Almost none of the community members could answer exit questions; rather they tried to explain about the project services and their benefits.
Though it was thought provoking, it was not surprising at all due to existing NGO culture, where in practice, interventions are initiated and decisions made by the NGOs or funders, rather than the community.
So, why does it happen? First and foremost, exit is not taken seriously. Rather than trying to ensure sustainability of the project from the beginning or designing the project to achieve it, it is added, as if, an auxiliary component; apparently to fulfil criteria by outsiders. The word ‘exit strategy’ was mentioned twice and ‘sustainability’ five times in the action plan, a 38- page document, and we saw few activities.
While it takes time, the trend of project cycle has been increasingly heading towards shorter duration. A decade ago most projects were planned for five years or longer but now one to three years is the norm. This is not only counterproductive for sustainability but also detrimental to bring about significant social change (outcome/impact). The community members’ reactions indicated that sustainability was unfamiliar; it appeared, they were not informed or there was no such discussion with them. Planning for sustainability was not taken seriously.
Hot Tips:
To ensure sustainability and proper exit:
There should be sincere commitment and effort towards sustainability and exit from NGOs
Sustainability and exit planning should be built in the project implementation process from design stage. Projects should be designed for sustainability and not ignored
Involvement of the community members, their voice and participation in decision making at all stages is very important for sustainability and proper exit
Except for emergency projects, development projects should be longer in duration and properly evaluated periodically
Rad Resources:
For exit and sustainability planning these resources could be helpful:
“NGO Research Culture and its Implication in Bangladesh: An Insider’s Perspective”, The Oriental Anthropologist, Vol. 14, No. 1, 2014, Pages 1-12, OICSR, Allahabad, India
Once malnutrition has decreased, students’ attendance has risen, or the number of small businesses has doubled, program implementers may be quick to pack their bags and leave. But the impact of their work may be undermined if not undone if they leave before implementing context-specific, comprehensive plans for post-project sustainability. It is for this reason that exit strategies – and particularly institution-wide mandates by donors – are crucial components of effective development programming. The United States Agency for International Development (USAID and the European Union (EU) Commission both have close-out protocols in place, but much to learn from one another in ensuring long-term progress in the wake of their close-outs.
Recommendations for Project Close-Out
Although every program should develop and follow context-specific exit measures, non-profits and ex-post evaluators have found that some exit strategies consistently lead to long-term, sustainable impact. The International NGO Training and Research Centre (INTRAC), for example, states that donors and implementers should actively plan for close-out throughout the project – from program design to the post-exit stage [1]. It recommends the following:
Implementers should “build exit thinking into the design of the project,” and during the program, representatives should be monitoring pre-determined exit indicators [1]
“Senior staff and management need to prioritize resources for exits in order to do them well, embed learning within organizations and across teams, and ensure mistakes are not repeated” [1]
“Practitioners should also consider going the extra mile to look after staff at headquarters and within partner organizations” [1]
“Document experiences and share them externally” [1]
According to USAID’s 2020 Program Cycle Operational Policy, the formal close-out process for specific projects are carried out by the USAID Mission team in the recipient country and Operating Units based in Washington, DC [2]. However, the Program Cycle does not reference the close-out process, let alone offer guidance [3]. Unlike country exits, which require thorough evaluation and international cooperation [4],ᴬ the sustainability measures of close-out procedures for activities – components of country-wide programs – seem to be delegated to implementing agencies.
Tellingly, the USAID formally defines “sustainability” as “the ability of a local system to produce desired outcomes” after its projects end, which the Agency may “contribute to” by strengthening capacity in their respective realms [5]. After close-out, USAID currently considers the sustainability of its projects to be a matter of “the country and/or targeted community’s commitment and capacity to achieve development solutions” [5]. Commitment and capacity, in turn, are gauged by 17 country-level “self-reliance metrics” ranging from “liberal democracy” to “export sophistication” [6,7].ᴮ Even post-project monitoring and evaluation are explicitly meant to “reveal whether implementation is on track and results are being achieved” with no mention of sustainability or long-term follow-up [3].
In effect, this framework not only puts the onus of project sustainability on recipients, but evaluates it based on large-scale and potentially tangential metrics. The stakes of this practice are raised by the fact that the resultant “Country Roadmaps” inform when USAID ends its programming in a given country, endangering a range of populations that these metrics do not account for, like seniors and prisoners [6].
USAID’s official Program Cycle [3]
During close-out, USAID representatives seem to focus on administrative tasks. When a Mission is preparing for activity close-out, for example, Mission Directors are responsible for developing plans that include the following:
“The retention of sufficient and suitable staff members and the delegation of authority and assignment of specific responsibility to each to carry to completion the required close-out action” [8]
“Estimates of the personnel, space, and funds required to complete the close-out” [8]
“The timetable for phasing out, transferring, or terminating U.S. direct-hire, participating agency, and contract personnel and the replacement of assistance from other resources” [8]
“Actions for terminating the services of cooperating country and third country national staff” [8]
In its 2003 version, but since removed:
“Missions must think in terms of leaving sustainable and useful units of assistance” [9]
“A monthly timeframe/activity projection for the entire close-out period [approximately six months] of major activities that must occur” [9]
Similarly, in the case of Interagency Agreements, USAID Agreement Officers’ (AO) personal and delegated close-out responsibilities are limited to the collection of financial reports and compliance-related documents [10]. AOs are expected to work in this same, limited capacity as they terminate Assistance and Acquisition Awards [11].
By contrast, when USAID gives grants to NGOs and has “little direct involvement,” recipients are strongly encouraged to adhere to six substantial, well-rounded “tips,” which include [12,13]:
*Staff refers to each NGO’s own staff; based on [13]
Among all of the terms and conditions reviewed, USAID’s exit strategies for foreign Missions, Agreements, and Awards have been almost exclusively administrative in nature. Although the Agency should not be solely responsible for ensuring the sustainability of its programming, its lack of regulation in this realm risks both undue pressure on implementing partners to do so and short-lived progress in the wake of its exit [14].
Fortunately, USAID has demonstrated its interest in sustainable close-out by commissioning studies and reports. In March 2020, USAID’s three-year “collaborative learning project” called Stopping as Success published a checklist for sustainable project transition [15,16]. The publication offers 16 concrete suggestions ranging from “build[ing] trust” with local communities to requiring implementers to report their progress in carrying out a pre-determined sustainability plan [16]. Similarly, the Food and Nutrition Technical Assistance III Project – a partnership between USAID and FHI 360 – studied the consequences of four food assistance programs and made thorough recommendations for ensuring the long-term impact of future projects [14]. These recommendations include accounting for “predictable external shocks and stressors” like natural disasters, and building “vertical linkages […] between community-based organizations or individuals and existing public or private sector institutions” [14]. Armed with this impressive base, USAID is well-equipped to account for new, evidence-based close-out practices.
Given the sheer scale of EU development programming – 28 countries providing 47% of the world’s development aid – no specific, all-encompassing mandates for the close-out process was found [17].ᶜ Perhaps as a result, there is notoriously little development policy coordination among Member States:
Due to inconsistencies among the eight documents outlining the EU’s development objectives and indicators, an internal audit characterized its collective approach as “mixed – at best” [18]
High-level meetings have explicitly prioritized “consensus on development” and “working better together,” producing appropriately vague guidelines that do not specifically address closure [19,20]
e.g. EU staff should “help to promote more inclusive, sustainable growth that does not compromise the ability of future generations to meet their needs” [19]
Its main development finance body, the Development Cooperation Instrument, has been critiqued for its “ambitio[us] desig[n] to address a very broad range of issues, from policy priorities in the agenda for change to EU policy concerns” [21]
For insight into the EU’s exit strategies, one must look to guidelines for specific types of development programming.
[33]
In the case of joint programming, when Member States work together to design and implement a country-wide, multi-sectorial project, donors must “reach agreement on a responsible exit strategy” for “phasing out sectors outside focal/centration areas, without creating a financing gap and with minimum disruption to the partner country” [20]. The policy offers two examples of such a strategy: “including scenarios in which other participating partners ‘plug the gap’ and/or other forms of cooperation beyond ODA [official development assistance] (e.g. the private sector)” [20]. The strategy must be drafted with input from the recipient country “as much as possible,” and may include either the intervention of other stakeholders or the private sector [20]. These discussions make take place at any of the following four “entry points” [20]:
“At the start of a new national development plan cycle, or a major change in the country” [20]
“At the start of a new strategic programming period or a review process for several important development partners” [20]
“During high-level country meetings […] where partner governments present their plans and solicit development partners support” [20]
“In situations of fragility or transition from humanitarian aid to more structured programmatic aid and more development partners” [20]
The development of context-specific exit strategies by Member States and recipients allows for practicable, fairly delegated, and concrete steps to exiting. However, this provision omits mention of enforcement or evaluation, may take place at the end of the implementation period, and does not encompass best practices as defined by INTRAC [1].
Outside of joint programming, the EU oversees projects funded by its executive implementing agencies, such as the Consumers, Health, Agriculture and Food Executive Agency and the Executive Agency for Small and Medium-sized Enterprises (EASME). In those cases, agency-wide final report guidelines could shed light on collective guidance but these are publicly inaccessible [22].ᴰ However, an early iteration of the Horizon 2020 Programme’s final report guide is still available online. The EU had required that implementers submit 20-30 lines of text addressing the following:
“Inform about activities undertaken during the project to ensure that your action
continues and reaches out further. Inform also about plans and intentions to continue
and sustain the project activities after the end of the project. Indicate the state
of advancement (e.g. agreed action, concrete proposal, idea)” [23].
Although field staff may not have been required to tangibly ensure the sustainability of their projects – and may not even make these assessments anymore – it is clear that at one time, the EU mandated assessments of expected sustainability at the end of program-specific project implementation.
With no overarching guidance for EU project exits, countries and individual NGOs have resorted to choosing or developing their own. The think tank E3G – funded in part by EASME [24] – has issued both internal and guidance for sustainable programming [25,26]. Its key points include “consultative engagement[s with] a broad range of stakeholders,” “integrat[ing programs] with long term national development plans,” and “strengthening city and local-level planning and institutions” [25]. These recommendations, however, are scattered among legal studies, commissioned reports, and briefing papers. By contrast, another EASME-funded program called Carbon Market Watch adheres to the project cycle guidelines of UN committees [27,28,29].ᴱ Although their Project Design Document and methodology forms request specific information about monitoring and evaluation, they do not ask for implementers’ plans for program close-out or long-term impact [30].
[32]
Other EU member-country NGOs end up developing sustainable exit measures mid-project. One example is a 2011-2013 agricultural project in Ethiopia implemented by People in Need, an international NGO based in the Czech Republic and funded in part by the EU’s Commission for Civil Protection and Humanitarian Aid [31]. Two months before its end, an evaluation of the program’s sustainability was conducted, receiving such recommendations as: 1) “increasing [recipients’] financial sustainability;” 2) “ensuring that FTCs’ [Farmer Training Centers] services benefit all, not only model farmers;” and 3) “providing books in Amharic language, not English” [32]. In the remaining time, 75% of these recommendations were met, leading to the internal assessment that the “sustainability of the project’s outcomes is likely to be largely positive” [32].
Learning from One Another
USAID and the EU operate at starkly different scales – one a country-specific bilateral donor, and the other an economic union of dozens of donor countries – but each of their exit strategies could be strengthened by incorporating elements from the other’s.
The EU could learn from the USAID’s multifaceted guidance for NGOs; the considerations listed in its final report guide are limited to the post-project involvement of EU field staff. It should explicitly promote local participation for sustainability during project close-out and beyond, such as transferring operations to local organizations or businesses.
In turn, USAID’s administration-centric strategy could benefit from the large-scale considerations of EU policies. Discussions with recipient governments, particularly in turbulent circumstances, could strengthen diplomatic relations with the US in addition to supporting reasonable transition plans. Formally soliciting exit plans from project implementers could also produce more sustainable project results.
NOTE: Valuing Voices thanks our excellent Harvard intern Rachel Sadoff for this research! We have a question: we found no EU/EC sites discussing exit strategies recommendations or guidance for global development projects. Do you have some? Please share in the comments below. Thank you.
Footnotes:
A) Projects and activities need to explain and demonstrate how their programming advances the objectives of USAID’s Country Development Cooperation Strategies for the given state (see ADS Chapter 201).
B) Current totals are publicly available on the Agency’s Country Roadmap website.
C) VV’s search spanned over five hours.
D) As of writing, final report guidelines are only available through an exclusive portal.
E) These are the Clean Development Mechanism and Joint Implementation Committee of the United Nations Framework Convention on Climate Change (UNFCCC).
[14] Food and Nutrition Technical Assistance. (2015, December). Effective Sustainability and Exit Strategies for USAID FFP Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
[20] European Commission. (2018, June). Joint Programming Guidance: Supporting EU Delegations to Work Better Together with Member States, Like-Minded Partners and Country Stakeholders. Retrieved from https://knowledge.effectivecooperation.org/node/77
[23] European Commission: Intelligent Energy Europe. (2012). Guidelines for Drafting the Final Technical Implementation Report. Retrieved June 2020 from https://ec.europa.eu/easme/en/technical-reporting
[31] Directorate-General for European Civil Protection and Humanitarian Aid Operations. (n.d.). Agreements for Humanitarian Aid and Emergency Support Within the EU Awarded in 2017 by DG ECHO. Retrieved June 2020 from https://ec.europa.eu/echo/files/funding/agreements/agreements_2017.pdf
Did we make enough time and measurement to foster sustainability, as we phased-down and phased-over an array of activities, alongside those remaining (white paper forthcoming)? Did we abruptly phase-out leaving partners and participants at a loss? Sharing power over all these decisions will influence what lasts.
There is an amazing breadth of local, ongoing resources, skills & capacities, linkages, motivation (thanks to Tufts FHI360’s work for USAID’s Food For Peace) that we can explore and learn from. There are local innovations and an array of unplanned collaborations (e.g., funding for health staff (Niger), training in small enterprise from the national government (Bangladesh), or private sector markets (Ethiopia) that can be accessed when partnerships are transparent and created one or more years pre-exit to collaborate on post-exit.
Ideally, we design and implement for exit from the onset. When we jointly set the timeframe and jointly assess risks to sustainability and adaptively manage exit, rather than exit based on pre-set timeframes, all sides win, with partners and participants able to foster sustainability. As USAID/ GIIN wrote about Responsible Exits for Impact Investors (2018), “the foundations for a responsible exit are laid even before an investment is made. To increase the likelihood of continued impact after exit, investors often select investees based on whether impact is embedded in their business model or inextricably linked to financial success. They also seek to understand the likely growth trajectory of the business, which has implications for which exit paths and options will become available.” They also note that a “growth strategy’ is needed throughout and at (investment) exit is “a company’s continued access to the right resources, networks, and knowledge” for sustained impact.
The need for a thoughtful approach to sustainability is shown by Hiller, Guthrie, and Jones in “Overcoming Ex-Post Development Stagnation“ (2016). The authors cite “limited evidence of program efficacies coupled with government and agency preference for planning, approval, and implementation processes rather than sustainment of outputs, outcomes, and impacts means that ex-post performance, scaling, and sustainability is not well understood or well pursued…. [There is a] lack of willingness to commit time and resources to rigorous evaluation of post-project effectiveness“. This affects a vast number of projects. For instance, they found 63,000 projects in 2003 alone, and “relative to the number of development projects undertaken, ex-post project [evaluations] are not commonly carried out, meaning that rates of success are often unknown and the complexity of causalities and ex-post dynamics of interactions and processes are not well understood.“ This limits our learning from what has (not) worked and what to do more (or less) of, including those that could not be sustained with only local resources.
We make sustainability assumptions are participants long for them, as Valuing Voices also found. Hiller et al. state that “whilst project documentation commonly conveys an expectation that some process of spread will occur ex-post, it rarely does, despite strong ex-post case-study evidence of stakeholder requests for further development opportunities.” This cautionary feedback could mean some project activities could be so resource-intensive that they could not feasibly be sustained or spread without long-term support, and retaining results may be limited to less costly activities. Valuing Voices found that other activities could be remunerative enough (financially, in health or education outcomes, for instance), as to be locally demanded and continued to be pursued. We have found in our Valuing Voices research and the Tufts research that activities where incentives did not continue, tended to die ex-post, while those which continued to bring benefits, such as cash crops and credit, water supply, and health, were prioritized ex-post, even in the absence of external funding continuing.
Hiller et al. outline that there are multiple ways in which scaling-up environmental sustainability over time, over area and, interestingly, scaling-within projects. The Ex-post Development Stagnation authors are clear that “creating conditions to support longer-term sustainability beyond project completion represents a recurring challenge, and it is not uncommon for activities and institutions to become inactive ex-post or for stakeholders to revert to previously unsustainable practices.” They note that some watershed studies have even found that participants actively destroy project measures in some cases. Certainly, the inability of locals to sustain often expensive activities without a project or larger organization’s support is common. “If it is assumed that development needs remain outstanding, then there may be merit in ensuring that development projects do not just remain “isolated, one-time interventions, like unconnected dots on a white page” or “islands of salvation.” The authors concur, “based on project subsidiarity and participatory principles, scaling-within management should be devolved to the local level (local authorities and local communities) to allow communities and individuals to filter out irrelevant practices and encourage adaptation and evolution of activities which are of greatest perceived livelihood benefit”.
As Valuing Voices research on exit has shown, it is a process, not an event(forthcoming, with thanks to I. Davies). The Hiller article notes that sustainability is best enhanced by capacity-building during implementation and with time for handover where “organizations adopt modes of functioning that allow local communities and organizations to build conceptual, operational, and institutional capacities. While scaling-down does not mean that governments disengage from processes such as community-driven development — it does, however, require it to be more flexible and responsive to locally generated demand to ensure the terrain is fertile for community organizations to emerge, learn, and grow.“
Let’s work together to extend the sustainability of impacts. Would love to hear your thoughts…
Assuming Sustainability and Impact is Dangerous to Development
(+ OECD/ DAC evaluation criteria)
We all do it; well, I used to do it too. I used to assume that if I helped my field staff and partners target and design funded projects well enough, and try to ensure a high quality of implementation and M&E, then it would result in sustainable programming. I assumed we would have moved our participants and partners toward projected long-term, top-of-logical-framework’s aspirational impact such as “vibrant agriculture leading to no hunger”, “locally sustained maternal child health and nutrition”, “self-sustained ecosystems”.
INTRAC nicely differentiates between what is typically measured (“outputs can only ever be the deliverables of a project or programme…that are largely within the control of an agency”) and what is not: “impact as the lasting or significant changes in people’s lives brought about by an intervention or interventions” [1]. They continue: “as few organisations are really judged on their impact, the OECD DAC impact definition (“positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended“) allows for long-term changes in institutional capacity or policy change to be classed as impact” [1]. Do we do this? Virtually never. 99% of the time we only evaluate what happened while the project and its results is under the control of the aid implementer. Yet the five OECD/DAC evaluation criteria asks us to evaluate relevance, effectiveness, efficiency (fair enough, this is important to know if a project was good) and also impact and sustainability. So in addition to the prescription to evaluate ‘long-term effects’ (impact), evaluators are to measure “whether the benefits of an activity are likely to continue after donor funding has been withdrawn… [including being] environmentally as well as financially sustainable” [2].
How do we know we are getting to sustained outcomes and impacts? We ask people on the receiving end ideally after projects end. It is dangerous to assume sustainability and impact, and assume positive development trajectories (Sridharan) unless we consistently do “ex-post” project evaluations such as these from our research or catalytic organizations that have done at least one ex-post. At very minimum we should evaluate projected sustainability at end of project with those tasked to sustain it before the same project is repeated. Unfortunately we rarely do so and the assumed sustainability is so often not borne out, as I presented at the European Evaluation Society conference Sustainability panel two weeks ago along with AusAid’s DFAT, the World Bank, University College London and UNFEM.
Will we ever know if we have gotten to sustained impacts? Not unless the OECD/DAC criteria are drastically updated and organizations evaluate most projects ex-post (not just good ones :)), learn from the results and fund and implement for country-led sustainability with the country nationals. We must, as Sanjeev Sridharan tells us in a forthcoming paper embed sustainability into our Theories of Change from the onset (“Till time (and poor planning) do us part: Programs as dynamic systems — Incorporating planning of sustainability into theories of change” (Canadian Journal of Program Evaluation, 2018).*
There are remarkable assumptions routinely made. Many projects put sustainability into the proposal, yet most close out projects in the last 6 months. Rarely do projects take the time to properly phase down or phase over (unlike CRS Niger); many exit ceremonially ‘handing over’ projects to country-nationals, disposing project assets, and leaving only a final report behind. Alternatively, this USAID Uganda CDCS Country Transition Plan which looks over 20 years in the future by when it assumes to have accomplished sustained impact for exit [3]. Maybe they will measure progress towards that goal and orient programs toward handover, as in the new USAID “Journey to Self-Reliance” – we hope! Truly, we can plan to exit, but only when data bears out our sustained impact, not when the money or political will runs out.
As OXFAM’s blog today on the evaluation criteria says, “Sustainability is often treated as an assessment of whether an output is likely to be sustained after the end of the project. No one, well, hardly anyone, ever measures sustainability in terms of understanding whether we are meeting the needs of the present, without compromising the ability of future generations to meet their own need” and “too often in development we evaluate a project or programme and claim impact in a very narrow sense rather than the broader ecology beyond project or programme parameters” [4]. In fact, most ‘impact evaluations’ actually test effectiveness rather than long-term impact. Too rarely do we test impact assumptions by returning 2-10 years later and gather proof of what impacted locals’ lives sustainably, much less – importantly – what emerged from their own efforts once we left (SEIEs)! Oh, our hubris.
if you’re interested in the European Evaluation Society’s DAC criteria update discussion, see flagship discussion and Zenda Offir’s blog which stresses the need for better design that include ownership, inclusivity, empowerment [5][6]. These new evaluation criteria need to be updated, including Florence Etta’s and AGDEN‘s additional criteria participation, non-discrimination and accountability!
We can no longer afford to spend resources without listening to our true clients – those tasked with sustaining the impacts after we pack up – our partners and participants. We can no longer fund what cannot be proven to be sustained that is impactful. We talk about effectiveness and country ownership (which is paramount for sustainability and long-term impact), with an OECD report (2018) found “increases [in[ aid effectiveness by reducing transaction costs and improving recipient countries ownership” [7]. Yet donor governments who ‘tie’ aid to their own country national’s contracts benefit a staggering amount from ‘aid’ given. “Australia and the United Kingdom both reported … 93 percent and 90 percent of the value of their contracts respectively went to their own firms” [7]. It is not so different in the USA where aid is becoming bureaucratically centralized in the hands of a few for-profit contractors and centralized hundreds of millions in a handful of contracts. We must Do Development Differently. We can’t be the prime beneficiaries of our own aid; accountability must be to our participants; is it their countries, not our projects, and we cannot keep dangerously assuming sustained impact. Please let us know what you think…
[3] USAID. (2016, December 6). USAID Uganda Country Development Cooperation Strategy 2016-2021. Retrieved October, 2018, from https://www.usaid.gov/uganda/cdcs
[6] Ofir, Z. (2018, October 13). Updating the DAC Criteria, Part 11 (FINAL). From Evaluation Criteria to Design Principles. Retrieved from https://zendaofir.com/dac-criteria-part-11/
Embedding Sustainability Everywhere – All Five Slices Now
It has been a tumultuous year, and next year does not look like we will have much stability as a respite. As domestic concerns grow larger in two huge economies, US and UK, the question of the place foreign aid will play abound in conversations around the world.
However 2017’s transitions transform our work, for now I do have some good news,
1) Sustainability can be cheap. Far cheaper, in fact to design for sustainability, create feedback loops checking on sustainability through the eyes of our partners and participants, monitor for sustainability than to assume it’ll happen and far cheaper than finding out funds could have had far greater impact if we had valued their voices in the first place.
In our work to help our clients and partners fund, design, implement and monitor/ evaluate with sustainability in mind, we created what we hope is a helpful tool (guidance forthcoming).
a) By Designing for Sustainability with those who will sustain them, their financial buy-in and commitment are far higher (see CRS/ Niger [1]), as is advocacy and community buy-in (see new post-project OXFAM/ DRC [2]) and there are indicators the costs of start-up later are more cost-effective.
b) By clarifying Sustainability Indicators we check assumptions about who will do so, how much of a priority our activities are before we scale them up (Federation/ Ethiopian Red Cross). Retrospective post-project sustainability evaluations also enable us to learning from past successes and do better.
c) Sustainability Monitoring and Adaptation involve those pesky but pivotal feedback loops which are vital to understanding if we have gone off the rails or not, especially in terms of unexpected shocks derailing logical frameworks of designed projects. USAID’s nice recent CLA (Collaborating, Learning, Adapting) process includes donor funding for adaptation mid-stream which fosters effectiveness and sustainability [3]. Even lovelier are the Doing Development Differently examples that are often very low-cost and high-effect.
d) Informed Exit, Stakeholder Sustainability Consultation should be done throughout the cycle, at least a year earlier than most projects can begin this (note; not the last few months please). Transitioning for success leverages sunk costs for ongoing results. It includes heavy knowledge management on how the implementers managed information and resources, tracked data, sustained outputs and outcomes which now local partners will need to do, etc (FFP Exit Strategies study, a Czech study, and a new UK Results study shows range of items to consider during and post-transition) [4] [5].
e) Post-Project Sustainability: Our Valuing Voices/ Better Evaluation/ Tufts joint presentation at AEA on SEIE has more on how learning from post-project evaluation lessons can change sustainability of future projects for the better [6]! Further, what capacities and systems have been built in-country to sustain the results after our funding and expertise leaves? What can we do differently?
Can it be done? Demand is rising. I recently presented at a conference about embedding sustainability in programming now, and an enterprising NGO took this idea to heart. They proposed joint design with communities, which is the very bedrock of designing for sustained impact. Kudos!
2) Secondly, donors are willing to pay for sustainability. Sustainability is ensconced in USAID/ Food For Peace’s (FFP) 2016 Strategy and CLA (above) [7]. In the section called New learning and Implications for FFP Programming, they present findings such as “actions that drive big results during the life of the project may actually undermine sustainability in the long run. It raises the question as to whether FFP is willing to accept more modest results in the near term if they can be delivered in a way that will yield more sustainable gains over time….”
They also point us in the direction of country-led development: “Sustained capacity, resources, motivation, and linkages all require a focus on catalysts for change beyond FFP. Facilitative approaches that rely on and strengthen local actors help ensure that resource and knowledge transfers, and the incentives and linkages that support them, will be self-perpetuating beyond project end”. Notably, while UK’s DFID focuses on maximizing impact through Value-for-Money, it is a shorter-term economy, efficiency and effectiveness rather than sustained impact for the end-users and DFID’s exit strategies have recently been critiqued [5].
There is an issue of disincentives for the new administration to heed (if the agricultural lobby for US food exports does not prevail): “In [USAID’s exit] study evaluating how sustainable the results of Title II development programs are 2–3 years after project closure, FFP found that “providing free resources can threaten sustainability, unless replacement of those resources both as project inputs and as incentives has been addressed” [8]. As the Natural Resources section notes, “Whether entirely in the hands of the community or linked to a formal institution, the incentives and resources necessary to maintain a community asset are part of the system that will sustain it. The lack of such systems is visible in rusted irrigation pumps, failed mangrove plantations, abandoned bore wells, eroded dikes, and silted-in fish ponds around the world” [7]. Yet the private and public sectors are important: “Sustainable, broad-based change is more likely to be achieved by supporting and strengthening existing community, private sector, and public sector mechanisms for product and service delivery, and by supporting the capacity, quality, and accountability of government institutions” [7].
FFP’s new strategy calls for taking a systems approach to change that emphasizes sustainable long-term gains over unsustainable short-term wins. Even more delightful, in a small meeting at MFAN with Dina Esposito, Director of Food For Peace in November, she announced that they were looking to do pilot funding of an additional three years to typical five-year DFAP development projects. One year would involve collaborative participatory design between partners, communities and FFP, the second additional years being evaluation post-project of sustained and emerging impact!
This is a sea shift that can hopefully withstand political winds. After all, US foreign aid accounts for less than 1% of our federal budget, even though many Americans believe it is over 15% (hence easy to cut)…. but fingers crossed the aid effectiveness value of our work is… Valued.
[4] Food and Nutrition Technical Assistance (FANTA). (n.d.). Effective Sustainability and Exit Strategies for USAID FFP Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
[5] Del Mese, F. (2016, November 16). When aid relationships change: DFID’s approach to managing exit and transition in its development partnerships. Retrieved from https://icai.independent.gov.uk/report/transition/
[8] Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
Jindra Cekan, Ph.D. has used participatory methods for 30 years to connect with participants, ranging from villagers in Africa, Central/ Latin America and the Balkans to policy makers and Ministers around the world for her international clients. Their voices have informed the new Sustained and Emerging Impacts Evaluation, other M&E, stakeholder analysis, strategic planning, knowledge management and organizational learning.
If you don’t find what you are looking for via the search, categories, or posts above, you can go to the Blog page, scroll to the bottom, and click “previous posts” to go through all of the posts (newest–>oldest).
1) Sustainability can be cheap. Far cheaper, in fact to design for sustainability, create feedback loops checking on sustainability through the eyes of our partners and participants, monitor for sustainability than to assume it’ll happen and far cheaper than finding out funds could have had far greater impact if we had valued their voices in the first place.
