by Jindra Cekan | Oct 19, 2018 | Doing Development Differently, Ethiopian Red Cross, European Evaluation Society (EES), ex-post evaluation, Exit strategies, foreign aid, Impact, impact evaluation, International aid, international development, NGOs, OECD, OXFAM, Participants, post-project evaluation, SDGs, Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation, Sustained and Emerging Impacts Evaluations (SEIE)
Assuming Sustainability and Impact is Dangerous to Development
(+ OECD/ DAC evaluation criteria)
We all do it; well, I used to do it too. I used to assume that if I helped my field staff and partners target and design funded projects well enough, and try to ensure a high quality of implementation and M&E, then it would result in sustainable programming. I assumed we would have moved our participants and partners toward projected long-term, top-of-logical-framework’s aspirational impact such as “vibrant agriculture leading to no hunger”, “locally sustained maternal child health and nutrition”, “self-sustained ecosystems”.
INTRAC nicely differentiates between what is typically measured (“outputs can only ever be the deliverables of a project or programme…that are largely within the control of an agency”) and what is not: “impact as the lasting or significant changes in people’s lives brought about by an intervention or interventions” [1]. They continue: “as few organisations are really judged on their impact, the OECD DAC impact definition (“positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended“) allows for long-term changes in institutional capacity or policy change to be classed as impact” [1]. Do we do this? Virtually never. 99% of the time we only evaluate what happened while the project and its results is under the control of the aid implementer. Yet the five OECD/DAC evaluation criteria asks us to evaluate relevance, effectiveness, efficiency (fair enough, this is important to know if a project was good) and also impact and sustainability. So in addition to the prescription to evaluate ‘long-term effects’ (impact), evaluators are to measure “whether the benefits of an activity are likely to continue after donor funding has been withdrawn… [including being] environmentally as well as financially sustainable” [2].
How do we know we are getting to sustained outcomes and impacts? We ask people on the receiving end ideally after projects end. It is dangerous to assume sustainability and impact, and assume positive development trajectories (Sridharan) unless we consistently do “ex-post” project evaluations such as these from our research or catalytic organizations that have done at least one ex-post. At very minimum we should evaluate projected sustainability at end of project with those tasked to sustain it before the same project is repeated. Unfortunately we rarely do so and the assumed sustainability is so often not borne out, as I presented at the European Evaluation Society conference Sustainability panel two weeks ago along with AusAid’s DFAT, the World Bank, University College London and UNFEM.
Will we ever know if we have gotten to sustained impacts? Not unless the OECD/DAC criteria are drastically updated and organizations evaluate most projects ex-post (not just good ones :)), learn from the results and fund and implement for country-led sustainability with the country nationals. We must, as Sanjeev Sridharan tells us in a forthcoming paper embed sustainability into our Theories of Change from the onset (“Till time (and poor planning) do us part: Programs as dynamic systems — Incorporating planning of sustainability into theories of change” (Canadian Journal of Program Evaluation, 2018).*
There are remarkable assumptions routinely made. Many projects put sustainability into the proposal, yet most close out projects in the last 6 months. Rarely do projects take the time to properly phase down or phase over (unlike CRS Niger); many exit ceremonially ‘handing over’ projects to country-nationals, disposing project assets, and leaving only a final report behind. Alternatively, this USAID Uganda CDCS Country Transition Plan which looks over 20 years in the future by when it assumes to have accomplished sustained impact for exit [3]. Maybe they will measure progress towards that goal and orient programs toward handover, as in the new USAID “Journey to Self-Reliance” – we hope! Truly, we can plan to exit, but only when data bears out our sustained impact, not when the money or political will runs out.
As OXFAM’s blog today on the evaluation criteria says, “Sustainability is often treated as an assessment of whether an output is likely to be sustained after the end of the project. No one, well, hardly anyone, ever measures sustainability in terms of understanding whether we are meeting the needs of the present, without compromising the ability of future generations to meet their own need” and “too often in development we evaluate a project or programme and claim impact in a very narrow sense rather than the broader ecology beyond project or programme parameters” [4]. In fact, most ‘impact evaluations’ actually test effectiveness rather than long-term impact. Too rarely do we test impact assumptions by returning 2-10 years later and gather proof of what impacted locals’ lives sustainably, much less – importantly – what emerged from their own efforts once we left (SEIEs)! Oh, our hubris.
if you’re interested in the European Evaluation Society’s DAC criteria update discussion, see flagship discussion and Zenda Offir’s blog which stresses the need for better design that include ownership, inclusivity, empowerment [5][6]. These new evaluation criteria need to be updated, including Florence Etta’s and AGDEN‘s additional criteria participation, non-discrimination and accountability!
We can no longer afford to spend resources without listening to our true clients – those tasked with sustaining the impacts after we pack up – our partners and participants. We can no longer fund what cannot be proven to be sustained that is impactful. We talk about effectiveness and country ownership (which is paramount for sustainability and long-term impact), with an OECD report (2018) found “increases [in[ aid effectiveness by reducing transaction costs and improving recipient countries ownership” [7]. Yet donor governments who ‘tie’ aid to their own country national’s contracts benefit a staggering amount from ‘aid’ given. “Australia and the United Kingdom both reported … 93 percent and 90 percent of the value of their contracts respectively went to their own firms” [7]. It is not so different in the USA where aid is becoming bureaucratically centralized in the hands of a few for-profit contractors and centralized hundreds of millions in a handful of contracts. We must Do Development Differently. We can’t be the prime beneficiaries of our own aid; accountability must be to our participants; is it their countries, not our projects, and we cannot keep dangerously assuming sustained impact. Please let us know what you think…
Footnotes:
[*] This paper is now available at https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/53055
Sources:
[1] Simister, N. (2015). Monitoring and Evaluation Series: Outcomes Outputs and Impact. Retrieved from https://www.intrac.org/wpcms/wp-content/uploads/2016/06/Monitoring-and-Evaluation-Series-Outcomes-Outputs-and-Impact-7.pdf
[2] OECD. (n.d.). DAC Criteria for Evaluating Development Assistance. Retrieved September, 2018, from https://web.archive.org/web/20180919035910/http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm
[3] USAID. (2016, December 6). USAID Uganda Country Development Cooperation Strategy 2016-2021. Retrieved October, 2018, from https://www.usaid.gov/uganda/cdcs
[4] Porter, S. (2018, October 18). DAC Criteria: The Hand That Rocks the Cradle. Retrieved from https://views-voices.oxfam.org.uk/2018/10/dac-criteria-the-hand-that-rocks-the-cradle/
[5] European Evaluation Society Biennial Conference: Flagship Symposia. (2018). Retrieved from http://www.ees2018.eu/1539782596-flagship-symposia.htm
[6] Ofir, Z. (2018, October 13). Updating the DAC Criteria, Part 11 (FINAL). From Evaluation Criteria to Design Principles. Retrieved from https://zendaofir.com/dac-criteria-part-11/
[7] OECD. (2018, June 11). 2018 Report On The DAC Untying Recommendation. Retrieved from http://www.oecd.org/dac/financing-sustainable-development/development-finance-standards/DCD-DAC(2018)12-REV2.en.pdf
by Jindra Cekan | Jan 17, 2017 | 3ie, Accountability, Aid effectiveness, Better Evaluation, CDA, Doing Development Differently, Evaluation, ex-post evaluation, impact evaluation, NGO, Participants, partners, post-project evaluation, SEIE, stakeholders, Sustainability, Sustainable development, Sustained and Emerging Impact Evaluation (SEIE), Sustained Impact Evaluation, USAID
Making up your mind. Prioritizing and making it happen
* As Dr. Martin Luther King Jr. said, "every man must decide whether he will walk in the light of creative altruism or in the darkness of destructive selfishness".
Source: https://shanemcdonnell45.wordpress.com/tag/darkness-into-light/
* Our President, Barack Obama said in his farewell speech, "change only happens when ordinary people get involved and they get engaged, and they come together to demand it".
* OXFAM International demanded changing shocking inequity: "just eight billionaires own the same wealth as the 3.6 billion people who make up the poorest half of humanity".
* Caroline Heider of the World Bank's IEG asked we examine how we evaluate long-term impacts: "current considerations of efficiency, cost savings, or cost-benefit analyses are challenged to take long-term impacts into account".
What do you want to prioritize and demand of international development? In these times of shifting priorities in powerful nations, where politicians are questioning the needs of those whom many of us have been serving, what do you want to demand? What issue do you prioritize, and want to move forward?
I choose to prioritize sustained impact driven by country-nationals. Why? I grew up in large cities, and when I first worked in Africa’s Sahel desert 25 years ago, the herders and farmers making a living from the arid pastures and sandy soil, with wells 100 feet deep astonished me.
Without them, I’d last 3 days out there. They were the experts.
I always assumed we measured ‘sustainable’ development in work with such herders and farmers, but in 2013 I founded Valuing Voices after I began to see how rarely we return to evaluate what remained after our foreign aid projects stopped.
Reviewing thousands of “ex-post” or “post-project” documents in 30 organizations’ public databases, Valuing Voices has found the vast majority of documents only suggested a post-project be done, a small proportion were desk studies and fewer than 1% were original fieldwork post-project evaluations of sustainability. In these 370 post-project (ex-post) evaluations, development workers asked partners and participants what was still standing, showed what succeeded or failed and what unexpected successes participants created themselves from what we left behind.
Returning to learn, consulting our participant-experts seems so common sense as they are the ones that can tell us what we should replicate, adapt or abandon. In 2015 research we found only three World Bank IEG evaluations that asked participants their views in a methodologically clear way (out of 33 post-project PPAR evaluations), and only one was perceived as successful. On the other hand, in 2014, IRIN highlighted Rwanda’s very successful community based nutrition solutions, replete with participant voices. We have found 23 ‘catalytic’ (mostly NGO) organizations having done one or more (ex-) post-project evaluation that include participant input and each of them is filled with excellent lessons for doing ‘development’ well now and after closeout. Yet what are any of these organizations doing differently and why are so few doing more? Why do donors seem to care so little about sustained impact that such studies are so rarely funded by them, and NGOs use private funds? That is what drives me.
A seminal book, Time to Listen asked 6,000 such participant-experts in 20 countries what they wanted foreign aid to look like. “Very few people call for more aid; virtually everyone says they want “smarter” aid…. A majority criticize the “waste” of money and other resources through programs they perceive as misguided or through the failure of aid providers to be sufficiently engaged… [it is] a supply-driven approach that squeezes out the views of the recipients, and a focus on spending – both volume and speed, which undermines aid’s ability to listen, learn and adapt to local contexts.”
Sobering stuff.
While Valuing Voices is not profitable (yet?) and growth is slow, I continue to evaluate and advocate, believing that designing, implementing, monitoring and evaluating for sustained impact by our true clients is key to successful work life well spent.
We need a sustained impact mindset.
We are getting there. Better Evaluation just featured our Sustained and Emerging Impact Evaluation (SEIE) approach as a new theme in evaluation. OXFAM and Save the Children recently wrote “The Power of Ownership: Transforming US Foreign Assistance” (2016). They ask: “country ownership is at the core of effective development… as the United States transitions to a new President and new leadership for development cooperation, how will the next administration build on current successes and chart a path forward?“ I fear the answer, as it takes trust and interest in countries’ capacity to chart their own way forward. USAID (and maybe other donors?) are ready to help. USAID alone has some done some exciting work recently through USAID Forward’s local partners (e.g. Afghanistan has done this in depth) and it has looked at Local Systems. Food For Peace’s strategy includes sustainability.
What will be a priority in 2017 onward? What each of us creates will remain.
The powerful Sidekick Manifesto beautifully proposes this new core belief which we can each espouse, that “Local leaders with local solutions to local problems” will end poverty. We will not.” We can, however, “always be listening, learning and seeking a deeper understanding…” I am delighted to be a sidekick in projects that prioritize participant and partner views, for that is how they end poverty.
What do you want to prioritize and create? What is so vital for you that you must work on it? What has been neglected? What difference do you want to make? GO!
by Jindra Cekan | Dec 12, 2016 | Aid effectiveness, DFID, Doing Development Differently, Evaluation, Exit strategies, Feedback loops, Food for Peace (FFP), International aid, international development, NGO, Sustainability, Sustainable development, Sustained and Emerging Impact Evaluation (SEIE), Sustained Impact Evaluation, USAID, Valuing Voices
Embedding Sustainability Everywhere – All Five Slices Now
It has been a tumultuous year, and next year does not look like we will have much stability as a respite. As domestic concerns grow larger in two huge economies, US and UK, the question of the place foreign aid will play abound in conversations around the world.
However 2017’s transitions transform our work, for now I do have some good news,
1) Sustainability can be cheap. Far cheaper, in fact to design for sustainability, create feedback loops checking on sustainability through the eyes of our partners and participants, monitor for sustainability than to assume it’ll happen and far cheaper than finding out funds could have had far greater impact if we had valued their voices in the first place.
In our work to help our clients and partners fund, design, implement and monitor/ evaluate with sustainability in mind, we created what we hope is a helpful tool (guidance forthcoming).
a) By Designing for Sustainability with those who will sustain them, their financial buy-in and commitment are far higher (see CRS/ Niger [1]), as is advocacy and community buy-in (see new post-project OXFAM/ DRC [2]) and there are indicators the costs of start-up later are more cost-effective.
b) By clarifying Sustainability Indicators we check assumptions about who will do so, how much of a priority our activities are before we scale them up (Federation/ Ethiopian Red Cross). Retrospective post-project sustainability evaluations also enable us to learning from past successes and do better.
c) Sustainability Monitoring and Adaptation involve those pesky but pivotal feedback loops which are vital to understanding if we have gone off the rails or not, especially in terms of unexpected shocks derailing logical frameworks of designed projects. USAID’s nice recent CLA (Collaborating, Learning, Adapting) process includes donor funding for adaptation mid-stream which fosters effectiveness and sustainability [3]. Even lovelier are the Doing Development Differently examples that are often very low-cost and high-effect.
d) Informed Exit, Stakeholder Sustainability Consultation should be done throughout the cycle, at least a year earlier than most projects can begin this (note; not the last few months please). Transitioning for success leverages sunk costs for ongoing results. It includes heavy knowledge management on how the implementers managed information and resources, tracked data, sustained outputs and outcomes which now local partners will need to do, etc (FFP Exit Strategies study, a Czech study, and a new UK Results study shows range of items to consider during and post-transition) [4] [5].
e) Post-Project Sustainability: Our Valuing Voices/ Better Evaluation/ Tufts joint presentation at AEA on SEIE has more on how learning from post-project evaluation lessons can change sustainability of future projects for the better [6]! Further, what capacities and systems have been built in-country to sustain the results after our funding and expertise leaves? What can we do differently?
Can it be done? Demand is rising. I recently presented at a conference about embedding sustainability in programming now, and an enterprising NGO took this idea to heart. They proposed joint design with communities, which is the very bedrock of designing for sustained impact. Kudos!
2) Secondly, donors are willing to pay for sustainability. Sustainability is ensconced in USAID/ Food For Peace’s (FFP) 2016 Strategy and CLA (above) [7]. In the section called New learning and Implications for FFP Programming, they present findings such as “actions that drive big results during the life of the project may actually undermine sustainability in the long run. It raises the question as to whether FFP is willing to accept more modest results in the near term if they can be delivered in a way that will yield more sustainable gains over time….”
They also point us in the direction of country-led development: “Sustained capacity, resources, motivation, and linkages all require a focus on catalysts for change beyond FFP. Facilitative approaches that rely on and strengthen local actors help ensure that resource and knowledge transfers, and the incentives and linkages that support them, will be self-perpetuating beyond project end”. Notably, while UK’s DFID focuses on maximizing impact through Value-for-Money, it is a shorter-term economy, efficiency and effectiveness rather than sustained impact for the end-users and DFID’s exit strategies have recently been critiqued [5].
There is an issue of disincentives for the new administration to heed (if the agricultural lobby for US food exports does not prevail): “In [USAID’s exit] study evaluating how sustainable the results of Title II development programs are 2–3 years after project closure, FFP found that “providing free resources can threaten sustainability, unless replacement of those resources both as project inputs and as incentives has been addressed” [8]. As the Natural Resources section notes, “Whether entirely in the hands of the community or linked to a formal institution, the incentives and resources necessary to maintain a community asset are part of the system that will sustain it. The lack of such systems is visible in rusted irrigation pumps, failed mangrove plantations, abandoned bore wells, eroded dikes, and silted-in fish ponds around the world” [7]. Yet the private and public sectors are important: “Sustainable, broad-based change is more likely to be achieved by supporting and strengthening existing community, private sector, and public sector mechanisms for product and service delivery, and by supporting the capacity, quality, and accountability of government institutions” [7].
FFP’s new strategy calls for taking a systems approach to change that emphasizes sustainable long-term gains over unsustainable short-term wins. Even more delightful, in a small meeting at MFAN with Dina Esposito, Director of Food For Peace in November, she announced that they were looking to do pilot funding of an additional three years to typical five-year DFAP development projects. One year would involve collaborative participatory design between partners, communities and FFP, the second additional years being evaluation post-project of sustained and emerging impact!
This is a sea shift that can hopefully withstand political winds. After all, US foreign aid accounts for less than 1% of our federal budget, even though many Americans believe it is over 15% (hence easy to cut)…. but fingers crossed the aid effectiveness value of our work is… Valued.
Sources:
[1] Cekan, J., PhD, Kagendo, R., & Towns, A. (2016). Participation by All: The Keys to Sustainability of a CRS Food Security Project in Niger. Retrieved from https://www.crs.org/our-work-overseas/research-publications/participation-all
[2] Lindley-Jones, H. (2016, November 16). ‘If we don’t do it, who will?’ A study into the sustainability of Community Protection Structures supported by Oxfam in the Democratic Republic of Congo (DRC). Retrieved from https://policy-practice.oxfam.org.uk/publications/if-we-dont-do-it-who-will-a-study-into-the-sustainability-of-community-protecti-620149
[3] USAID Learning Lab. (n.d.). Collaborating, Learning, and Adapting (CLA)? Retrieved from https://usaidlearninglab.org/faq/collaborating%2C-learning%2C-and-adapting-cla
[4] Food and Nutrition Technical Assistance (FANTA). (n.d.). Effective Sustainability and Exit Strategies for USAID FFP Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
[5] Del Mese, F. (2016, November 16). When aid relationships change: DFID’s approach to managing exit and transition in its development partnerships. Retrieved from https://icai.independent.gov.uk/report/transition/
[6] Cekan, J., Rogers, B. L., Rogers, P., & Zivetz, L. (2016, October 26). Barking Up a Better Tree: Lessons about SEIE (Sustained and Emerging Impact Evaluation). Retrieved from https://valuingvoices.com/wp-content/uploads/2016/11/Barking-up-a-Better-Tree-AEA-Oct-26-FINAL.pdf
[7] USAID. (2016, October 6). 2016–2025 Food Assistance and Food Security Strategy. Retrieved from https://www.usaid.gov/ffpstrategy#:~:text=FFP’s%20new%20strategy%2C%20the%202016,USG)%20food%20assistance%20as%20a
[8] Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
