Accountability: Are we responsible for meeting the SDGs? Yes, personally and by programming for more sustainability
In a 2014 article about mutual accountability for the SDGs, Dr. Paul Zeitz states that “Sustainable development is the most urgent challenge facing humanity. The fundamental question is how the world economy can continue to develop in a way that is socially inclusive, advances human rights, and ensures environmental sustainability.” Today, the UN Secretary General’s report on SDG Progress on progress to meeting them is unveiled. As many of us work in global development, we need to consider our accountability for its findings.
First, Dr. Zeitz points to key aspects to such accountability, of which three are most relevant to sustainability:
- Universal, Voluntary and Commitment-Based Approach
For the SDG agenda to be successful, “Shared and joint commitments by partners from governments, civil society and the private sector can inspire faster and bolder action, can garner enhanced citizen and media attention; and can contribute to the mobilization of resources from internal and external sources.”
Our industry has pushed for greater investments for decades.
- Broad-Based Youth and Citizen Engagement
Local youth- and citizen-driven monitoring and accountability mechanisms are essential for improving budget transparency and service delivery outcomes. If citizens are enabled to pay attention, respond and engage, and then take responsibility and action, then everyone can be empowered to foster an enabling environment for “mutual accountability” and measurable results.”
While most of our work does M&E and listen to ciitzens, even foster voice through civic engagement and feedback loops, we are far from done.
- Call for a Multi-Stakeholder SDG Monitoring and Accountability Mechanism
“… it is more challenging and more complex to ensure ‘mutual accountability’ for results. Given the advances in human cooperation and technology, we know that the SDG era can usher in and foster a new culture of ‘mutual accountability.’
It is here where we fail quite badly in global aid. We rarely talk about our reciprocal accountability with our participants and partners, with the countries themselves. Too often we push money, extract data, claim success and leave abruptly. These SDGs push us to think about “global accountability” and how our actions at work and home need to change to (un)affect others. In a piece on mutual accountability, “Accountability for Development Cooperation under the 2030 Agenda” by Timo Mahn Jones explores “global accountability”, based on “mutual accountability, by which two partners agree to be held responsible for the commitments they voluntarily made.”
- He warns of the danger that “development cooperation stakeholders do not follow through with their commitments, and are not held accountable.
- He suggests that existing donors need to honor ODA/aid targets and new partners, especially the private sector are vital for the implementation of the 2030 Agenda, “to move from billions to trillions” in funding.
- Equally in need of revision is the roles of ‘donor’ and ‘recipient’ as we have entered a new world of “sharing of risk and ‘mutual pain’ where all of us are affected if we do not reach the SDG targets – although unequally.
How do those of us in our industry foster such accountability? We can say, rightly that we have played a role in sectors like health, agriculture, natural resource management through many projects over the last century or more. Absolutely but we know, our projects are piecemeal and often scattered, more short than long in implementation (typically 5 years) and as we rarely return after clcose-out, we do not know how long will results will be sustained. Yet we have tried to do good and there are many public and private players. Is it adding up? .
The Sept 2019 UN report shows promising if mixed results.
Immunizations are increasing, thereby saving millions of lives (SDG3).
There is also good news from investments in renewable energy growth.
However, while those living below minimum living wage is falling on every continent, still, “one child in five lives in extreme poverty” (SDG1).
Hunger, unfortunately, is rising (SDG2), partly due to a 150% increase in “direct economic losses from disasters… over the past 20 years, with losses disproportionately borne by vulnerable developing countries” and 68.5 million people have been forcibly displaced, sometimes from wars donor governments supply arms to or refugees are forced to remain in other developing countries, the UN report tells us, with shrinking refugee funds.
Climate is the most worrisome of the UN reports. Reversing CO2 emissions and fostering the sustainability of climate adaptation and mitigation projects is imperative given today’s UNEP report that includes this graph. Collective accountability is key to reining in CO2, for our emissions are leading to unprecedented ice melt, sea-level rise, and high pollution, not globally sustainable.
While we in ‘development’ could say we affect the earlier SDGs, climate is a global problem, one that each one of us affects with our consumptive actions… or aren’t all of them? Don’t we affect hunger through our food waste and food purchases from afar? Don’t we affect child survival and immunization through our advocacy for aid and even private donations to other health and food security charities?
Some of us disproportionally affect emissions by our wealth, population, or both. The Global Carbon Atlas from 2017 shows how wealthier and more populous countries emit far more than the poor ones. As CO2 rises, climate is affected, storms are more severe, yet the poorest countries have the fewest means to prepare or respond. In their World Disasters Report, The International Federation of the Red Cross found that “Between 1991 and 2010, the impact of recorded disaster events in poor countries resulted in over $840 billion of financial losses. Yet, over the same period, only 0.4% of the $3.3 trillion spent on aid was dedicated to prevention or risk reduction“. Our industry does some Disaster Risk Reduction (DRR) but why have we not advocated to prevent more suffering?
Thinking about “collective accountability” makes this graphic from the UN SDG report uncomfortable, as the burden of climate change falls disproportionately on the poor through economic losses from disasters.
The UN Secretary General states: “The world will soon enter a decade that will be decisive for both current and future generations and for all life on this planet. It is the world’s responsibility and within its power to make it a decade of action and delivery for sustainable development.”
This will require both a clear accountability to them – not just us and ‘our’ projects. This will require us seeing such projects as our continued responsibility to sustain, namely, to design them collaboratively enough, led by local partners ranging from governments and private sector to communities, with the means to sustain what they value. As we return to evaluate sustainability less than 1% of the time, learning opportunities have been scarce to improve current and future projects.
USAID has talked about long-term transparent and accountable investments in “local solutions” partners. Has it worked? Not yet. While President Obama and former United States Agency for International Development (USAID) Administrator Raj Shah promised up to 30% of all contracts would go to ‘local solutions’ that “promote sustainable development through high-impact partnerships and local solutions”, little of that was met, given bureaucracy and the need for fast success, rather than investing more in long-term capacity development of partners. While there seem to be good examples such as Haiti, Afghanistan is a poorer example. While most international non-profits implement projects through local sub-contractors, certainly building their capacity to manage and account for foreign taxpayer dollars spent, like this MSI in Lebanon example, is important. But if we extend the measure of ‘success’ beyond our project implementation, then policies and programming needs to change to sustain capacity and implementation post-exit (INTRAC report). Too often we still exit when funds are spent. USAID’s new Journey to Self-Reliance does promise to listen, to “ support partners to become self-reliant and capable of leading their own development journeys.“
It requires listening to those whom our aid, aids. Time to Listen talks movingly of a desire for collaboration during design, implementation, monitoring and evaluation. It requires we take the time to listen, have an openness to learn from our partners and participants about what they could – and could not- sustain and why. We need to do so before, during and well-after projects close, learn quickly and do better and be collectively accountable for longer. It requires seeing such people as real experts, not abstractions. How? Listening to those living with hunger and climate change is vital. While we read about Amazonian fires threatening over 2 million acres of rainforest, contextualizing statistics with stories that illustrate that those whose Development Goals donors are ‘sustaining’ know best what works there. In this case, understanding the range of why the Amazon rainforest reserves are endangered is important for design, so that better approaches to achieving SDGs become a driving force to change all our lives. We need to be accountable to them.
Let me know your thoughts on bringing the SDGs to our work and lives…
Presenting Lessons on (post-project) Sustained and Emerging Impact Evaluations from the U.S. AEA Conference
Dear readers, attached please find the Barking up a Better Tree: Lessons about SEIE Sustained and Emerging Impact Evaluation presentation we did last week at the American Evaluation Association (AEA) conference in Atlanta GA . I had the pleasure of co-presenting with Beatrice Lorge Rogers PhD, Professor, Friedman Nutrition School, Tufts University (aka the famous Food for Peace/ Tufts Exit Strategy study ), Patricia Rogers PhD, Director, BetterEvaluation, Professor, Australia and New Zealand School of Government (where we recently published guidance on SEIE ) and Laurie Zivetz PhD, International Development Consultant and Valuing Voices evaluator.
We integrated our presentations from Africa, Asia and Latin America into this fascinating overview:
1.Sustained and Emerging Impact Evaluation: global context
2.SEIE: definitions and methods
3.Case studies: findings from post-project evaluations
4.Designing an SEIE: Considerations
5.Q&A — which fostered super comments, but since you couldn’t come, please tell us what you think and what questions you have…
There are amazing lessons to learn about design, implementation, M&E from doing post-project evaluation. We have also grown in appreciating that sustainability can be tracked throughout the project cycle, not just during post-project SEIE evaluation.
We’ll be building this into a white paper or a … (toolkit? webinar series? training? something else?). What’s your vote ___? (I know in this US election season, so… :)).
What would you like to get to support your learning about Sustained and Emerging Impact Evaluations? Look forward to hearing from you- Jindra@ValuingVoices.com
The full presentation is available here:
 Cekan, J., Rogers, B. L., Rogers, P., & Zivetz, L. (2016, October 26). Barking Up a Better Tree: Lessons about SEIE (Sustained and Emerging Impact Evaluation). Retrieved from https://valuingvoices.com/wp-content/uploads/2016/11/Barking-up-a-Better-Tree-AEA-Oct-26-FINAL.pdf
 Food and Nutrition Technical Assistance (FANTA). (n.d.). Effective Sustainability and Exit Strategies for USAID FFP Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
 Zivetz, L., Cekan, J., & Robbins, K. (2017, May). Building the Evidence Base for Post-Project Evaluation: Case Study Review and Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf
Maximizing what we've got… Time is now!
We had a stirring conversation here in D.C. with someone very knowledgeable about sustainability; this person is a strong proponent of local ownership of all development. They also said vehemently, why evaluate the sustainability of projects after closeout; we all know what that will show! What was implied is that our system of international development and aid is so flawed, so broken, that the inevitable result of not focusing on local ownership as the fundamental basis for our work means Nothing. Will. Be. Left…. All. Is. Lost.
We disagree. Our development industry does some good, some bad, and is ever-changing (albeit slowly). Billions of dollars each year are spent trying to improve people's lives and livelihoods around the world, and we've seen great good be done. While. We. Remain.
We know far, far less about what remains after our projects end because less than 1% of the time we return post-project (ex-post) to evaluate anything.
Our problem with time begins with fixed timelines within projects that say we have 1, 3, 5 years to get to success. They work with participants and partners who need to make substantial changes to how they use their resources and beliefs over a relatively short time of a few months to a few years. We expect immediate results from them, changing how they farm (use new seeds, new methods, new ways of interacting with markets) and save money (learn new concepts of profit and interest, repayment and re-lending), and improve their health and that of their families (get prenatal exams, vaccinate your children, exclusively breastfeed without adding water or tea). Everyone in our projects is 'on the clock' from the donor and implementer to partners and participants. This clock ticks down irrevocably as project closeout looms, promised-successes-to-donors at hand or a mirage in the distance. We assume sustained results.
How many of us have ever gone on a diet? How many have learned a new language? How many of us have transferred jobs and had to learn new skills on the job? How quickly have we managed to do all that successfully, all at once?! Probably many. How many of you have had to do this on a fixed timeline? Were you successful when there was a limited, fixed time and you did not set your own pace?
It takes time to implement projects well enough to ensure that most participants ‘got it’, not just the 'early adopters'. It takes time to hand over projects so well that our partners and participants are ready to take over at least some of what we worked so hard to transfer. It took leadership and staff two years in the very successful participatory USAID/ Food For Peace food security project by CRS Niger that was a continuation of similar programming for 15 years. It also takes time to pass for conditions to be ready for our return, to isolate what people could self-sustain from what the project supplied, to learn what was so well designed and implemented during projects that to 'took root' in people's lives, that they have made it their own. We estimate optimal evaluation time is 2-7 years after closeout. Valuing Voices also believes we should not just evaluate the sustainability of outputs and outcomes of what we put in place that we thought they would continue, and the sustained impact of those cumulative investments, but also the emerging, unintended new activities and impacts we never imagined people would innovate from our projects. We are doing just such evaluations in Zimbabwe and Uganda now and hope to do and catalyze much more fieldwork around the world.
And why does it matter? Why shouldn't we write off our time-limited donor-funded projects? Because:
1) It's all we've got. Our current development system is not going anywhere soon, and there is success to learn from.
2) We need to quickly learn from what worked sustainably best and stop wasting time and resources on what we refuse to admit fails because we are too scared to return to see. Go back with the intention to learn what does and focus on doing more of what works.
3) Such analysis – and design of new projects – must have country ownership as a centerpiece throughout the project cycle assumptions, but to throw out decades of good work simply because we are just learning the value of country ownership is foolish.
Finally, here's a lovely example from Brazil of how local, participation (and yes, as my colleague thought, local ownership) works best. And. It. Takes. Time.
"Our results also show that Participatory Budgeting’s influence strengthens over time… Participatory Budgeting’s increasing impact indicates that governments, citizens, and civil society organizations are building new institutions… cities incorporate citizens at multiple moments of the policy process, allowing community leaders and public officials to exchange better information." How often do we return to do what are called longitudinal reviews of our work abroad, using the same rigorous standards we evaluate our domestic projects? Not often. Shouldn't that change?
Only by working together, honoring the value of our participants, that they deserve the same chance at change that we take for granted will things change. We must value both the voices of our participants and our own expertise for development to improve for true aid effectiveness…. Let us begin anew!
This blog post by Jindra Cekan and Laurie Zivetz of Valuing Voices discusses the need for post-project impact evaluations. An area that needs more attention, BetterEvaluation will be working with Valuing Voices over the next couple of months to expand the available information and resources for this type of evaluation.
The question of what is impact is one that has been coming up again and again recently. Simon Hearn’s recent blog on BetterEvaluation (and his accompanyingMethods Lab paper) began unravelling the many ways we use the term. Simon notes that in addition to the range of definitions of impact, our industry has a growing appetite to demonstrate the impact of our international development projects with evidence. In a blog post last year, Developing a Research Agenda for Impact Evaluation, Patricia Rogers and Greet Peersman note that current impact evaluations tend to look “only at relatively short-term, intended direct effects” and attention to unintended or unexpected (emerging) impacts of our projects remains undeveloped. While organisations such as 3ie and J-Pal have risen to prominence for their work on impact evaluation, the different views surrounding what constitutes a sustained impact in these organisations are important (read the Valuing Voices blog post on the differences between our definition and theirs, and the implications of this).
The preoccupation with short-term impacts represents a serious gap in evaluation practice, theory and design. Returning 2-10 years post-project offers an opportunity to assess whether improvements- such as those in organisational efficiencies, community infrastructure, knowledge, behaviour change, livelihoods- that may have been planned and shown at the end of the project cycle, actually endured. It also provides a chance to understand whether other unintended impacts emerged over time as a result of a project or program or participants’ efforts in the intervening years after the project ended. How can we claim we are doing “sustainable development” if we do not return to assess sustainability that we either envisioned or that emerged from people’s own efforts?
This gap presents serious challenges to our ability to be genuinely accountable or to learn and improve – also fundamental to effectively meeting the Sustainable Development Goals. We believe that funding, doing and learning from Sustained and Emerging Impact Evaluations (SEIE) needs to become a standard part of the program cycle. This is key to doing sustainable development. Building an evidence base and generating lessons across projects and sectors, and from design to implementation, monitoring to evaluation promise benefits in terms of better design and implementation of projects, more robust, accountable partnerships, and a path to foster true country ownership.
The numbers are staggering. The OECD tells us that over $5 trillion in foreign aid has been given since 1970 (excluding private donations). Over $1.5 trillion of that has been given by the EU and USAID since 2000 alone and even last year $137 billion was spent worldwide. At Valuing Voices, we have reviewed hundreds of documents tagged as ‘post-project’ and ‘ex-post’ evaluation from nearly 30 international development and implementer organisations. Most of these documents only suggest that such evaluations be done; a few are desk studies. Only a handful (some 1% of all projects actually evaluated) have actually returned to the field post-project, and systematically asked participants and partners what they had sustained themselves 2-10 years after the project ended. Even fewer of these have delved into why some outcomes were sustained, and asked questions about what other impacts or innovations emerged. It seems that much of the ‘sustainability’ focus is on the continuation of funding for projects, rather than assessing impact, and including the efforts and hopes of our stakeholders. Curiosity about what might be done better seems to stop at the end of the resourced project cycle.
Valuing Voices Full-Cycle Development
The 19 organisations that we have found that have done one or more such evaluations deserve to be strongly celebrated as catalytic learning organisations. They are at the vanguard of redefining ‘success’ through our participants’ and partners‘ eyes and improving our programming accordingly. We have piloted this approach in a number of post-project sustained impact evaluations that found important lessons for design with follow-on funding as well as for monitoring and implementation. We have documented the tip of the sustained impact iceberg, including an excellent study by USAID that highlighted four essential elements for successful exit (resources, capacity, linkages and motivation). We discuss some of the assumptions many of us make about barriers to doing more such evaluations, Including needed improvements to our M&E systems, how to forge stronger partnerships and country ownership.
We ask you to join Valuing Voices and BetterEvaluation in advocating creating new success measure in international development: sustained and emerging impact evaluations. We ask you to fund and do these SEIEs and share them for inclusion in our database. Such learning should not only be standard in our industry, but the lessons from them to be a globally available resource. We envision creating regional centres of analysis of learning and country-ownership, with data available long after any project ends. We ask you to help us put participant feedback at the centre of all of our work so we can better learn about sustained and emerging impact, foster design for success, implement for national ownership of projects long after we are gone. That is why we do what we do, isn’t it?
Have you been involved in a post-project evaluation of sustained and emerging impacts? Or do you know of any good examples or have some advice on conducting one? Let us know in the comments below or send us a message! We’ll be developing a new BetterEvaluation option for this over the coming months so your comments and examples will help contribute to this.
Learning about Sustainability and Exit Strategies
from USAID’s Food Assistance Projects
USAID overall and Food for Peace (FFP) specifically have become far more progressive in the Obama Administration and under Administrator Rajiv Shah, with a much greater focus on accountability and results. Those of you unfamiliar with USAID’s Food For Peace will learn it has been a large channel of international assistance for over 60 years and is not a small funding instrument. For 2016 alone, they have proposed spending $1.75 billion to feed 47 million people through humanitarian and development programs implemented by non-profits, for-profits and the UN’s World Food Program . Given this scale of resources, it is highly surprising that while many documents in their archive ask for post-project evaluation and there are a handful of desk reviews, they have done only two actual ones with new fieldwork in the last 30 years (these four countries and a recently published one on Uganda, see Catalysts page). This recent and excellent 2015 synthesis report by authors Rogers and Coates is presented below .
Commissioned by USAID, Tufts University and FHI360 have done a remarkably thorough two to three year post-project evaluation of four (Title II) food-assisted programs containing 12 projects in Bolivia, Honduras, India, and Kenya that closed out in 2009. The methodologies used are clearly outlined (itself a boon to our fledgling field) as are limitations and comments on context, findings and recommendations. It was no small feat to compare activities across four countries and so many sectors (some of which were supported by provision of US food aid resources, others with in-kind or cash inputs): maternal and child health and nutrition; water and sanitation; agriculture, livestock, and rural income generation; natural resource management; school feeding; and micro-savings and loans. Also this covered many implementers, from CARE, ADRA and Save the Children to World Vision, CRS and Feed the Hungry.
In this document, Dina Esposito, the Director of FFP states “We commissioned this report with the objective of determining what factors enhanced the likelihood of sustained project benefits, in order to improve our guidance for future food assistance development projects.… FFP development projects are designed to reduce the long-term need for food assistance by strengthening the capacity of developing societies to ensure access to nutritious food for their most vulnerable communities and individuals, especially women and children. The study team looked at 12 FFP development projects across four countries and asked not only what was achieved by each project’s end?, but also, what of those achievements remained one year after project close-out? and two years after? This rigorous, retrospective approach is not widely done, but is essential if we are to understand the true impacts of our investments. To be effective, development projects must result in changes that last beyond the duration of the project themselves.”
Process and findings:
The researchers compared baseline, midline and endline evaluations and exit strategy documents to new mixed-method data collection. There were four main findings:
1) Impact vs. Sustainability trade-offs: Evidence of project success at the time of exit (as assessed by impact indicators) did not necessarily imply sustained benefit over time. Just because projects were deemed successful at exit does not mean that those continued after closeout. “Moreover, the study found that focusing exclusively on demonstrating impact at exit may jeopardize investment in longer-term sustainability.” Valuing Voices found the same in Ethiopia in research done in 2013 .
2) Preconditions to successful sustainability: In addition to an ongoing source of resources, good technical and managerial capacity, and sustained motivation of participants and partners, linkages to governmental organizations and/or other entities were key to continuity and sustainability of outcomes and new impacts. “No project in this study achieved sustainability without [the first] three of them in place before the project ended,” and linkages between community partners and the public/private sector were critical for handover (Figure 1, below). Further, a gradual transition from project-supported activities to independent operation was important for sustainability. “Sustainability was more likely when projects withdrew gradually, allowing community-based organizations to develop the capacity to operate independently.”
3) Free resources can threaten sustainability, unless replaced while there is no one-size fits all for resources:
Using incentives has costs. “Free supplementary food in maternal and child health and nutrition projects or free marketing services in agriculture projects created expectations in many projects that could not be sustained once resources were withdrawn”. Valuing Voices found the same in research in Niger (report imminent). But other financing options, free health care or fee for service are still unsystematically studied regarding fostering sustainability in differing sectors.
4) External factors (climate, economy) can affect sustainability: The operating context and exogenous shocks (e.g., economic, legal and climatic) also affected the sustainability of project benefits, positively or negatively.
Most tellingly, the authors warned that “sustainability plans cannot be based on the hope that activities and benefits will continue in the absence of the key factors identified in this study.” Throughout the report and in pending country-specific studies, they outlined the assumptions that projects made about sustainability in order to exit and closeout, which were variably disproved, such as:
- Community health workers would continue to provide services although without remuneration,
- Households could continue to access nutritious food from their own (increased) production or purchases and have time, and know how to prepare such food,
- Farmers will pay for inputs with profits from increased production and commercialization and can meet the quantity and quality requirements of long-term contracts
- Community members will recognize the tangible benefit of Natural Resource Management activities and will be motivated to continue them without further inputs or remuneration
- Water committees will have sufficient administrative capacity and resources to manage their budgets effectively
- Community-based organizations have strong institutional capacity
- Partner organizations will continue to provide teacher training
- Government will have the resources and commitment to support future needs
The country studies with detailed findings are still forthcoming but these examples may illustrate the range of sustainability. There were some very well-sustained positive results in Food Production (India by area) and Child Health Growth Monitoring (Bolivia by consortium implementers) between baseline or enline and followup 2-3 years later:
As well as some far more mixed or negative results in examples across all the Water and Sanitation projects:
And far less stellar results in Maternal Child Health’s Community Health Workers (Kenya):
The authors recommended not only ensuring resources, capacity, motivation and linkages are present before exiting but also institutionalizing sustainable approaches to project design and evaluation including in solicitations and applications, project assessments, project management and knowledge management. They also recommended not only phasing down exit but also extending more such evaluations beyond the 5 years of implementation and assessing impacts as long as 10 years after. This requires some sizeable revisions to how development is done at Food For Peace.
All of these findings recommendations are near and dear to those of us at Valuing Voices. We strongly commend Food For Peace and ask for many more such studies, for unless we know what worked best and why, how do we know what to design next together with our partners and participants for real sustainability?
 InterAction. (2015). Choose to Invest 2016: Food For Peace Title II. Retrieved from https://web.archive.org/web/20150307160559/https://www.interaction.org/choose-to-invest-2016/food-for-peace-title-II
 Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
 Cekan, J., PhD. (2014, April 7). Evaluation of ERCS/Tigray’s “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity”. Retrieved from http://adore.ifrc.org/Download.aspx?FileId=147802&.pdf