Living in a Well-meaning Lie: Valuing all Voices? – The Solutions Journal

Living in a Well-meaning Lie: Valuing all Voices?

By Jindra Cekan


Solar panels in Cap Haitien, Haiti.

Poor villagers like Edith, Aminata, Rituu, and Juan don’t appear much on the nightly news.  You might never know they exist unless you stop and read your mail from some charity asking you to help them.  On the brochures, they can look scared or sad; maybe surrounded by their thin children, with a parched land or dying animals behind them. Our foreign aid programs should be helping them, but are they?

 

I have met these people while they are working in their fields, growing corn and peas, millet and coffee, raising their chickens and goats. I have talked to them outside of health centers where they have brought their babies to be vaccinated or their parents for medical care.  I got to know them when I interviewed them under the big tree in the middle of their village, or in empty school rooms, asking them what they need from us, and how we can design foreign aid projects to better help them.

 

I have worked in international development as a technical expert in project design and monitoring and evaluation for international non-profits such as Catholic Relief Services, Save the Children, the Red Cross, and many others including the Bill and Melinda Gates Foundation and the US Agency for International Development (USAID). I estimate that I have designed and evaluated over 200 projects in 28 countries over the last 29 years. I have felt lucky to do this work, and foreign aid does achieve some very good work while we are there: helping farmers to farm better, or helping men and women to care for their family’s health, wealth, and future with new knowledge, tools, and items they need for daily living. My colleagues do wonderful work as well, in hard conditions, within countries with few resources, and for donors with unrealistic expectations of how much can be done well in short timeframes.  In 2010, USAID stated that they would aim for 30 percent of funding to be spent by national partners under USAID Forward.1 This is an excellent step toward the country-led development that the Paris Accords promised, yet as of 2016, there is no list of local partners, other than a handful of examples.2 The only ‘country partners’ list posted to the USAID website includes 80 organizations doing programming via USAID in Afghanistan alone, 55 of which are American firms, four US agencies, nine Afghan government-affiliated organizations, six foreign governments, six UN agencies, and two MENA firms.3 Not quite the national civil-society-NGO partners we envisioned in 2010. Under the new U.S. administration, these are likely to shrink even more as the 0.5 percent of our GNP we allocate to foreign aid is redirected inward—that much more reason to make it as sustained as possible.  European aid as well as other rising world nations need this approach just as much.

Large parts of international aid system remain broken. We design too many projects outside of the countries themselves. We have fixed funding and leave in pre-set times rather than when participants are actually ready to take over. We ‘handover’ without partnering throughout the whole project so that partners can determine what they are able to sustain. Even worse, we leave and do not look back to learn from our Ediths and Juans after our projects have closed.  Sometimes, we disparage their knowledge, and at other times we don’t make enough time to ask but wish we did. Mostly, our aid industry is designed around measuring success while we intervene, and then abruptly leave because funding ended. Yet development is, as international evaluator Ian Davies says, “A process, not a result.”

 

Our policies say we are doing “sustainable development”, that we are helping our ‘beneficiaries’ (really our partners and participants) feed themselves over the long-term, that our projects are almost all successful, and that all we need to do is to scale up the great projects out there.  But the numbers prove we are not, in fact, achieving sustainable development. Nine times out of ten, we rarely go back to talk to our participants and partners after our project end, and we move on.

 

The numbers are staggering:

  • Of the US$5 trillion dollars of international foreign aid spent since 1945, we have evaluated the long-term sustainability far less than one percent of the time.
  • Since 2000, for example, USAID and the Millennium Challenge Corporation have only done three such evaluations apiece, yet they spent well over US$300 billion.
  • The EU evaluated only a few dozen of its projects and programs, in spite of spending US$1.5 trillion in the last 15 years. The United Nations Development Program may do up to six a year, and the World Bank more, but how often do any of them talk to project participants and design anew based on what we learned that succeeded and failed?
  • The Japan International Cooperation Agency, and to some degree EU bilateral countries (through the Organization for Economic Cooperation and Development), have evaluated the sustainability of over 300 projects.
  • Despite this, tens of thousands of new projects are launched every year.
Cekan 2
Satellites atop homes in a slum in Tigray, Ethopia.

 

This is why I founded Valuing Voices—to analyze what little we know to make development better.4 Not to destroy international development, but to change how we fund, design, and implement it. We need to design for sustainability of the activities by the country nationals themselves, rather than designing for results we can show to get more funding. We also must jointly implement, monitor, and evaluate our projects so countries can continue after we leave. Smaller organizations can do even simple activities, designing projects based on what the participants feel they can self-sustain, and partnering with those who will take over while they are still there.

 

Having spread the word for the last three years, to mostly little response, I now turn to you, readers.  Our analysis – and a wonderful 2012 book, Time to Listen by Mary Anderson and Dayna Brown, shows that sometimes when our projects partner with country nationals, their people become – and stay – better off.5They want to be engaged, yet our very structure of delivering aid prevents this. Often we are not there long enough to make a lasting difference, or we invest scarce time on untested innovations that work in some places but don’t in others. Even worse, sometimes we design activities so badly that villages are left with irrelevant technology and trainings, wasted funding, and lost hopes. At other times, there are successes as well, but not returning robs us of the chance to replicate those. We do ‘impact evaluations,’ but only on successes during project implementation, and not on what people can self-sustain after we leave. Our vision is so limited. Our well-meaning self-interest blinds us.

 

Across the board, our development projects make one massive and incorrect assumption that once we ‘handover’ the project, the local government, community, and households have the means to sustain our multi-million dollar investments.6 We assume that technical knowledge will still be locally available to the villagers, that inputs like seeds and tools, data and vaccines will be accessible both physically and financially, that the government staff have the means to get to villages or that new NGOs and donors will appear to fill the gaps. The Huffington Post has stated, “as long-term projects and action-plans are established, more investment must go into financing locally designed solutions and projects that ensures ownership is placed back into local communities.”7 While more project have begun using feedback loops of listening to participants during implementation, virtually all good work stops when project funding stops.

 

Don’t we want development to be sustained after resources leave, and the opinions of these aid recipients to be heard? Don’t we want the next project to address the needs better?  Don’t we, as taxpayers, want to demand that agencies using our tax dollars learn what is really sustainable and what is not?  And shouldn’t we demand that all projects costing more than US$1 million over the past 10 years be examined now for lessons learned by sector (agriculture, health, credit, education, etc) and region? Shouldn’t post-project sustainability evaluations be included in all new projects?  Don’t our participants and partners deserve the dignified futures they hope for, our creating channels for their voices that enable them to evaluate us and teach us how we can help them to be successful?

 

In fact, a radical Foreign Aid Transparency Act was just passed in the U.S. in June 2016.8 The bill calls for the President, within 18 months of enactment, to “set forth guidelines…for the establishment of measurable goals, performance metrics, and monitoring and evaluation plans that can be applied with reasonable consistency to covered United States foreign assistance.” These include ‘ex-post’ (sustainability) evaluations, and “can have enormous value when it comes to making programming and budgeting decisions.”  Yet while there is a call for guidance to be developed, no funding came along with this bill.  Without the funds to make this happen, this may be more ‘window dressing’ for sustainable development than excellent policy.

Cekan 3
A small enterprise in Cap Haitien, Haiti.

 

There is some hope coming from the corporate sector. While impact investors are often more focused on return from emerging economies than fostering sustainable development, corporate social responsibility is building bridges in lovely ways.  Tsikululu Social Investment of South Africa has thought about what advice to give to the companies they advise on such investments, as well as exiting from them.9

 

We argue that our budgeting needs a basic business metric: Return on Investment. In a time of huge demands on our resources worldwide from refugee flows, terrorism and climate change, we currently do little or no analysis of:

  • How much actual investment: What percent of allocated funds went to the activities that benefitted the partners and participants themselves, rather than being used as overhead for operations?
  • How much return: What is the value of what remains used 3-10 years after we leave? What was the value-added that communities and other funders (including the national governments themselves) who followed catalyzed based on our earlier investments? What were unexpected new results that emerged?

 

We envision a beautiful future, one where Edith, Aminata, Rituu, and Juan and their local partners are at the center of development.  We imagine a world in which we listen to what people in need can sustain for themselves. Through these approaches data is shared widely on what has worked best and why; aid projects invest in country-systems and staff that boost their ability to self-sustain; and only sustainable projects are designed and funded that foster country-led development. The global adoption of the Sustainable Development Goals finally puts the focus on all we can do to foster sustainability of our work.

 

By valuing voices and focusing on sustainable solutions for excellent impacts, this will promote truly sustainable development from our aid organizations, government and non-governmental alike. We have much to learn, and there is not a moment to waste before we start Valuing Voices of those we serve and partner with on country-led development.

 

References

  1. USAID Forward. USAID [online] (2017). https://www.usaid.gov/usaidforward.
  2. In-country Partners. USAID [online] (2016). https://www.usaid.gov/partnership-opportunities/in-country-partners.
  3. Implementing Partners. USAID [online] (2017). https://www.usaid.gov/afghanistan/implementing-partners.
  4. Valuing Voices [online]. https://valuingvoices.com/.
  5. Jacobs, A. Time to Listen by Dayna Brown and Mary B Anderson. NGO Performance [online] (December 4, 2012). https://ngoperformance.org/2012/12/04/time-to-listen-by-dayna-brown-and-….
  6. Cekan, J. What happens after the project ends? Lessons about funding, assumptions and fears (Part 3). Valuing Voices [online] (February 29, 2016). https://valuingvoices.com/what-happens-after-the-project-ends-lessons-abo….
  7. Zuabi, V. Investing in Locally Designed Solutions for Syria and the Middle East. The Huffington Post [online] (May 24, 2016). http://www.huffingtonpost.com/vanessa-zuabi/investing-in-locally-desi_b_….
  8. Ingram, G, Miles, C & Veillette, C. The Foreign Aid Transparency and Accountability Act is Law! Now What? Modernizing Foreign Assistance Network [online] (August 2, 2016). http://modernizeaid.net/2016/08/foreign-aid-transparency-accountability-….
  9. Cekan, J. Towards responsible donor exiting strategies and practices: Reblog from Tshikululu. Valuing Voices [online] (October 8, 2016). https://valuingvoices.com/towards-responsible-donor-exiting-strategies-an….

Source: Living in a Well-meaning Lie: Valuing all Voices? – The Solutions Journal

Making up your mind. Prioritizing and making it happen

Making up your mind. Prioritizing and making it happen

 

* As Dr. Martin Luther King Jr. said, "every man must decide whether he will walk in the light of creative altruism or in the darkness of destructive selfishness".

 

 

 

 

 

 

 

 

Source: https://shanemcdonnell45.wordpress.com/tag/darkness-into-light/

* Our President, Barack Obama said in his farewell speech, "change only happens when ordinary people get involved and they get engaged, and they come together to demand it".

* OXFAM International demanded changing shocking inequity: "just eight billionaires own the same wealth as the 3.6 billion people who make up the poorest half of humanity".

* Caroline Heider of the World Bank's IEG asked we examine how we evaluate long-term impacts: "current considerations of efficiency, cost savings, or cost-benefit analyses are challenged to take long-term impacts into account".

 

What do you want to prioritize and demand of international development? In these times of shifting priorities in powerful nations, where politicians are questioning the needs of those whom many of us have been serving, what do you want to demand? What issue do you prioritize, and want to move forward?

 

I choose to prioritize sustained impact driven by country-nationals. Why? I grew up in large cities, and when I first worked in Africa’s Sahel desert 25 years ago, the herders and farmers making a living from the arid pastures and sandy soil, with wells 100 feet deep astonished me.

 

Without them, I’d last 3 days out there. They were the experts.

 

I always assumed we measured ‘sustainable’ development in work with such herders and farmers, but in 2013 I founded Valuing Voices after I began to see how rarely we return to evaluate what remained after our foreign aid projects stopped.

 

Reviewing  thousands of “ex-post” or “post-project” documents in 30 organizations’ public databases, Valuing Voices has found the vast majority of documents only suggested a post-project be done, a small proportion were desk studies and fewer than 1% were original fieldwork post-project evaluations of sustainability. In these 370 post-project (ex-post) evaluations, development workers asked partners and participants what was still standing, showed what succeeded or failed and what unexpected successes participants created themselves from what we left behind.

 

Returning to learn, consulting our participant-experts seems so common sense as they are the ones that can tell us what we should replicate, adapt or abandon.  In 2015 research we found only three World Bank IEG evaluations that asked participants their views in a methodologically clear way (out of 33 post-project PPAR evaluations), and only one was perceived as successful.  On the other hand, in 2014, IRIN highlighted Rwanda’s very successful community based nutrition solutions, replete with participant voices.  We have found 23 ‘catalytic’ (mostly NGO) organizations having done one or more (ex-) post-project evaluation that include participant input and each of them is filled with excellent lessons for doing ‘development’ well now and after closeout. Yet what are any of these organizations doing differently and why are so few doing more? Why do donors seem to care so little about sustained impact that such studies are so rarely funded by them, and NGOs use private funds? That is what drives me.

 

A seminal book, Time to Listen asked 6,000 such participant-experts in 20 countries what they wanted foreign aid to look like. “Very few people call for more aid; virtually everyone says they want “smarter” aid…. A majority criticize the “waste” of money and other resources through programs they perceive as misguided or through the failure of aid providers to be sufficiently engaged… [it is] a supply-driven approach that squeezes out the views of the recipients, and a focus on spending – both volume and speed, which undermines aid’s ability to listen, learn and adapt to local contexts.”

Sobering stuff.

 

While Valuing Voices is not profitable (yet?) and growth is slow, I continue to evaluate and advocate, believing that designing, implementing, monitoring and evaluating for sustained impact by our true clients is key to successful work life well spent.

 

We need a sustained impact mindset.

 

We are getting there. Better Evaluation just featured our Sustained and Emerging Impact Evaluation (SEIE) approach as a new theme in evaluation. OXFAM and Save the Children recently wrote “The Power of Ownership: Transforming US Foreign Assistance” (2016). They ask: “country ownership is at the core of effective development… as the United States transitions to a new President and new leadership for development cooperation, how will the next administration build on current successes and chart a path forward?“  I fear the answer, as it takes trust and interest in countries’ capacity to chart their own way forward.  USAID (and maybe other donors?) are ready to help. USAID alone has some done some exciting work recently through USAID Forward’s local partners (e.g. Afghanistan has done this in depth) and it has looked at Local Systems. Food For Peace’s strategy includes sustainability.

 

What will be a priority in 2017 onward? What each of us creates will remain.

 

The powerful Sidekick Manifesto beautifully proposes this new core belief which we can each espouse, that “Local leaders with local solutions to local problems” will end poverty. We will not.” We can, however, “always be listening, learning and seeking a deeper understanding…” I am delighted to be a sidekick in projects that prioritize participant and partner views, for that is how they end poverty.

 

What do you want to prioritize and create? What is so vital for you that you must work on it?  What has been neglected?  What difference do you want to make?  GO!

Maximizing what we’ve got… Time is now!

Maximizing what we've got… Time is now!

 

We had a stirring conversation here in D.C. with someone very knowledgeable about sustainability; this person is a strong proponent of local ownership of all development. They also said vehemently, why evaluate the sustainability of projects after closeout; we all know what that will show!  What was implied is that our system of international development and aid is so flawed, so broken, that the inevitable result of not focusing on local ownership as the fundamental basis for our work means Nothing. Will. Be. Left…. All. Is. Lost. 

 

We disagree. Our development industry does some good, some bad, and is ever-changing (albeit slowly). Billions of dollars each year are spent trying to improve people's lives and livelihoods around the world, and we've seen great good be done. While. We. Remain.

 

We know far, far less about what remains after our projects end because less than 1% of the time we return post-project (ex-post) to evaluate anything.

 

Our problem with time begins with fixed timelines within projects that say we have 1, 3, 5 years to get to success. They work with participants and partners who need to make substantial changes to how they use their resources and beliefs over a relatively short time of a few months to a few years. We expect immediate results from them, changing how they farm (use new seeds, new methods, new ways of interacting with markets) and save money (learn new concepts of profit and interest, repayment and re-lending), and improve their health and that of their families (get prenatal exams, vaccinate your children, exclusively breastfeed without adding water or tea). Everyone in our projects is 'on the clock' from the donor and implementer to partners and participants. This clock ticks down irrevocably as project closeout looms, promised-successes-to-donors at hand or a mirage in the distance. We assume sustained results.

 

How many of us have ever gone on a diet? How many have learned a new language? How many of us have transferred jobs and had to learn new skills on the job? How quickly have we managed to do all that successfully, all at once?!  Probably many. How many of you have had to do this on a fixed timeline? Were you successful when there was a limited, fixed time and you did not set your own pace?

Timeline_Wylio7739861570_ef1a5c745f_m

https://www.flickr.com/photos/psd/7739861570

 

It takes time to implement projects well enough to ensure that most participants ‘got it’, not just the 'early adopters'. It takes time to hand over projects so well that our partners and participants are ready to take over at least some of what we worked so hard to transfer. It took leadership and staff two years in the very successful participatory USAID/ Food For Peace food security project by CRS Niger that was a continuation of similar programming for 15 years.  It also takes time to pass for conditions to be ready for our return, to isolate what people could self-sustain from what the project supplied, to learn what was so well designed and implemented during projects that to 'took root' in people's lives, that they have made it their own.  We estimate optimal evaluation time is 2-7 years after closeout. Valuing Voices also believes we should not just evaluate the sustainability of outputs and outcomes of what we put in place that we thought they would continue, and the sustained impact of those cumulative investments, but also the emerging, unintended new activities and impacts we never imagined people would innovate from our projects.  We are doing just such evaluations in Zimbabwe and Uganda now and hope to do and catalyze much more fieldwork around the world.

 

And why does it matter? Why shouldn't we write off our time-limited donor-funded projects? Because:

 

1) It's all we've got. Our current development system is not going anywhere soon, and there is success to learn from.

 

2) We need to quickly learn from what worked sustainably best and stop wasting time and resources on what we refuse to admit fails because we are too scared to return to see. Go back with the intention to learn what does and focus on doing more of what works.

 

3) Such analysis – and design of new projects – must have country ownership as a centerpiece throughout the project cycle assumptions, but to throw out decades of good work simply because we are just learning the value of country ownership is foolish.

 

Finally, here's a lovely example from Brazil of how local, participation (and yes, as my colleague thought, local ownership) works best. And. It. Takes. Time.

 

"Our results also show that Participatory Budgeting’s influence strengthens over time… Participatory Budgeting’s increasing impact indicates that governments, citizens, and civil society organizations are building new institutions… cities incorporate citizens at multiple moments of the policy process, allowing community leaders and public officials to exchange better information."  How often do we return to do what are called longitudinal reviews of our work abroad, using the same rigorous standards we evaluate our domestic projects? Not often. Shouldn't that change?

 

Only by working together, honoring the value of our participants, that they deserve the same chance at change that we take for granted will things change. We must value both the voices of our participants and our own expertise for development to improve for true aid effectiveness…. Let us begin anew!

 

What happens after the project ends?  Lessons about Funding, Assumptions and Fears (Part 3)

 

What happens after the project ends?
Lessons about Funding, Assumptions and Fears (Part 3)

 

In part 1 and part 2 of this blog, we showcased 11 of the 18 organizations that have done post-project evaluations.  While this was scratching the surface of all that is to be learned, we shared a few insights on How we do it Matters, Expect Unexpected Results and Country-national Ownership. We gained some champions in this process of sharing our findings, including Professor Zenda Ofir of South Africa, who said “we cannot claim to have had success in development interventions if the outcomes and/or impacts are not durable, or at least have a chance to sustain or endure.”

 

In this third blog of Lessons Learned from What Happens After the Project Ends, we turn to some of the curious factors that hold us back from undertaking more post project evaluations: Funding, Assumptions, and Fears.

 

Funding

  • Why haven’t we gone back? For the last 2+years Valuing Voices has been researching the issue, we have heard from colleagues: ‘we would love to evaluate post-project but we don’t have any money, ‘donors don’t fund this’, ‘it is too expensive’[*].  Funding currently from bilateral donors such as USAID is given in 1-5 year tranches with fixed terms for completion of results and learning from them and one-year close-out processes [1]. Much of the canon of evaluations conducted after close out that we amassed was from international NGOs that had used their private funds to evaluate large donor-funded projects for their own learning.  Many aimed also to show leadership in sustainability and admittedly dazzle their funders – join them!.
  • We fund capacity building during projects but if we do not return to evaluate how well we have supported our partners and communities to translate this to sustainability, then we fall short. Meetings convened by INTRAC on civil society sustainability are opening new doors for joint learning about factors such as “legitimacy… leadership, purpose, values, and structures” within organizations well beyond any project’s end [2]. The OECD’s DAC criteria for evaluating development assistance define sustainability as: “concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable“ [3]. We need to extend our view beyond typical criterion for sustainability being a focus primarily on continued funding.
  • We need funding to explore whether certain sectors lend themselves to sustainability. In addition to the cases in blog 1, a study by CARE/ OXFAM/ PACT on Cambodian Savings groups finds that we have some revisions to make on how we design and implement with communities to foster sustainability in this sector which typically promises greater sustainability because capital can be recycled [4]. Valuing Voices blogs show indications that once we amass a greater range of post-project evaluations (funders unite), the insights gleaned can illuminate cost efficient paths to more sustained programming, possibly leading to revisions in programming or interventions which have greater likelihood for country-ownership
  • Extend the program cycle to include post-project sustainability evaluation. Rare are donors such as the Australian government (forthcoming) and USAID’s Food For Peace that commission such studies. Rare is the initiative such as 3ie that has research funds allocated by major donors to explore an aspect of impact. We miss out on key opportunities to learn from the past for improved project design if we do not return to learn how sustained our outcomes and impacts have been. We miss learning how we could better implement so more national partners could take on the task of sustaining the changes we catalyzed.
  • We call on donors to fund a research initiative to comprehensively review sustainability evaluations.
  • We call on governments to ask for this in their discussion with donors. 
  • We call on implementers to invest in such learning to improve the quality of implementation today and sustained impact in the years to come.

 

 

Assumptions

Development assistance makes many assumptions about what happens after projects end in terms of people’s self-sufficiency,   partners’ capacity to continue to support activities, and projects’ financial independence and people’s ability to step into the shoes of donors and carry on.   Unless we take a hard look at our assumptions, we will not move from proving what we expect to learning what is actually there.

 

evaluation_-_Hledat_Googlem

 

Among them are these six assumptions:

  • All will be well once we exit; we have implemented so well that of course national participants and partners will be ready and able to carry on without us. We may assume the only important outcomes and impacts are within our Logical Frameworks and Theories of Change. Thus there is no need to return to explore unexpected negative ones, or ways in which the people we strengthened may have innovated in unexpectedly wonderful ways. Aysel Vazirova, a fellow international consultant wrote me: “Post-project evaluations provide data for a deeper analysis of sustainability and help to appreciate numerous avenues taken by the beneficiaries in incorporating development projects into their lives. The theory of change narratives presented by a majority of development programs and projects have a rather disturbing resemblance to the structure of magic tales: (from) Lack – (to) change – (to) happy ending. Post project evaluations have a power to change a rigid structure of this narrative.”
  • We assume evaluations are often used to inform new designs, yet dozens of colleagues have lamented that too often this does not happen in the race to new project design. But there is hope. World Wildlife Fund/UK M&E expert Clare Crawford says when following its new management standards, WWF “expects to see the recommendations of an evaluation before the next phase of design can happen (hence evaluations happen a little before the end of a strategic period).  WWF-UK, when reading new program plans is mandated to verify if – and how – the recommendations of the last evaluation(s) were made use of in the new design phase.  Equally we track management responses to evaluations to see how learning has been applied in current or in future work.” Such a link across the program cycle is not common in our experience and none of the post-project sustained impact evaluations we reviewed said how learning would be used.
  • We may assume data continues accessible from the projects we have evaluated, yet our team member Siobhan Green has found that until recently, with the move toward open data, often project data remains the province of the donors and implementers and to the best of our knowledge leaves the country when projects close. While some sectoral data such as health and education data remains local, we are finding in fieldwork that household level data has been rolled up or discarded once projects close, which makes interviews difficult.
  • We may assume that the participants and partners are not able to evaluate  projects, particularly after the fact.  Being vulnerable does not mean that people are not able to share insights or assess how projects helped or not. Methods such as empowerment evaluation and evaluative thinking are powerful supports [5] [6].
  • Some may assume that the situation has changed in the intervening years, that there is no benefit in returning to see what results remain. Change is inevitable and sometimes more rapid or dramatic than others.  But does that mean we shouldn’t want to understand what happened? This is the greatest disservice of all, for we are selling “sustainable development” so how well have we designed it to be so?
  • We assume that learning for our own benefit is enough. A potential client brought me in to discuss my working on a rare post-project evaluation last year. It was to cost hundreds of thousands of dollars and while would occur in several countries. What I discovered was that while the donor really wanted to learn what results remained more than a decade on, I asked ‘how would the countries themselves benefit from this research and findings?’  There was a long silence. Turns out, nothing from the research would benefit or even remain in country. No one had considered the learning needs of the countries themselves. This simply cannot continue if we are to be accountable to those we serve.

 

Fears

This may be the greatest barrier of all to returning to assess sustainability.

  • We assume our projects continue. We may be afraid to look for what will this tell us about the sustainability of our efforts to save lives and livelihoods so we only choose to publicly study what is successful. Valuing Voices has found that across most the post project studies there is some ‘selection bias’, as we repeatedly learned in our research from colleagues that organizations choose to evaluate projects that are most likely to be successfully sustained.  For instance, USAID Food For Peace’ study notes, “The countries included in this study—Bolivia, Honduras, India, and Kenya—were also chosen because of their attention to sustainability and exit.” Yet as an Appreciative Inquiry practitioner, I would argue that learning what worked best to know what to do more of may be the best way forward.
  • All too often the choice of evaluation design, and sensitivity to findings fly in the face of learning—particularly when findings are negative.  This raises fears around a discontinuation of funding (an implementer fear; a beneficiary fear; could also be a recipient government’s fear). Yet as Bill Gates says, “your most unhappy customers are your greatest source of learning.”>
  • Participants asked during the project cycle about interventions may be fearful of truth telling because of perceived vulnerabilities around promised future resources, local power imbalances in control over resources, or even political imperatives to adopt a particular position. Alternatively we may not believe them, thinking they may not tell us the truth were that to stop resources.

 

Those are ours.

  • Peter Kimeu, my wise advisor and 20-year friend and colleague from Kenya tells us some fears of the that we need to listen to – those that haunt our national partners and participants.

 

They are afraid we do not see their real desires:

  • “It is ‘not how many have you (the NGO) fed, but how many of us have the capability to feed ourselves and our community?’
  • ‘How can we (country national) support our fellow citizens to take our lives and livelihoods into our own hands and excel, sustainably?’
  • What is sustainability if it isn’t expanded opportunities, Isn’t the capability of one to make a choice of value/quality life out of the many choices that the opportunities present?”

 

Will you help us address these challenges? Will you join us in advocating filling the gap in the program cycle, and looking beyond it to how we design and implement with country nationals? Will you, in your own work foster their ownership throughout and beyond? We need to fund learning from sustained impact to transparently discuss assumptions and face our fears.  This is a sustained purpose we need to and can fill.

 

 

Sources:

[1] Capable Partners Program & FHI 36. (2010). Essential NGO Guide to Managing Your USAID Award: Chapter 6 – Close Out. Retrieved from https://www.ngoconnect.net/sites/default/files/resources/Essential%20NGO%20Guide%20-%20Chapter%206%20-%20Close%20Out.pdf

[2] Hayman, R. (2014, November 5). Civil society sustainability: Stepping up to the challenge. Retrieved from https://www.intrac.org/civil-society-sustainability-stepping-challenge/

[3] OECD. (n.d.). DAC Criteria for Evaluating Development Assistance. Retrieved 2015, from https://web.archive.org/web/20151206171605/http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm

[4] Emerging Markets Consulting. (2013). Sustainability of Savings Group Programs in Cambodia for CARE, Oxfam, and Pact. Retrieved from https://mangotree.org/Resource/Sustainability-of-Savings-Group-Programs-in-Cambodia-for-CARE-Oxfam-and-Pact

[5] Better Evaluation. (n.d.). Empowerment Evaluation. Retrieved from https://www.betterevaluation.org/plan/approach/empowerment_evaluation

[6] Griñó, L., Levine, C., Porter, S., & Roberts, G. (Eds.). (2016). Embracing Evaluative Thinking for Better Outcomes: Four NGO Case Studies. Retrieved from https://www.theclearinitiative.org/resources/embracing-evaluative-thinking-for-better-outcomes-four-ngo-case-studies

 


[*] It does not have to be. We have done these evaluations for under $170,000, all-inclusive.

 

What happens after the project ends?  Country-national ownership lessons from post-project sustained impacts evaluations (Part 2)

 

What happens after the project ends? Country-national ownership lessons from post-project sustained impacts evaluations (Part 2)

 

In Part 1 of our blog on lessons learned from post-project evaluations, we explored:

  • How we do it matters for great results
  • Expect unexpected results

This time we turn to who continues after closeout, and what conditions foster both successful handover and ownership from the onset in order to foster sustained impact.

 

Who Takes Over? Country nationals

When project handover is integral to the design, development projects needn’t be long-term or expensive. What they need to be is increasingly community-driven. Unless exit strategies are explicit and thorough, sustained impacts are less likely.

 

1. USAID/ Food for Peace (FANTA/ Tufts)

An ‘exit strategies’ evaluation of 12 projects in four USAID Food for Peace (FFP) countries of Bolivia, Honduras, India, and Kenya carried out in 2009, three to four years after close out detailed mixed results, described here [1]. These were complex food security projects across multiple sectors of: maternal and child health and nutrition; water and sanitation; agriculture, livestock, and rural income generation; natural resource management; school feeding; and micro-savings and loans.

FANTA/Tufts found “providing free resources, such as supplementary food as an incentive for growth monitoring participation or free agricultural marketing services to promote sales, created expectations that could not be sustained once the free resources were no longer offered.” Valuing Voices found similar issues in Niger’s PROSAN (see part 1), with the lack of continued incentives (food and in-kind inputs) led to activities not being continued by community members.

On the other hand, in India, the government took over FFP food ration distribution after closeout. “This phase-over of responsibility to national government programs was effective in the case of supplementary feeding but not in the case of school-feeding (the latter through the midday meals program), due to varying levels of government commitment. India’s government had the resources, capacity (an already existing supply chain), and motivation (commitment) to provide this benefit.” Again, CRS/Niger showed us that decades-long investments in partnerships and 2+ years for phase-over pays off; 80% of outcomes were self-sustained three years on. CIDA Peru also found that “Shared responsibilities and participatory process were instrumental in ensuring sustainability….a shared understanding of project objectives and counterpart interventions was established with Peruvian sector authorities, between donors and local communities” [2].

The lesson learned about close coordination with the partners such as the national government during design and implementation in order for transition to country-ownership and responsibility to be smooth also appeared among multilaterals that Valuing Voices has examined. Three multilateral agencies stand out as having conducted multiple post project evaluation (OECD, JICA and the Asian Development Bank).

We posit that too often in international development, the accountability focus is on fulfilling funder (donor) requirements, rather than accountability to project participants and what is needed to achieve sustained impact for them (Figure 1, below). The optimal case has project funders, implementers and national governments aligning to support those we ostensibly serve: women, men, youth, elders in need of assistance.

 

Valuing_Voices_Accountability_Capabilities_2015_pdf

@ValuingVoices2015

 

Are there certain kinds of projects or implementers that manifest optimal accountability? Far more examination is needed, but a promising path is microenterprise.

 

2. Pact’s WORTH project in Nepal

PACT’s project illuminates that local ownership and structures sustain results and even multiply impact. Implemented from 1999-2001, Pact worked with many local NGOs to reach 125,000 women in 6,000 economic groups across Nepal; of those, one quarter chose to implement village banks.  Village Banks cultivated women as agents of change and development in their communities—promoting grassroots sustainability   The post project evaluation in 2006 found that:

  • Almost two thirds of the original 1,536 village banks were still active eight years after the program began and assets of an average village bank has tripled in the last three years post-project (from $1000 to over $3000 at the time of the evaluation) [3]
  • 83% reported that because of WORTH they are able to send more of their children to school [3]
  • Women’s economic groups helped start an estimated 425 new groups involving another 11,000 women with neither external assistance nor prompting from the project [3].

Why? The post-project report tells that the banks were not an end in and of themselves, women’s empowerment was: “WORTH groups and banks were explicitly envisaged as more than just microfinance providers; they were seen as organizations that would build up women as agents of change and development in their communities” [3]. Thus local Nepalese sustained and grew their own development.

 

3. CARE Zanzibar’s Village Savings and Loan Associations were evaluated four years post-project. Similar to PACT, they found the model was sustained and grew:

  • Total membership rose from 1,272 in 2002’s closeout to an estimated membership of 4,552 in July 2006, an increase of 258% [4]
  • During the most recent payout for all 25 groups, the mean rate of return was 53%, with individual groups’ rates ranging from 10% to 92% [4]
  • Participants said that the main changes in the lives as a result of the program were an improved standard of living (22%), improved housing (21%) and increased incomes (20%) [4].

While wonderful, can it only be the responsibility of communities to sustain their gains? How well are we designing for country-ownership and handover to the state?

 

4. The UN’s OECD has dozens of post project evaluations on its website, funded by member governments.

One study illuminates that while communities may manage to sustain some of the outcomes, structural investment in national capacity to takeover is key. This example evaluated four 10-15 year-long projects funded largely by Germany that were carried out in Indonesia, Sri Lanka, Tanzania and Zambia with a cumulative value of Euro 145.1 million ($180 million). The study was done in 2004, evaluating activities an astonishing 30 years after inception. Results?

  • The good news: “living conditions of the target groups have improved in all four project regions,” with specific sustainable project outcomes observed in the “health and education sector, food security, increase in income and employment and the ensuing rise in the standard of living”.  Links were made to project-supported improvements in infrastructure, enhanced private sector economy, and the project’s innovations in agriculture.
  • The bad news:there was low institutional sustainability at the level of state executing organizations for all four projects due to inadequate funds, inefficient organizational structures and a lack of coordination”. Thus, viable exit and handover was limited. Structures advanced as part of a development project ran a high risk of not being sustainable.”

 

5. Three years ago, the Asian Development Bank reviewed 491 project completion reports (desk studies) and undertook a handful of field visits to projects financed between 2001-09. Similar results:

  • “Some early evidence suggests that as many as 40% of all new activities are not sustained beyond the first few years after disbursement of external funding” [5]
  • “National government ownership, commitment to and financing of the projects were vital to sustainability“ [5]
  • “Neither governments nor other international agencies benefit from systematic information on whether projects reached their intended economic or social objectives over the full life of the intervention or in the decade afterwards” [5].

This is a clarion call to all funders to invest in future excellence by returning to the past, learning what worked best and what failed to do so, examine why and begin anew with accountability to our participants!

 

Conclusion

What can be done?

A) Foster ownership of the process of development through empowerment to begin with, as PACT’s WORTH project did in Nepal and elsewhere. InterAction’s lovely “A Missing Piece in Local Ownership: Evaluation” reminds us “the local ownership agenda must extend to all parts of the program cycle – from design all the way through evaluation.  Including those meant to benefit from international assistance (we use the term “participants”) in deciding what should be done and how it should be done is critically important for effectiveness and sustainability” [6].

Ask yourselves how well we involve governments in collaborative design of what they feel they can sustain of our programming after we leave, how and for how long with what resources, linkages, capacity-built and motivation (see FFP study #1).

B) Design and implement in the present while considering sustaining outcomes and impacts in the long-term, as we learned in Part 1 of this blog as well as taking lessons from some emerging guides such as the systematic guidance of PCI’s Resource Guide for Enhancing Potential for Sustainable Impact [7].

C) Dare to return to learn. As Dina Esposito, the Director of USAID/Food For Peace stated, “this rigorous, retrospective [ex-post] approach is not widely done, but is essential if we are to understand the true impacts of our investments. To be effective, development projects must result in changes that last beyond the duration of the project themselves” [8].

Imagine the sustained cost-efficiencies of learning certain sectoral programs lent itself best to sustainability by communities, others needed Ministries to take over, others still needed different support such as private sector – or all of the above. If we look at sustained impact as our true goal, how differently could we work together? How much more efficiently would we use our global resources?

 

What can we say about the sustained impact post-project evaluations we have featured?

We have covered lessons about how matters in design and implementation; expect unexpected results and who takes over? Country nationals. Much more research and analysis is needed, many more case studies need to be created for us to understand how to foster the best handover as well as national ownership at the beginning, middle and end. Maybe you drew some of the same conclusions we have:

  • Post project evaluations provide valuable insights about sustainability.
  • Lessons from such evaluations can lead to better programming in current and future projects.
  • The voice of national stakeholders—participants and partners, including governments is essential.
  • Donors lose amazing opportunities to learn what works now and continues to work unless they fund more sustainable impact evaluations and support investing resources in fostering sustainability during design and implementation.

 

In Part 3, we will look at what is keeping us from looking to the past for the future (hint: funding, assumptions and fears) and how we can move ahead together…

Please join us in advocating for and funding this vital approach!

 

 

Sources:

[1] Food and Nutrition Technical Assistance (FANTA). (n.d.). Effective Sustainability and Exit Strategies for USAID FFP Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp

[2] Canadian International Development Agency (CIDA). (2012). Evaluation of CIDA’s Peru Program. Retrieved 2014, from https://web.archive.org/web/20140807174641/http://www.acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/Evaluations2/$file/peru-eng.pdf

[3] Mayoux, L. (2008, June). Women Ending Poverty: The WORTH Program in Nepal – Empowerment through Literacy, Banking and Business 1999-2007. Retrieved from https://www.findevgateway.org/case-study/2008/06/women-ending-poverty-worth-program-nepal-empowerment-through-literacy-banking

[4] Anyango, E., Esipisu, E., Opoku, L., Johnson, S., Malkamaki, M., & Musoke, C. (2006, January). Village Savings and Loan Associations: Experiences from Zanzibar. Retrieved from https://www.findevgateway.org/case-study/2006/01/village-savings-and-loan-associations-experiences-zanzibar

[5] Asian Development Bank. (2010, October 31). Post-Completion Sustainability of Asian Development Bank-Assisted Projects. Retrieved from https://www.adb.org/documents/post-completion-sustainability-asian-development-bank-assisted-projects

[6] Grino, L. (2015, February 19). A Missing Piece in Local Ownership: Evaluation. Retrieved 2015, from https://web.archive.org/web/20150502162547/https://www.interaction.org/blog/missing-piece-local-ownership-evaluation

[7] Choi-Fitzpatrick, J., Schooley, J., Eder, C., & Lomeli, B. (2014). A Resource Guide for Enhancing Potential for Sustainable Impact: Food and Nutrition Security. Retrieved from https://www.fsnnetwork.org/resource-guide-enhancing-potential-sustainable-impact

[8] Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fsnnetwork.org/ffp-sustainability-and-exit-strategies-study-synthesis-report

 

Learning about Sustainability and Exit Strategies from USAID’s Food Assistance Projects

 

Learning about Sustainability and Exit Strategies
from USAID’s Food Assistance Projects

 

USAID overall and Food for Peace (FFP) specifically have become far more progressive in the Obama Administration and under Administrator Rajiv Shah, with a much greater focus on accountability and results. Those of you unfamiliar with USAID’s Food For Peace will learn it has been a large channel of international assistance for over 60 years and is not a small funding instrument. For 2016 alone, they have proposed spending $1.75 billion to feed 47 million people through humanitarian and development programs implemented by non-profits, for-profits and the UN’s World Food Program [1]. Given this scale of resources, it is highly surprising that while many documents in their archive ask for post-project evaluation and there are a handful of desk reviews, they have done only two actual ones with new fieldwork in the last 30 years (these four countries and a recently published one on Uganda, see Catalysts page). This recent and excellent 2015 synthesis report by authors Rogers and Coates is presented below [2].

Commissioned by USAID, Tufts University and FHI360 have done a remarkably thorough two to three year post-project evaluation of four (Title II) food-assisted programs containing 12 projects in Bolivia, Honduras, India, and Kenya that closed out in 2009. The methodologies used are clearly outlined (itself a boon to our fledgling field) as are limitations and comments on context, findings and recommendations. It was no small feat to compare activities across four countries and so many sectors (some of which were supported by provision of US food aid resources, others with in-kind or cash inputs): maternal and child health and nutrition; water and sanitation; agriculture, livestock, and rural income generation; natural resource management; school feeding; and micro-savings and loans. Also this covered many implementers, from CARE, ADRA and Save the Children to World Vision, CRS and Feed the Hungry.

In this document, Dina Esposito, the Director of FFP states “We commissioned this report with the objective of determining what factors enhanced the likelihood of sustained project benefits, in order to improve our guidance for future food assistance development projects.… FFP development projects are designed to reduce the long-term need for food assistance by strengthening the capacity of developing societies to ensure access to nutritious food for their most vulnerable communities and individuals, especially women and children. The study team looked at 12 FFP development projects across four countries and asked not only what was achieved by each project’s end?, but also, what of those achievements remained one year after project close-out? and two years after? This rigorous, retrospective approach is not widely done, but is essential if we are to understand the true impacts of our investments. To be effective, development projects must result in changes that last beyond the duration of the project themselves.”

 

Process and findings:

The researchers compared baseline, midline and endline evaluations and exit strategy documents to new mixed-method data collection. There were four main findings:

1) Impact vs. Sustainability trade-offs: Evidence of project success at the time of exit (as assessed by impact indicators) did not necessarily imply sustained benefit over time. Just because projects were deemed successful at exit does not mean that those continued after closeout. “Moreover, the study found that focusing exclusively on demonstrating impact at exit may jeopardize investment in longer-term sustainability.” Valuing Voices found the same in Ethiopia in research done in 2013 [3].

2) Preconditions to successful sustainability: In addition to an ongoing source of resources, good technical and managerial capacity, and sustained motivation of participants and partners, linkages to governmental organizations and/or other entities were key to continuity and sustainability of outcomes and new impacts. “No project in this study achieved sustainability without [the first] three of them in place before the project ended,” and linkages between community partners and the public/private sector were critical for handover (Figure 1, below). Further, a gradual transition from project-supported activities to independent operation was important for sustainability.Sustainability was more likely when projects withdrew gradually, allowing community-based organizations to develop the capacity to operate independently.”

 

[2]

 

3) Free resources can threaten sustainability, unless replaced while there is no one-size fits all for resources:

Using incentives has costs. “Free supplementary food in maternal and child health and nutrition projects or free marketing services in agriculture projects created expectations in many projects that could not be sustained once resources were withdrawn”. Valuing Voices found the same in research in Niger (report imminent). But other financing options, free health care or fee for service are still unsystematically studied regarding fostering sustainability in differing sectors.

4) External factors (climate, economy) can affect sustainability: The operating context and exogenous shocks (e.g., economic, legal and climatic) also affected the sustainability of project benefits, positively or negatively.

 

Most tellingly, the authors warned that “sustainability plans cannot be based on the hope that activities and benefits will continue in the absence of the key factors identified in this study.” Throughout the report and in pending country-specific studies, they outlined the assumptions that projects made about sustainability in order to exit and closeout, which were variably disproved, such as:

  • Community health workers would continue to provide services although without remuneration,
  • Households could continue to access nutritious food from their own (increased) production or purchases and have time, and know how to prepare such food,
  • Farmers will pay for inputs with profits from increased production and commercialization and can meet the quantity and quality requirements of long-term contracts
  • Community members will recognize the tangible benefit of Natural Resource Management activities and will be motivated to continue them without further inputs or remuneration
  • Water committees will have sufficient administrative capacity and resources to manage their budgets effectively
  • Community-based organizations have strong institutional capacity
  • Partner organizations will continue to provide teacher training
  • Government will have the resources and commitment to support future needs

 

The country studies with detailed findings are still forthcoming but these examples may illustrate the range of sustainability. There were some very well-sustained positive results in Food Production (India by area) and Child Health Growth Monitoring (Bolivia by consortium implementers) between baseline or enline and followup 2-3 years later:

FFP-Sustainability-Exit-Strategies-Synthesis-Dec2015_docx

[2]

FFP-Sustainability-Exit-Strategies-Synthesis-Dec2015_docx

[2]

As well as some far more mixed or negative results in examples across all the Water and Sanitation projects:

FFP-Sustainability-Exit-Strategies-Synthesis-Dec2015_docx

[2]

And far less stellar results in Maternal Child Health’s Community Health Workers (Kenya):

FFP-Sustainability-Exit-Strategies-Synthesis-Dec2015_docx

[2]

The authors recommended not only ensuring resources, capacity, motivation and linkages are present before exiting but also institutionalizing sustainable approaches to project design and evaluation including in solicitations and applications, project assessments, project management and knowledge management. They also recommended not only phasing down exit but also extending more such evaluations beyond the 5 years of implementation and assessing impacts as long as 10 years after. This requires some sizeable revisions to how development is done at Food For Peace.

All of these findings recommendations are near and dear to those of us at Valuing Voices. We strongly commend Food For Peace and ask for many more such studies, for unless we know what worked best and why, how do we know what to design next together with our partners and participants for real sustainability?

 

Sources:

[1] InterAction. (2015). Choose to Invest 2016: Food For Peace Title II. Retrieved from https://web.archive.org/web/20150307160559/https://www.interaction.org/choose-to-invest-2016/food-for-peace-title-II

[2] Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp

[3] Cekan, J., PhD. (2014, April 7). Evaluation of ERCS/Tigray’s “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity”. Retrieved from http://adore.ifrc.org/Download.aspx?FileId=147802&.pdf