Learning about Sustainability and Exit Strategies from USAID’s Food Assistance Projects

 

Learning about Sustainability and Exit Strategies
from USAID’s Food Assistance Projects

 

USAID overall and Food for Peace (FFP) specifically have become far more progressive in the Obama Administration and under Administrator Rajiv Shah, with a much greater focus on accountability and results. Those of you unfamiliar with USAID’s Food For Peace will learn it has been a large channel of international assistance for over 60 years and is not a small funding instrument. For 2016 alone, they have proposed spending $1.75 billion to feed 47 million people through humanitarian and development programs implemented by non-profits, for-profits and the UN’s World Food Program [1]. Given this scale of resources, it is highly surprising that while many documents in their archive ask for post-project evaluation and there are a handful of desk reviews, they have done only two actual ones with new fieldwork in the last 30 years (these four countries and a recently published one on Uganda, see Catalysts page). This recent and excellent 2015 synthesis report by authors Rogers and Coates is presented below [2].

Commissioned by USAID, Tufts University and FHI360 have done a remarkably thorough two to three year post-project evaluation of four (Title II) food-assisted programs containing 12 projects in Bolivia, Honduras, India, and Kenya that closed out in 2009. The methodologies used are clearly outlined (itself a boon to our fledgling field) as are limitations and comments on context, findings and recommendations. It was no small feat to compare activities across four countries and so many sectors (some of which were supported by provision of US food aid resources, others with in-kind or cash inputs): maternal and child health and nutrition; water and sanitation; agriculture, livestock, and rural income generation; natural resource management; school feeding; and micro-savings and loans. Also this covered many implementers, from CARE, ADRA and Save the Children to World Vision, CRS and Feed the Hungry.

In this document, Dina Esposito, the Director of FFP states “We commissioned this report with the objective of determining what factors enhanced the likelihood of sustained project benefits, in order to improve our guidance for future food assistance development projects.… FFP development projects are designed to reduce the long-term need for food assistance by strengthening the capacity of developing societies to ensure access to nutritious food for their most vulnerable communities and individuals, especially women and children. The study team looked at 12 FFP development projects across four countries and asked not only what was achieved by each project’s end?, but also, what of those achievements remained one year after project close-out? and two years after? This rigorous, retrospective approach is not widely done, but is essential if we are to understand the true impacts of our investments. To be effective, development projects must result in changes that last beyond the duration of the project themselves.”

 

Process and findings:

The researchers compared baseline, midline and endline evaluations and exit strategy documents to new mixed-method data collection. There were four main findings:

1) Impact vs. Sustainability trade-offs: Evidence of project success at the time of exit (as assessed by impact indicators) did not necessarily imply sustained benefit over time. Just because projects were deemed successful at exit does not mean that those continued after closeout. “Moreover, the study found that focusing exclusively on demonstrating impact at exit may jeopardize investment in longer-term sustainability.” Valuing Voices found the same in Ethiopia in research done in 2013 [3].

2) Preconditions to successful sustainability: In addition to an ongoing source of resources, good technical and managerial capacity, and sustained motivation of participants and partners, linkages to governmental organizations and/or other entities were key to continuity and sustainability of outcomes and new impacts. “No project in this study achieved sustainability without [the first] three of them in place before the project ended,” and linkages between community partners and the public/private sector were critical for handover (Figure 1, below). Further, a gradual transition from project-supported activities to independent operation was important for sustainability.Sustainability was more likely when projects withdrew gradually, allowing community-based organizations to develop the capacity to operate independently.”

 

[2]

 

3) Free resources can threaten sustainability, unless replaced while there is no one-size fits all for resources:

Using incentives has costs. “Free supplementary food in maternal and child health and nutrition projects or free marketing services in agriculture projects created expectations in many projects that could not be sustained once resources were withdrawn”. Valuing Voices found the same in research in Niger (report imminent). But other financing options, free health care or fee for service are still unsystematically studied regarding fostering sustainability in differing sectors.

4) External factors (climate, economy) can affect sustainability: The operating context and exogenous shocks (e.g., economic, legal and climatic) also affected the sustainability of project benefits, positively or negatively.

 

Most tellingly, the authors warned that “sustainability plans cannot be based on the hope that activities and benefits will continue in the absence of the key factors identified in this study.” Throughout the report and in pending country-specific studies, they outlined the assumptions that projects made about sustainability in order to exit and closeout, which were variably disproved, such as:

  • Community health workers would continue to provide services although without remuneration,
  • Households could continue to access nutritious food from their own (increased) production or purchases and have time, and know how to prepare such food,
  • Farmers will pay for inputs with profits from increased production and commercialization and can meet the quantity and quality requirements of long-term contracts
  • Community members will recognize the tangible benefit of Natural Resource Management activities and will be motivated to continue them without further inputs or remuneration
  • Water committees will have sufficient administrative capacity and resources to manage their budgets effectively
  • Community-based organizations have strong institutional capacity
  • Partner organizations will continue to provide teacher training
  • Government will have the resources and commitment to support future needs

 

The country studies with detailed findings are still forthcoming but these examples may illustrate the range of sustainability. There were some very well-sustained positive results in Food Production (India by area) and Child Health Growth Monitoring (Bolivia by consortium implementers) between baseline or enline and followup 2-3 years later:

FFP-Sustainability-Exit-Strategies-Synthesis-Dec2015_docx

[2]

FFP-Sustainability-Exit-Strategies-Synthesis-Dec2015_docx

[2]

As well as some far more mixed or negative results in examples across all the Water and Sanitation projects:

FFP-Sustainability-Exit-Strategies-Synthesis-Dec2015_docx

[2]

And far less stellar results in Maternal Child Health’s Community Health Workers (Kenya):

FFP-Sustainability-Exit-Strategies-Synthesis-Dec2015_docx

[2]

The authors recommended not only ensuring resources, capacity, motivation and linkages are present before exiting but also institutionalizing sustainable approaches to project design and evaluation including in solicitations and applications, project assessments, project management and knowledge management. They also recommended not only phasing down exit but also extending more such evaluations beyond the 5 years of implementation and assessing impacts as long as 10 years after. This requires some sizeable revisions to how development is done at Food For Peace.

All of these findings recommendations are near and dear to those of us at Valuing Voices. We strongly commend Food For Peace and ask for many more such studies, for unless we know what worked best and why, how do we know what to design next together with our partners and participants for real sustainability?

 

Sources:

[1] InterAction. (2015). Choose to Invest 2016: Food For Peace Title II. Retrieved from https://web.archive.org/web/20150307160559/https://www.interaction.org/choose-to-invest-2016/food-for-peace-title-II

[2] Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp

[3] Cekan, J., PhD. (2014, April 7). Evaluation of ERCS/Tigray’s “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity”. Retrieved from http://adore.ifrc.org/Download.aspx?FileId=147802&.pdf

 

Who is responsible for sustaining development?

Whose responsibility is it to sustain project activities?

Billions of dollars are pumped into development activities in developing countries all over the world. Communities getting involved in these projects have a clear objective, which is to have their lives improved in the sectors that the projects target. As to whether this is the main objective of the development partners is not clear. What is clear is that the development partners focus more on numbers than on getting people to participate.

 

We note that majority of these projects are designed to last between 2-5 years. Delays occasioned poor planning or other unforeseen factors eat into the implementation time to an extent that in some projects it takes about 1-2 years to get a program running. This means that the planned implementation time is reduced.  Baselines, midlines and end lines studies are conducted to inform changes that may have occurred within the program life, and in most cases they happen shortly after the program has started or just before it ends. In fact, some baselines are conducted after programs have started.

Considering the reduced implementation time and the fact that it takes a much longer time to get concrete behavior change related results, questions emerge whether indeed the reported changes are solid enough during implementation to be sustained. There is also a difference between measuring what can be referred to artificial changes (activities that community members adopt as a way of short-term trial in their excitement, but don't find useful afterward) and long lasting changes that community members adopt because they are useful part and parcel of their lives.

 

Almost all projects have logical frameworks (logframes) that show how project activities will be implemented and to some extent there are also exit strategies for closing out the project. This can be an illusion long-term. In most cases donors and implementers assume that communities will adopt activities that are being implemented within a specified period of time, and so projects close down at the end of the specified period of implementation assuming things will continue, but have no proof. Valuing Voices has done projected sustainability work in Ethiopia which points to possible differences between what donors expect to the sustained versus what communities are able to sustain.

 

Communitywomansorting

 

The big questions remain: "whose responsibility is it to ensure that whatever has been adopted is continued? Whose responsibility is it to sustain project activities post project implementation?" It is silently assumed that communities can take up this responsibility and a key question is "what guarantees are there that this is possible and is happening?" Project sustainability should not be seen as a community-alone responsibility but rather a responsibility for all those who are involved in program activities.  Sustainability studies should be planned for and executed in the same breath that the baselines, midlines, end-lines and in the rare cases impact assessments in real time should be planned for. We must do sustainability studies as they provide an additional realistic opportunity to inform us on actual community development post project implementation. Communities should not be left alone with it.

 

What’s likely to ‘stand’ after we go? A new consideration in project design and evaluation

What’s likely to ‘stand’ after we go? A new consideration in project design and evaluation

This spring I had the opportunity to not only evaluate a food security project but also to use the knowledge gleaned for the follow-on project design.  This Ethiopian Red Cross (ERCS) project “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity” (with Federation and Swedish Red Cross support) was targeted to 2,259 households in Dedba, Dergajen and Shibta through provision of crossbreed cows, ox fattening, sheep/goats, beehives and poultry which were to be repaid in cash over time as well water and agriculture/ seedlings for environmental resilience.   ERCS had been working with the Ethiopian government to provide credit for these food security inputs to households in Tigray which were to be repaid in cash over time.  During this evaluation, we met with 168 respondents (8% of total project participants).

 

Not only were we looking for food consumption impacts (which were very good), and income impacts (good), we also probed for self-sustainability of activities. My evaluation team and I asked 52 of these participants more in-depth questions on income and self-sustainability preferences. We used participatory methods to learn what they felt they could most sustain themselves after they repaid the credit and the project moved on to new participants and communities. 

VV_AEA_Finaldraft101314_pptx

We also asked the to rank what input provided the greatest source of income.  The largest income (above 30,000 birr or $1,500) was earned from dairy and oxen fattening, while a range of dairy, oxen, shoats and beehives provided over half of our respondents (40 people) smaller amounts between 1,000-10,000 birr ($50 to $500).

And even while 87% of total loans were for ox fattening, dairy cows (and beehives) which brought in farm more income, and only 11% of loans were sheep/goats (shoats) and 2% for poultry, the self-sustainability feedback was clear. In the chart below, poultry and shoats (and to a lesser degree, ox fattening) were what men and women felt they could self-sustain. In descending order, the vast majority of participants prioritized these activities:

To learn more about how we discussed that Ethiopian participants can self-monitor, see blog.

So how can such a listening and learning approach feed program success and sustainability? We need to sit with communities to discuss the project’s objectives during design plus manage our/ our donors’ impact expectations:

1) If raising income in the short-term is the goal, the project could only have offered dairy and ox fattening to the communities as their incomes gained the most. Note, fewer took this risk as the credit for these assets was costly.

2) If they took a longer view, investing in what communities felt they could self-sustain, then poultry and sheep/goats were the activities to promote. This is because more people (especially women, who preferred poultry 15:1 and shoats 2:1 compared to men ) could afford these smaller amounts of credit as well as the feed to sustain them.

3) In order to learn about true impacts we must return post-project close to confirm the extent to which income increases continued, as well as the degree to which communities were truly able to self-sustain the activities the project enabled them to launch. How do our goals fit with the communities’?

What is important is seeing community actors, our participants as the experts. It is their lives and livelihoods, and not one of us in international development is living there except them…

What are your questions and thoughts? Have you seen such tradeoffs? We long to know…

[*NB: There were other inputs (water, health, natural resource conservation) which are separate from this discussion.]

Sustainability SPRINGing out all over the place… and Disrupting!

Sustainability SPRINGing out all over the place… and Disrupting

 

So what is sustainability? You may think it's the climate's long-term wellbeing and how to gauge changes to that.  You may think it's linked to sustainable development regarding consumption, trade, education and environment and how to assess it. You may think it's data-driven organizational success as Chelsea Clinton describes, or is it Michael Porter's business' view of Creating Shared Value on social and environmental concerns or is it about people, as hallowed University of Cambridge trains experts in its Institute for Sustainability Leadership (I revel that I was a Fellow there in the '90s). Finally, is it WCED’s lovely definition "Sustainable development is development that meets the needs of the present without compromising the ability of future generations to meet their own needs"? Yes, when applied to communities' abilities to self-sustainably and resiliently chart their own development! 

So how are we to get there? A Sustainable Brands Conference this year gets us there through being clear about their own consumption, and USAID is no different. USAID Forward is putting their money where their keyboards are (so to speak), toward more sustainable local delivery by directing a huge 30 percent of its funding to “local solutions” through procurement in coming years.  This framework is to “support the ‘new model of development’ that USAID Administrator Rajiv Shah has touted, which entails a shift away from hiring U.S.-based development contractors and NGOs to implement projects, and toward channeling money through host-country governments and local organizations to build their capacity to do the work themselves and sustain programs after funding dries up. I, and others celebrate the investments this will enable local firms to make in their own capacity, in leading development!

Of course all sorts of safeguards are needed, and ideally US firms would be providing capacity development, but shouldn’t we have been doing this all along, to move toward transferring ‘development’ to the countries themselves?

 GAO_Sustain_LocalSyst0514

Source: GAO report

Also vital to sustainable development is learning from what works and doing more of it. USAID is finally planning to incorporate more ex-post evaluations into its toolkit of evaluating sustainability!  Two weeks ago, PPL/LER shared their great new policy document- “Local systems: A framework for supporting sustained development on how they can better incorporate local systems thinking into policy as well as DIME (Design, Implementation, Monitoring and Evaluation).  Industry insider DevEx tells us "even though the agency plans to use ex-post evaluations to measure whether development projects are successful or not, these evaluations will not focus on “specific contractor performance” but instead consider the “types of approaches that contribute to more sustainable outcomes…to inform USAID’s country strategies and project design." While PVO implementing partners will not [yet?] be required to do ex-post evaluations as part of their projects, having this door cracked open is excitingly opening. Notably, it is a ‘back to the future’ moment, as 30 years ago USAID led the development world in post-project evaluations, yet in the last 24 years has done none (or at least not published any) except for the Food for Peace retrospective below, as I found in our Valuing Voices research of USAID's Development Experience Clearinghouse.

There is far more to watch. In our view, the whole development industry needs to grapple with the perceived barrier that funding ends with projects (note: a trust could be set up to document post-project impact 1, 3, 5 years later and results retained, much as 3ie does now for impact evaluations) and the view that one cannot discern attributable project impact with a time-lag of several years. Yet even the General Accounting Office is asking for longitudinal data; they reviewed USAID’s document and wants to see clear measures of success at Mission and HQ level by different indicators of local institutional sustainability and impact four years on.

Why should we care? As Chelsea Clinton of the Clinton Global Initiative puts it, "you can't measure everything, but you can measure almost everything through quantitative or qualitative means, so that we know what we're disproportionately good at. And, candidly, what we're not so good at, so we can stop doing that.

Yes! Development should be about doing more of what works, sustainably, and less of what doesn’t. USAID’s Local Systems Framework found the best could also be free, as in this one Food For Peace evaluation shows:

FFPBox8

Returning to Chelsea Clinton, I’ll conclude by stating something obvious. She "wants to see some evidence of why we're making decisions, as opposed to the anecdotes” which is what getting post-project evaluation data from our true clients, our participants, is all about. Clinton says this will transform CGI into a smart, accountable, and sustainable support system for philanthropic disrupters around the world. USAID is radical for me, today, with their Local Systems investments… my neighborhood disrupter.

 

Are you such a disrupter too? Who else is one whom we can celebrate together? 

Stepping up community self-sustainability, one [Ethiopian] step at a time

 

Stepping up community self-sustainability, one [Ethiopian] step at a time

 

Having just come back from evaluation and design fieldwork for an Ethiopian Red Cross (ERCS)/ Swedish Red Cross/ Federation of the Red Cross and Red Crescent project, the power of communities is still palpable in my mind. They know what great impact looks like. They know what activities they can best sustain themselves. It’s up to us to ask, listen and learn from them and support their own monitoring/ evaluating/ reporting. It’s up to us to share such learning with others and to act on it everywhere.

There are a myriad of possible sustainability indicators, and the outcome indicators below, suggested by 116 rural participants from Tigray, Ethiopia seem to fall into two categories of expected changes: Assets and Life Quality (Table 1). As the food security/ livelihood project extended credit for animal purchases, it is logical that tracking increased income, savings, assets, and home investments plus expenditures on food and electricity appeared.

We gleaned this from discussions with participants, asking them “what can we track together that would show that we had impact”? Our question led to a spirited discussion of not only what was traceable, but also what could be publicly posted and ‘ground-truthed’ by the community. Discussing indicators led to even deeper conversations about the causes of food insecurity which were illuminating to staff.  What was surprising, for instance, was the extent to which families saw changing seasonal child-field labor practices in favor of 100% child-school attendance as great indicators.  School attendance (or lack thereof) was dependent on families’ need for children’s seasonal labor in the fields. Community members said they knew who sent their children or not, which no only ‘cleaned’ the publicly posted data but triangulated implementer surveys and opened room for discussions of vulnerability.

 

self-sustain-indicators

 

 

Not only is this exciting for the project’s outcome tracking but even more importantly, our team proposed to create a community self-monitoring system, suggested in by Causemann/Gohl in an IIED PLA Notes article– “Tools for measuring change: self-assessment by communities” used in Africa and Asia. This learning, management and reporting process will fill a gaping need as current “monitoring systems serve only for donor accountability, but neither add value for poor people nor for the implementing NGOs because they do not improve effectiveness on the ground.” The authors found that not only “participatory data collection produces higher quality data in some fields than standard extractive methodologies [as] understanding the context leads to a higher accuracy of data and learning processes [which] increase the level of accountability… “ but also that such shared collaboration builds mutual learning and bridge-building.” While our community members may have offered to track this publicly to make this partner happy, men and women discussed this excitedly and embraced the idea of self-monitoring happily. ERCS will be discussing with communities to either track data monthly in notebooks or on a large chart hung in the woreda office for transparency.  Data (Chart 1) would include these asset and quality of life indicators as well as loan repayments (tracked vertically) while households (tracked horizontally) could see who was meeting the goal (checked boxes), not meeting it fully (dashed boxes) or not meeting it at all yet (blank boxes).  Community members corrected each other as they devised the indicators during our participatory research and this openness reassures us that the public monitoring will be quite transparent as well.

 

ParticipTracjking

[1]


 

Further, what was especially satisfying was getting feedback from across the three tibias (sub-regions) on what activities they felt they could sustain themselves irrespective of the project’s continuation. Table 2 shows us which activities communities felt were most self-sustainable by households; these could form the core of the follow on project. Sheet/goats, poultry and oxen for fattening were highly prioritized by both women and men, in addition to a few choosing improved dairy cows. The convergence of similar responses was gratifying and somewhat unexpected, as there were several other project activities.  The communities’ own priorities need to be seriously considered as currently they get only one loan per family and thus self-sustainable activities are key.

 

self-sustain-activities

 

There is more to incorporate in future project planning by NGOs like ERCS. The NGO-IDEAs concept mentioned above also includes involving project participants in setting goals and targets themselves, differentiating between who achieved them and why, and brainstorming who/what contributes to it and what they should do next. Peer groups, development agencies and any actors could collect and learn from the data. Imagine the empowerment were communities to design, monitor and evaluate and tell us as their audience!

And they must, according to ODI UK’s Watkins, who has a clear vision on how to achieve a global equity agenda for the post-2015 MDG goalsHe suggests converting the principle of ‘leave no one behind’ into measurable targets. He argues that, by introducing a series of ‘stepping stone’ benchmarks, the world can set ambitious goals on equity by 2030. He writes, wisely, that “narrowing these equity deficits is not just an ethical imperative but a condition for accelerated progress towards the ambitious 2030 targets. There are no policy blueprints. However, the toolkit for governments actively seeking to narrow disparities …has to include some key elements [such as] identifying who is being left behind and why is an obvious starting point. That’s why improvements to the quality of data available to policy-makers is an equity issue in its own right”. Valuing Voices believes who creates that data is an equally compelling equity issue.

 

So how will we reach these ambitious targets by 2030? By putting in stepping stone targets, returning project design functions to the ultimate clients – the communities themselves- and matching their wants with what we long to transfer to them. In this way we will be Valuing their Voices so much that they evaluate our projects jointly and we can respond. That’s how it should always have been.

What are your thoughts on this? We long to know.

 

 

Sources:

[1] Ashley, H., Kenton, N., & Milligan, A. (Eds.). (2013). Tools for supporting sustainable natural resource management and livelihoods. Participatory Learning and Action, (66). Retrieved from https://pubs.iied.org/14620IIED/

[2] Watkins, K. (2013, October 17). Leaving no-one behind: An equity agenda for the post- 2015 goals. Retrieved from https://www.odi.org/blogs/7924-leaving-no-one-behind-equity-agenda-post-2015-goals