Are We Done Yet?
When are we off the hook, so to speak, for the well-being of the participants whom we said we'd make healthier, better fed, more educated, safer, etc?
America’s Agency for International Development (USAID) is the main channel for international development aid. It is also an organization interested in learning from its programming and numerous contracts support such work. One such contract by FHI360/FANTA was Food for Peace tasking them to review the agency’s Title II development food aid from 2003-2009 covering 28 countries. This Second Food Aid and Food Security Assessment (FAFSA-2) Summary found that such programs can “reduce undernutrition in young children, improve health and nutrition outcomes, and increase access to income and food” and also found practices that did not work well.
While USAID has made enormous strides in the intervening six years on monitoring and evaluation (I was a consultant to USAID/PPL/LER from 2013-14), excellent recommendations that would support great, sustainable programs are unfulfilled:
Recommendations #1, 4 “USAID/FFP should develop an applied research agenda and sponsor studies that focus on the implementation of Title II programs in the field to better define what works and what does not…. [and] should select the review panel for new Title II applications… and give reviewers a ‘cheat sheet’ on interventions and approaches that USAID/FFP is and is not interested in funding because they work better or do not work as well, [and] provide this same information in the Request for Assistance” [Request for proposals].
Yes, all across our industry there is little learning from past evaluations for future design and Valuing Voices believes local participants and stakeholders need to be consulted to tell us what (still) works and what they want more of not only during implementation but long after. Their voices must support great design, as it’s their lives we go there to improve; they must be involved in the design of these original requests that non-profits design and fulfill. Further, the study found that only 1/3 of all evaluations were included in USAID’s database, and as Valuing Voices’ partner Sonjara has written in our blog, aid transparency requires data retention and access for learning to happen.
Recommendation #3 “USAID/FFP should include options for extensions of awards or separate follow-on awards to enable USAID/FFP to continue to support high-performing programs beyond five years and up to ten years… [as] longer implementation periods are associated with greater impact.”
This would address the ‘how much impact can we accomplish in 1, 3, 5 years” question that many of us in international non-profits ask ourselves. Finally, the graphic below is self-explanatory – USAID sees its role ending at close-out.
The crux lies in their honest statement: "It was beyond the scope and resources of the FAFSA-2 to explore in any depth the sustainability of Title II development programs after they ended." While they state that there is merit in having impact while you intervene, such as "having a positive impact on the nutritional status of the first cohort of children is of immense benefit in its own right", they go on to say that "ideally, one would like to see mothers continuing positive child feeding practices and workers continuing to deliver services long after programs end… [yet] whether the [maternal child health and nutrition] interventions are sustainable beyond one generation is unknown and would require research." This is because funding is pre-programmed, fixed to end within set 1, 3, 5 year increments, and no one goes back to learn how it all turned out. This is what most needs to change, this illusion that what happens after closeout is no longer our issue, that the ‘positive impact’ we had while there is enough.
They are not alone. I think of NORAD, the Government of Norway's development arm as very progressive. So I went to NORAD's website and searched for 'ex-post' (we do a lot of that at ValuingVoices). So like our World Bank blog on finding real ex-post evaluations, many many things are considered 'ex-post', including one actual evaluation in Palestine with fieldwork which asked participants and a few that looked at institutional sustainability. Many of the 100+ 'finds' were actually documents recommending ex-post. Typical of our other searches of other donors. I emailed NORAD whether there were more with participant voices, yet they assured me they did them. Maybe our problem is in definitions and taxonomy again. Maybe we should call them post-project participant feedback?
Most of my colleagues would agree that the sustainability of activities aimed at making communities food secure in the long-term and independent of aid is a shared goal, one which short-term assistance aimed at huge impacts such as to ‘make communities food secure’ and ‘sustainably decrease malnutrition’ (common proposal goals) is unrealistic. We need participant voices to teach us how well we served them. We need to return, learn “what works and what does not”, and Value Voices in true sustained partnership. We all look forward to being done.
 “Another major obstacle to transparency and learning from the Title II program experience was the fact that only one-third of the final evaluations were publicly available on the Development Experience Clearinghouse (DEC), despite the requirement that Awardees post them to the DEC…. [There was a lack of] cross-cutting studies or in-depth analyses of Title II evaluation results to advance organizational learning [and] much greater use could be made of the evaluation data for systematic reviews, meta-analyses, secondary analyses, and learning.”