Stepping up community self-sustainability, one [Ethiopian] step at a time
Having just come back from evaluation and design fieldwork for an Ethiopian Red Cross (ERCS)/ Swedish Red Cross/ Federation of the Red Cross and Red Crescent project, the power of communities is still palpable in my mind. They know what great impact looks like. They know what activities they can best sustain themselves. It’s up to us to ask, listen and learn from them and support their own monitoring/ evaluating/ reporting. It’s up to us to share such learning with others and to act on it everywhere.
There are a myriad of possible sustainability indicators, and the outcome indicators below, suggested by 116 rural participants from Tigray, Ethiopia seem to fall into two categories of expected changes: Assets and Life Quality (Table 1). As the food security/ livelihood project extended credit for animal purchases, it is logical that tracking increased income, savings, assets, and home investments plus expenditures on food and electricity appeared.
We gleaned this from discussions with participants, asking them “what can we track together that would show that we had impact”? Our question led to a spirited discussion of not only what was traceable, but also what could be publicly posted and ‘ground-truthed’ by the community. Discussing indicators led to even deeper conversations about the causes of food insecurity which were illuminating to staff. What was surprising, for instance, was the extent to which families saw changing seasonal child-field labor practices in favor of 100% child-school attendance as great indicators. School attendance (or lack thereof) was dependent on families’ need for children’s seasonal labor in the fields. Community members said they knew who sent their children or not, which no only ‘cleaned’ the publicly posted data but triangulated implementer surveys and opened room for discussions of vulnerability.
Not only is this exciting for the project’s outcome tracking but even more importantly, our team proposed to create a community self-monitoring system, suggested in by Causemann/Gohl in an IIED PLA Notes article– “Tools for measuring change: self-assessment by communities” used in Africa and Asia. This learning, management and reporting process will fill a gaping need as current “monitoring systems serve only for donor accountability, but neither add value for poor people nor for the implementing NGOs because they do not improve effectiveness on the ground.” The authors found that not only “participatory data collection produces higher quality data in some fields than standard extractive methodologies [as] understanding the context leads to a higher accuracy of data and learning processes [which] increase the level of accountability… “ but also that such shared collaboration builds mutual learning and bridge-building.” While our community members may have offered to track this publicly to make this partner happy, men and women discussed this excitedly and embraced the idea of self-monitoring happily. ERCS will be discussing with communities to either track data monthly in notebooks or on a large chart hung in the woreda office for transparency. Data (Chart 1) would include these asset and quality of life indicators as well as loan repayments (tracked vertically) while households (tracked horizontally) could see who was meeting the goal (checked boxes), not meeting it fully (dashed boxes) or not meeting it at all yet (blank boxes). Community members corrected each other as they devised the indicators during our participatory research and this openness reassures us that the public monitoring will be quite transparent as well.
Further, what was especially satisfying was getting feedback from across the three tibias (sub-regions) on what activities they felt they could sustain themselves irrespective of the project’s continuation. Table 2 shows us which activities communities felt were most self-sustainable by households; these could form the core of the follow on project. Sheet/goats, poultry and oxen for fattening were highly prioritized by both women and men, in addition to a few choosing improved dairy cows. The convergence of similar responses was gratifying and somewhat unexpected, as there were several other project activities. The communities’ own priorities need to be seriously considered as currently they get only one loan per family and thus self-sustainable activities are key.
There is more to incorporate in future project planning by NGOs like ERCS. The NGO-IDEAs concept mentioned above also includes involving project participants in setting goals and targets themselves, differentiating between who achieved them and why, and brainstorming who/what contributes to it and what they should do next. Peer groups, development agencies and any actors could collect and learn from the data. Imagine the empowerment were communities to design, monitor and evaluate and tell us as their audience!
And they must, according to ODI UK’s Watkins, who has a clear vision on how to achieve a global equity agenda for the post-2015 MDG goals. He suggests converting the principle of ‘leave no one behind’ into measurable targets. He argues that, by introducing a series of ‘stepping stone’ benchmarks, the world can set ambitious goals on equity by 2030. He writes, wisely, that “narrowing these equity deficits is not just an ethical imperative but a condition for accelerated progress towards the ambitious 2030 targets. There are no policy blueprints. However, the toolkit for governments actively seeking to narrow disparities …has to include some key elements [such as] identifying who is being left behind and why is an obvious starting point. That’s why improvements to the quality of data available to policy-makers is an equity issue in its own right”. Valuing Voices believes who creates that data is an equally compelling equity issue.
So how will we reach these ambitious targets by 2030? By putting in stepping stone targets, returning project design functions to the ultimate clients – the communities themselves- and matching their wants with what we long to transfer to them. In this way we will be Valuing their Voices so much that they evaluate our projects jointly and we can respond. That’s how it should always have been.
What are your thoughts on this? We long to know.
 Ashley, H., Kenton, N., & Milligan, A. (Eds.). (2013). Tools for supporting sustainable natural resource management and livelihoods. Participatory Learning and Action, (66). Retrieved from https://pubs.iied.org/14620IIED/
 Watkins, K. (2013, October 17). Leaving no-one behind: An equity agenda for the post- 2015 goals. Retrieved from https://www.odi.org/blogs/7924-leaving-no-one-behind-equity-agenda-post-2015-goals
Aminata and I both want to be proud…
I met Aminata in Mali in 1990 during my doctoral research. She was a Bambara farmer and an impressive woman, with pride in her community. She was a helpful informant during my research on how communities cope with famine, and how famine early-warning systems could support them to do so more effectively (Note: 'early' warning is often far too late to prevent anything, I found, as donors don't want to intervene until far too late).
She's stayed in my mind these 24 years, and I want her to be proud of our international development work. I too want to be proud of what we accomplish to alleviate poverty and ill health by improving lives and livelihoods. We do so much in agriculture/ natural resource management, health and nutrition, education, etc. Yet do we start with communities' burning needs? End with their evaluating us? Or do we mostly use them to implement our ideas?
I've been part of the problem, part of the development 'industry' for 25 years. While I've used participatory methods such as PRA/ RRA and Appreciative Inquiry, I've mainly been focused on answering donor questions as to how successful we were, rather than helping communities ask the questions they want to answer and get the resources to succeed. I am no longer as proud of that work. We've left African Aminatas, Asian Aainas, Latina Adrienna's and millions like them behind in our rush to implement, to get more funds, to succeed, win more, 'develop' more…..
In founding ValuingVoices I put myself in their shoes, seeing projects come, projects go, temporary changes coming, some unintended results coming, some building on projects coming, but mostly going. It's never hard to celebrate local capacity when surrounded by vibrant voices, teaching us about their communities and their needs. What is hard is when so much of those decisions are not in their hands, but are at the mercy of donors who often decided not to support them (after our fieldwork showing both need and great capacity was presented): 'this region was not a priority' or 'we are now working on a new sectoral focus or a new initiative and this is no longer in our priorities.'
I am that much more deeply encouraged by a recent article on nutrition that points the way toward building these women’s pride as well as my own by starting with their voices and with donors being willing to listen. Nutrition research in Rwanda suggests that scaling up community-based solutions is the way forward. They decreased (stunting) malnutrition by 18% in five years which is very fast by "'setting up an almost universal community-based health insurance scheme… with the help of [each village determining its own way to tackle malnutrition… and not packaged interventions provided by donors" said Fidele Ngabo, director of Maternal Child Health. The article says "the Rwandan model could be used in other African countries, where foreign donor-driven initiatives tend to focus on treatment and technical solutions…. Change will only come when nutrition research is led by Africa, and interventions are designed to meet a country’s priorities, according to the findings of a two-year European Union-funded SUNRAY (SUstainable Nutrition Research for Africa in the Years to come)."
(a future Aminata?)
This groundbreaking research highlights the issue so many of us encounter. "Researchers from developed countries search for African partners for joint research, based on funding and research priorities defined outside Africa… so, despite enormous amounts of money spent on nutrition research and interventions, malnutrition rates have not fallen [in Benin]. Instead, the research agenda should be based on needs identified within the continent. Calls for research proposals of donors should match this agenda… [and prioritize] the locally identified needs and priorities." That sounds almost heretical in some circles.
In Benin, "researchers and policymakers wait for 'the dictate of donors before taking action. Hence, donor-funded programmes aren’t sustainable. As soon as they end, all activities are stopped, and acquired benefits and good practices are lost,' said Eunice Nago Koukoubou of the Université d'Abomey-Calavi in Benin, an author of the published findings." Yet the crux of the problem is that donor funding is filling a gap not filled by national governments. It isn't even so much that they are unwilling, rather in the case of Benin, “the government is trying to raise funds for the [strategic plan where nutrition should be central in development]", which also needs technical capacity building, a means of carrying on data retention and learning and a means of sharing findings with each other. What is needed is a "partnership between African researchers, 'who have more credibility and knowledge of the context', and Western researchers with the resources and opportunities (e.g. the African Economic Research Consortium).
Until we get to national partnership, let's improve our local ones. Let's really listen to participants in co-designing activities in projects we still think are best. Let's build on and share SUNRAY's approach of starting with what Aminantas, Aainas, Adriennas want and get Rwanda's great results. Let's partner in the truest sense of the world. After all, this excellent article underscores that "ultimately it is about political will…. [some] who feel they lack resources to tackle their long-standing battle with chronic malnutrition have to realize that “your children are not the donors' children, they are yours." They are our children, at least we need to treat them with respect as if they were.
What can we learn from Ex-Post Evaluations?
In trying to learn more about sustainable development solutions, the first place to look for information is in ex-post evaluations, also commonly called post-project evaluations, which are conducted by either development organizations themselves or by independent external evaluators. Unlike final project evaluations, which are completed at the time of a project’s conclusion to assess whether or not it has achieved its intended goals, an ex-post evaluation is conducted in the years after a project’s official end date – maybe one, three, or five years after the fact. An ex-post evaluation is a highly valuable tool for determining not just how successful a development project may have been after resources and international funding were withdrawn, but rather the long-term sustainability of the outcomes for the community members who were being ‘developed’.
With the seemingly obvious necessity for ex-post evaluations to gaining a better understanding of both positive and negative development practice, I was surprised by how hard it was to actually find any. Some organizations are diligent about conducting post-project evaluations and documenting the results for future reference, namely the development assistance organization Japan International Cooperation Agency (JICA), which has an extensive reference database to search its ex-post evaluations. However, this is certainly not the norm (yet), or if organizations are conducting ex-post evaluations they are not making the information widely available to the public. My research process included search terms such as, “ex-post evaluations by international development organizations”, “post-project evaluations”, and “impact evaluations.” In using these generic search terms, I was only moderately successful in finding helpful evaluations for my reserch, which suggests the need for more readily accessible information to the public about development outcomes.
We also found that some organizations had completed these evaluations, but they were at times too vague to obtain much useful information from. Out of about 10-15 evaluations that we found so far, there were only around 7 that were clear and organized enough to include in my table of summaries. (My search was limited to projects that were conducted predominantly at the community level, rather than at the municipal or state level.) The variable quality of these evaluations has a negative impact on their usefulness – if an ex-post evaluation is in an unsearchable format or doesn’t follow a fairly standardized organization, how will it be able to inform future projects efficiently? Additionally, it would be much easier for project coordinators to learn from past projects, and even other organizations, if there existed a more accessible and methodical database to make searching for ex-post evaluations simple. Despite these challenges, I have included five different evaluations from my preliminary research with which I was able to compare results for a better understanding of how to achieve sustainable project outcomes. The framework used for analyzing these evaluations considered:
- The sector of the development project (i.e. food security, poverty reduction, agricultrual development);
- the implementing organization and the evaluating organization (if it was different);
- the dates and gap between the project and the ex-post evaluation;
- the project objectives;
- specific ex-post evaluation methods;
- the positive/sustainable outcomes;
- the negative/unsustainable outcomes;
- the transfer to authorities;
- the amount of money invested overall;
- and the level of local participation.
The five evaluations analyzed include:
For a full summary of these evaluations, please see the Ex-Post Evaluations Summary Table. Here, are brief synopses of the most pertinent information for the above framework of analysis, and the table provides a better context for our conclusions.
Here are the key findings from the various ex-post evaluations that we found to be most significant:
- Over 18 million USD were spent on the five combined projects, but most projects did not explicitly enumerate how many people/households were impacted by the individual projects. An exception to this is the project in Mauritius, which reported reaching around 3,500 people. Without understanding the scale of the program, it is difficult to compare projects directly to one another.
- Mercy Corps’ MILK Project in Niger was inclusive and participatory in its ex-post evaluation process, which resulted in hard data that can easily be analyzed, compared, and learned from in future projects. In addition, this evaluation utilized a unique pictoral tool developed specifically to include all project participants in the feedback loop, despite widespread illiteracy, so that every individual had the opportunity to provide their insight on project impacts.
- JICA’s Ethiopian agricultural development program involved community participation in the project from the earliest planning phases, with 100% of members reporting that they had “participated” or “actively participated” in the process. This resulted in feelings of greater personal ownership of the project, and heightened local understanding of their responsibilities.
- Evaluations that included direct community feedback in their analyses were by far the most helpful when trying to determine sustainability. For instance, in JICA’s Agricultural Development Project in the Kambia District of Sierra Leone there was no mention of local level involvement throughout any of the stages of project planning, implementation, or evaluation, which could have influenced why the project only “somewhat” achieved its objectives
- Projects that have flexible agendas, willing to change with the changing needs of the population during the planning/implementation phases, are viewed positively by the developing community and achieve more successful outcomes. This willingness to adapt was what characterized the project in GVC OLNUS Argentine Puna. Considering the true, up-to-date needs of the community allowed for greater local participation that enabled the strengthening of local autonomy (and thus, sustainability).
- None of the project evaluations provided a breakdown of how successful budget allocation was. The JICA projects included a breakdown of the overall budget into equipment and local costs, however despite some evaluations noting who provided certain funding, none mentioned if parts of the budget were inefficiently used. We believe it would be helpful to include not just how much money was invested in a project, but also how much of that budget either prompted direct growth or failed to produce an effective outcome.
Local community members are often referred to as ‘beneficiaries’ in the development process, yet they are the ones who governments, NGOs, and multilateral organizations are trying to empower through their various socioeconomic development missions. So, when we need to understand what worked with a project, and as importantly what didn’t work for a project, it is the voices of the community that need to be heard. A lot of great work is being done in international development, but it is clear that after her initial research that ex-post evaluations are essential to determining project sustainability and that projects that propose community-level development must also take the time to directly involve those community members in their own evaluation process. This feedback loop has the power to inform and influence future projects, while also creating the opportunity to actually listen to what participants (not beneficiaries) can sustain for themselves to achieve a better life.
Where have you found feedback loops that work? What excellent programming can you share?
 Nishimaki, R., Kunihiro, H., & Tahashi, S. (2008, July 8). Evaluation Result Summary: The Project for Irrigation Farming Improvement. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/tech_and_grant/project/term/africa/c8h0vm000001rp75-att/ethiopia_2008_01.pdf
 Kumagai, M., Otsuka, M., & Sakagami, J. (2009, September 26). Evaluation Result Summary: The Agricultural Development Project in Kambia in the Republic of Sierra Leone. Retrieved from https://www.jica.go.jp/english/our_work/evaluation/tech_and_grant/project/term/africa/c8h0vm000001rp75-att/ethiopia_2008_01.pdf
 The Improve Group. (2012, December). Post Project Evaluation of Mercy Corps’ MILK Program in Niger: Examining Contributions to Resilience. Retrieved from https://www.yumpu.com/en/document/read/35930718/niger-milk-post-project-evaluation-final-report-mercy-corps
 Proatec SRL. (2013, March). Ex Post Evaluation of Projects Managed by NGOs in Argentina. Retrieved from https://www.oecd.org/derec/italy/Evalutation-of-Projects-Managed-by-NGOs-in-Argentina.pdf
 International Fund for Agricultural Development (IFAD). (1997, August). Small-Scale Agricultural Development Project – Ex-post Evaluation. Retrieved from https://www.ifad.org/en/web/ioe/evaluation/asset/39828071
Time is ripe for incentives to change….
There is so much promise now in returning international development to the hands of those who we mean to ‘develop’. Here are a few examples:
•Accountability has been proposed as a core feature of the new post-2015 development agenda, according to UNICEF.
•The UN’s 2011 Busan Declaration focused on improving sustainable development, aid effectiveness through international cooperation
•Major foundations (Rockefeller, Gates) are promoting participatory evaluation and program effectiveness (respectively), looking for long-term impact.
•An industry acceptance of how mobile technologies and knowledge sharing democratizes information and power, e.g. USAID’s mobile applications and the power of crowd-sourcing, evidenced by amazingly robust African applications of Kenya’s Ushahidi, and sectorally specific ones such as Ghana’s Esoko for agricultural value chains and Microplanet's microcredit, to name a few.
Many of us have come to sense that we’ve had the incentives wrong in development. While many of us go into international development feeling the strong need to work with communities, to support their own path to sustained well-being, it doesn’t often work that way once we begin. Following on another blog, we push ourselves to get more, do more, but we end up doing work for- or to- rather than with- our participants. Many of our projects still refer to them as beneficiaries (a more passive term was never coined). Many of our project timelines and result expectations are so rigid that there is no luxury to involve communities in the detailed design, implementation much less getting their feedback on how sustained impact of the project is likely to be (or how the project could be adapted to yield sustained results).
This is not to say our projects don’t do good! We do. We train participants and transfer a dizzying array of resources to them during the 2, 3, 5 even 10 years we intervene. We work within often dysfunctional national systems with scant resources. But how often have we built capacity for communities, regions and country-level nationals to continue doing development ‘right’ after we go?
Ironically, this lack of sustained capacity is in large part this is due to our deep desire to do it ‘right’. Right means getting as many resources out there as possible, getting to as many people as possible, getting as many projects done, and getting as many new awards for our firms to do more good as possible. And therein lies the problem. In our desire to do things right for ourselves, we leave our participants behind.
And most egregiously, we stack the deck against sustainability from the start. The incentives are framed by the Requests for Proposals written in donor offices in Washington, London, Paris, Tokyo and even Beijing. Today, USAID alone has 1075 opportunities on FedBixOpps.gov. Were these drafted with national governments, local community based organizations or communities? Unlikely. Mostly these opportunities are informed by updated versions of past work, sometimes by new policy directions, possibly by new think-tanks research or international non-profit fieldwork, yet the country Missions, implementer offices or community-based organizations can only do so much to ‘fix’ things in the field to retrofit what communities may actually want to do specifically but what was not a priority in the RFP. Having also been part of a few proposal development teams that wrote proposals answering RFPs based on little field research, this sadly exacerbates the original RFP-wasn't-drafted-in-country problem. It is again our lack of time, our rush to do good (including getting our overhead from projects to maintain our organizations) that is our barrier to actually doing it sustainably. With communities leading.
But "times are a changing" and new incentives and directions are appearing at USAID such as IDEA's Local Solutions or PPL's Evidence Summit. New opportunities have been used to engage countries in public-private-partnership agricultural livelihood collaborative programming such as via NEPAD's CAADP (and USAID's Feed the Future programs in a dozen countries). Combined with above-mentioned new funding for capacity building evaluation by Rockefeller Foundation, and new mobile technology ways to value and pass on grassroots voices are all promising. These provide ways to dramatically tweak incentives, to feed in local voices. Hallelujah.
What do you think? Are our incentives righter than I see? Are our Requests for Proposal actually mirroring more closely country-national desires for development their way? Please share your learning… so we can truly support their development their way.
What could we know… about international development, ourselves, 'them' and the intersection?
How would international development look from the eyes of the participants? What works best? What fails? Who decides what success and failure is, and how much do we even know what they think, 3, 5, 10 years after projects end?
And who is 'we' in the sentence anyway?
This blog will stream my view and those of and my colleagues in my field, many of whom have been working as I have over 20 years across Africa, Latin America, Asia, Balkans and many corners of the world, as well as those from these very corners. I hope to be a bridge for knowledge sharing between 'old timers', those new in my field, as well as between the 'North' and "South', sharing evidence, trends, stories, and promoting ideas like:
* Using Appreciative Inquiry to celebrate what works best and how to do more of it, as I did for the Bill and Melinda Gates Foundation, USAID and Johns Hopkins University et al, leads to more knowledge sharing, excitement and nurtured souls than a focus on what is broken.
* Country nationals should evaluate their own projects and programs – it's their countries, after all
* Participants best know what success looks like in their own communities – including them in their own discovery, design, monitoring and evaluation is paramount for success and sustainability
* No 'development' project with behavior change as any part of it should be shorter than 10 years, heck it takes 3 for communities and non-profits to get to know and trust each other…
* Donors should go back 3, 5, 10 years later to see what exists now, what communities valued enough to keep up themselves after project funds and staff left.
* There is great need for compassion in development. Parts of the system are broken when we development workers push projects designed abroad, don't have time we know is vital to take to involve participants, feel such pressure to perform, to prove impact that we work 80 hours a week, race to meet donor reporting requirements…
There are so many fascinating trends out there right now which we need to address mindfullly, and understand how our participants need our help to address:
* Over 50% of the world's population is under the age of 30! So should we design youth-centric development? Have them as our main respondents and designers?
* Even though we produce 1 ½ times enough food for every man, woman and child on the planet, nearly a billion people go hungry while over a billion are malnourished …. but social movements are changing that. How to address redistribution issues, water scarcity, food waste, etc, together?
* Women produce 60% of the world's food, get 10% of its income and own 1% of the world's property. We are making progress on women's rights by deeply embedding women in development, looking at activities through a 'gender lens'. How much more could we do if they designed programming to fit their circumstances?
* Financial and Youth demographics in Africa show will tip the scales in terms of country-led development as "Africans are perfectly capable of representing themselves and developing in ways of their own choosing. The African diaspora is making massive contributions to their countries-of-origin, not just in terms of sending back money (about $50 billion annually), but also in terms of reclaiming the development discourse."
How are we addressing, including these as well as 'lessons learned' from the past and envisioning a collaborative, kind future for all of us, where happiness reigns?
What else could we know?…