Aid providers: More puzzle pieces, including unexpected outcomes; ours is not the whole picture
When we did our first ex-post evaluation/ delayed final evaluation in 2006 in Niger for Lutheran World Relief (LWR) funded by the Bill & Melinda Gates Foundation (pg75 on), we found all sorts of unexpected/ unintended outcomes and impacts that far outweighed the original aid’s expectations. The project measured success by the livelihoods rebuilding post-drought ravaged sheep herds and water points for them. Instead, while the LWR aid was beautifully based on ‘habbanaye,’ a pastoralist practice of lending or giving small-stock offspring to poorer family members (and was expanded to passing on animals to poorer community members), and results showed a majority of the poor benefitted, our respondents showed it was far more nuanced. We aid providers (and our expectations) are only a part of any ‘impact’, which needs to be defined by the communities themselves:
- While many families benefitted from the sheep, enabling some young boys to shepherd several at a time, it turns out the poorest chosen by the communities were not necessarily ones who lost small livestock during the drought but were, in fact, the ultra-poor who had never had them. Therefore, proof of successful ‘restocking’ post-drought left these poor who were helped the most, out, as they were… unexpected;
- Holding onto the donated sheep was not as important an indicator for some: one woman told us that selling her aid-received sheep to buy her daughter dowry to marry a wealthier husband was a far better investment for their financial future than the sheep would have been. Our main measure of success was not nuanced enough;
- The provision of water through well-rehabilitation and building in the five villages was a vital resource. Women reported they saved 8 hours every two days by having potable water in their villages. Before then, they spent three hours walking each way to the far-off well and waited two hours to fetch 50l water, which they head-carried back. With the well in place, they generated household income through weaving mats, cooking food for sale, etc., which amounted to as much as 20% of increased household income – a boon! Also, having both time and water access enabled them to bathe themselves and their children, make their husbands lunch, and make their mothers-in-law tea, which led to far more household harmony. No ‘impact’ measures outside of livestock and water were included or could be added;
- Finally, the resulting show-stopper: in the last village where we interviewed women participants, they said that the groups of fellow recipients were a boon for community solidarity across ethnic groups. In their meetings, thanks to the sheep, water, and collective moral support, they said the conversations turned from conflict to collaboration, and best of all, women reported, “our husbands don’t beat us anymore.” Peace among ethnic groups, within and between households was completely unexpected.
Such a highly unexpected outcome would fall under #2 and #4 of the Netherlands study below. Unfortunately, the Foundation seemed less interested in these unexpected but stellar results. Yet at the same time I have empathy for their position, as so many of us in global development want to help solve problems, and demand proof we have.…. so we can leave and help others, equally deserving. Taking complexity into account, seeing lives in a wider context where our aid can be helping differently or even harming makes garnering more aid hard.
As the brilliant Time to Listen series by CDA showed, aid intervenes in people’s lives in complex ways and we need to listen to our participants and partners who always share more complex views than our reports can honor..A few of hundreds of quotes of 6000 interviews, this from the Solomon Islands:
Some appreciate the aid as it is given:
“People in my village are very grateful for the road because now with trucks coming into our village, the women can now take their vegetables to the market. Before, the tomatoes just rotted in the gardens. Tomatoes go bad quickly and despite our attempts in the past to take them to the market to sell, we always lost.” Woman from East Malaita
But for others there are great caveats:
“Donors should send their officers to Solomon Islands to implement activities in urban and rural areas. This will help them understand the difficulties we often face with people, environment, culture, geography, etc. ‘no expectem evri ting bae stret’ [Don’t expect everything to go right].” Man, Honiara
“They have their own charters, sometimes we might want to go another way but they don’t want to touch that. So sometimes there is some conflict there; some projects are not really what we would like to address – because the donors only want to do one component, and not another, because it is sensitive, or because they want quick results and to get out.” Government official, Honiara
“What changes have I noticed since independence? Whatever development you see here is due to individual struggles. No single aid program is sustainable. NGOs are created by donors and are comfortable with who they know. NGOs eat up the bulk of help intended for the communities. NGOs become international employers. They do their own thing in our province. Most projects have no impact. I want to say stop all aid except for education and health. If international assistance concentrates on quality education and health, the educated and healthy people will take care of themselves.” Government official, Auki, Malaita
“The most important impacts of aid people do not think about – they are not listed, not planned, they are remote, but these are the longest lasting. Often they are the opposite of the stated objectives. So remote, unintended, unexpected impacts are very often more important and more lasting and more dramatic than the short term intended, measured ones.” Aid consultant, Honiara
How widespread is our myopic focus on our intended results? A recent Netherlands Foreign Aid IOB study found unintended effects were an evaluator’s blindspot as across 664 evaluations over 20 years, “The ‘text miners’ found that only 1 in 6 IOB evaluation documents pay attention to unintended effects.” This dearth of attention to all the other things happening in projects led to 10 micro, macro, meso, and multiple level effects, from negative price effects such as food aid on local food producers or nationalist backlash to Afghan projects to positive effects (they found 40% of projects had this) such as a harbor built happening to expand beach tourism as well.
But if we don’t look for such effects, we don’t know the true impact of our aid programming. We also don’t honor the breadth of people’s lives rather than just as narrow ‘aid beneficiaries’ (ugh), not even honoring them with the terms’ participants’ much less ‘partners’ in their own development).
In our ex-post work, we find a wide array of ways ownership and implementation of activities is done after donors leave and without additional or with different resources, capacities, and partnerships. Taking emerging outcomes and impacts examples from a different Niger project, and one from Nepal:
1. Partnerships & Ownership: Half of the members of the all-women Village Banks reported helping one another deal with domestic disputes and violence. (Pact/Nepal)
2. Capacities: Trained local women charged rates to sell course materials onward (PACT/Nepal)
3. Ownership: Participants valued clinic-based birthing and sustained it by introducing locally-created social punishments and incentives (CRS/Niger)
4. Resources: New Ministry funding reallocated to sustain [health] investments, and private traders generated large crop purchases and contracts (CRS/Niger)
The assets and capacities we bring to help people and their country systems help only a sliver of their lives, and often in unexpected ways that sadly we aid donors and implementers don’t seem interested in. There are other puzzle pieces to add…
Let us not forget, as a Sustainable Development Goals Evaluation colleague said in 2017 on a call:
I am sure you, my dear colleagues, have reams of similar findings from your fieldwork. Please share yours!