What’s likely to ‘stand’ after we go? A new consideration in project design and evaluation
This spring I had the opportunity to not only evaluate a food security project but also to use the knowledge gleaned for the follow-on project design. This Ethiopian Red Cross (ERCS) project “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity” (with Federation and Swedish Red Cross support) was targeted to 2,259 households in Dedba, Dergajen and Shibta through provision of crossbreed cows, ox fattening, sheep/goats, beehives and poultry which were to be repaid in cash over time as well water and agriculture/ seedlings for environmental resilience. ERCS had been working with the Ethiopian government to provide credit for these food security inputs to households in Tigray which were to be repaid in cash over time. During this evaluation, we met with 168 respondents (8% of total project participants).
Not only were we looking for food consumption impacts (which were very good), and income impacts (good), we also probed for self-sustainability of activities. My evaluation team and I asked 52 of these participants more in-depth questions on income and self-sustainability preferences. In Tigray, Ethiopia, we used participatory methods to learn what they felt they could most sustain themselves after they repaid the credit and the project moved on to new participants and communities.
We also asked the to rank what input provided the greatest source of income. The largest income (above 30,000 birr or $1,500) was earned from dairy and oxen fattening, while a range of dairy, oxen, shoats and beehives provided over half of our respondents (40 people) smaller amounts between 1,000-10,000 birr ($50 to $500).
And even while 87% of total loans were for ox fattening, dairy cows (and beehives) which brought in farm more income, and only 11% of loans were sheep/goats (shoats) and 2% for poultry, the self-sustainability feedback was clear. In the chart below, poultry and shoats (and to a lesser degree, ox fattening) were what men and women felt they could self-sustain. In descending order, the vast majority of participants prioritized these activities:
To learn more about how we discussed that Ethiopian participants can self-monitor, see blog.
So how can such a listening and learning approach feed program success and sustainability? We need to sit with communities to discuss the project’s objectives during design plus manage our/ our donors’ impact expectations:
1) If raising income in the short-term is the goal, the project could only have offered dairy and ox fattening to the communities as their incomes gained the most. Note, fewer took this risk as the credit for these assets was costly.
2) If they took a longer view, investing in what communities felt they could self-sustain, then poultry and sheep/goats were the activities to promote. This is because more people (especially women, who preferred poultry 15:1 and shoats 2:1 compared to men ) could afford these smaller amounts of credit as well as the feed to sustain them.
3) In order to learn about true impacts we must return post-project close to confirm the extent to which income increases continued, as well as the degree to which communities were truly able to self-sustain the activities the project enabled them to launch. How do our goals fit with the communities’?
What is important is seeing community actors, our participants as the experts. It is their lives and livelihoods, and not one of us in international development is living there except them…
What are your questions and thoughts? Have you seen such tradeoffs? We long to know…
[*NB: There were other inputs (water, health, natural resource conservation) which are separate from this discussion.]
Excellent point about a key element missing in international development projects – iterative design. Humans have a long history of learning from our history – heck, we even invented writing as a technology to share lessons learned across time and space.
One approach to implementing better foreign assistance is to borrow what works from other "high risk" endeavors – software. Software is notorious for being high risk – look at healthcare.gov for a recent example. So how does software reduce the risk? Through constant iterative design.
In software, the "agile" method of development means you spiral towards higher risk by starting with lower risk interventions. You start your first phase with building just enough software to get it into the hands of users to solve immediate needs that they have clearly identified – the "low hanging fruit". You get real time feedback both from asking them what they think, as well as observing what they are actually doing (since people can sometimes say things to be nice or think reflects well on them). You capture this feedback – what is working and what is not working – and build that into your phase two.
Phase two is easier because you already have developed a relationship with your users, have a solid infrastructure to build on, and you are building engagement. When users see their ideas reflected in your design, they feel listened to and they feel valued. They then are much more likely to be advocates for the project and promote it to others. It also goes more easily because if something is NOT working, you have the opportunity to drop or restructure at each phase – do more of what is working and less of what is not.
The problem for international development to use this approach is donors often want to see the full picture of what they get for their funds too early. They are not used to the "we will see what works and do more of that" approach – it makes them nervous that nothing will work or they are funding a high risk venture. But this approach actually lowers risk to the project, because it is responsive to the very people it is meant to benefit. It isn't about pleasing the funder, but rather the beneficiaries.
Siobhan, this is excellent! Exactly what risk-averse participants need, an iterative approach. And if we can involve them early enough, e.g. during design or midterm evaluations to give feedback on what's working or not, and through two-way feedback loops (they tell us what's working best or worst, we reflect, adapt/ are 'agile' as you say and give them feedback on what we fixed as a result), confidence is built that it's a joint enterprise… that builds up their confidence that we not only Value their Voices but also that they're empowered to tell us more about how to serve their needs best – during implementation, close-out and new design. Imagine having such lessons in accessible repositories (as you've told me before!) and folks learning who are coming anew to the project. Wow. Impact.