by Jindra Cekan | Feb 13, 2016 | Aid effectiveness, Bill and Melinda Gates Foundation, Catholic Relief Services (CRS), Central Asia, Ethiopia, Ethiopian Red Cross, Evaluation, Federation of the Red Cross, Lutheran World Relief, Mercy Corps, Niger, PACT, Partners for Democratic Change, post-project evaluation, Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation, USAID
What happens after the project ends?
Lessons from post-project sustained impacts evaluations (Part 1)
We talk a lot about impact of our interventions, but far less is analyzed about the sustained impact of our work in the years after projects close out. We take for granted that successful strategies will continue after projects shut down. Do they always? Maybe they do but we don’t know. Maybe there are innovations and impacts to be learned from…
To answer these question Valuing Voices spent 2 ½ years looking for and analyzing ‘post-project’ evaluations of projects undertaken 2-10 years after projects ended. The result: in our $137 billon international development industry, some 99% of projects remain unevaluated after project close out [1]. Only 17 agencies we have found so far have publically available post-project evaluations; most of them have one, while the OECD and JICA have dozens. Hundreds of studies recommend such learning that is missing from our industry’s program cycle (green slice).
Six decades on, this astonishing finding raises serious questions about stewardship of resources and commitment to learning—particularly learning from participant and partner stakeholders for whom sustainability matters most, and who are tasked with it over the long-term.
A review of post project evaluations generate food for thought about good program design and illustrate the value added of post project perspectives. This rapid review of select ‘ex-post’ evaluations points to three early lessons:
How we do it matters for great results
Expect unexpected results
Who takes over? Country Nationals
First, organizations go back to see how well their projects results were sustained. What we learned was that how well they used participatory processes in how they implemented and handed over mattered a lot for sustainability and that we must expect unexpected results.
How we do it matters for great results
1. Catholic Relief Services/ USAID PROSAN food security project in Niger
Valuing Voices evaluated this food security (agriculture, health and resilience project in 2015 which ran from 2007-2012 (report forthcoming). The $32 million project was implemented in a consortium of CRS, CARE and Helen Keller but this evaluation focused only on CRS areas.
- Interviews with over 500 participants found that three years after project closeout, 80% of project activities still continued, as did many village committees and there were a variety of community innovations [2]
- On average, households can feed themselves through their own production or purchase of food for 8-12 months three years after closeout compared to 6-9 months at closeout three years earlier. Such impact was unexpected. [2]
- 91% of respondents reported improved household health, hygiene, and nutrition [2]
- A 2.5-year participatory exit process from CRS to country stakeholders (local government, an array of local and international NGOs and the private sector ensured continuity and boosted local ownership [2]
- 20% of the activities did not continue, mostly food-assisted NRM and resilience-related [2]
- Youth make up 50% of the population and need to be engaged during the project for long term sustainability to occur. [2]
2. PartnersGlobal (formerly Partners for Democratic Change)
Their mission is to “to build sustainable capacity to advance civil society and a culture of change and conflict management worldwide” uses an approach that is “bottom-up, locally-led rather than foreign-led, based on the belief that change comes from sustainable efforts led by local people, organizations and institutions invested in their own long-term future.” They went back and to review 55 case studies of projects through 22 centers they founded in central and eastern Europe from 1989-2011. They found:
- In 80% of cases, there was advancement of good governance by influencing the participation of civil society working with government [3]
- In 50% of the cases there was increased access to justice and managing and resolving disputes/conflicts, thereby strengthening civil society [3]
- 18 of 22 of the centers that had been established still exist today (82%) [3].
3. Mercy Corps
They did a post-project evaluation in Central Asia in 2007, one and three years after two conflict resolution projects ended which were worth $18 million. These complex community mobilization programs with aims “to empower communities to work together in a participatory manner to address the infrastructure and social needs [while] developing sustainable skills [and] empowers communities to identify and utilize existing resources within the communities and not to depend only on external assistance.”
- 72% of youth report that they continue to use at least one skill they learned during the programs (e.g. teamwork and communication, and skills such as sewing, construction, roofing, journalism and cooking) [4]
- 68% of community members witnessed local government becoming more involved in community activities after the end of the programs as compared with before the programs [4]
- 57% of the communities studied continuing to use one or more of the decision-making practices promoted during the program [4]
- 42% of members, representing 70% of communities, reported that the community had worked collectively on new projects or repairs to existing infrastructure. Participants and partners had implemented almost 100 infrastructure projects by themselves independent of donor funds. [4]
These are terrific expected results.
We also learned to Expect unexpected results
4. Federation of the Red Cross and Ethiopian Red Cross
Valuing Voices combined a final evaluation of “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity” that had funding over $3 million in 2009 from the Swedish Red Cross with an assessment of projected sustainability. It was an IFRC/ERCS collaboration with the Ethiopian government to provide credit for food security inputs to 2,259 households, which were to be repaid in cash over time as well water and agriculture/ seedlings for environmental resilience. We answered the DAC criteria for evaluation and found the project overall to be quite good, albeit with weak data tracking systems.
In terms of sustainability, we used participatory methods to learn about what people felt they could self-sustain once the project left their area, so we could shape a similar follow-on project design to be moved elsewhere in Tigray, particularly around the credit for animals.
- While 87% of the loans had been promoted by the government and given for large animals (oxen and cows), and 13% was for small animals (sheep, goats, chickens)…
- But project participants we interviewed, strongly preferred the small animals in terms of being able to sustain them on their own. They felt they could afford these smaller amounts of credit as well as the feed to sustain them, without taking the risk of animal death leaving them with large debt. This was especially true for women, who preferred poultry to all other animals 15:1.
In our quest for fast results, are we asking participants to bear too much risk? As one of our Valuing Voices team asks, Who is responsible for sustainability?
5. Lutheran World Relief
From 2005-2007 Lutheran World Relief intervened in Niger, the world’s poorest country, with a $500,000 Pastoralist Survival and Recovery Programme (ARVIP) drought rehabilitation project funded by the Bill and Melinda Gates Foundation. There were numerous outcomes from targeting sheep, wells and animal fodder to 600 of the poorest women in 10 communities in northern Niger, among them:
- Women’s share of household income increased from 5% to 25% in some households. This was due to the value of the sheep grants, as well as time-savings used for income generation. Access to wells in five of the villages saved women a staggering 7-10 hours every other day from not having to go fetch water 3.5 hours away each way for household and animal needs and were free to weave mats or cook food for sale [5]
- Many said they didn’t have to resort to worse survival strategies during the next hungry season after they received the sheep [5]
What was not expected were these results:
- Many women in several villages reported an impact that was completely unexpected to the implementer and donor which was “our husbands don’t beat us anymore” [5]. This was thanks to both increased respect and income from the sheep as well as access to well-water which led to cleanliness and their ability to be home for their husbands, children and mothers-in-law, rather than fetching water whole days 2-3 times a week. The same was found in PACT’s WORTH empowerment and village banking project in Nepal that wrote, “one in 10 reported that WORTH has actually helped “change her life” because of its impact on domestic violence” [6].
- We defined success too narrowly. Many interviewees were content with the project even though prospects for project-expected drought resilience or sustained food security were less likely. Some women sold the sheep to buy food, pay their children’s school fees or their daughters’ dowries, while some had their sheep sold by their husbands who used them to buy other animals, pay for ceremonies or other expenses. Participants saw the project as bringing them resources and considered it a success. Spending assets on immediate needs is not at all illogical for a community who can feed itself only 4 months a year; for some households, their pressing needs far outweighed the luxury to wait and buffer seasonal food insecurity far down the line.
We hope you agree that allocating funds and attention to post-project sustained impacts evaluations is necessary for the remaining 99% of international development projects as it offers a fantastic learning opportunity about how to ‘do development’ well now and for the future. Without returning to look for what participants and partners valued enough to continue on their own, without returning to learn about unexpected sustained impacts, we rob ourselves of pivotal learning needed for success.
In part two, we look at ownership onward, planning for handover and lessons from who takes over? Country Nationals. In part three, we focus on Funding, Assumptions and Fears.
Please join us in advocating for this! Please think about your own projects… and whether you have considered these things, or need our help. We’re listening!
Sources:
[1] OECD. (2015, December 22). Detailed final 2014 aid figures released by OECD/DAC. Retrieved from http://www.oecd.org/dac/stats/final2014oda.htm
[2] Cekan, J., PhD, Kagendo, R., Towns, A. (2016). Participation by All: The Keys to Sustainability of a CRS Food Security Project in Niger. Retrieved from https://www.crs.org/our-work-overseas/research-publications/participation-all
[3] Carstarphen, N., PhD. (2013, November). Sustainable Investment in Local Capacity for Democracy and Peace: A Global Evaluation of Partners for Democratic Change. Retrieved from https://www.partnersglobal.org/resource/sustainable-investment-in-local-capacity-for-democracy-and-peace/
[4] Westerman, B., & Sheard, S. (2007, December). Sustainability Field Study – Understanding What Promotes Lasting Change at the Community Level. Retrieved from https://reliefweb.int/report/world/sustainability-field-study-understanding-what-promotes-lasting-change-community-level
[5] Cekan, J. (2013). Increasing women’s incomes, increasing peace: Unexpected lessons from Niger. Participatory Learning and Action, (66), 75-82. Retrieved from https://pubs.iied.org/G03661/
[6] Mayoux, L. & Valley Research Group (2008, June). Women Ending Poverty: The WORTH Program in Nepal – Empowerment through Literacy, Banking and Business 1999-2007. Retrieved from https://www.findevgateway.org/case-study/2008/06/women-ending-poverty-worth-program-nepal-empowerment-through-literacy-banking
by valuingvoicesjin | Feb 16, 2015 | Bolivia, Catholic Relief Services (CRS), Ethiopia, ex-post evaluation, Exit strategies, Food for Peace (FFP), Food security, Honduras, India, International aid, Kenya, post-project evaluation, Project design, self-sustainability, Sustainable development
Learning from the Past… for Future Sustainability
Heading up Food Security for Catholic Relief Services (CRS) was my first international development job in 1995-1999 and I have watched this organization grow in its commitment to program quality and learning/ knowledge management ever since. At the time I oversaw 17 of of CRS' USAID/ Food for Peace (FFP) programs. So I was delighted that not only has CRS done an ex-post evaluation and used the findings for programming (e.g. the effectiveness of investing in a particular sector—for example, the importance of supporting girls’ education within a food security program) and also for advocacy (e.g. evaluation lessons from Rwandan peace-building projects seven years after the genocide informed CRS’ evolving approach to peace and justice strategies), but I get to celebrate FFP learning too. In addition to having consulted to USAID/PPL (Policy, Planning and Learning) and the FANTA project, all featured below, I went to Tufts University’s Fletcher School. Super to see great organizations learning about sustainability!
CRS’ 2007 project package guidance for implementation and guidance (ProPackII), described ex-post evaluation/sustainable impact evaluation’s aim “to determine which project interventions have been continued by project participants on their own [which] may contribute to future program design…. it is fair to say that NGOs rarely evaluate what remains following the withdrawal of project funding [which] is unfortunate [as] important lessons can be generated regarding factors that help to ensure project sustainability.”
A 2004 Catholic Relief Services excellent ex-post evaluation in Ethiopia was featured: “Looking at the past for better programming: dap I Ex-Post Assessment Report”. It assessed sustainability of Agriculture Natural Resource Management, and Food-Assisted Child Survival/Community Based Health Care programming, done as an internal evaluation by CRS staff and partners with document review, partner, government and community interviews. Results were mixed.
-
Some activities generated enough food and income that households could eat throughout the year and have some savings, making them more resilient against drought
-
Almost 100% of cropland bunding and irrigation practices for improved crop production were still being applied and buffered them during a subsequent drought
-
Health practices had also continued (e.g. trained traditional birth attendants had continued to provide services with high levels of enthusiasm and commitment, and increased levels of health care-seeking behaviors existed).
However, many other benefits and services had severely deteriorated:
-
Nearly all water committees had dissolved, fee collection was irregular or had been discontinued
-
Many water schemes were not operational
-
The centrally managed [tree] nurseries had been abandoned (given the existing management capacity of communities and government).
CRS/Ethiopia and its partners came to see that:
-
“The potable water strategy had over-focused on the technical aspects (“hardware”) while not paying enough attention to the community organizing dimensions and support by existing government services (“software”).
-
Even limited post-project follow-up by partners and government staff might have gone a long way towards mitigating the deterioration of project benefits and services.
What was terrific was that they “went on to use these findings and lessons learned from this ex-post evaluation to inform the design of similar projects in Ethiopia… while also raising awareness of these issues among partner staff”. The ex-post recommended increased planning for sustainability, setting up village management for post-project and incentive maintenance. Great learning, yet we have found few ex-posts at CRS or elsewhere. Our industry needs to explore issues such as those the evaluators posed: Was the lack of sustainability due to technical, institutional or financial faults in the programming? In other words, was the lack of self-sustainability due to the design/ aim of the activity itself or how it was implemented?
In 2013, USAID’s Food For Peace commissioned fascinating research on Exit Strategies. Tufts University went to Bolivia, Honduras, India and Kenya which were phasing out of Title II food aid to look at how to “ensure that the benefits of interventions are sustained after they end, [as] there is little rigorous evidence on the effectiveness of different types of exit strategies used in USAID Office of Food for Peace Title II development food aid prog
rams.” The research is to “assess the extent to which the programs’ impacts were maintained or improved and to help understand factors of success or failure in the specific exit strategies that were used.” They have made the important discernment that the effectiveness of Title II programs depends on both short-term impact and long-term sustainability.
The FANTA project (contractor) made the following preliminary results available:
-
Impact assessment at exit does not consistently predict sustained impact two years later…. It can be misleading.
-
Many activities, practices, and impacts across sectors declined over the two years after exit. These declines are related to inadequate design and implementation of sustainability strategies and exit processes.
-
There are specific ways to increase the likelihood of sustainability: Sustaining service provision and beneficiary utilization of services and practices depends on three critical factors: Resources, Technical and Management Capacity, Motivation
-
Withdrawal of free food rations or any other free input (as incentive) jeopardizes sustainability without consideration of substitute incentives. For instance,
-
Withdrawal of food was a disincentive for participation in and provision of [child] growth monitoring…. Resources and health system linkages are needed to sustain health activities
-
Motivation, capacity and resources are all needed to maintain water systems
-
Agriculture and Natural Resource Management suffered greatly when resource incentives disappeared
Their main recommendations are that sustainability should be built into the design from the beginning, program cycles are longer and exit is gradual.
CRS found the same issue of incentives as a barrier, as they did technical and (institutional) capacity/ motivation/ management issues. We have much to learn… at least we’ve started Valuing Voices and asking… and eventually designing for sustainability!
by valuingvoicesjin | Nov 18, 2014 | Accountability, Africa, Credit, Ethiopia, Evaluation, ex-post evaluation, Food security, impact evaluation, International aid, LIvelihoods, Local Participants, Participation, post-project evaluation, Project design, Rural Development, self-sustainability, Sustainability, Sustainable development
What’s likely to ‘stand’ after we go? A new consideration in project design and evaluation
This spring I had the opportunity to not only evaluate a food security project but also to use the knowledge gleaned for the follow-on project design. This Ethiopian Red Cross (ERCS) project “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity” (with Federation and Swedish Red Cross support) was targeted to 2,259 households in Dedba, Dergajen and Shibta through provision of crossbreed cows, ox fattening, sheep/goats, beehives and poultry which were to be repaid in cash over time as well water and agriculture/ seedlings for environmental resilience. ERCS had been working with the Ethiopian government to provide credit for these food security inputs to households in Tigray which were to be repaid in cash over time. During this evaluation, we met with 168 respondents (8% of total project participants).
Not only were we looking for food consumption impacts (which were very good), and income impacts (good), we also probed for self-sustainability of activities. My evaluation team and I asked 52 of these participants more in-depth questions on income and self-sustainability preferences. In Tigray, Ethiopia, we used participatory methods to learn what they felt they could most sustain themselves after they repaid the credit and the project moved on to new participants and communities.
We also asked the to rank what input provided the greatest source of income. The largest income (above 30,000 birr or $1,500) was earned from dairy and oxen fattening, while a range of dairy, oxen, shoats and beehives provided over half of our respondents (40 people) smaller amounts between 1,000-10,000 birr ($50 to $500).
![]()
And even while 87% of total loans were for ox fattening, dairy cows (and beehives) which brought in farm more income, and only 11% of loans were sheep/goats (shoats) and 2% for poultry, the self-sustainability feedback was clear. In the chart below, poultry and shoats (and to a lesser degree, ox fattening) were what men and women felt they could self-sustain. In descending order, the vast majority of participants prioritized these activities:
![]()
To learn more about how we discussed that Ethiopian participants can self-monitor, see blog.
So how can such a listening and learning approach feed program success and sustainability? We need to sit with communities to discuss the project’s objectives during design plus manage our/ our donors’ impact expectations:
1) If raising income in the short-term is the goal, the project could only have offered dairy and ox fattening to the communities as their incomes gained the most. Note, fewer took this risk as the credit for these assets was costly.
2) If they took a longer view, investing in what communities felt they could self-sustain, then poultry and sheep/goats were the activities to promote. This is because more people (especially women, who preferred poultry 15:1 and shoats 2:1 compared to men ) could afford these smaller amounts of credit as well as the feed to sustain them.
3) In order to learn about true impacts we must return post-project close to confirm the extent to which income increases continued, as well as the degree to which communities were truly able to self-sustain the activities the project enabled them to launch. How do our goals fit with the communities’?
What is important is seeing community actors, our participants as the experts. It is their lives and livelihoods, and not one of us in international development is living there except them…
What are your questions and thoughts? Have you seen such tradeoffs? We long to know…
[*NB: There were other inputs (water, health, natural resource conservation) which are separate from this discussion.]
by valuingvoicesjin | Jul 25, 2014 | Aid effectiveness, Ethiopia, Evaluation, Feedback Labs, Feedback loops, Food security, LIvelihoods, Participants, Red Cross, Sustainability
Reposted from Feedback Labs: http://feedbacklabs.org/see-how-it-turned-out-feedback-loops-for-implementation-and-sustainability/
When I met Aminata in a central Malian village, she asked me whether I was with the people with the yellow trucks or the white trucks. That was her way of differentiating between development projects. I explained to her I was doing (PhD) research, with neither. She asked me whether they (the trucks) would come back. I couldn’t answer, and therein lies the problem. “Country-led development” begins with communities being involved at every stage of a project. Ongoing community input during project design, implementation, and monitoring is needed for impact, local community ownership and sustainability. Developing “feedback loops” that facilitate two-way communication are key for building cultures of collaboration, adaptation and learning into international development programs. Valuing Voices sees data as another resource to deliver development, beyond serving the needs of donors and international non-profits.
The distance between our intentions and our reality
Too often, data is extracted from communities by development organizations, in order to evaluate how well they fulfilled the project, rather than communitiesevaluating how well international development projects supported community needs. The best projects co-design interventions and monitoring with communities. Too often communities have no mechanism to learn how their feedback during implementation or during evaluations led to real changes. Such feedback leads to community buy-in, as there is proof that their voices matter and that they are co-driving development. Outside of submitting a format report in a PDF to a donor, the development field has consistently inadequately retained the data from projects once they close, much less leave it in-country, and rarely return afterwards for follow-up evaluations. This further reinforces the idea that these communities are being exploited for resources or development experimentation, with no thought to long-term capacity, learning or sustainability.
Ever farther from this vision of collaborative partnership in project conceptualization, design, and monitoring is what happens once these short and long term projects are closed out. Due to budget restrictions and bureaucratic habits, too often the task of sustaining these projects is handed over to local partners, without funding, staff or data continuity. Cursed with “pilot-itis”, development initiatives too often lose sight of achieving scalability and sustainability. ‘Sustainability’ is often incorrectly defined only as whether the specific project got follow-on funding, rather than by asking communities what activities they were able to continue long after the project ends. Communities provide feedback only 1% of the time, missing great learning opportunities to be had from returning to assess these same projects 1-10 years later to see what was expected, unexpected and what could be sustained. We therefore never truly learn what had the greatest prospect for replicability and scaling elsewhere.
How do we really show that we Value Voices?
At Valuing Voices, we believe that community members are our true clients. We have identified two kinds of feedback loops that are needed to make international development far more effective.
-
Country-Led Project Implementation Feedback Loops: Valuing Voices wants to create standardized methodologies and data collection processes that can be integrated into most international development projects to create feedback loops that continue working long after projects close. Valuing Voices believes key elements to developing collaborative feedback loops are:
-
Identification of different feedback loop methodologies. Based on what is appropriate for the population, Valuing Voices identifies different methodologies to create these loops through a mix of traditional and digital tools. This means explicitly targeting the building, capturing and sharing of feedback during a specific project, to test and document different methodologies and create standard processes and infrastructure. This includes participatory methodologies and studies such as Empowerment Evaluation and “Who Counts? The Power of Participatory Statistics, as well as ways of evaluating qualitativeStorytelling. We must of course protect our respondents as well.
-
Use of a franchise approach to replicate and scale. The identified methodologies can be taught to local and in-country evaluators and development experts who can then “sell” those services to governments, local NGOs, international donors, and the private sector. Valuing Voices is a catalyst for this country-based franchise approach which strengthens national evaluation, capture and learning. It also allows for feedback loops to exist within a program as local evaluators provide feedback on impact and lessons for improvement. This leads to local empowerment, sustainability and aid transparency.
-
Use of digital tools- capture once, share forever. Following evaluations on the local level, Valuing Voices then rolls up national evaluations by sector, analyzes them for what’s most sustained, and shares that learning around the world, influencing project design, implementation, funding and empowerment. In addition to generating feedback loops, this valuable feedback is entered into a curated database of findings for comparison across projects such as agriculture, livelihoods, credit, natural resources, health and education. By using structured data analysis, we can compare feedback data and actual behavior. We will look at whether we are capturing the right associated metadata (date, location, project) to contextualize feedback and bring forth lessons our partners can learn from, analyzing similarities and learning from them to improve program quality, organizational learning (of all partners) and sustainability prospects.
-
Post-Project evaluation Feedback Loops: The eagerly awaited Local Systems: A Framework for Supporting Sustainable Development was published by USAID in May:
“More ‘ex-post’ evaluations — which are meant to determine the impact of a project after it is completed, sometimes years later… are needed to design and implement projects.”
This is great news, as in the last 24 years, USAID has not published a single post-project evaluation, and the World Bank has only published one. This could indicate that none were written, or none were seen as ‘publishable’.
Post-project (ex-post) evaluations should look at:
-
The resilience of expected impacts of the project two, five, and ten years after close-out;
-
The communities’ and NGOs’ ability to self-sustain those activities;
-
Positive and negative unintended impacts of the project, both immediately in in the long term;
-
Which activities the community and NGOs felt were successes which could not be maintained without further funding;
-
Lessons for other projects on what interventions were most resilient which communities valued enough to continue themselves or for which NGOs valued enough to obtain additional funding, as well as what was not resilient.
These evaluations are rarer than snow in Sri Lanka. Feedback loops post-implementation are key to understanding the sustainability of projects and to improve transparency, efficacy and learning. Valuing Voices has a handful of examples of ex-post (post-project) evaluation done by organizations such as Mercy Corps and Plan International as well as bilateral JICA, but shockingly 99.9% of all development projects just don’t go back and check.
Voices to be valued
These are just a few examples of areas where we can value the voices of our communities, to be more effective and impactful with our efforts, and show our respect and value of communities’ input.
The time is ripe for growth, for these voices to be heard more loudly, telling us what they want development to be. Join us in advocating for funding for such feedback, join us as partners in our field sites, as partners in ICT database creation, as voices for communities.
Communities want their voices heard. Sustained development depends on it.
Jindra Cekan, PhD is the founder of Valuing Voices at Cekan Consulting LLC. She has worked in international development for 25 years in participatory design, monitoring & evaluation and knowledge management in over 20 countries. Her PhD was in Mali, “Listening to One’s Clients” and she consults to non-profit, public (USAID), foundation and private sectors.
Siobhan Green, MA is the founder of Sonjara, Inc, and a member of ValuingVoices. She has worked in international development since 1992, and specializes in ICTs for development, knowledge management, and technology for monitoring and evaluation. Her master’s thesis was on “The Internet in Africa: Policy Perspectives and Approaches” in 1997, and she works with USAID and other USG clients, non-profits, and for-profit partners.
by Jindra Cekan | Mar 26, 2014 | Accountability, Africa, Bilateral organizations, Credit, Donors, Ethiopia, Evaluation, Food security, International aid, International non-profits, LIvelihoods, Local Participants, MDG, Milenium Development Goals, NGO, Participation, Results, Rural Development, Sustainability, Sustainable development, Transparency, Women and gender
Stepping up community self-sustainability, one [Ethiopian] step at a time
Having just come back from evaluation and design fieldwork for an Ethiopian Red Cross (ERCS)/ Swedish Red Cross/ Federation of the Red Cross and Red Crescent project, the power of communities is still palpable in my mind. They know what great impact looks like. They know what activities they can best sustain themselves. It’s up to us to ask, listen and learn from them and support their own monitoring/ evaluating/ reporting. It’s up to us to share such learning with others and to act on it everywhere.
There are a myriad of possible sustainability indicators, and the outcome indicators below, suggested by 116 rural participants from Tigray, Ethiopia seem to fall into two categories of expected changes: Assets and Life Quality (Table 1). As the food security/ livelihood project extended credit for animal purchases, it is logical that tracking increased income, savings, assets, and home investments plus expenditures on food and electricity appeared.
We gleaned this from discussions with participants, asking them “what can we track together that would show that we had impact”? Our question led to a spirited discussion of not only what was traceable, but also what could be publicly posted and ‘ground-truthed’ by the community. Discussing indicators led to even deeper conversations about the causes of food insecurity which were illuminating to staff. What was surprising, for instance, was the extent to which families saw changing seasonal child-field labor practices in favor of 100% child-school attendance as great indicators. School attendance (or lack thereof) was dependent on families’ need for children’s seasonal labor in the fields. Community members said they knew who sent their children or not, which no only ‘cleaned’ the publicly posted data but triangulated implementer surveys and opened room for discussions of vulnerability.
Not only is this exciting for the project’s outcome tracking but even more importantly, our team proposed to create a community self-monitoring system, suggested in by Causemann/Gohl in an IIED PLA Notes article– “Tools for measuring change: self-assessment by communities” used in Africa and Asia. This learning, management and reporting process will fill a gaping need as current “monitoring systems serve only for donor accountability, but neither add value for poor people nor for the implementing NGOs because they do not improve effectiveness on the ground.” The authors found that not only “participatory data collection produces higher quality data in some fields than standard extractive methodologies [as] understanding the context leads to a higher accuracy of data and learning processes [which] increase the level of accountability… “ but also that such shared collaboration builds mutual learning and bridge-building.” While our community members may have offered to track this publicly to make this partner happy, men and women discussed this excitedly and embraced the idea of self-monitoring happily. ERCS will be discussing with communities to either track data monthly in notebooks or on a large chart hung in the woreda office for transparency. Data (Chart 1) would include these asset and quality of life indicators as well as loan repayments (tracked vertically) while households (tracked horizontally) could see who was meeting the goal (checked boxes), not meeting it fully (dashed boxes) or not meeting it at all yet (blank boxes). Community members corrected each other as they devised the indicators during our participatory research and this openness reassures us that the public monitoring will be quite transparent as well.
[1]
Further, what was especially satisfying was getting feedback from across the three tibias (sub-regions) on what activities they felt they could sustain themselves irrespective of the project’s continuation. Table 2 shows us which activities communities felt were most self-sustainable by households; these could form the core of the follow on project. Sheet/goats, poultry and oxen for fattening were highly prioritized by both women and men, in addition to a few choosing improved dairy cows. The convergence of similar responses was gratifying and somewhat unexpected, as there were several other project activities. The communities’ own priorities need to be seriously considered as currently they get only one loan per family and thus self-sustainable activities are key.
There is more to incorporate in future project planning by NGOs like Ethiopia’s ERCS. The NGO-IDEAs concept mentioned above also includes involving project participants in setting goals and targets themselves, differentiating between who achieved them and why, and brainstorming who/what contributes to it and what they should do next. Peer groups, development agencies and any actors could collect and learn from the data. Imagine the empowerment were communities to design, monitor and evaluate and tell us as their audience!
And they must, according to ODI UK’s Watkins, who has a clear vision on how to achieve a global equity agenda for the post-2015 MDG goals. He suggests converting the principle of ‘leave no one behind’ into measurable targets. He argues that, by introducing a series of ‘stepping stone’ benchmarks, the world can set ambitious goals on equity by 2030. He writes, wisely, that “narrowing these equity deficits is not just an ethical imperative but a condition for accelerated progress towards the ambitious 2030 targets. There are no policy blueprints. However, the toolkit for governments actively seeking to narrow disparities …has to include some key elements [such as] identifying who is being left behind and why is an obvious starting point. That’s why improvements to the quality of data available to policy-makers is an equity issue in its own right”. Valuing Voices believes who creates that data is an equally compelling equity issue.
So how will we reach these ambitious targets by 2030? By putting in stepping stone targets, returning project design functions to the ultimate clients – the communities themselves- and matching their wants with what we long to transfer to them. In this way we will be Valuing their Voices so much that they evaluate our projects jointly and we can respond. That’s how it should always have been.
What are your thoughts on this? We long to know.
Sources:
[1] Ashley, H., Kenton, N., & Milligan, A. (Eds.). (2013). Tools for supporting sustainable natural resource management and livelihoods. Participatory Learning and Action, (66). Retrieved from https://pubs.iied.org/14620IIED/
[2] Watkins, K. (2013, October 17). Leaving no-one behind: An equity agenda for the post- 2015 goals. Retrieved from https://www.odi.org/blogs/7924-leaving-no-one-behind-equity-agenda-post-2015-goals
