Blog

PARTICIPATION BY ALL: The Key To Sustainability of CRS/ Niger’s Food Security Project

Posted by on Apr 13, 2016 in Catholic Relief Services (CRS), Exit strategies, Food for Peace (FFP), international development, LIvelihoods, Niger, Participation, Sustainability, Sustainable development, USAID | 0 comments

PARTICIPATION BY ALL: The Key To Sustainability of CRS/ Niger’s Food Security Project

Valuing Voices is delighted to share our sustainability evaluation of Catholic Relief Service Niger’s PROSAN project.  This project that ran from 2006-2012 in Niger and was implemented by three NGOs: CRS, Cooperative for Assistance and Relief Everywhere (CARE), and Helen Keller International (HKI) under the direction of United States Agency for International Development (USAID) Office of Food for Peace (FFP) as a multi-year assistance program (MYAP) to support food security activities in the Dosso, Tahoua, and Zinder regions. PROSAN focused on increasing agricultural production and agro-enterprise, improving household health and nutrition status, reinforcing the capacities of health agents, and enhancing community resiliency.

Here are the highlights from the report which itself is an excerpt from a longer analysis we did. Also please note one Annex highlights the similarlties/ differences we found to USAID/ FFP's 4 elements of sustainability:

AIM, METHODS, AND RESEARCH QUESTIONS

The aim of this sustainability evaluation was to explore perceptions of sustainability from Nigeriens involved in PROSAN, former CRS staff and donors. It focused on evaluating participants’ adherence to project outcomes and their creation of new innovations. It also evaluated partners’ involvement in sustaining project outcomes.

 

This evaluation used qualitative and quantitative methods including community mapping, focus group discussions, beneficiary interviews, and key stakeholder interviews. The evaluation was carried out in six communities in the Dosso region, with more than 500 interviewees, focusing on the following research questions:

 

1. Sustainability of activities and groups: Are the communities sustaining the activities three to five years after the end of the project? What can we learn from the communities and their post-project implementation partners?

2. Spread and unexpected outcomes: If the project was considered a success in the eyes of the community, how well did it spread?

3. Fostering Sustainability: What are the long-term prospects for continued sustainability?

 

FINDINGS

Three years after PROSAN’s conclusion, the project was considered a success by community members, national partners, the implementer (CRS), and donor (USAID) staff. The main findings include:

 

1. SUSTAINABILITY OF ACTIVITIES AND GROUPS

Eighty percent (80%)[1] of all activities were reported to have become self-sustained and community innovations have emerged:

 

• On average, households reported moving from being food secure for 3-6 months per year during PROSAN to 8-12 months at the time of this evaluation, which is a remarkable impact.

.CRS_Niger_PROSAN_Sustainability_Evaluation_pdf

• Women reported greater income through the increase in sales of food that was produced and processed due to the grain mills.

• Respondents also reported improved household health, hygiene, and nutrition, with 91% of survey respondents indicating that their health and sense of well being had improved, especially through the efforts of the health posts and clinics that CRS helped build and the government of Niger’s efforts in sustaining them with resources and staff.

 

Community groups/committees have continued and are well-supported by NGO partners:

 

• 81% of the committees set up by PROSAN were functioning at the time of this evaluation, with many participants discussing ways to sustain best practices within their communities, and members still receiving regular trainings or updates.

• Several new and refresher trainings come through national partners, NGOs, and new channels such as radio programs.

• Some new NGOs and international organizations have built upon PROSAN’s success, for instance, by using land previously managed by PROSAN for a new vegetable gardening training program, building hygiene programs on past health awareness efforts, or extending agricultural credit for further inputs.

 

Twenty percent (20%) of implemented activities were not sustained or have stagnated:

 

• While hygiene practices were sustained by households and there was widespread latrine construction, sanitation was poor in the villages, and most latrines had fallen into disrepair.

• Fewer than 50% of women reported practicing exclusive breastfeeding for children less than six months of age.

• While almost half of all health committees no longer exist, new health clinics staff have replaced some of the work of the committees with health and agricultural promotion messages now being sent via radio, television, and cell phones.

• Literacy training and theater groups have completely ceased.

• With the exception of the Système Communautaire d’Alerte Précoce-Réponses aux Urgences’ (SCAP-RU) SCAP-RU early warning system which has expanded, other resilience activities such as roadwork and caring for the environment are a lesser priority due in part to the lack of food and cash-incentives to continue doing them.

 

2. SPREAD AND UNEXPECTED OUTCOMES

New innovations and ceased activities reflected the project’s legacy:

 

• Community innovations have emerged such as collective funds paying for cleaners of the new health center, community-imposed sanctions for births occurring outside of the health centers, and the monitoring of savings from well water sales.

• National partners have praised the project, with many lamenting its withdrawal. One non-PROSAN village told an Agriculture Ministry staff and potential NGO partner that “No one should bring a program here unless it is like PROSAN.”

• PROSAN-trained masons, well repair technicians, and village youth have learned land recuperation techniques (zai holes, bunds and demi-lunes) that helped generate income beyond project communities.

• Project activities that received free inputs have largely stopped being implemented once the incentives were withdrawn such as Food for Training (FFT), Food for Work (FFW), or Cash for Work (CFW) (e.g. literacy, seedlings, latrines, theater etc.); nonetheless the inputs were highly valued and have continued to support agriculture and health (carts, bicycles).

 

3. FOSTERING SUSTAINABILITY

The following areas were identified as potential barriers to sustainability that could be systematically explored in other projects:

 

• Although most committees are still functioning, there are no processes in place to engage and train youth and new inhabitants of the villages.

• While village communities have been maintained, there is an increasing lack of ministry resources (e.g., staff, transportation, and communications) to take the place of NGOs like CRS after a program ends.

• There is little management of knowledge around project data, which is further exacerbated by staff changes in NGOs, government ministries, and donors. Project data (proposal content, monitoring data, evaluation results, participant lists, partner names, and exit agreements) must be managed ethically, locally and be held online, accessible for future projects to use and for villages to conduct self-evaluations.

 

 

 

 

 

 

 

 

 

 

 


[1] Percentages were calculated as a combination of the number of activities that had been continued and the percentage of participants which continued them.

 

 

 

 

 

 

 

 

 

 

 

 

 

What happens after the project ends?  Lessons about Funding, Assumptions and Fears (Part 3)

Posted by on Feb 29, 2016 in 3ie, Accountability, Appreciative Inquiry, Assumptions, CARE, Fears, Food for Peace (FFP), Funding, Learning, microcredit, OECD, OXFAM, PACT, Sustainable development, USAID, Village Banks | 0 comments

What happens after the project ends?  Lessons about Funding, Assumptions and Fears (Part 3)

 

In part 1 and part 2 of this blog, we showcased 11 of the 18 organizations that have done post-project evaluations.  While this was scratching the surface of all that is to be learned, we shared a few insights on How we do it Matters, Expect Unexpected Results and Country-national Ownership. We gained some champions in this process of sharing our findings, including Professor Zenda Ofir of South Africa, who said “we cannot claim to have had success in development interventions if the outcomes and/or impacts are not durable, or at least have a chance to sustain or endure."

 

In this third blog of Lessons Learned from What Happens After the Project Ends, we turn to some of the curious factors that hold us back from undertaking more post project evaluations: Funding, Assumptions, and Fears

 

Funding

  1. Why haven’t we gone back? For the last 2+years Valuing Voices has been researching the issue, we have heard from colleagues: ‘we would love to evaluate post-project but we don’t have any money, ‘donors don’t fund this’, ‘it is too expensive’[1].  Funding currently from bilateral donors such as USAID is given in 1-5 year tranches with fixed terms for completion of results and learning from them and one-year close-out processes. Much of the canon of evaluations conducted after close out that we amassed was from international NGOs that had used their private funds to evaluate large donor-funded projects for their own learning.  Many aimed also to show leadership in sustainability and admittedly dazzle their funders – join them!.

 

  1. We fund capacity building during projects but if we do not return to evaluate how well we have supported our partners and communities to translate this to sustainability, then we fall short. Meetings convened by INTRAC on civil society sustainability are opening new doors for joint learning about factors such as “legitimacy… leadership, purpose, values, and structures” within organizations well beyond any project’s end. The OECD’s DAC criteria for evaluating development assistance define sustainability as: “concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable.“ We need to extend our view beyond typical criterion for sustainability being a focus primarily on continued funding.

 

  1. We need funding to explore whether certain sectors lend themselves to sustainability. In addition to the cases in blog 1, a study by CARE/ OXFAM/ PACT on Cambodian Savings groups finds that we have some revisions to make on how we design and implement with communities to foster sustainability in this sector which typically promises greater sustainability because capital can be recycled. Valuing Voices blogs show indications that once we amass a greater range of post-project evaluations (funders unite), the insights gleaned can illuminate cost efficient paths to more sustained programming, possibly leading to revisions in programming or interventions which have greater likelihood for country-ownership

 

  1. Extend the program cycle to include post-project sustainability evaluation. Rare are donors such as the Australian government (forthcoming) and USAID’s Food For Peace that commission such studies. Rare is the initiative such as 3ie that has research funds allocated by major donors to explore an aspect of impact. We miss out on key opportunities to learn from the past for improved project design if we do not return to learn how sustained our outcomes and impacts have been. We miss learning how we could better implement so more national partners could take on the task of sustaining the changes we catalyzed.

 

  • We call on donors to fund a research initiative to comprehensively review sustainability evaluations.

 

  • We call on governments to ask for this in their discussion with donors. 

 

  • We call on implementers to invest in such learning to improve the quality of implementation today and sustained impact in the years to come.

 

 

Assumptions

Development assistance makes many assumptions about what happens after projects end in terms of people’s self-sufficiency,   partners’ capacity to continue to support activities, and projects’ financial independence and people’s ability to step into the shoes of donors and carry on.   Unless we take a hard look at our assumptions, we will not move from proving what we expect to learning what is actually there.

 

evaluation_-_Hledat_Googlem

 

Among them are these six assumptions:

  1. All will be well once we exit; we have implemented so well that of course national participants and partners will be ready and able to carry on without us. We may assume the only important outcomes and impacts are within our Logical Frameworks and Theories of Change. Thus there is no need to return to explore unexpected negative ones, or ways in which the people we strengthened may have innovated in unexpectedly wonderful ways. Aysel Vazirova, a fellow international consultant wrote me: “Post-project evaluations provide data for a deeper analysis of sustainability and help to appreciate numerous avenues taken by the beneficiaries in incorporating development projects into their lives. The theory of change narratives presented by a majority of development programs and projects have a rather disturbing resemblance to the structure of magic tales: (from) Lack – (to) change – (to) happy ending. Post project evaluations have a power to change a rigid structure of this narrative.”

 

  1. We assume evaluations are often used to inform new designs, yet dozens of colleagues have lamented that too often this does not happen in the race to new project design. But there is hope. World Wildlife Fund/UK M&E expert Clare Crawford says when following its new management standards, WWF “expects to see the recommendations of an evaluation before the next phase of design can happen (hence evaluations happen a little before the end of a strategic period).  WWF-UK, when reading new program plans is mandated to verify if – and how – the recommendations of the last evaluation(s) were made use of in the new design phase.  Equally we track management responses to evaluations to see how learning has been applied in current or in future work." Such a link across the program cycle is not common in our experience and none of the post-project sustained impact evaluations we reviewed said how learning would be used.

 

  1. We may assume data continues accessible from the projects we have evaluated, yet our team member Siobhan Green has found that until recently, with the move toward open data, often project data remains the province of the donors and implementers and to the best of our knowledge leaves the country when projects close. While some sectoral data such as health and education data remains local, we are finding in fieldwork that household level data has been rolled up or discarded once projects close, which makes interviews difficult.

 

  1. We may assume that the participants and partners are not able to evaluate  projects, particularly after the fact.  Being vulnerable does not mean that people are not able to share insights or assess how projects helped or not. Methods such as empowerment evaluation and evaluative thinking are powerful supports.

 

  1. Some may assume that the situation has changed in the intervening years, that there is no benefit in returning to see what results remain. Change is inevitable and sometimes more rapid or dramatic than others.  But does that mean we shouldn’t want to understand what happened? This is the greatest disservice of all, for we are selling “sustainable development” so how well have we designed it to be so?

 

  1. We assume that learning for our own benefit is enough. A potential client brought me in to discuss my working on a rare post-project evaluation last year. It was to cost hundreds of thousands of dollars and while would occur in several countries. What I discovered was that while the donor really wanted to learn what results remained more than a decade on, I asked ‘how would the countries themselves benefit from this research and findings?’  There was a long silence. Turns out, nothing from the research would benefit or even remain in country. No one had considered the learning needs of the countries themselves. This simply cannot continue if we are to be accountable to those we serve.

 

Fears

This may be the greatest barrier of all to returning to assess sustainability.

 

  1. We assume our projects continue. We may be afraid to look for what will this tell us about the sustainability of our efforts to save lives and livelihoods so we only choose to publicly study what is successful. Valuing Voices has found that across most the post project studies there is some ‘selection bias’, as we repeatedly learned in our research from colleagues that organizations choose to evaluate projects that are most likely to be successfully sustained.  For instance, USAID Food For Peace’ study notes, “The countries included in this study—Bolivia, Honduras, India, and Kenya—were also chosen because of their attention to sustainability and exit.” Yet as an Appreciative Inquiry practitioner, I would argue that learning what worked best to know what to do more of may be the best way forward.

 

  1. All too often the choice of evaluation design, and sensitivity to findings fly in the face of learning—particularly when findings are negative.  This raises fears around a discontinuation of funding (an implementer fear; a beneficiary fear; could also be a recipient government’s fear). Yet as Bill Gates says, “your most unhappy customers are your greatest source of learning.”

 

  1. Participants asked during the project cycle about interventions may be fearful of truth telling because of perceived vulnerabilities around promised future resources, local power imbalances in control over resources, or even political imperatives to adopt a particular position. Alternatively we may not believe them, thinking they may not tell us the truth were that to stop resources.

 

Those are ours.

 

  1. Peter Kimeu, my wise advisor and 20-year friend and colleague from Kenya tells us some fears of the that we need to listen to – those that haunt our national partners and participants.

 

They are afraid we do not see their real desires:

  • “It is ‘not how many have you (the NGO) fed, but how many of us have the capability to feed ourselves and our community?’

 

  • ‘How can we (country national) support our fellow citizens to take our lives and livelihoods into our own hands and excel, sustainably?’

 

  • What is sustainability if it isn’t expanded opportunities, Isn’t the capability of one to make a choice of value/quality life out of the many choices that the opportunities present?”

 

Will you help us address these challenges? Will you join us in advocating filling the gap in the program cycle, and looking beyond it to how we design and implement with country nationals? Will you, in your own work foster their ownership throughout and beyond? We need to fund learning from sustained impact to transparently discuss assumptions and face our fears.  This is a sustained purpose we need to and can fill.

 

 

 

 

 

 

 

 

 

 

 

 

 

 


[1] It does not have to be. We have done these evaluations for under $170,000, all-inclusive.

 

 

 

 

 

 

 

 

 

 

 

 

 

What happens after the project ends?  Country-national ownership lessons from post-project sustained impact evaluations (Part 2)

Posted by on Feb 19, 2016 in Accountability, Aid effectiveness, Asian Development Bank (ADB), CARE, ex-post evaluation, Exit strategies, Food for Peace (FFP), InterAction, Microenterprise, OECD, PACT, Participation, Project Concern International (PCI), Sustainability, Sustainable development, Sustained Impact Evaluation, USAID | 0 comments

What happens after the project ends?  Country-national ownership lessons from post-project sustained impact evaluations (Part 2)

 

In Part 1 of our blog on lessons learned from post-project evaluations, we explored:

– How we do it matters for great results

– Expect unexpected results

 

This time we turn to who continues after closeout, and what conditions foster both successful handover and ownership from the onset in order to foster sustained impact.

 

Who Takes Over? Country nationals

When project handover is integral to the design, development projects needn’t be long-term or expensive. What they need to be is increasingly community-driven. Unless exit strategies are explicit and thorough, sustained impacts are less likely.

 

1. USAID/ Food for Peace (FANTA/ Tufts)


An ‘exit strategies’ evaluation of 12 projects in four USAID Food for Peace (FFP) countries of Bolivia, Honduras, India, and Kenya carried out in 2009, three to four years after close out detailed mixed results, described here. These were complex food security projects across multiple sectors of: maternal and child health and nutrition; water and sanitation; agriculture, livestock, and rural income generation; natural resource management; school feeding; and micro-savings and loans.

 

FANTA/Tufts found “providing free resources, such as supplementary food as an incentive for growth monitoring participation or free agricultural marketing services to promote sales, created expectations that could not be sustained once the free resources were no longer offered.” Valuing Voices found similar issues in Niger’s PROSAN (see part 1), with the lack of continued incentives (food and in-kind inputs) led to activities not being continued by community members.

 

On the other hand, in India, the government took over FFP food ration distribution after closeout. “This phase-over of responsibility to national government programs was effective in the case of supplementary feeding but not in the case of school-feeding (the latter through the midday meals program), due to varying levels of government commitment. India’s government had the resources, capacity (an already existing supply chain), and motivation (commitment) to provide this benefit.” Again, CRS/Niger showed us that decades-long investments in partnerships and 2+ years for phase-over pays off; 80% of outcomes were self-sustained three years on. CIDA Peru also found that "Shared responsibilities and participatory process were instrumental in ensuring sustainability….a shared understanding of project objectives and counterpart interventions was established with Peruvian sector authorities, between donors and local communities."

 

The lesson learned about close coordination with the partners such as the national government during design and implementation in order for transition to country-ownership and responsibility to be smooth also appeared among multilaterals that Valuing Voices has examined. Three multilateral agencies stand out as having conducted multiple post project evaluation (OECD, JICA and the Asian Development Bank).

 

We posit that too often in international development, the accountability focus is on fulfilling funder (donor) requirements, rather than accountability to project participants and what is needed to achieve sustained impact for them (Figure 1, below). The optimal case has project funders, implementers and national governments aligning to support those we ostensibly serve: women, men, youth, elders in need of assistance.

 

 

Valuing_Voices_Accountability_Capabilities_2015_pdf

@ValuingVoices2015

 

Are there certain kinds of projects or implementers that manifest optimal accountability? Far more examination is needed, but a promising path is microenterprise.

 

2. Pact’s WORTH project in Nepal

PACT’s project illuminates that local ownership and structures sustain results and even multiply impact. Implemented from 1999-2001, Pact worked with many local NGOs to reach 125,000 women in 6,000 economic groups across Nepal; of those, one quarter chose to implement village banks.  Village Banks cultivated women as agents of change and development in their communities—promoting grassroots sustainability   The post project evaluation in 2006 found that:

  • Almost two thirds of the original 1,536 village banks were still active eight years after the program began and assets of an average village bank has tripled in the last three years post-project (from $1000 to over $3000 at the time of the evaluation) 
  • 83% reported that because of WORTH they are able to send more of their children to school
  • Women’s economic groups helped start an estimated 425 new groups involving another 11,000 women with neither external assistance nor prompting from the project.

 

Why? The post-project report tells that the banks were not an end in and of themselves, women’s empowerment was: “WORTH groups and banks were explicitly envisaged as more than just microfinance providers; they were seen as organizations that would build up women as agents of change and development in their communities.” Thus local Nepalese sustained and grew their own development.

 

3. CARE Zanzibar’s Village Savings and Loan Associations were evaluated four years post-project. Similar to PACT, they found the model was sustained and grew:

  • Total membership rose from 1,272 in 2002’s closeout to an estimated membership of 4,552 in July 2006, an increase of 258%
  • During the most recent payout for all 25 groups, the mean rate of return was 53%, with individual groups’ rates ranging from 10% to 92%.
  • Participants said that the main changes in the lives as a result of the program were an improved standard of living (22%), improved housing (21%) and increased incomes (20%).

 

While wonderful, can it only be the responsibility of communities to sustain their gains? How well are we designing for country-ownership and handover to the state?

 

4. The UN's OECD has dozens of post project evaluations on its website, funded by member governments.

 

One study illuminates that while communities may manage to sustain some of the outcomes, structural investment in national capacity to takeover is key. This example evaluated four 10-15 year-long projects funded largely by Germany that were carried out in Indonesia, Sri Lanka, Tanzania and Zambia with a cumulative value of Euro 145.1 million ($180 million). The study was done in 2004, evaluating activities an astonishing 30 years after inception. Results?

  • The good news: “living conditions of the target groups have improved in all four project regions,” with specific sustainable project outcomes observed in the “health and education sector, food security, increase in income and employment and the ensuing rise in the standard of living”.  Links were made to project-supported improvements in infrastructure, enhanced private sector economy, and the project’s innovations in agriculture.
  • The bad news:there was low institutional sustainability at the level of state executing organizations for all four projects due to inadequate funds, inefficient organizational structures and a lack of coordination”. Thus, viable exit and handover was limited. Structures advanced as part of a development project ran a high risk of not being sustainable.”

 

5. Three years ago, the Asian Development Bank reviewed 491 project completion reports (desk studies) and undertook a handful of field visits to projects financed between 2001-09. Similar results:

  • “Some early evidence suggests that as many as 40% of all new activities are not sustained beyond the first few years after disbursement of external funding.”
  • “National government ownership, commitment to and financing of the projects were vital to sustainability.“
  • “Neither governments nor other international agencies benefit from systematic information on whether projects reached their intended economic or social objectives over the full life of the intervention or in the decade afterwards.” 

 

This is a clarion call to all funders to invest in future excellence by returning to the past, learning what worked best and what failed to do so, examine why and begin anew with accountability to our participants!

 

Conclusion

What can be done?

A. Foster ownership of the process of development through empowerment to begin with, as PACT’s WORTH project did in Nepal and elsewhere. InterAction’s lovely “A Missing Piece in Local Ownership: Evaluation” reminds us “the local ownership agenda must extend to all parts of the program cycle – from design all the way through evaluation.  Including those meant to benefit from international assistance (we use the term “participants”) in deciding what should be done and how it should be done is critically important for effectiveness and sustainability.

Ask yourselves how well we involve governments in collaborative design of what they feel they can sustain of our programming after we leave, how and for how long with what resources, linkages, capacity-built and motivation (see FFP study #1).

 

B. Design and implement in the present while considering sustaining outcomes and impacts in the long-term, as we learned in Part1 of this blog as well as taking lessons from some emerging guides such as the systematic guidance of PCI’s Resource Guide for Enhancing Potential for Sustainable Impact.

 

C. Dare to return to learn. As Dina Esposito, the Director of USAID/Food For Peace stated, “this rigorous, retrospective [ex-post] approach is not widely done, but is essential if we are to understand the true impacts of our investments. To be effective, development projects must result in changes that last beyond the duration of the project themselves.”

Imagine the sustained cost-efficiencies of learning certain sectoral programs lent itself best to sustainability by communities, others needed Ministries to take over, others still needed different support such as private sector – or all of the above. If we look at sustained impact as our true goal, how differently could we work together? How much more efficiently would we use our global resources?

 

What can we say about the sustained impact post-project evaluations we have featured?

We have covered lessons about how matters in design and implementation; expect unexpected results and who takes over? Country nationals. Much more research and analysis is needed, many more case studies need to be created for us to understand how to foster the best handover as well as national ownership at the beginning, middle and end. Maybe you drew some of the same conclusions we have:

 

  • Post project evaluations provide valuable insights about sustainability.
  • Lessons from such evaluations can lead to better programming in current and future projects.
  • The voice of national stakeholders—participants and partners, including governments is essential.
  • Donors lose amazing opportunities to learn what works now and continues to work unless they fund more sustainable impact evaluations and support investing resources in fostering sustainability during design and implementation.

 

In Part 3, we will look at what is keeping us from looking to the past for the future (hint: funding, assumptions and fears) and how we can move ahead together…

 

Please join us in advocating for and funding this vital approach!

What happens after the project ends?  Lessons from post-project sustained impact evaluations (Part 1)

Posted by on Feb 13, 2016 in Aid effectiveness, Bill and Melinda Gates Foundation, Catholic Relief Services (CRS), Central Asia, Ethiopia, Ethiopian Red Cross, Evaluation, Federation of the Red Cross, Lutheran World Relief, Mercy Corps, Niger, PACT, Partners for Democratic Change, post-project evaluation, Sustainability, Sustainable development, Sustained Impact Evaluation, USAID | 1 comment

What happens after the project ends?  Lessons from post-project sustained impact evaluations (Part 1)


We talk a lot about impact of our interventions, but far less is analyzed about the sustained impact of our work in the years after projects close out. We take for granted that successful strategies will continue after projects shut down.  Do they always? Maybe they do but we don’t know. Maybe there are innovations and impacts to be learned from…

 

To answer these question Valuing Voices spent 2 ½ years looking for and analyzing ‘post-project’ evaluations of projects undertaken 2-10 years after projects ended.  The result: in our $137 billon international development industry, some 99% of projects remain unevaluated after project close out.  Only 17 agencies we have found so far have publically available post-project evaluations; most of them have one, while the OECD and JICA have dozens.  Hundreds of studies recommend such learning that is missing from our industry’s program cycle (green slice).

What_have_we_learned_from_postproject_sustainability_impact_Pt1and2_0216_docx

Six decades on, this astonishing finding raises serious questions about stewardship of resources and commitment to learning—particularly learning from participant and partner stakeholders for whom sustainability matters most, and who are tasked with it over the long-term

 

A review of post project evaluations generate food for thought about good program design and illustrate the value added of post project perspectives. This rapid review of select ‘ex-post’ evaluations points to three early lessons:

– How we do it matters for great results

– Expect unexpected results

– Who takes over? Country Nationals

First, organizations go back to see how well their projects results were sustained. What we learned was that how well they used participatory processes in how they implemented and handed over mattered a lot for sustainability and that we must expect unexpected results.

How we do it matters for great results

1. Catholic Relief Services/ USAID PROSAN food security project in Niger

Valuing Voices evaluated this food security (agriculture, health and resilience project in 2015 which ran from 2007-2012 (report forthcoming). The $32 million project was implemented in a consortium of CRS, CARE and Helen Keller but this evaluation focused only on CRS areas.

  • Interviews with over 500 participants found that three years after project closeout, 80% of project activities still continued, as did many village committees and there were a variety of community innovations
  • On average, households can feed themselves through their own production or purchase of food for 8-12 months three years after closeout compared to 6-9 months at closeout three years earlier. Such impact was unexpected.
  • 91% of respondents reported improved household health, hygiene, and nutrition
  • A two-year participatory exit process from CRS to country stakeholders (local government, an array of local and international NGOs and the private sector ensured continuity and boosted local ownership
  • 20% of the activities did not continue, mostly food-assisted NRM and resilience-related
  • Youth make up 50% of the population and need to be engaged during the project for long term sustainability to occur.

 

2. Partners for Democratic Change

Their mission is to “to build sustainable capacity to advance civil society and a culture of change and conflict management worldwide” uses an approach that is “bottom-up, locally-led rather than foreign-led, based on the belief that change comes from sustainable efforts led by local people, organizations and institutions invested in their own long-term future.” They went back and to review 55 case studies of projects through 22 centers they founded in central and eastern Europe from 1989-2011. They found:

* In 80% of cases, there was advancement of good governance by influencing the participation of civil society working with government
* In 50% of the cases there was increased access to justice and managing and resolving disputes/conflicts, thereby strengthening civil society

* 18 of 22 of the centers that had been established still exist today (82%).

 

3. Mercy Corps 

They did a post-project evaluation in Central Asia in 2007, one and three years after two conflict resolution projects ended which were worth $18 million. These complex community mobilization programs with aims "to empower communities to work together in a participatory manner to address the infrastructure and social needs [while] developing sustainable skills [and] empowers communities to identify and utilize existing resources within the communities and not to depend only on external assistance."

  • 72% of youth report that they continue to use at least one skill they learned during the programs (e.g. teamwork and communication, and skills such as sewing, construction, roofing, journalism and cooking)
  • 68% of community members witnessed local government becoming more involved in community activities after the end of the programs as compared with before the programs
  • 57% of the communities studied continuing to use one or more of the decision-making practices promoted during the program
  • 42% of members, representing 70% of communities, reported that the community had worked collectively on new projects or repairs to existing infrastructure. Participants and partners had implemented almost 100 infrastructure projects by themselves independent of donor funds.

 

These are terrific expected results. We also learned to Expect unexpected results

4. Federation of the Red Cross and Ethiopian Red Cross

Valuing Voices combined a final evaluation of "Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity" that had funding over $3 million in 2009 from the Swedish Red Cross with an assessment of projected sustainability.  It was an IFRC/ERCS collaboration with the Ethiopian government to provide credit for food security inputs to 2,259 households, which were to be repaid in cash over time as well water and agriculture/ seedlings for environmental resilience.  We answered the DAC criteria for evaluation and found the project overall to be quite good, albeit with weak data tracking systems.

In terms of sustainability, we used participatory methods to learn about what people felt they could self-sustain once the project left their area, so we could shape a similar follow-on project design to be moved elsewhere in Tigray, particularly around the credit for animals.

* While 87% of the loans had been promoted by the government and given for large animals (oxen and cows), and 13% was for small animals (sheep, goats, chickens)

* But project participants we interviewed, strongly preferred the small animals in terms of being able to sustain them on their own. They felt they could afford these smaller amounts of credit as well as the feed to sustain them, without taking the risk of animal death leaving them with large debt. This was especially true for women, who preferred poultry to all other animals 15:1.

In our quest for fast results, are we asking participants to bear too much risk? As one of our Valuing Voices team asks, Who is responsible for sustainability?

 

5. Lutheran World Relief

From 2005-2007 Lutheran World Relief intervened in Niger, the world’s poorest country, with a $500,000 Pastoralist Survival and Recovery Programme (ARVIP) drought rehabilitation project funded by the Bill and Melinda Gates Foundation. There were numerous outcomes from targeting sheep, wells and animal fodder to 600 of the poorest women in 10 communities in northern Niger, among them:

  • Women's share of household income increased from 5% to 25% in some households. This was due to the value of the sheep grants, as well as time-savings used for income generation. Access to wells in five of the villages saved women a staggering 7-10 hours every other day from not having to go fetch water 3.5 hours away each way for household and animal needs and were free to weave mats or cook food for sale
  • Many said they didn't have to resort to worse survival strategies during the next hungry season after they received the sheep

What was not expected were these results:

  • Many women in several villages reported an impact that was completely unexpected to the implementer and donor which was “our husbands don’t beat us anymore.” This was thanks to both increased respect and income from the sheep as well as access to well-water which led to cleanliness and their ability to be home for their husbands, children and mothers-in-law, rather than fetching water whole days 2-3 times a week. The same was found in PACT’s WORTH empowerment and village banking project in Nepal that wrote, “one in 10 reported that WORTH has actually helped “change her life” because of its impact on domestic violence”.
  •  We defined success too narrowly. Many interviewees were content with the project even though prospects for project-expected drought resilience or sustained food security were less likely. Some women sold the sheep to buy food, pay their children’s school fees or their daughters' dowries, while some had their sheep sold by their husbands who used them to buy other animals, pay for ceremonies or other expenses.  Participants saw the project as bringing them resources and considered it a success. Spending assets on immediate needs is not at all illogical for a community who can feed itself only 4 months a year; for some households, their pressing needs far outweighed the luxury to wait and buffer seasonal food insecurity far down the line.

 

We hope you agree that allocating funds and attention to post-project sustained impact evaluations is necessary for the remaining 99% of international development projects as it offers a fantastic learning opportunity about how to ‘do development’ well now and for the future. Without returning to look for what participants and partners valued enough to continue on their own, without returning to learn about unexpected sustained impacts, we rob ourselves of pivotal learning needed for success.

 

In part two, we look at  ownership onward, planning for handover and lessons from who takes over? Country Nationals. In part three, we focus on Funding, Assumptions and Fears.

 

Please join us in advocating for this! Please think about your own projects… and whether you have considered these things, or need our help. We're listening!

Learning about Sustainability and Exit Strategies from USAID’s Food Assistance Projects

Posted by on Jan 20, 2016 in Accountability, Aid effectiveness, ex-post evaluation, Exit strategies, Food for Peace (FFP), Participants, partners, Rural Development, self-sustainability, Sustainability, Sustainable development, USAID | 4 comments

Learning about Sustainability and Exit Strategies from USAID’s Food Assistance Projects

 

USAID overall and Food for Peace (FFP) specifically have become far more progressive in the Obama Administration and under Administrator Rajiv Shah, with a much greater focus on accountability and results. Those of you unfamiliar with USAID’s Food For Peace will learn it has been a large channel of international assistance for over 60 years and is not a small funding instrument. For 2016 alone, they have proposed spending $1.75 billion to feed 47 million people through humanitarian and development programs implemented by non-profits, for-profits and the UN’s World Food Program. Given this scale of resources, it is highly surprising that while many documents in their archive ask for post-project evaluation and there are a handful of desk reviews, they have done only two actual ones with new fieldwork in the last 30 years (these four countries and a recently published one on Uganda, see catalysts). This recent and excellent 2015 synthesis report by authors Rogers and Coates is presented below.

 

Commissioned by USAID, Tufts University and FHI360 have done a remarkably thorough two to three year post-project evaluation of four (Title II) food-assisted programs containing 12 projects in Bolivia, Honduras, India, and Kenya that closed out in 2009. The methodologies used are clearly outlined (itself a boon to our fledgling field) as are limitations and comments on context, findings and recommendations. It was no small feat to compare activities across four countries and so many sectors (some of which were supported by provision of US food aid resources, others with in-kind or cash inputs): maternal and child health and nutrition; water and sanitation; agriculture, livestock, and rural income generation; natural resource management; school feeding; and micro-savings and loans. Also this covered many implementers, from CARE, ADRA and Save the Children to World Vision, CRS and Feed the Hungry.

 

In this document, Dina Esposito, the Director of FFP states “We commissioned this report with the objective of determining what factors enhanced the likelihood of sustained project benefits, in order to improve our guidance for future food assistance development projects.… FFP development projects are designed to reduce the long-term need for food assistance by strengthening the capacity of developing societies to ensure access to nutritious food for their most vulnerable communities and individuals, especially women and children. The study team looked at 12 FFP development projects across four countries and asked not only what was achieved by each project’s end?, but also, what of those achievements remained one year after project close-out? and two years after? This rigorous, retrospective approach is not widely done, but is essential if we are to understand the true impacts of our investments. To be effective, development projects must result in changes that last beyond the duration of the project themselves.”

Process and findings:

The researchers compared baseline, midline and endline evaluations and exit strategy documents to new mixed-method data collection. There were four main findings:

1) Impact vs. Sustainability trade-offs: Evidence of project success at the time of exit (as assessed by impact indicators) did not necessarily imply sustained benefit over time. Just because projects were deemed successful at exit does not mean that those continued after closeout. “Moreover, the study found that focusing exclusively on demonstrating impact at exit may jeopardize investment in longer-term sustainability.” Valuing Voices found the same in Ethiopia in research done in 2013.

 

2) Preconditions to successful sustainability: In addition to an ongoing source of resources, good technical and managerial capacity, and sustained motivation of participants and partners, linkages to governmental organizations and/or other entities were key to continuity and sustainability of outcomes and new impacts. “No project in this study achieved sustainability without [the first] three of them in place before the project ended,” and linkages between community partners and the public/private sector were critical for handover (Figure 1, below). Further, a gradual transition from project-supported activities to independent operation was important for sustainability.Sustainability was more likely when projects withdrew gradually, allowing community-based organizations to develop the capacity to operate independently.”

FFP-Sustainability-Exit-Strategies-Synthesis-Dec2015_docx

3) Free resources can threaten sustainability, unless replaced while there is no one-size fits all for resources: Using incentives has costs. “Free supplementary food in maternal and child health and nutrition projects or free marketing services in agriculture projects created expectations in many projects that could not be sustained once resources were withdrawn”. Valuing Voices found the same in research in Niger (report imminent). But other financing options, free health care or fee for service are still unsystematically studied regarding fostering sustainability in differing sectors.

4) External factors (climate, economy) can affect sustainability: The operating context and exogenous shocks (e.g., economic, legal and climatic) also affected the sustainability of project benefits, positively or negatively.

 

Most tellingly, the authors warned that “sustainability plans cannot be based on the hope that activities and benefits will continue in the absence of the key factors identified in this study.” Throughout the report and in pending country-specific studies, they outlined the assumptions that projects made about sustainability in order to exit and closeout, which were variably disproved, such as:

  • Community health workers would continue to provide services although without remuneration,
  • Households could continue to access nutritious food from their own (increased) production or purchases and have time, and know how to prepare such food,
  • Farmers will pay for inputs with profits from increased production and commercialization and can meet the quantity and quality requirements of long-term contracts
  • Community members will recognize the tangible benefit of Natural Resource Management activities and will be motivated to continue them without further inputs or remuneration
  • Water committees will have sufficient administrative capacity and resources to manage their budgets effectively
  • Community-based organizations have strong institutional capacity
  • Partner organizations will continue to provide teacher training
  • Government will have the resources and commitment to support future needs

 

The country studies with detailed findings are still forthcoming but these examples may illustrate the range of sustainability. There were some very well-sustained positive results in Food Production (India by area) and Child Health Growth Monitoring (Bolivia by consortium implementers) between baseline or enline and followup 2-3 years later:

FFP-Sustainability-Exit-Strategies-Synthesis-Dec2015_docx

FFP-Sustainability-Exit-Strategies-Synthesis-Dec2015_docx

As well as some far more mixed or negative results in examples across all the Water and Sanitation projects:

FFP-Sustainability-Exit-Strategies-Synthesis-Dec2015_docx

and far less stellar results in Maternal Child Health’s Community Health Workers (Kenya):

FFP-Sustainability-Exit-Strategies-Synthesis-Dec2015_docx

 

The authors recommended not only ensuring resources, capacity, motivation and linkages are present before exiting but also institutionalizing sustainable approaches to project design and evaluation including in solicitations and applications, project assessments, project management and knowledge management. They also recommended not only phasing down exit but also extending more such evaluations beyond the 5 years of implementation and assessing impacts as long as 10 years after. This requires some sizeable revisions to how development is done at Food For Peace.

 

All of these findings recommendations are near and dear to those of us at Valuing Voices. We strongly commend Food For Peace and ask for many more such studies, for unless we know what worked best and why, how do we know what to design next together with our partners and participants for real sustainability?

Sustained Impact post-project (ex-post)? Little proof at 3ie

Posted by on Dec 15, 2015 in 3ie, Aid effectiveness, DFID, Evaluation, ex-post evaluation, post-project evaluation, Sustainability, Sustainable development, USAID | 14 comments

Sustained Impact post-project (ex-post)? Little proof at 3ie

Find_Evidence-3ie_International_Initiative_for_Impact_Evaluation___Evaluating_Impact__Informing_Policy__Improving_Lives

Surely an organization that has received tens of millions of dollars of funding must track 'impact' as the actual long-term impact of projects. That is what I thought when I first began exploring post-project sustainability. If you look at the International Initiative on Impact Evaluation (3ie), so respected in RCTs (randomized control trials), they do valuable work comparing international development interventions among participants versus control groups during implementation, showing which interventions work best. Valuable stuff. Yet in an earlier post on impact vs. sustainability, I looked at 3ie, and found that much of the impact seems to be projected, during implementation. Our projects' 'logical frameworks' are designed to measure progress to get to outputs (e.g. farmers trained), outcomes (e.g. hectares planted with new seeds), impacts (e.g. higher harvests and lower malnutrition) and goals (e.g. hunger sustainably eliminated). So did we get to sustained impact and goals? Who is looking if we got there?

 

This kind of impact appears in OECD Post-2015: “how cost-effective is it? To what extent does aid help nurture solutions that are sustainable over time? In other words, is aid being delivered effectively and is it having an impact?“ Post-project evaluations that would assess sustained impact don't seem to widely be done or get much traction at the otherwise respected 3ie either. Returning a year later to 3ie since my first post, I found 18 documents under searches of ‘ex-post’ and ‘post project’.  Sadly, again and again, their definition of ex-post seems to be post the onset of implementation, ranging from 1-2 years after the baseline rather than the typical definition of 2-10 years after projects actually closed, where long-term impact can be found.

 

When I searched by the more technical term 'ex-post' on 3ie, I found 17 that were titled ex-post but in fact were not. This ex-post evaluation of a conditional transfer in Costa Rica in fact was not one as they were comparing the efficacy of the project on school attendees in 2002 while the project was still underway. The French study of Cambodian healthcare was done firmly within implementation yet was titled ex-post. Another, evaluating a women’s group project in Kenya says they have a post-project evaluation but how can it be post project if it was only one and a half years after he baseline while still during implementation? A third “ex-post” on Job Youth training in the Dominican Republic only looks at the efficacy one year after the training, hence again during implementation and firmly not ex-post. 

 

The only true ex-post was a 15-yea.r-old study by the World Bank in Nicaragua, comparing the impact of an emergency social investment fund for primary education, rural health and water/sanitation projects that were done from January 1994 to June 1997.  It was evaluated two years after completion regarding targeting the poorest, community priorities and participation, projects' utilization rates, and operational and physical sustainability and impact on beneficiaries' health and education status.  The rest of the documents on 3ie under ex-post were secondary systemic reviews of other project evaluations and a hopeful post that a new UK collaboration will track and improve aid impact. While DIFD is doing terrific work on value for money, it focuses on donor money well spent rather than country-national participants and partners’ views of how well they felt our project money served them. But I quibble.

 

So is it a matter only of definition of what impact we’re looking for and expectations that we can find it during implementation rather than the dusty ankles post-project/ ex-post research Valuing Voices promotes?  As a PhD I firmly expected to find many definitions and examples of real, sustained research on actual long-term impact and the 3ie site and on many donors’ sites.  I am flabbergasted today at not only the lack of post-project/ ex-post evaluations but also that these terms seem not to be defined. There are no definitions or examples on Betterevaluation, nowhere on the American Evaluation site or within USAID's Evaluation Policy, none at all in UK's DFID's annals on overseas aid effectiveness. Other Valuing Voices blogs have outlined the dearth of post-project evaluations. Only today did I realize even their definitions are scarce. One definition I did find was from the UK's Department of Education:

 

The purpose of this post-project evaluation (PPE) is to:

  • Evaluate the effectiveness of the project in realizing the proposed benefits
  • Compare planned costs and benefits with actual costs and benefits to allow an assessment of the project's overall value for money to be made
  • Identify particular aspects of the project which have affected benefits either positively or negatively; [and make] recommendations for future projects
  • Reveal opportunities for increasing the project's yield of benefits, whether they were planned or became apparent during or after implementation, and to recommend the actions required to achieve their maximisation

 

Sustainability is the hoped-for outcome of ex-post evaluation work; so I have long turned to the OECD/ DAC have five related criteria for evaluating development assistance. They define sustainability as concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable. When evaluating the sustainability of a programme or a project, it is useful to consider the following questions:

  • To what extent did the benefits of a programme or project continue after donor funding ceased?
  • What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project?

 

While this definition is 15 years old, it works quite well. Are you equally surprised at how hard it is to find definitions and examples of post-project and ex-post evaluations? 3ie doesn’t seem to have it down; USAID has hundreds of documents listed as ex-post. Some are secondary analyses of previous evaluations and most simply say post-project evaluation should be done. At least USAID has now two new ex-post studies (one in 2014, one in 2015). They are the first studies done in 30 years by USAID.  The latter is a brave and fascinating one, where Tufts evaluates Food For Peace projects ex-post in four countries this month, with publicly mixed results.  Even J-Pal, the highly-respected Abdul Lateef Jameel Poverty Action Lab does rigorous impact evaluations, comparing the outcomes of project activities to non-participant control groups to ascertain the effectiveness of the interventions, but it too seems to have no documents on their impact evaluation site that show the sustainability of impacts.

Putting ourselves in the shoes of those we indend to serve… the sustained impact is what they actually want.

When will our organizations actually go back to the field and learn from those living with the projects years after donors and implementers leave? When will we walk our accountability talk? DO TELL!

Who is responsible for sustaining development?

Posted by on Nov 19, 2015 in Beneficiaries, Evaluation, Exit strategies, International aid, Local Participants, Participants, post-project evaluation, Project design, Rural Development, Sustainability, Sustainable development | 0 comments

Whose responsibility is it to sustain project activities?

Billions of dollars are pumped into development activities in developing countries all over the world. Communities getting involved in these projects have a clear objective, which is to have their lives improved in the sectors that the projects target. As to whether this is the main objective of the development partners is not clear. What is clear is that the development partners focus more on numbers than on getting people to participate.

 

We note that majority of these projects are designed to last between 2-5 years. Delays occasioned poor planning or other unforeseen factors eat into the implementation time to an extent that in some projects it takes about 1-2 years to get a program running. This means that the planned implementation time is reduced.  Baselines, midlines and end lines studies are conducted to inform changes that may have occurred within the program life, and in most cases they happen shortly after the program has started or just before it ends. In fact, some baselines are conducted after programs have started.

Considering the reduced implementation time and the fact that it takes a much longer time to get concrete behavior change related results, questions emerge whether indeed the reported changes are solid enough during implementation to be sustained. There is also a difference between measuring what can be referred to artificial changes (activities that community members adopt as a way of short-term trial in their excitement, but don't find useful afterward) and long lasting changes that community members adopt because they are useful part and parcel of their lives.

 

Almost all projects have logical frameworks (logframes) that show how project activities will be implemented and to some extent there are also exit strategies for closing out the project. This can be an illusion long-term. In most cases donors and implementers assume that communities will adopt activities that are being implemented within a specified period of time, and so projects close down at the end of the specified period of implementation assuming things will continue, but have no proof. Valuing Voices has done projected sustainability work in Ethiopia which points to possible differences between what donors expect to the sustained versus what communities are able to sustain.

 

Communitywomansorting

 

The big questions remain: "whose responsibility is it to ensure that whatever has been adopted is continued? Whose responsibility is it to sustain project activities post project implementation?" It is silently assumed that communities can take up this responsibility and a key question is "what guarantees are there that this is possible and is happening?" Project sustainability should not be seen as a community-alone responsibility but rather a responsibility for all those who are involved in program activities.  Sustainability studies should be planned for and executed in the same breath that the baselines, midlines, end-lines and in the rare cases impact assessments in real time should be planned for. We must do sustainability studies as they provide an additional realistic opportunity to inform us on actual community development post project implementation. Communities should not be left alone with it.

 

The Disruptive Potential of Feedback (reblog from NonprofitChronicles.com)

Posted by on Oct 29, 2015 in Accountability, Feedback Labs, Feedback loops, Fund for Shared Insight, Giving Evidence, Hewlett Foundation, Innovations for Poverty Action, International aid, Keystone, Participation, VOTO Mobile | 0 comments

The Disruptive Potential of Feedback

by MARC GUNTHER, OCTOBER 18, 2015 ( reblogged http://nonprofitchronicles.com/2015/10/18/the-disruptive-potential-of-feedback/)

Few institutions in the US are as undemocratic as endowed foundations. The executives in charge of foundations answer to, er, no one. They give money away, so people tend to laugh at their jokes, tell them they look well, nod in agreement at their banal remarks. What’s not to like?

As for nonprofits, they pay heed to foundations and donors, but they need not listen to their “beneficiaries,” unless they feel a moral obligation to do so. What if, goodness knows, the people they are trying to serve turn out to be unhappy with the service? Talk about inconvenient truths.

fbl-final-logoLast week in Washington, a group of about 70 people — the generals and foot soldiers of a growing movement to devolve power to mostly poor recipients of aid in the US and abroad — came together to talk about how to turn that power dynamic of philanthropy upside down. They believe that feedback from constituents “has the potential to unleash massive, timely and necessary changes in the way social change and development are pursued,” in the words of Feedback Labs, a DC-based NGO that convened the firstFeedback Summit.

As Dennis Whittle, the executive director of Feedback Labs, has written:

Will aid and philanthropy democratize themselves? Will aid agencies and foundations cede power and sovereignty to the people they are trying to serve?

It’s too soon to say but there were signs during the two-day confab that a half-dozen or so forward-thinking foundations, along with a growing number of nonprofits, are starting to figure how to create tight feedback loops that will enable them to solicit feedback from citizens, listen, analyze and, most important, change their practices as a result of what they learn.

“It’s the right thing to do, morally and ethically, philosophically. It’s the smart thing to do,” Whittle said. Now the goal is to make it “the feasible thing to do, financially and operationally.”

How do feedback loops differ from conventional monitoring and evaluation (M&E)? One attendee told me that feedback loops are the equivalent of diagnosing and treating a disease; a conventional evaluation is more like an autopsy, and thus of limited value to the patient.

Here are three signs that the feedback movement is gathering momentum:

A group called the Fund for Shared Insight, a collaboration of foundations that makes grants to improve philanthropy, has launched an initiative called Listen for Good that intends to fund 50 nonprofits to seek feedback from the people they are trying to help. They will use the now-famous Net Promoter System methodology developed for business by Bain & Co., which is working with the fund to make sure that the simple, elegant and yet rigorous system is deployed effectively. “It needs to be a high velocity loop of feedback, learning and action,” said Vikki Tam, a Bain partner. To the extent possible, the feedback results will be made public, enabling nonprofits to compare their net promoter scores with peers. “It’s so important for foundations to be open about what they do, and what they’re learning,” said Lindsay Austin Louie, a program officer at William and Flora Hewlett Foundation who works closely with the fund. Supporters of the fund include the David and Lucile Packard Foundation, the Ford Foundation, the Gordon and Betty Moore Foundation, the JPB Foundation, Liquidnet (which I wrote about here), the Rita Allen Foundation and the W.K. Kellogg Foundation.

FeedBackLoopBanner-300x300Efforts to build “good enough” feedback loops are underway, aimed at helping small or midsize NGOs measure their impact without having to undertake expensive, long-term randomized control trials.  Thoai Ngo, a senior director of research at Innovations for Poverty Action, talked about an effort called The Goldilocks Project that aims to build “right-fit” evaluation systems, focusing on collecting credible, actionable data in a timely way–that is, feedback to help an NGO change course if needed. Ken Berger, the former chief executive at Charity Navigator, described his work at a firm called Algorhythm which offers impact measurement to small NGOs for as little as $750. “These are organizations that never before had an opportunity to measure what matters most,” Berger said.

Technology is making it much easier to gather feedback, and make sense of it. Louis Dorval is the co-founder of VOTO Mobile, a Ghana-based tech startup that aims to “amplify the voice of the under-heard” by using voice and text messages on mobile devices to survey citizens, as well as send one-way messages. Less than three years old, Voto Mobile has already worked with about 250 organizations, including Unicef and Innovations for Poverty Action. David Bonbright of Keystone Accountability, who is a pioneer in the feedback arena, talked about Feedback Commons, an online platform designed to allow  organizations to “share and compare” the feedback they collect from their constituents.

SUPPLY AND DEMAND

All these initiatives are designed to improve the development of feedback loops. Before long, NGOs should be able to show how collecting feedback generates better outcomes for their clients. That’s all to the good. Think of that as the supply side of the feedback “business.”

But what about the demand side? Who’s going to fund feedback loops? I’m still a newcomer to the development world but my impression (from reporting on water and sanitation projects, and on cookstoves) is that most foundations and NGOs are not as rigorous as they could and should be about measuring their impact.

This brings us back to the fundamental power dynamic of philanthropy, as Caroline Fiennes of Giving Evidence explained during the Feedback Summit.

Why don’t foundations ask for feedback?” Fiennes asked. “Because they don’t have to.” 

“It’s difficult and it’s a bit painful,” she went on. “If you have made a chunk of money and you want to give it away, in general you will feel good about that, and everybody will love you. Once you start asking questions” — questions designed to find out if the work is making as much of a difference as it can–“you might not like the answers.”

In an essay called What do they want? at Aeon, Claire Melamed elaborates:

While most individual aid workers do care, very much, about the people they work with and for, the actual structure of the aid business offers few reasons for anyone to worry about what aid recipients think or want. Staff in aid agencies need to think about what their funders want to pay for. For their own performance reviews, they need to think about how to demonstrate that what they are doing is achieving the best possible results with the smallest amount of money. So the incentives for spending money on expensive surveys to find out what representative samples of poor people think of their operations are just not there.

And besides, the information might be a threat. What if it turned out that people feel patronised by aid workers? Or that they would rather their food didn’t arrive with logos announcing their indebtedness to foreign governments? Or that they resent being given a T-shirt when really they would sooner just have the money? What if people don’t really want another agricultural programme, and they’d rather have a bus ticket to the nearest town and somewhere to stay when they get there? These kinds of discoveries could be quite discomfiting for the agencies themselves – though in the long run, they would presumably do a better job.”

This is why those funders (like the foundations behind the Fund for Shared Insight) who push for feedback loops and rigorous evaluation deserve a lot of credit. Let’s hope their numbers grow.

 

Sustainable Development Goals and Foreign Aid– How Sustainable and Accountable to whom?  Reposting Blog from LinkedIn Pulse

Posted by on Sep 24, 2015 in Accountability, Brookings Institute, Center for Global Development, Evidence-based policy, ex-post evaluation, Exit strategies, foreign aid, IEG, Milenium Development Goals, Millennium Challenge Corporation, Participants, post-project evaluation, self-sustainability, Sustainable development, USAID, USDA, World Bank | 0 comments

Sustainable Development Goals and Foreign Aid– How Sustainable and Accountable to whom?       

Jindra Cekan, PhD of ValuingVoices

World leaders will paint New York City red next week at the UN Summit adopting the new post-2015 development agenda.  The agreed plans set 17 new ‘Sustainable’ Development Goals (SDGs) to be achieved by 2030. These are successors to the Millennium Development Goals (MDGs).

While many of us have heard of them, how many of us know whether we met them and what prospects are there for the Sustainable Development Goals to do well or better?  According to Bill Gates the MDGs were “the best idea for focusing the world on fighting global poverty that I have ever seen.” The Brookings Institute goes on to praise the eight MDG targets for aiming high by setting targets such as halving world poverty and reducing child mortality, improving universal primary education and gender equality and empower women etc. Donor countries pledged three times more than they had until that time (raising the percentage of gross national income for international development assistance from 0.2% to 0.7%, not huge amounts but laudable).  From 2000 to 2015 extreme poverty did fall by half (although some argue China and Asia were well on their way before this aid came) and in some countries (Senegal, Cambodia) child deaths fell by half.  Global health improved via huge coalitions on immunization and HIV/AIDS. Yet while poverty dropped and health increased, hunger, environment and sanitation targets were not met, for instance, and there are 850 million people still hungry worldwide (11% of all people). Yet gains far outweigh losses.  The new Sustainable Development Goals are to be achieved in 15 years, by 2030. The UN and member nations will track a remarkable 169 indicators,monitoring progress towards the SDGs at the local, national, regional, and global levels…[1]

BusinessWorld___INFOGRAPHIC__MDGs__It’s_2015__Now_what_

Overall, one would feel rather tickled by these results — not 100% but still amazing given global disparities.  Nancy Birdsall of the Center for Global Development thinks measurement is not the goal: “Growing global interconnectedness means that the problems the world faces, that hold back development, are increasingly shared… we’re making a promise to ourselves that we are one world, one planet, one society, one people, who look out for each other…”

But I’m a fan of measurable results, I must admit. One would logically think that our international development projects funded by U.S. Agency for International Development (USAID), the U.S. Department of Agriculture (USDS), the Millennium Challenge Corporation and others could outline how the caused the good results that some MDGs showed. USAID’s website links its work to the MDGs clearly: “in September 2010, President Obama called for the elevation of development as a key pillar of America’s national security and foreign policy. This set forth a vision of an empowered and robust U.S. Agency for International Development that could lead the world in solving the greatest development challenges of our time and, ultimately, meet the goal of ending extreme poverty in the next generation.” It goes on to talk about work to “Promote sustainable development through high-impact partnerships and local solutions.USDA’s Foreign Agricultural Service states their “non-emergency food aid programs help meet recipients’ nutritional needs and also support agricultural development and education. These food assistance programs, combined with trade capacity building efforts, support long-term economic development.” Finally, MCC states they are “committed to delivering results throughout the entire lifecycle of its investments. From before investments begin to their completion and beyond… MCC’s continuum of results is designed to foster learning and accountability.”

Maybe. We don’t actually know because 99% of the time we never return to projects after they end to learn how sustainable they actually were. We could be fostering super-sustainability. Or not.

For international development programming works on 1-5 year programming cycles. Multi-million dollar project requests for proposals are designed and sent out by these funders to non-profit or for-profit implementers. These are awarded to one or more organization, quite rigorously monitored, and most have very good results. Then they end. Since 2000, the US Agency for International Development has spent $280 billion on country-to-country development and humanitarian aid projects as well as funding multilateral aid and in spite of much work evaluating the final impact of projects at the end, they never go back..  The EU has spent a staggering $1.4 trillion. USAID has funded one evaluation that has gone back to see what communities and partners could sustain… in the last 30 years, and that is about to be published. A handful of international non-profits have taken matters in their own hands and funded such studies privately. The EU’s track record is even more dismal, with policies being proposed but not done. The World Bank, which has funded over 12,000 projects has an independent evaluation arm, the Independent Evaluation Group. They returned after projects closed out to evaluate results only 33 times and we found only three of them systematically talked to project participants about what was sustained.

The bottom line is, how do we know anything we’ve done in international development or SDGs is sustainable unless we go back to see? What amazing or awful results must we know for future design? If we do not return, are we really accountable to our taxpayers and our real clients: the participants and the national countries foreign aid recipients themselves?

The UN has pledged to have an SGD report card to “measure progress towards sustainable development and help ensure the accountability of all stakeholders for achieving the SDGs….[and a] Global Partnership for Sustainable Development Data, to help drive the Data Revolution….by using data we can ensure accountability for the policies that are implemented to reduce global and local inequities.”  I completely agree that having citizen generated data at the local, national, regional, and global levels is so very important “to fill gaps in our knowledge, establish global norms and standards and…help countries develop robust national strategies for data development.”  And as the World Bank IEG’s Caroline Heider states, measuring them is complex (e.g. agriculture affected by climate change and measuring changes across sectors is hard) but worthwhile.

While SDG data tells us what donor-funded activities and policies work, very few in international development know how sustainably our programming works for our ultimate clients, our participants and partners.  And the price needn’t be high—a recent post-project evaluation we did cost under $120,000 which is a pittance given the project cost over $30 million and reached 500,000 folks.  We found clear (mostly successful) lessons. USAID has, after 30 years, funded one post-project evaluation that also has clear cost-effective lessons (forthcoming). Really, in this era of cost-effectiveness, don’t we want data on what worked best (Note to self: do that more) and what worked least (Note to self: stop doing that)?

Learning what participants and partners could self-sustain after we left is actually all we should care about. They want to get beyond aid. Shouldn’t we know if we are getting them there? Self-sustainability of outcomes is a clear indicator of good Return on Investment of our resources and expertise and their time, effort and expertise. It shows us we want to put ourselves out of a job, having built country-led development that really has a future in-country with their own resources.

Two steps are:

1) Donors to add a funding equivalent to 1% of program value for five years after closeout for all projects over $10 million to support local capacity-building of NGOs and national partners to take over implementation plus to evaluate lessons across different sector’s sustainability outcomes.

2) A cross-donor fund for country-led analysis of such learning plus lessons for what capacity needs to be built in-country to take over programming. This needs support from regionally-based knowledge repositories and learning centers in Africa, Asia, etc. Online and tangible centers could house both implementer reports/ evaluations and analyze/ share lessons learned across sectors and countries from post-project evaluations for projects that closed out 2-7 years ago for future design.

Now that is accountability. Let’s advocate for sustainability funding, data and learning now.

What are your suggestions? How can we improve sustainability?


[1] http://valuingvoices.com/wp-content/uploads/2015/09/BusinessWorld___INFOGRAPHIC__MDGs__It’s_2015__Now_what_.png

https://www.linkedin.com/pulse/sustainable-development-goals-foreign-aid-how-whom-jindra-cekan-phd?published=t

Face our fears! Learn from failure…

Posted by on Sep 16, 2015 in Accountability, Aid effectiveness, Community Driven Development, Effective Philanthropy, Failure, Global Giving, International aid, Participants, Sustainability | 0 comments

Face our fears! Learn from failure…

Global Giving has a nice example about getting participant feedback success/ failrure of a project their fundraising funded. The organization failed which was sorrowful to the players in West Africa and funders worldwide (I too am a soccer-mom). Yes, it failed. It is hard to read those words and all involved faced it and they learned from it. Thanks to a sprited discussion on Pelican Platform for Evidence-based Learning about a plethora of unintended impacts, I learned about the organization committed to learning from failures: Admitting Failure.  

 

Maybe some of you have heard of FailFests, such as the one in Raleigh N.C US this year that answered "Why are we doing this?" with "We’re on a mission to erase the stigma around failure. The more we talk about it, the more we can learn from it. Failure doesn’t have to be fatal!" You may know Engineers without Borders Canada has been an early proponent of "openly acknowledging failure is often a catalyst for innovation that takes our work from good to great."  Such examples are heartening, especially for participatory sustainability evaluation.

Admitting_Failure___Engineers_Without_Borders_Canada

In 2014 a met a fellow evaluator at the American Evaluation Association Conference where I was presenting on Post-Project Sustainability Evaluation. They told me their organization was waiting for a really successful project to end so they could evaluate its sustainability. Another evaluator told me of a post-project evaluation that was hidden away, unpublished, after findings were negative. And that's where the problem lies: our international development industry's neurosis about presenting anything unsuccessful.

 

No wonder so few projects have been evaluated post-project, as we fear:


* What if activities and outcomes aren't sustained (note: we didn't design them that way, often we opt for quicker wins achievable only with large resource inflows of funds and technical help)?

 

* What if our funders find out that funds had limited impact or partners had no means to continue good programming?

 

* What if we had entirely unintended impacts that favored some over others (well beyond our expected logical frameworks and Theories of Change)?

 

Hallelujah 🙂 Why can't we see this as good news that propels us to improve our projects by listening to what worked? To learn not to do what didn't work again?! To design in locally sustainable ways now that we have learned?


* What if we learn that our resources and empowerment led them to succeed on their own terms in ways we couldn't imagine, far exceeding the planned impacts we had expected?

 

* What if we find that unexpected outcomes showcased ways groups within communities stood on their feet, making development work for them on their terms?

 

Would we design and implement, monitor and evaluate projects differently? We'll see. Organizations such as USAID’s Food for Peace funded a four-country study on exit-strategies (forthcoming 2015), are looking at success and failure. Catholic Relief Services hired Valuing Voices to do sustainability evaluation in Africa. Others may follow…

 

Post-project sustainability evaluations expose us to learning the full range. Let's be brave and ask, learn and innovate, making aid and philanthropy more effective, and learn from failure for success!