by Jindra Cekan | Feb 19, 2016 | Accountability, Aid effectiveness, Asian Development Bank (ADB), CARE, ex-post evaluation, Exit strategies, Food for Peace (FFP), InterAction, Microenterprise, OECD, PACT, Participation, Project Concern International (PCI), Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation, USAID
What happens after the project ends? Country-national ownership lessons from post-project sustained impacts evaluations (Part 2)
In Part 1 of our blog on lessons learned from post-project evaluations, we explored:
- How we do it matters for great results
- Expect unexpected results
This time we turn to who continues after closeout, and what conditions foster both successful handover and ownership from the onset in order to foster sustained impact.
Who Takes Over? Country nationals
When project handover is integral to the design, development projects needn’t be long-term or expensive. What they need to be is increasingly community-driven. Unless exit strategies are explicit and thorough, sustained impacts are less likely.
1. USAID/ Food for Peace (FANTA/ Tufts)
An ‘exit strategies’ evaluation of 12 projects in four USAID Food for Peace (FFP) countries of Bolivia, Honduras, India, and Kenya carried out in 2009, three to four years after close out detailed mixed results, described here [1]. These were complex food security projects across multiple sectors of: maternal and child health and nutrition; water and sanitation; agriculture, livestock, and rural income generation; natural resource management; school feeding; and micro-savings and loans.
FANTA/Tufts found “providing free resources, such as supplementary food as an incentive for growth monitoring participation or free agricultural marketing services to promote sales, created expectations that could not be sustained once the free resources were no longer offered.” Valuing Voices found similar issues in Niger’s PROSAN (see part 1), with the lack of continued incentives (food and in-kind inputs) led to activities not being continued by community members.
On the other hand, in India, the government took over FFP food ration distribution after closeout. “This phase-over of responsibility to national government programs was effective in the case of supplementary feeding but not in the case of school-feeding (the latter through the midday meals program), due to varying levels of government commitment. India’s government had the resources, capacity (an already existing supply chain), and motivation (commitment) to provide this benefit.” Again, CRS/Niger showed us that decades-long investments in partnerships and 2+ years for phase-over pays off; 80% of outcomes were self-sustained three years on. CIDA Peru also found that “Shared responsibilities and participatory process were instrumental in ensuring sustainability….a shared understanding of project objectives and counterpart interventions was established with Peruvian sector authorities, between donors and local communities” [2].
The lesson learned about close coordination with the partners such as the national government during design and implementation in order for transition to country-ownership and responsibility to be smooth also appeared among multilaterals that Valuing Voices has examined. Three multilateral agencies stand out as having conducted multiple post project evaluation (OECD, JICA and the Asian Development Bank).
We posit that too often in international development, the accountability focus is on fulfilling funder (donor) requirements, rather than accountability to project participants and what is needed to achieve sustained impact for them (Figure 1, below). The optimal case has project funders, implementers and national governments aligning to support those we ostensibly serve: women, men, youth, elders in need of assistance.
@ValuingVoices2015
Are there certain kinds of projects or implementers that manifest optimal accountability? Far more examination is needed, but a promising path is microenterprise.
2. Pact’s WORTH project in Nepal
PACT’s project illuminates that local ownership and structures sustain results and even multiply impact. Implemented from 1999-2001, Pact worked with many local NGOs to reach 125,000 women in 6,000 economic groups across Nepal; of those, one quarter chose to implement village banks. Village Banks cultivated women as agents of change and development in their communities—promoting grassroots sustainability The post project evaluation in 2006 found that:
- Almost two thirds of the original 1,536 village banks were still active eight years after the program began and assets of an average village bank has tripled in the last three years post-project (from $1000 to over $3000 at the time of the evaluation) [3]
- 83% reported that because of WORTH they are able to send more of their children to school [3]
- Women’s economic groups helped start an estimated 425 new groups involving another 11,000 women with neither external assistance nor prompting from the project [3].
Why? The post-project report tells that the banks were not an end in and of themselves, women’s empowerment was: “WORTH groups and banks were explicitly envisaged as more than just microfinance providers; they were seen as organizations that would build up women as agents of change and development in their communities” [3]. Thus local Nepalese sustained and grew their own development.
3. CARE Zanzibar’s Village Savings and Loan Associations were evaluated four years post-project. Similar to PACT, they found the model was sustained and grew:
- Total membership rose from 1,272 in 2002’s closeout to an estimated membership of 4,552 in July 2006, an increase of 258% [4]
- During the most recent payout for all 25 groups, the mean rate of return was 53%, with individual groups’ rates ranging from 10% to 92% [4]
- Participants said that the main changes in the lives as a result of the program were an improved standard of living (22%), improved housing (21%) and increased incomes (20%) [4].
While wonderful, can it only be the responsibility of communities to sustain their gains? How well are we designing for country-ownership and handover to the state?
4. The UN’s OECD has dozens of post project evaluations on its website, funded by member governments.
One study illuminates that while communities may manage to sustain some of the outcomes, structural investment in national capacity to takeover is key. This example evaluated four 10-15 year-long projects funded largely by Germany that were carried out in Indonesia, Sri Lanka, Tanzania and Zambia with a cumulative value of Euro 145.1 million ($180 million). The study was done in 2004, evaluating activities an astonishing 30 years after inception. Results?
- The good news: “living conditions of the target groups have improved in all four project regions,” with specific sustainable project outcomes observed in the “health and education sector, food security, increase in income and employment and the ensuing rise in the standard of living”. Links were made to project-supported improvements in infrastructure, enhanced private sector economy, and the project’s innovations in agriculture.
- The bad news: “there was low institutional sustainability at the level of state executing organizations for all four projects due to inadequate funds, inefficient organizational structures and a lack of coordination”. Thus, viable exit and handover was limited. Structures advanced as part of a development project ran a high risk of not being sustainable.”
5. Three years ago, the Asian Development Bank reviewed 491 project completion reports (desk studies) and undertook a handful of field visits to projects financed between 2001-09. Similar results:
- “Some early evidence suggests that as many as 40% of all new activities are not sustained beyond the first few years after disbursement of external funding” [5]
- “National government ownership, commitment to and financing of the projects were vital to sustainability“ [5]
- “Neither governments nor other international agencies benefit from systematic information on whether projects reached their intended economic or social objectives over the full life of the intervention or in the decade afterwards” [5].
This is a clarion call to all funders to invest in future excellence by returning to the past, learning what worked best and what failed to do so, examine why and begin anew with accountability to our participants!
Conclusion
What can be done?
A) Foster ownership of the process of development through empowerment to begin with, as PACT’s WORTH project did in Nepal and elsewhere. InterAction’s lovely “A Missing Piece in Local Ownership: Evaluation” reminds us “the local ownership agenda must extend to all parts of the program cycle – from design all the way through evaluation. Including those meant to benefit from international assistance (we use the term “participants”) in deciding what should be done and how it should be done is critically important for effectiveness and sustainability” [6].
Ask yourselves how well we involve governments in collaborative design of what they feel they can sustain of our programming after we leave, how and for how long with what resources, linkages, capacity-built and motivation (see FFP study #1).
B) Design and implement in the present while considering sustaining outcomes and impacts in the long-term, as we learned in Part 1 of this blog as well as taking lessons from some emerging guides such as the systematic guidance of PCI’s Resource Guide for Enhancing Potential for Sustainable Impact [7].
C) Dare to return to learn. As Dina Esposito, the Director of USAID/Food For Peace stated, “this rigorous, retrospective [ex-post] approach is not widely done, but is essential if we are to understand the true impacts of our investments. To be effective, development projects must result in changes that last beyond the duration of the project themselves” [8].
Imagine the sustained cost-efficiencies of learning certain sectoral programs lent itself best to sustainability by communities, others needed Ministries to take over, others still needed different support such as private sector – or all of the above. If we look at sustained impact as our true goal, how differently could we work together? How much more efficiently would we use our global resources?
What can we say about the sustained impact post-project evaluations we have featured?
We have covered lessons about how matters in design and implementation; expect unexpected results and who takes over? Country nationals. Much more research and analysis is needed, many more case studies need to be created for us to understand how to foster the best handover as well as national ownership at the beginning, middle and end. Maybe you drew some of the same conclusions we have:
- Post project evaluations provide valuable insights about sustainability.
- Lessons from such evaluations can lead to better programming in current and future projects.
- The voice of national stakeholders—participants and partners, including governments is essential.
- Donors lose amazing opportunities to learn what works now and continues to work unless they fund more sustainable impact evaluations and support investing resources in fostering sustainability during design and implementation.
In Part 3, we will look at what is keeping us from looking to the past for the future (hint: funding, assumptions and fears) and how we can move ahead together…
Please join us in advocating for and funding this vital approach!
Sources:
[1] Food and Nutrition Technical Assistance (FANTA). (n.d.). Effective Sustainability and Exit Strategies for USAID FFP Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
[2] Canadian International Development Agency (CIDA). (2012). Evaluation of CIDA’s Peru Program. Retrieved 2014, from https://web.archive.org/web/20140807174641/http://www.acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/Evaluations2/$file/peru-eng.pdf
[3] Mayoux, L. (2008, June). Women Ending Poverty: The WORTH Program in Nepal – Empowerment through Literacy, Banking and Business 1999-2007. Retrieved from https://www.findevgateway.org/case-study/2008/06/women-ending-poverty-worth-program-nepal-empowerment-through-literacy-banking
[4] Anyango, E., Esipisu, E., Opoku, L., Johnson, S., Malkamaki, M., & Musoke, C. (2006, January). Village Savings and Loan Associations: Experiences from Zanzibar. Retrieved from https://www.findevgateway.org/case-study/2006/01/village-savings-and-loan-associations-experiences-zanzibar
[5] Asian Development Bank. (2010, October 31). Post-Completion Sustainability of Asian Development Bank-Assisted Projects. Retrieved from https://www.adb.org/documents/post-completion-sustainability-asian-development-bank-assisted-projects
[6] Grino, L. (2015, February 19). A Missing Piece in Local Ownership: Evaluation. Retrieved 2015, from https://web.archive.org/web/20150502162547/https://www.interaction.org/blog/missing-piece-local-ownership-evaluation
[7] Choi-Fitzpatrick, J., Schooley, J., Eder, C., & Lomeli, B. (2014). A Resource Guide for Enhancing Potential for Sustainable Impact: Food and Nutrition Security. Retrieved from https://www.fsnnetwork.org/resource-guide-enhancing-potential-sustainable-impact
[8] Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fsnnetwork.org/ffp-sustainability-and-exit-strategies-study-synthesis-report
by Jindra Cekan | Feb 19, 2015 | Accountability, Evaluation, ex-post evaluation, foreign aid, International aid, International non-profits, Local Participants, Microenterprise, Nepal, Participants, Participation, post-project evaluation, Results, SEEP, self-sustainability, Sustainability, Sustainable development, Village Banks
Making money– microenterprise– is this a way to sustainable livelihoods? PACT’s Nepalese Lessons
Many Americans are steeped in the belief that we must ‘pull ourselves up by our bootstraps’, that hard work and especially faith in small businesses is the way to success. This is one of the many reasons why microfinance so appeals to donors as an investment. Does it work?
The US NGO-umbrella, Interaction, posted some “Aid Works” global results, including “the percentage of USAID-funded microfinance institutions that achieved financial sustainability jumped from 38% in 2000 to 76% in 2012.” Yet there have been numerous detractors of the model and the unsustainability of control over resources/ empowerment [1] [2].
What does one ex-post evaluations that we have on hand tell us? PACT’s USAID-funded WORTH program in Nepal was focused on women ending poverty through business, banking and literacy/ bookkeeping [3]. The project, implemented between 1999 and 2001 worked with 240 local NGOs to reach 125,000 women in 6,000 economic groups across Nepal’s southern Terai (in 2001 a Maoist insurgency led to the groups being on their own) [3]. By then, 1,500 of these groups led by the women themselves (35,000-strong) received training to become informal-sector Village Banks [3]. Working with local NGOs enabled them to reach 100,000 women in a few months due to the NGOs’ presence and connections in the communities. The collaboration worked well due to a shared belief by PACT and the NGOs that dependency is not empowering. As the report says “WORTH groups and banks were explicitly envisaged as more than just microfinance providers; they were seen as organizations that would build up women as agents of change and development in their communities” [3].
In 2006, PACT and Nepalese Valley Research Group looked to see sustainability of the banks, the extent of retained income by the women as well as any effect on community development and broader issues such as domestic abuse [3]. They went to 272 Banks from a random sample of 450 from seven of the 21 WORTH districts. Remarkably, they found even more functioning: 288 (16 more) of them were thriving and – wow- WORTH women had spawned another 400 more groups on their own [3]. Participant interviews were done with members and management as well as those women who had left their Banks and members of groups that had dissolved plus they interviewed a ‘control group’ of poor, non-WORTH women in Village Bank communities.
Was it a universal success? Almost. See the bar chart below showing what impacts the management committee felt the village banks had had on members, which is mostly better off, some the same, some far better off. This held true for the original village bank members and the new bank members.
[3]
The SEEP network reviewed WORTH’s ex-post and found five key findings:
- Wealth creation: A Village Bank today holds average total assets of over Rs. 211,000, or $3,100, more than three times its holdings in 2001. Each woman member of WORTH now has an average equity stake of $116 in her Village Bank [3].
- Sustainability: Approximately two-thirds (64 percent) of the original 1,536 Village Banks are still active eight and a half years after the program began and five to six years after all WORTH-related support ended. That means there are nearly 1,000 surviving groups with approximately 25,000 members [3].
- Replication: A quarter of the existing WORTH groups has helped start an estimated 425 new groups involving another 11,000 women with neither external assistance nor prompting from WORTH itself. If all these groups are currently operating, then more Village Bankers are conducting business today in Nepal than when formal WORTH programming ended in 2001. The report also said 63% of the Village Bank members derived the income from agriculture/ sale of food versus 17% in commerce/ retail trade and the rest in miscellaneous trades. Over 40% of the participants said they borrowed to pay for education and health costs and another 20% to pay off other loans plus for festivals (e.g. birth, death) [3].
- Literacy: 97 percent of respondents reported that literacy is “very important” to their lives; 83 percent reported that because of WORTH they are able to send more of their children to school [3].
- Domestic disputes and violence: Two-thirds of groups reported that members bring their personal or family problems to the group for advice or help. Of these, three-quarters reported helping members deal with issues of domestic disputes and related problems. Forty-three percent of women said that their degree of freedom from domestic violence has changed because of their membership in a WORTH group. One in 10 reported that WORTH has actually helped “change her life” because of its impact on domestic violence [3].
The report outlines other impacts, including self-help actions such as two-thirds of groups being engaged in community action, and three-quarters said that the group has done something to help others in the community. Speaking of community, it is notable that the self-selected women were primarily from wealthier groups (60%), 15% from the middle class, with only 20% from the most disadvantaged castes [3]. Frankly this is not as surprising, as those most willing to take on risk are rarely the poorest until later; 67% of the very poor later wanted to join such a bank (once the risk was shown not to be too high versus income) [3].
The study’s author asks “Yet for all this documented success, WORTH and other savings-led microfinance programs remain among the best kept secret in the world of international development and poverty alleviation. Although together such programs reach some two million poor people, they go almost unnoticed by the $20 billion credit-led microfinance industry… The empowered women in this study—like WORTH women elsewhere in Asia and Africa— have proved themselves equipped to lead a new generation of entrepreneurs who can take WORTH [onward] through a model of social franchising now being pilot-tested [which is] as creative and potentially groundbreaking as is WORTH…WORTH has the potential to become an “international movement that supports women’s efforts to lift themselves, their families, and their communities out of poverty” [3].
So why aren’t are we learning from such projects and scaling them up everywhere? PACT is [4]. They have reached 365,000 women in 14 countries – including Myanmar, Cambodia, Colombia, Swaziland, DRC, Ethiopia, with Nigeria and Malawi starting this year [4]. Coca-Cola awarded $400,000 to PACT in 2013 to replicate WORTH in Vietnam with 2,400 women [5]. Who else is replicating this model? It’s not clear from many excellent microenterprise sites I visited except one tells me that Mastercard Foundation and Aga Khan are looking into wider replication as well. Let’s track their results and ask participants!
Sources:
[1] Bateman, M. (2011, September 20). Microcredit doesn’t work – it’s now official. Retrieved from https://opinion.bdnews24.com/2011/09/20/microcredit-doesn%E2%80%99t-work-%E2%80%93-it%E2%80%99s-now-official/
[2] Vaessan, J., Rivas, A., & Duvendack, M. (2014, November). The Effects of Microcredit on Women’s Control Over Household Spending in Developing Countries: A Systematic Review and Meta-analysis. Retrieved from https://www.findevgateway.org/paper/2014/11/effects-microcredit-womens-control-over-household-spending-developing-countries
[3] Mayoux, L. (2008, June). Women Ending Poverty: The WORTH Program in Nepal – Empowerment through Literacy, Banking and Business 1999-2007. Retrieved from https://www.findevgateway.org/case-study/2008/06/women-ending-poverty-worth-program-nepal-empowerment-through-literacy-banking
[4] PACT. (n.d.). WORTH. Retrieved 2015, from https://web.archive.org/web/20141106013639/http://www.pactworld.org/worth
[5] PACT. (2013, August 13). The Coca-Cola Foundation awards $400,000 grant to Pact. Retrieved from https://www.pactworld.org/article/coca-cola-foundation-awards-400000-grant-pact
