by Jindra Cekan | Jul 7, 2020 | Accountability, ex-post evaluation, International aid, Participation, post-project evaluation, Sustainable development
On June 24th under GLocal’s UNConference, “Co-creating our future stories of hope and action”, Jindra Cekan, Holta Trandafili, and Isabella Jean presented their work on sustainability evaluations and exit strategies via local voices. We chaired a 2-hour discussion session on the following topics:
- Sustainability of global development projects and exit from them,
- The importance of valuing local partners’ and participants’ voices,
- How to embed ex-post evaluation of sustainability into the project cycle,
- How expectations, benchmarking, and early joint planning in exit strategies, as well as considering long-term ownership & relevance will support projects to be sustained locally, including questioning who will maintain results,
- Considering power dynamics between donors and ultimate ‘beneficiaries’ and
the value of the impact of the project from a variety of perspectives, and more
Here is the recording of our presentation or see just the PowerPoint presentation. We harvested lessons from our three presentations:
Jindra Cekan:
· Fear of learning about failure in our global development industry – INGOs are “waiting for a successful enough project” to commission ex-post project sustainability evaluation
· It needs to be a culture of learning, not a culture of success.
· Lack of transparency in sharing program evaluation results with communities and local government is widespread, and even more rarely do we come back after many years and share learnings
· Participatory approaches are vital, listening to participants about sustained impacts is key
· It is never attribution, always contribution. To isolate impacts, we need to look for project sites that haven’t had multiple other organizations overlapping through all phases.
· Building sustainability planning throughout the project cycle is key – but often doesn’t happen
Holta Trandafili:
· Sustainability needs to be planned to be researched, including evaluating why or why not were project elements sustained, and why? What has the project done to enable communities to sustain improvements?
· Expectations of sustainability need to be more modest (as most results are mixed good/bad)
· We need to ask: How are you defining and measuring sustainability – for how long should the results last? Among how many participants? Have you set benchmarks for success?
· We should expand your toolbox on methodology to investigate sustainability. Stories of success are one of a myriad of methods used, including mixed-methods, cost-benefit, etc.
· Start with the need for learning not [just] accountability
Isabella Jean:
· Sustainability investigation/evaluation/learning should be mindful that this is NOT about projects. It is about people.
· We have a system that focuses on gaps and needs to be filled vs. existing capacities’ structures to be reinforced. How can our work on measuring sustainability bring this to light and call it out, so that we change the norm?
· Planning for sustainability requires the insight to integrate resources and experiences of outsiders with the assets and capacities of insiders to develop context-appropriate strategies for change
If you would like to discuss this with any of us, please send us comments and we’re happy to respond. Thanks again to the G-Local UnConference team!
Below please find our bios:
Jindra Cekan/ova has worked in global development for 33 years focused on participatory design and M&E for global non-profits. She founded Valuing Voices 7 years ago. For details, see: Valuing Voices Founder
Holta Trandafili is the Research, Learning, and Analytics Manager with World Vision US and has been leading field research, monitoring, and evaluation since 2007. She has led sustainability measurement studies for World Vision programs in Uganda, Kenya, Sri Lanka, India, Burma, and Bolivia. Her areas of expertise and interest include program and community groups’ sustainability measurements; agency-level measurements; empowerment approaches to development; integrated programming; local capacities for peace; gender analysis; and outcome monitoring. Currently, Mrs. Trandafili serves as an Advisory Committee member for InterAction’s Effectiveness and Program Evaluation Working Group and chairs one of the sub-working groups under The Movement for Community-led Development.
Isabella Jean supports international and local organizations and funders to document promising practices, facilitate learning and strengthen capacities for conflict sensitivity, peacebuilding and humanitarian effectiveness. She has facilitated action research, collaborative learning and advisory engagements in over 25 countries, and serves as an advisor to policymakers, senior leadership and program teams. Isabella co-authored the book, Time to Listen: Hearing People on the Receiving End of Aid and developed practical guidance to support accountability to communities, listening and feedback loops, and responsible INGO exits. She teaches graduate-level courses on aid effectiveness, program strategies and M&E of peacebuilding at Brandeis University’s Heller School for Social Policy and Management. Previously, Isabella directed training at a community organizing network and conducted policy research for the Institute for Responsive Education, UNDP, and Coexistence International.
by Jindra Cekan | May 16, 2018 | Accountability, Aid effectiveness, CGDev, Evaluation, ex-post evaluation, Feed the Future, Feedback Labs, foreign aid, impact investing, Impact Investors, International aid, international development, Local Participants, Participants, Participation, post project evaluation, post-project evaluation, Return on Investment (ROI), SALT, Sidekick Manifesto, SUSRoi, Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation, Sustained and Emerging Impacts Evaluations (SEIE), USAID, Valuing Voices
Setting a higher bar: Sustained Impacts are about All of us
Global development aid has a problem which may already affect impact investing as well.
It is that we think it’s really all about us (individuals, wealthy donors and INGO implementers) not all of us (you, me, and project participants, their partners and governments). It’s also about us for a short time.
All too often, the measurable results we in global development aid and Corporate Social Responsibility (CSR) funded projects that last 1-5 years track and report data for two reasons:
1) Donors have Compliance for grantees to meet (money spent, not lost, and results met by fixed deadlines of 1-5 years – look at some of the European Commission Contracting rules) and
2) Fund recipients and the participants they serve are accountable to ‘our’ donors and implementers who take what happened through their philanthropic grants as ‘their’ results.
Both can skew how sustainably we get to create impacts. An example of such strictures on sustainability from USAID. As respected CGDev Elliot and Dunning researchers found in 2016 when assessing the ‘US Feed the Future Initiative: A New Approach to Food Security?‘ the $10.15 billion leveraged $20 billion from other funders for disbursement over three years (2013-16). “We are concerned that pressure to demonstrate results in the short term may undermine efforts to ensure any impact is sustainable…. Unfortunately, the pressure to show immediate results can encourage pursuit of agricultural investments unlikely to be sustained. For example, a common response to low productivity is to subsidize or facilitate access to improved inputs… it can deliver a quick payoff… however, if the subsidies become too expensive and are eliminated or reduced, fertilizer use and yields often fall…..
With so much focus on reporting early and often about the progress in implementing the initiative, there is a risk that it increases the pressure to disburse quickly and in ways that may not produce sustainable results. For example, for 2014, Feed the Future reports that nearly 7 million farmers applied “improved technologies or management practices as a result of U.S. Government assistance,” but only 1,300 received “long-term agricultural sector productivity.” Are the millions of others that are using improved inputs or management practices because of subsidies likely to have these practices sustained? And how likely are they to continue using improved practices once the project ends?”
3) Impact investors stick to the same two paths-to-results and add a new objective: market-competitive financial returns. They also need to show short-term results to their investors, albeit with social, environmental and governance results like non-profits (future blog).
4) Altruists create things we want ‘beneficiaries’ (our participants) to have. For instance a plethora of apps for refugees cropped up in recent years, over 5,000 it is estimated, which can be appropriate, nor not so helpful. Much like #2 above, ‘we’re’ helping ‘them’ but again, it seems to be a ‘give a man a fish’… and my fish is cool sort of solution… but do our participants want/ need this?
How often is our work-for-change mostly about us/by us/ for us... when ideally it is mostly about ‘them’ (OK, given human self-interest, shouldn’t changes we want at least be about all of us?).
All too often we want to be the solution but really, our ‘grassroots’ clients who are our true customers need to generate their own solution. Best if we listen and we design for long-term sustainability together?
As the Brilliant Sidekick Manifesto stated in two of its ten steps:
a) “I will step out of the spotlight: Sustainable solutions to poverty come from within are bottom-up, and flow from local leaders who are taking the risks of holding their politicians accountable and challenging the status quo.”
b) “I will read “To Hell with Good Intentions” again and again: Politicians, celebrities and billionaire philanthropists will tell me that I can be a hero. I cannot. The poor are not powerless or waiting to be saved. Illich will check my delusions of grandeur.”
We have examples of where we have stepped away and participants had to fend for themselves. At Valuing Voices, we’ve done post project-exit evaluations 2-15 years afterward. What did participants value so much that they sustained it themselves (all about them, literally)? These Sustained and Emerging Impacts Evaluations (SEIE) also give us indications of Sustained ROI (Sustained Return on Investment (SusROI) is a key missing metric. As respected evaluator Ricardo Wilson-Grau said in an email, “I think calculating cost-effectiveness of an intervention’s outcomes would be a wonderful challenge for a financial officer searching for new challenges — if not a Nobel prize in economics!”)
Most of these evaluations are pretty bad news mixed with some good news about what folks could sustain after we left, couldn’t and why not. (These are the ones folks expect to have great results, otherwise they wouldn’t share them!) While most clients are understandably interested in what of ‘theirs’ was still standing, and it was interesting disentangling where the results were attributable by implementation or design or partnership flaws or something else, what was mesmerizing was what came from ‘them’.
The key is looking beyond ‘unexpected’ results to look at emerging impacts that are about ‘them’ (aka what we didn’t expect that was a direct result of our project, e.g. spare parts were no longer available to fix the water well pump once we left or a drought rehabilitation water project that decreased violence against women), to what emerging results are attributable not to use but only to our participants and partners who took over after our projects closed. One example is a Nepalese project ended yet the credit groups of empowered women spawned groups of support groups for battered women. Another is a child maternal health project changed how it worked as women reverted to birthing at home after NGOs left; community leaders punished both parents with incarceration in the health clinic for a week if they didn’t given birth there (wow did that work to sustain behavior change of both parents!).
Many of us at Valuing Voices are shocked that funders don’t seem that interested in this, as this is where they not only take over (viz picture, sustaining the project themselves), but they are making it theirs, not ours. Imagine assuming the point of development is to BE SUSTAINABLE.
Source: Community Life Competence
Our participants and national stakeholder partners are our true clients, yet… Feedback Labs tell us Americans alone gave $358 billion to charities (equivalent to the 2014 GDP of 20 countries) – in 2014 but how much of this was determined by what ‘beneficiaries’ want? Josh Woodard, a development expert, suggests a vouchers approach where our true clients, our participants, who would “purchase services from those competing organizations… [such an] approach to development would enable us all to see what services people actually value and want. And when we asked ourselves what our clients want, we would really mean the individuals in the communities we are in the business of working with and serving. Otherwise we’d be out of business pretty quickly.”
This opens the door to client feedback – imagine if participants could use social media to rate the sustained impacts on them of the projects they benefited from? A customer support expert wrote in Forbes, “Today, every customer has, or feels she has, a vote in how companies do business and treat customers. This is part of a new set of expectations among customers today that will only grow ... you can’t control product ratings, product discussions or much else in the way of reviews, except by providing the best customer experience possible and by being proactive in responding to negative trends that come to the surface in your reviews and ratings stronger.”
So how well are we working with our participants for ‘development’ to be about them?
What do you think?
by Jindra Cekan | Aug 2, 2017 | evaluability checklist, ex-post evaluation, Faster Forward Fund (3F), measuring sustainability checklist, Participation, post-project evaluation, Project cycle, SEIE, Sustainability, Sustained and Emerging Impact Evaluation (SEIE), Sustained and Emerging Impacts Evaluation, Valuing Voices
Building the Evidence Base for Post Project Evaluation:
A report to the Faster Forward Fund
We are delighted to share Valuing Voices’ report on the value added of post-project evaluation, which compares findings from eight end-of-project and subsequent post project evaluations [1]. Many of you are aware of how rarely post project evaluations are undertaken. As a result, there is little real evidence about project impact on long-term sustainability. Valuing Voices received a grant from Michael Scriven’s Faster Forward Fund to begin to address this gap.
Our findings show that post project evaluations can contribute to better understanding of sustainability impacts, and reveal unexpected and emerging outcomes years after project close. They also indicate ways in which we can design and implement for sustainability.
Finding suitable projects for this review was difficult because so few post project evaluations are done, fewer are publically available, and fewer still had comparable final evaluations and included local voices. Agencies that fund post project evaluations offer a range of reasons for doing so: to learn, to promote a success, to inform replication or scale, to provide justification for future funding, to promote accountabilities. However, many funding agencies consider post project evaluation a luxury or not necessary. JICA and OECD are notable exceptions in this regard.
Highlights include:
- The review highlights the range of methods that have been used in post project evaluations, and point to the advantages of planning for sustainability measurement from the outset of the project.
- The cases reviewed in the study highlight the (sometime dramatic) difference between the anticipated trajectory of a project, what is happening as the project ends, and what actually continued, was adapted, ceased or changed course after close out.
- Taxonomies, knowledge management about evaluation, data retrieval/ retention, analysis, use and dissemination are elements of sustained impact evaluation that require attention.
- Little documentation is available about how post project evaluations have actually informed and influenced organizational learning, sectoral dialogue or future programming.
- Post project evaluations shed particularly interesting light on what emerged post-project that was entirely due to the efforts and resources of participants and partners after project investments stopped. More on these Sustained and Emerging Impacts Evaluations (SEIEs) at Better Evaluation.
As part of this report, Valuing Voices created an evaluability checklist for assessing whether a post project evaluation is viable, as well as a checklist for measuring sustainability starting at the beginning of the project cycle [2].
We welcome your comments on this report and checklists, and encourage you to share it in your networks and get us feedback on their use. Please use the report and findings to advocate for more post project sustainability impact evaluations which will contribute to greater evidence-based learning about project sustainability. Valuing Voices is among a handful of organizations who do post-project evaluations and we can either conduct one or refer you to another who does.
Thank you,
Laurie Zivetz, MPH, PhD and Jindra Cekan, PhD, with Kate Robins, MPH, PhD of Valuing Voices
The full report is available here:
https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf
Sources:
[1] Zivetz, L., Cekan, J., & Robbins, K. (2017, May). Building the Evidence Base for Post-Project Evaluation: Case Study Review and Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2013/11/The-case-for-post-project-evaluation-Valuing-Voices-Final-2017.pdf
[2] Zivetz, L., & Cekan, J. (n.d.). Evaluability Checklists. Retrieved from https://valuingvoices.com/wp-content/uploads/2017/08/Valuing-Voices-Checklists.pdf
by valuingvoicesjin | Jun 25, 2016 | Accountability, Aid effectiveness, Catholic Relief Services (CRS), Evaluation, ex-post evaluation, foreign aid, International aid, International non-profits, Local Participants, Niger, Participation, post-project evaluation, SEIE, Sustainability, USAID
Maximizing what we've got… Time is now!
We had a stirring conversation here in D.C. with someone very knowledgeable about sustainability; this person is a strong proponent of local ownership of all development. They also said vehemently, why evaluate the sustainability of projects after closeout; we all know what that will show! What was implied is that our system of international development and aid is so flawed, so broken, that the inevitable result of not focusing on local ownership as the fundamental basis for our work means Nothing. Will. Be. Left…. All. Is. Lost.
We disagree. Our development industry does some good, some bad, and is ever-changing (albeit slowly). Billions of dollars each year are spent trying to improve people's lives and livelihoods around the world, and we've seen great good be done. While. We. Remain.
We know far, far less about what remains after our projects end because less than 1% of the time we return post-project (ex-post) to evaluate anything.
Our problem with time begins with fixed timelines within projects that say we have 1, 3, 5 years to get to success. They work with participants and partners who need to make substantial changes to how they use their resources and beliefs over a relatively short time of a few months to a few years. We expect immediate results from them, changing how they farm (use new seeds, new methods, new ways of interacting with markets) and save money (learn new concepts of profit and interest, repayment and re-lending), and improve their health and that of their families (get prenatal exams, vaccinate your children, exclusively breastfeed without adding water or tea). Everyone in our projects is 'on the clock' from the donor and implementer to partners and participants. This clock ticks down irrevocably as project closeout looms, promised-successes-to-donors at hand or a mirage in the distance. We assume sustained results.
How many of us have ever gone on a diet? How many have learned a new language? How many of us have transferred jobs and had to learn new skills on the job? How quickly have we managed to do all that successfully, all at once?! Probably many. How many of you have had to do this on a fixed timeline? Were you successful when there was a limited, fixed time and you did not set your own pace?
https://www.flickr.com/photos/psd/7739861570
It takes time to implement projects well enough to ensure that most participants ‘got it’, not just the 'early adopters'. It takes time to hand over projects so well that our partners and participants are ready to take over at least some of what we worked so hard to transfer. It took leadership and staff two years in the very successful participatory USAID/ Food For Peace food security project by CRS Niger that was a continuation of similar programming for 15 years. It also takes time to pass for conditions to be ready for our return, to isolate what people could self-sustain from what the project supplied, to learn what was so well designed and implemented during projects that to 'took root' in people's lives, that they have made it their own. We estimate optimal evaluation time is 2-7 years after closeout. Valuing Voices also believes we should not just evaluate the sustainability of outputs and outcomes of what we put in place that we thought they would continue, and the sustained impact of those cumulative investments, but also the emerging, unintended new activities and impacts we never imagined people would innovate from our projects. We are doing just such evaluations in Zimbabwe and Uganda now and hope to do and catalyze much more fieldwork around the world.
And why does it matter? Why shouldn't we write off our time-limited donor-funded projects? Because:
1) It's all we've got. Our current development system is not going anywhere soon, and there is success to learn from.
2) We need to quickly learn from what worked sustainably best and stop wasting time and resources on what we refuse to admit fails because we are too scared to return to see. Go back with the intention to learn what does and focus on doing more of what works.
3) Such analysis – and design of new projects – must have country ownership as a centerpiece throughout the project cycle assumptions, but to throw out decades of good work simply because we are just learning the value of country ownership is foolish.
Finally, here's a lovely example from Brazil of how local, participation (and yes, as my colleague thought, local ownership) works best. And. It. Takes. Time.
"Our results also show that Participatory Budgeting’s influence strengthens over time… Participatory Budgeting’s increasing impact indicates that governments, citizens, and civil society organizations are building new institutions… cities incorporate citizens at multiple moments of the policy process, allowing community leaders and public officials to exchange better information." How often do we return to do what are called longitudinal reviews of our work abroad, using the same rigorous standards we evaluate our domestic projects? Not often. Shouldn't that change?
Only by working together, honoring the value of our participants, that they deserve the same chance at change that we take for granted will things change. We must value both the voices of our participants and our own expertise for development to improve for true aid effectiveness…. Let us begin anew!
by Jindra Cekan | Apr 13, 2016 | Catholic Relief Services (CRS), Exit strategies, Food for Peace (FFP), international development, LIvelihoods, Niger, Participation, Sustainability, Sustainable development, USAID
PARTICIPATION BY ALL:
The Key To Sustainability of CRS/ Niger’s Food Security Project
Valuing Voices is delighted to share our sustainability evaluation of Catholic Relief Service Niger’s PROSAN project [1]. This project that ran from 2006-2012 in Niger and was implemented by three NGOs: CRS, Cooperative for Assistance and Relief Everywhere (CARE), and Helen Keller International (HKI) under the direction of United States Agency for International Development (USAID) Office of Food for Peace (FFP) as a multi-year assistance program (MYAP) to support food security activities in the Dosso, Tahoua, and Zinder regions. PROSAN focused on increasing agricultural production and agro-enterprise, improving household health and nutrition status, reinforcing the capacities of health agents, and enhancing community resiliency.
Here are the highlights from the report which itself is an excerpt from a longer analysis we did. Also please note one Annex highlights the similarlties/ differences we found to USAID/ FFP’s 4 elements of sustainability:
AIM, METHODS, AND RESEARCH QUESTIONS
The aim of this sustainability evaluation was to explore perceptions of sustainability from Nigeriens involved in PROSAN, former CRS staff and donors. It focused on evaluating participants’ adherence to project outcomes and their creation of new innovations. It also evaluated partners’ involvement in sustaining project outcomes.
This evaluation used qualitative and quantitative methods including community mapping, focus group discussions, beneficiary interviews, and key stakeholder interviews. The evaluation was carried out in six communities in the Dosso region, with more than 500 interviewees, focusing on the following research questions:
- Sustainability of activities and groups: Are the communities sustaining the activities three to five years after the end of the project? What can we learn from the communities and their post-project implementation partners?
- Spread and unexpected outcomes: If the project was considered a success in the eyes of the community, how well did it spread?
- Fostering Sustainability: What are the long-term prospects for continued sustainability?
FINDINGS
Three years after PROSAN’s conclusion, the project was considered a success by community members, national partners, the implementer (CRS), and donor (USAID) staff. The main findings include:
1. SUSTAINABILITY OF ACTIVITIES AND GROUPS
Eighty percent (80%)[*] of all activities were reported to have become self-sustained and community innovations have emerged:
- On average, households reported moving from being food secure for 3-6 months per year during PROSAN to 8-12 months at the time of this evaluation, which is a remarkable impact [1].
[1]
- Women reported greater income through the increase in sales of food that was produced and processed due to the grain mills [1].
- Respondents also reported improved household health, hygiene, and nutrition, with 91% of survey respondents indicating that their health and sense of well being had improved, especially through the efforts of the health posts and clinics that CRS helped build and the government of Niger’s efforts in sustaining them with resources and staff [1].
Community groups/committees have continued and are well-supported by NGO partners:
- 81% of the committees set up by PROSAN were functioning at the time of this evaluation, with many participants discussing ways to sustain best practices within their communities, and members still receiving regular trainings or updates [1].
- Several new and refresher trainings come through national partners, NGOs, and new channels such as radio programs [1].
- Some new NGOs and international organizations have built upon PROSAN’s success, for instance, by using land previously managed by PROSAN for a new vegetable gardening training program, building hygiene programs on past health awareness efforts, or extending agricultural credit for further inputs [1].
Twenty percent (20%) of implemented activities were not sustained or have stagnated:
- While hygiene practices were sustained by households and there was widespread latrine construction, sanitation was poor in the villages, and most latrines had fallen into disrepair [1].
- Fewer than 50% of women reported practicing exclusive breastfeeding for children less than six months of age [1].
- While almost half of all health committees no longer exist, new health clinics staff have replaced some of the work of the committees with health and agricultural promotion messages now being sent via radio, television, and cell phones [1].
- Literacy training and theater groups have completely ceased [1].
- With the exception of the Système Communautaire d’Alerte Précoce-Réponses aux Urgences’ (SCAP-RU) SCAP-RU early warning system which has expanded, other resilience activities such as roadwork and caring for the environment are a lesser priority due in part to the lack of food and cash-incentives to continue doing them [1].
2. SPREAD AND UNEXPECTED OUTCOMES
New innovations and ceased activities reflected the project’s legacy:
- Community innovations have emerged such as collective funds paying for cleaners of the new health center, community-imposed sanctions for births occurring outside of the health centers, and the monitoring of savings from well water sales.
- National partners have praised the project, with many lamenting its withdrawal. One non-PROSAN village told an Agriculture Ministry staff and potential NGO partner that “No one should bring a program here unless it is like PROSAN.”
- PROSAN-trained masons, well repair technicians, and village youth have learned land recuperation techniques (zai holes, bunds and demi-lunes) that helped generate income beyond project communities.
- Project activities that received free inputs have largely stopped being implemented once the incentives were withdrawn such as Food for Training (FFT), Food for Work (FFW), or Cash for Work (CFW) (e.g. literacy, seedlings, latrines, theater etc.); nonetheless the inputs were highly valued and have continued to support agriculture and health (carts, bicycles).
3. FOSTERING SUSTAINABILITY
The following areas were identified as potential barriers to sustainability that could be systematically explored in other projects:
- Although most committees are still functioning, there are no processes in place to engage and train youth and new inhabitants of the villages.
- While village communities have been maintained, there is an increasing lack of ministry resources (e.g., staff, transportation, and communications) to take the place of NGOs like CRS after a program ends.
- There is little management of knowledge around project data, which is further exacerbated by staff changes in NGOs, government ministries, and donors. Project data (proposal content, monitoring data, evaluation results, participant lists, partner names, and exit agreements) must be managed ethically, locally and be held online, accessible for future projects to use and for villages to conduct self-evaluations.
The full report is available here:
https://www.crs.org/sites/default/files/tools-research/participation-by-all.pdf
Sources:
[1] Cekan, J., PhD, Kagendo, R., & Towns, A. (2016). Participation by All: The Keys to Sustainability of a CRS Food Security Project in Niger. Retrieved from https://www.crs.org/our-work-overseas/research-publications/participation-all
[*] Percentages were calculated as a combination of the number of activities that had been continued and the percentage of participants which continued them.
by Jindra Cekan | Feb 19, 2016 | Accountability, Aid effectiveness, Asian Development Bank (ADB), CARE, ex-post evaluation, Exit strategies, Food for Peace (FFP), InterAction, Microenterprise, OECD, PACT, Participation, Project Concern International (PCI), Sustainability, Sustainable development, Sustained and Emerging Impacts Evaluation, USAID
What happens after the project ends? Country-national ownership lessons from post-project sustained impacts evaluations (Part 2)
In Part 1 of our blog on lessons learned from post-project evaluations, we explored:
- How we do it matters for great results
- Expect unexpected results
This time we turn to who continues after closeout, and what conditions foster both successful handover and ownership from the onset in order to foster sustained impact.
Who Takes Over? Country nationals
When project handover is integral to the design, development projects needn’t be long-term or expensive. What they need to be is increasingly community-driven. Unless exit strategies are explicit and thorough, sustained impacts are less likely.
1. USAID/ Food for Peace (FANTA/ Tufts)
An ‘exit strategies’ evaluation of 12 projects in four USAID Food for Peace (FFP) countries of Bolivia, Honduras, India, and Kenya carried out in 2009, three to four years after close out detailed mixed results, described here [1]. These were complex food security projects across multiple sectors of: maternal and child health and nutrition; water and sanitation; agriculture, livestock, and rural income generation; natural resource management; school feeding; and micro-savings and loans.
FANTA/Tufts found “providing free resources, such as supplementary food as an incentive for growth monitoring participation or free agricultural marketing services to promote sales, created expectations that could not be sustained once the free resources were no longer offered.” Valuing Voices found similar issues in Niger’s PROSAN (see part 1), with the lack of continued incentives (food and in-kind inputs) led to activities not being continued by community members.
On the other hand, in India, the government took over FFP food ration distribution after closeout. “This phase-over of responsibility to national government programs was effective in the case of supplementary feeding but not in the case of school-feeding (the latter through the midday meals program), due to varying levels of government commitment. India’s government had the resources, capacity (an already existing supply chain), and motivation (commitment) to provide this benefit.” Again, CRS/Niger showed us that decades-long investments in partnerships and 2+ years for phase-over pays off; 80% of outcomes were self-sustained three years on. CIDA Peru also found that “Shared responsibilities and participatory process were instrumental in ensuring sustainability….a shared understanding of project objectives and counterpart interventions was established with Peruvian sector authorities, between donors and local communities” [2].
The lesson learned about close coordination with the partners such as the national government during design and implementation in order for transition to country-ownership and responsibility to be smooth also appeared among multilaterals that Valuing Voices has examined. Three multilateral agencies stand out as having conducted multiple post project evaluation (OECD, JICA and the Asian Development Bank).
We posit that too often in international development, the accountability focus is on fulfilling funder (donor) requirements, rather than accountability to project participants and what is needed to achieve sustained impact for them (Figure 1, below). The optimal case has project funders, implementers and national governments aligning to support those we ostensibly serve: women, men, youth, elders in need of assistance.
@ValuingVoices2015
Are there certain kinds of projects or implementers that manifest optimal accountability? Far more examination is needed, but a promising path is microenterprise.
2. Pact’s WORTH project in Nepal
PACT’s project illuminates that local ownership and structures sustain results and even multiply impact. Implemented from 1999-2001, Pact worked with many local NGOs to reach 125,000 women in 6,000 economic groups across Nepal; of those, one quarter chose to implement village banks. Village Banks cultivated women as agents of change and development in their communities—promoting grassroots sustainability The post project evaluation in 2006 found that:
- Almost two thirds of the original 1,536 village banks were still active eight years after the program began and assets of an average village bank has tripled in the last three years post-project (from $1000 to over $3000 at the time of the evaluation) [3]
- 83% reported that because of WORTH they are able to send more of their children to school [3]
- Women’s economic groups helped start an estimated 425 new groups involving another 11,000 women with neither external assistance nor prompting from the project [3].
Why? The post-project report tells that the banks were not an end in and of themselves, women’s empowerment was: “WORTH groups and banks were explicitly envisaged as more than just microfinance providers; they were seen as organizations that would build up women as agents of change and development in their communities” [3]. Thus local Nepalese sustained and grew their own development.
3. CARE Zanzibar’s Village Savings and Loan Associations were evaluated four years post-project. Similar to PACT, they found the model was sustained and grew:
- Total membership rose from 1,272 in 2002’s closeout to an estimated membership of 4,552 in July 2006, an increase of 258% [4]
- During the most recent payout for all 25 groups, the mean rate of return was 53%, with individual groups’ rates ranging from 10% to 92% [4]
- Participants said that the main changes in the lives as a result of the program were an improved standard of living (22%), improved housing (21%) and increased incomes (20%) [4].
While wonderful, can it only be the responsibility of communities to sustain their gains? How well are we designing for country-ownership and handover to the state?
4. The UN’s OECD has dozens of post project evaluations on its website, funded by member governments.
One study illuminates that while communities may manage to sustain some of the outcomes, structural investment in national capacity to takeover is key. This example evaluated four 10-15 year-long projects funded largely by Germany that were carried out in Indonesia, Sri Lanka, Tanzania and Zambia with a cumulative value of Euro 145.1 million ($180 million). The study was done in 2004, evaluating activities an astonishing 30 years after inception. Results?
- The good news: “living conditions of the target groups have improved in all four project regions,” with specific sustainable project outcomes observed in the “health and education sector, food security, increase in income and employment and the ensuing rise in the standard of living”. Links were made to project-supported improvements in infrastructure, enhanced private sector economy, and the project’s innovations in agriculture.
- The bad news: “there was low institutional sustainability at the level of state executing organizations for all four projects due to inadequate funds, inefficient organizational structures and a lack of coordination”. Thus, viable exit and handover was limited. Structures advanced as part of a development project ran a high risk of not being sustainable.”
5. Three years ago, the Asian Development Bank reviewed 491 project completion reports (desk studies) and undertook a handful of field visits to projects financed between 2001-09. Similar results:
- “Some early evidence suggests that as many as 40% of all new activities are not sustained beyond the first few years after disbursement of external funding” [5]
- “National government ownership, commitment to and financing of the projects were vital to sustainability“ [5]
- “Neither governments nor other international agencies benefit from systematic information on whether projects reached their intended economic or social objectives over the full life of the intervention or in the decade afterwards” [5].
This is a clarion call to all funders to invest in future excellence by returning to the past, learning what worked best and what failed to do so, examine why and begin anew with accountability to our participants!
Conclusion
What can be done?
A) Foster ownership of the process of development through empowerment to begin with, as PACT’s WORTH project did in Nepal and elsewhere. InterAction’s lovely “A Missing Piece in Local Ownership: Evaluation” reminds us “the local ownership agenda must extend to all parts of the program cycle – from design all the way through evaluation. Including those meant to benefit from international assistance (we use the term “participants”) in deciding what should be done and how it should be done is critically important for effectiveness and sustainability” [6].
Ask yourselves how well we involve governments in collaborative design of what they feel they can sustain of our programming after we leave, how and for how long with what resources, linkages, capacity-built and motivation (see FFP study #1).
B) Design and implement in the present while considering sustaining outcomes and impacts in the long-term, as we learned in Part 1 of this blog as well as taking lessons from some emerging guides such as the systematic guidance of PCI’s Resource Guide for Enhancing Potential for Sustainable Impact [7].
C) Dare to return to learn. As Dina Esposito, the Director of USAID/Food For Peace stated, “this rigorous, retrospective [ex-post] approach is not widely done, but is essential if we are to understand the true impacts of our investments. To be effective, development projects must result in changes that last beyond the duration of the project themselves” [8].
Imagine the sustained cost-efficiencies of learning certain sectoral programs lent itself best to sustainability by communities, others needed Ministries to take over, others still needed different support such as private sector – or all of the above. If we look at sustained impact as our true goal, how differently could we work together? How much more efficiently would we use our global resources?
What can we say about the sustained impact post-project evaluations we have featured?
We have covered lessons about how matters in design and implementation; expect unexpected results and who takes over? Country nationals. Much more research and analysis is needed, many more case studies need to be created for us to understand how to foster the best handover as well as national ownership at the beginning, middle and end. Maybe you drew some of the same conclusions we have:
- Post project evaluations provide valuable insights about sustainability.
- Lessons from such evaluations can lead to better programming in current and future projects.
- The voice of national stakeholders—participants and partners, including governments is essential.
- Donors lose amazing opportunities to learn what works now and continues to work unless they fund more sustainable impact evaluations and support investing resources in fostering sustainability during design and implementation.
In Part 3, we will look at what is keeping us from looking to the past for the future (hint: funding, assumptions and fears) and how we can move ahead together…
Please join us in advocating for and funding this vital approach!
Sources:
[1] Food and Nutrition Technical Assistance (FANTA). (n.d.). Effective Sustainability and Exit Strategies for USAID FFP Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp
[2] Canadian International Development Agency (CIDA). (2012). Evaluation of CIDA’s Peru Program. Retrieved 2014, from https://web.archive.org/web/20140807174641/http://www.acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/Evaluations2/$file/peru-eng.pdf
[3] Mayoux, L. (2008, June). Women Ending Poverty: The WORTH Program in Nepal – Empowerment through Literacy, Banking and Business 1999-2007. Retrieved from https://www.findevgateway.org/case-study/2008/06/women-ending-poverty-worth-program-nepal-empowerment-through-literacy-banking
[4] Anyango, E., Esipisu, E., Opoku, L., Johnson, S., Malkamaki, M., & Musoke, C. (2006, January). Village Savings and Loan Associations: Experiences from Zanzibar. Retrieved from https://www.findevgateway.org/case-study/2006/01/village-savings-and-loan-associations-experiences-zanzibar
[5] Asian Development Bank. (2010, October 31). Post-Completion Sustainability of Asian Development Bank-Assisted Projects. Retrieved from https://www.adb.org/documents/post-completion-sustainability-asian-development-bank-assisted-projects
[6] Grino, L. (2015, February 19). A Missing Piece in Local Ownership: Evaluation. Retrieved 2015, from https://web.archive.org/web/20150502162547/https://www.interaction.org/blog/missing-piece-local-ownership-evaluation
[7] Choi-Fitzpatrick, J., Schooley, J., Eder, C., & Lomeli, B. (2014). A Resource Guide for Enhancing Potential for Sustainable Impact: Food and Nutrition Security. Retrieved from https://www.fsnnetwork.org/resource-guide-enhancing-potential-sustainable-impact
[8] Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fsnnetwork.org/ffp-sustainability-and-exit-strategies-study-synthesis-report
