How ‘new’ are our projects… and who is aiming at the right outcomes?

 

How ‘new’ are our projects… and who is aiming at the right outcomes?

 

Valuing Voices exciting news is we have received research grant funding from the esteemed evaluator, Michael Scriven’s Faster Forward Fund. We’re looking into the value-added of (ex-post) Sustained and Emerging Impacts Evaluations (SEIEs) and we are doing the research now. We will be documenting methods used and discuss how best to evaluate such sustained impacts after project close-out. Very exciting stuff in this staggeringly ‘new’ field of evaluation.

During this research, a senior international development evaluation expert told us that they can’t return to evaluate now-closed projects because they aren’t the same projects anymore (after closeout) and we are no longer responsible for the results. That took my breath away.

All new projects come from old projects… we recycle old project design most of the time, occasionally making substantive changes in targeting or design but much of how we design and implement remains the same. And while we thoroughly evaluate them during implementation, learning ex-post is a key missing link which all projects in the future can benefit from as we do similar interventions and track similar outcomes year after year but we rarely know which ones were sustained or emerged anew. There absolutely are aspects that get adapted but there are only so many ways to heal the sick, improve crop growth, save money, learn to read and so on, and there is a world we need to learn about what enabled some to be sustained and even morph into new results!

This excellent article, Do NGOs (non-governmental organization) help?, notes that “due to donor pressure [NGOs] are increasingly forced to respond with a discrete project with x number of deliverable outcomes” [1]. It goes on to cite D. Sriskandarajah, the secretary-general of Civicus, a global network of civil society organizations and activists, wrote: ‘We have become a part of the problem rather than the solution… Since demonstrating bang for your buck has become all-important, we divide our work into neat projects, taking on only those endeavours that can produce easily quantifiable outcomes. Reliant on funding to service our own sizeable organizations, we avoid approaches or issues that might threaten our brand or upset our donors. We trade in incremental change’” [1].

We also settle for results while we control them, and don’t ask unpopular questions about who is to sustain these results, with what resources, and for goodness sake, why sustained impact was not funded, designed, implemented and monitored/ evaluated from the very onset in our rush to measurable results?

 

DontSettleForLess

 

As this great NGO article by Dinyar Godrej goes on to say, “most media scrutiny of NGO accountability is of how they use funds, their accountability to donors. But what of their accountability towards the recipients of their interventions” [1]? They have no lobbyists to persuade our funders they would like this but nto that, and often such lobbying for their needs falls to the very NGOs that have won these large contracts and tasked with implementing a dizzying array of mandatory input, output, outcome and some impact indicators. We do care deeply about results! US State Department/ USAID has a “Standard Foreign Assistance Master Indicator List” of 2,300 lines in an excel spreadsheet [2]. (There are more indicators still– custom and cross-cutting indicators, the mind boggles).

Wow. But are we asking the right questions? Are we asking what was sustained after all this hard work was done and ended? Rarely. Who should be?! “It is perhaps unrealistic to expect such large structural changes to be delivered by NGOs when governments don’t tackle them either.”

For the rub is this. When we take development over from national governments, largely do not involve country nationals in the funding, design and M&E of projects, then how sustained can these projects still be after we go? Millions are invested, then disappear… Last year, at local debrief at the end of one SEIE Valuing Voices did, the state of affairs became crystal clear when a government official asked us “Can you ever find some funds to fund us to do our own independent evaluations? Even if it is not the projects that they did themselves? We would be happy to get that support…”

When are we no longer responsible for doing great, sustained work? Valuing Voices will let you know what we found regarding the best ways to do SEIEs more. Stay tuned.

What do you think?

P.S. This blog topic prompted me to look for statistics on the number or percentage of funded projects that were renewed. Nothing.  Does anyone know how many or what % of projects were extended/ funded again after showing good results? (Often this happens in the form that a successful project in one area of the country gets either funded again or repeated elsewhere in the country or in the world, as have two of our own SEIEs, Niger and Ethiopia). For that matter, what made them so excellent to be replicated? What can we learn?

 

 

Sources:

[1] Godrej, D. (2014, December 1). NGOs – Do They Help? Retrieved from https://newint.org/features/2014/12/01/ngos-keynote/

[2] US Department of State. (n.d.). Standard Foreign Assistance Indicators. Retrieved 2017, from https://web.archive.org/web/20170404072145/https://www.state.gov/f/indicators/index.htm

 

Embedding Sustainability Everywhere – All Five Slices Now

 

Embedding Sustainability Everywhere – All Five Slices Now

 

It has been a tumultuous year, and next year does not look like we will have much stability as a respite. As domestic concerns grow larger in two huge economies, US and UK, the question of the place foreign aid will play abound in conversations around the world.

However 2017’s transitions transform our work, for now I do have some good news,

1) Sustainability can be cheap. Far cheaper, in fact to design for sustainability, create feedback loops checking on sustainability through the eyes of our partners and participants, monitor for sustainability than to assume it’ll happen and far cheaper than finding out funds could have had far greater impact if we had valued their voices in the first place.

In our work to help our clients and partners fund, design, implement and monitor/ evaluate with sustainability in mind, we created what we hope is a helpful tool (guidance forthcoming).

a) By Designing for Sustainability with those who will sustain them, their financial buy-in and commitment are far higher (see CRS/ Niger [1]), as is advocacy and community buy-in (see new post-project OXFAM/ DRC [2]) and there are indicators the costs of start-up later are more cost-effective.

b) By clarifying Sustainability Indicators we check assumptions about who will do so, how much of a priority our activities are before we scale them up (Federation/ Ethiopian Red Cross). Retrospective post-project sustainability evaluations also enable us to learning from past successes and do better.

c) Sustainability Monitoring and Adaptation involve those pesky but pivotal feedback loops which are vital to understanding if we have gone off the rails or not, especially in terms of unexpected shocks derailing logical frameworks of designed projects.  USAID’s nice recent CLA (Collaborating, Learning, Adapting) process includes donor funding for adaptation mid-stream which fosters effectiveness and sustainability [3]. Even lovelier are the Doing Development Differently examples that are often very low-cost and high-effect.

d) Informed Exit, Stakeholder Sustainability Consultation should be done throughout the cycle, at least a year earlier than most projects can begin this (note; not the last few months please). Transitioning for success leverages sunk costs for ongoing results. It includes heavy knowledge management on how the implementers managed information and resources, tracked data, sustained outputs and outcomes which now local partners will need to do, etc (FFP Exit Strategies study, a Czech study, and a new UK Results study shows range of items to consider during and post-transition) [4] [5].

e) Post-Project Sustainability: Our Valuing Voices/ Better Evaluation/ Tufts joint presentation at AEA on SEIE has more on how learning from post-project evaluation lessons can change sustainability of future projects for the better [6]! Further, what capacities and systems have been built in-country to sustain the results after our funding and expertise leaves? What can we do differently?

Can it be done? Demand is rising. I recently presented at a conference about embedding sustainability in programming now, and an enterprising NGO took this idea to heart. They proposed joint design with communities, which is the very bedrock of designing for sustained impact. Kudos!

 

2) Secondly, donors are willing to pay for sustainability.  Sustainability is ensconced in USAID/ Food For Peace’s (FFP) 2016 Strategy and CLA (above) [7]. In the section called New learning and Implications for FFP Programming, they present findings such as “actions that drive big results during the life of the project may actually undermine sustainability in the long run. It raises the question as to whether FFP is willing to accept more modest results in the near term if they can be delivered in a way that will yield more sustainable gains over time….”

They also point us in the direction of country-led development: “Sustained capacity, resources, motivation, and linkages all require a focus on catalysts for change beyond FFP.  Facilitative approaches that rely on and strengthen local actors help ensure that resource and knowledge transfers, and the incentives and linkages that support them, will be self-perpetuating beyond project end”.  Notably, while UK’s DFID focuses on maximizing impact through Value-for-Money, it is a shorter-term economy, efficiency and effectiveness rather than sustained impact for the end-users and DFID’s exit strategies have recently been critiqued [5].

There is an issue of disincentives for the new administration to heed (if the agricultural lobby for US food exports does not prevail): “In [USAID’s exit] study evaluating how sustainable the results of Title II development programs are 2–3 years after project closure, FFP found thatproviding free resources can threaten sustainability, unless replacement of those resources both as project inputs and as incentives has been addressed” [8]. As the Natural Resources section notes, “Whether entirely in the hands of the community or linked to a formal institution, the incentives and resources necessary to maintain a community asset are part of the system that will sustain it. The lack of such systems is visible in rusted irrigation pumps, failed mangrove plantations, abandoned bore wells, eroded dikes, and silted-in fish ponds around the world” [7]. Yet the private and public sectors are important: “Sustainable, broad-based change is more likely to be achieved by supporting and strengthening existing community, private sector, and public sector mechanisms for product and service delivery, and by supporting the capacity, quality, and accountability of government institutions” [7].

FFP’s new strategy calls for taking a systems approach to change that emphasizes sustainable long-term gains over unsustainable short-term wins. Even more delightful, in a small meeting at MFAN with Dina Esposito, Director of Food For Peace in November, she announced that they were looking to do pilot funding of an additional three years to typical five-year DFAP development projects. One year would involve collaborative participatory design between partners, communities and FFP, the second additional years being evaluation post-project of sustained and emerging impact!

This is a sea shift that can hopefully withstand political winds. After all, US foreign aid accounts for less than 1% of our federal budget, even though many Americans believe it is over 15% (hence easy to cut)…. but fingers crossed the aid effectiveness value of our work is… Valued.

 

 

Sources:

[1] Cekan, J., PhD, Kagendo, R., & Towns, A. (2016). Participation by All: The Keys to Sustainability of a CRS Food Security Project in Niger. Retrieved from https://www.crs.org/our-work-overseas/research-publications/participation-all

[2] Lindley-Jones, H. (2016, November 16). ‘If we don’t do it, who will?’ A study into the sustainability of Community Protection Structures supported by Oxfam in the Democratic Republic of Congo (DRC). Retrieved from https://policy-practice.oxfam.org.uk/publications/if-we-dont-do-it-who-will-a-study-into-the-sustainability-of-community-protecti-620149

[3] USAID Learning Lab. (n.d.). Collaborating, Learning, and Adapting (CLA)? Retrieved from https://usaidlearninglab.org/faq/collaborating%2C-learning%2C-and-adapting-cla

[4] Food and Nutrition Technical Assistance (FANTA). (n.d.). Effective Sustainability and Exit Strategies for USAID FFP Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp

[5] Del Mese, F. (2016, November 16). When aid relationships change: DFID’s approach to managing exit and transition in its development partnerships. Retrieved from https://icai.independent.gov.uk/report/transition/

[6] Cekan, J., Rogers, B. L., Rogers, P., & Zivetz, L. (2016, October 26). Barking Up a Better Tree: Lessons about SEIE (Sustained and Emerging Impact Evaluation). Retrieved from https://valuingvoices.com/wp-content/uploads/2016/11/Barking-up-a-Better-Tree-AEA-Oct-26-FINAL.pdf

[7] USAID. (2016, October 6). 2016–2025 Food Assistance and Food Security Strategy. Retrieved from https://www.usaid.gov/ffpstrategy#:~:text=FFP’s%20new%20strategy%2C%20the%202016,USG)%20food%20assistance%20as%20a

[8] Rogers, B. L., & Coates, J. (2015, December). Sustaining Development: A Synthesis of Results from a Four-Country Study of Sustainability and Exit Strategies among Development Food Assistance Projects. Retrieved from https://www.fantaproject.org/research/exit-strategies-ffp

 

Maximizing what we’ve got… Time is now!

Maximizing what we've got… Time is now!

 

We had a stirring conversation here in D.C. with someone very knowledgeable about sustainability; this person is a strong proponent of local ownership of all development. They also said vehemently, why evaluate the sustainability of projects after closeout; we all know what that will show!  What was implied is that our system of international development and aid is so flawed, so broken, that the inevitable result of not focusing on local ownership as the fundamental basis for our work means Nothing. Will. Be. Left…. All. Is. Lost. 

 

We disagree. Our development industry does some good, some bad, and is ever-changing (albeit slowly). Billions of dollars each year are spent trying to improve people's lives and livelihoods around the world, and we've seen great good be done. While. We. Remain.

 

We know far, far less about what remains after our projects end because less than 1% of the time we return post-project (ex-post) to evaluate anything.

 

Our problem with time begins with fixed timelines within projects that say we have 1, 3, 5 years to get to success. They work with participants and partners who need to make substantial changes to how they use their resources and beliefs over a relatively short time of a few months to a few years. We expect immediate results from them, changing how they farm (use new seeds, new methods, new ways of interacting with markets) and save money (learn new concepts of profit and interest, repayment and re-lending), and improve their health and that of their families (get prenatal exams, vaccinate your children, exclusively breastfeed without adding water or tea). Everyone in our projects is 'on the clock' from the donor and implementer to partners and participants. This clock ticks down irrevocably as project closeout looms, promised-successes-to-donors at hand or a mirage in the distance. We assume sustained results.

 

How many of us have ever gone on a diet? How many have learned a new language? How many of us have transferred jobs and had to learn new skills on the job? How quickly have we managed to do all that successfully, all at once?!  Probably many. How many of you have had to do this on a fixed timeline? Were you successful when there was a limited, fixed time and you did not set your own pace?

Timeline_Wylio7739861570_ef1a5c745f_m

https://www.flickr.com/photos/psd/7739861570

 

It takes time to implement projects well enough to ensure that most participants ‘got it’, not just the 'early adopters'. It takes time to hand over projects so well that our partners and participants are ready to take over at least some of what we worked so hard to transfer. It took leadership and staff two years in the very successful participatory USAID/ Food For Peace food security project by CRS Niger that was a continuation of similar programming for 15 years.  It also takes time to pass for conditions to be ready for our return, to isolate what people could self-sustain from what the project supplied, to learn what was so well designed and implemented during projects that to 'took root' in people's lives, that they have made it their own.  We estimate optimal evaluation time is 2-7 years after closeout. Valuing Voices also believes we should not just evaluate the sustainability of outputs and outcomes of what we put in place that we thought they would continue, and the sustained impact of those cumulative investments, but also the emerging, unintended new activities and impacts we never imagined people would innovate from our projects.  We are doing just such evaluations in Zimbabwe and Uganda now and hope to do and catalyze much more fieldwork around the world.

 

And why does it matter? Why shouldn't we write off our time-limited donor-funded projects? Because:

 

1) It's all we've got. Our current development system is not going anywhere soon, and there is success to learn from.

 

2) We need to quickly learn from what worked sustainably best and stop wasting time and resources on what we refuse to admit fails because we are too scared to return to see. Go back with the intention to learn what does and focus on doing more of what works.

 

3) Such analysis – and design of new projects – must have country ownership as a centerpiece throughout the project cycle assumptions, but to throw out decades of good work simply because we are just learning the value of country ownership is foolish.

 

Finally, here's a lovely example from Brazil of how local, participation (and yes, as my colleague thought, local ownership) works best. And. It. Takes. Time.

 

"Our results also show that Participatory Budgeting’s influence strengthens over time… Participatory Budgeting’s increasing impact indicates that governments, citizens, and civil society organizations are building new institutions… cities incorporate citizens at multiple moments of the policy process, allowing community leaders and public officials to exchange better information."  How often do we return to do what are called longitudinal reviews of our work abroad, using the same rigorous standards we evaluate our domestic projects? Not often. Shouldn't that change?

 

Only by working together, honoring the value of our participants, that they deserve the same chance at change that we take for granted will things change. We must value both the voices of our participants and our own expertise for development to improve for true aid effectiveness…. Let us begin anew!

 

Who is responsible for sustaining development?

Whose responsibility is it to sustain project activities?

Billions of dollars are pumped into development activities in developing countries all over the world. Communities getting involved in these projects have a clear objective, which is to have their lives improved in the sectors that the projects target. As to whether this is the main objective of the development partners is not clear. What is clear is that the development partners focus more on numbers than on getting people to participate.

 

We note that majority of these projects are designed to last between 2-5 years. Delays occasioned poor planning or other unforeseen factors eat into the implementation time to an extent that in some projects it takes about 1-2 years to get a program running. This means that the planned implementation time is reduced.  Baselines, midlines and end lines studies are conducted to inform changes that may have occurred within the program life, and in most cases they happen shortly after the program has started or just before it ends. In fact, some baselines are conducted after programs have started.

Considering the reduced implementation time and the fact that it takes a much longer time to get concrete behavior change related results, questions emerge whether indeed the reported changes are solid enough during implementation to be sustained. There is also a difference between measuring what can be referred to artificial changes (activities that community members adopt as a way of short-term trial in their excitement, but don't find useful afterward) and long lasting changes that community members adopt because they are useful part and parcel of their lives.

 

Almost all projects have logical frameworks (logframes) that show how project activities will be implemented and to some extent there are also exit strategies for closing out the project. This can be an illusion long-term. In most cases donors and implementers assume that communities will adopt activities that are being implemented within a specified period of time, and so projects close down at the end of the specified period of implementation assuming things will continue, but have no proof. Valuing Voices has done projected sustainability work in Ethiopia which points to possible differences between what donors expect to the sustained versus what communities are able to sustain.

 

Communitywomansorting

 

The big questions remain: "whose responsibility is it to ensure that whatever has been adopted is continued? Whose responsibility is it to sustain project activities post project implementation?" It is silently assumed that communities can take up this responsibility and a key question is "what guarantees are there that this is possible and is happening?" Project sustainability should not be seen as a community-alone responsibility but rather a responsibility for all those who are involved in program activities.  Sustainability studies should be planned for and executed in the same breath that the baselines, midlines, end-lines and in the rare cases impact assessments in real time should be planned for. We must do sustainability studies as they provide an additional realistic opportunity to inform us on actual community development post project implementation. Communities should not be left alone with it.

 

The Disruptive Potential of Feedback (reblog from NonprofitChronicles.com)

The Disruptive Potential of Feedback

by MARC GUNTHER, OCTOBER 18, 2015 ( reblogged http://nonprofitchronicles.com/2015/10/18/the-disruptive-potential-of-feedback/)

Few institutions in the US are as undemocratic as endowed foundations. The executives in charge of foundations answer to, er, no one. They give money away, so people tend to laugh at their jokes, tell them they look well, nod in agreement at their banal remarks. What’s not to like?

As for nonprofits, they pay heed to foundations and donors, but they need not listen to their “beneficiaries,” unless they feel a moral obligation to do so. What if, goodness knows, the people they are trying to serve turn out to be unhappy with the service? Talk about inconvenient truths.

fbl-final-logoLast week in Washington, a group of about 70 people — the generals and foot soldiers of a growing movement to devolve power to mostly poor recipients of aid in the US and abroad — came together to talk about how to turn that power dynamic of philanthropy upside down. They believe that feedback from constituents “has the potential to unleash massive, timely and necessary changes in the way social change and development are pursued,” in the words of Feedback Labs, a DC-based NGO that convened the firstFeedback Summit.

As Dennis Whittle, the executive director of Feedback Labs, has written:

Will aid and philanthropy democratize themselves? Will aid agencies and foundations cede power and sovereignty to the people they are trying to serve?

It’s too soon to say but there were signs during the two-day confab that a half-dozen or so forward-thinking foundations, along with a growing number of nonprofits, are starting to figure how to create tight feedback loops that will enable them to solicit feedback from citizens, listen, analyze and, most important, change their practices as a result of what they learn.

“It’s the right thing to do, morally and ethically, philosophically. It’s the smart thing to do,” Whittle said. Now the goal is to make it “the feasible thing to do, financially and operationally.”

How do feedback loops differ from conventional monitoring and evaluation (M&E)? One attendee told me that feedback loops are the equivalent of diagnosing and treating a disease; a conventional evaluation is more like an autopsy, and thus of limited value to the patient.

Here are three signs that the feedback movement is gathering momentum:

A group called the Fund for Shared Insight, a collaboration of foundations that makes grants to improve philanthropy, has launched an initiative called Listen for Good that intends to fund 50 nonprofits to seek feedback from the people they are trying to help. They will use the now-famous Net Promoter System methodology developed for business by Bain & Co., which is working with the fund to make sure that the simple, elegant and yet rigorous system is deployed effectively. “It needs to be a high velocity loop of feedback, learning and action,” said Vikki Tam, a Bain partner. To the extent possible, the feedback results will be made public, enabling nonprofits to compare their net promoter scores with peers. “It’s so important for foundations to be open about what they do, and what they’re learning,” said Lindsay Austin Louie, a program officer at William and Flora Hewlett Foundation who works closely with the fund. Supporters of the fund include the David and Lucile Packard Foundation, the Ford Foundation, the Gordon and Betty Moore Foundation, the JPB Foundation, Liquidnet (which I wrote about here), the Rita Allen Foundation and the W.K. Kellogg Foundation.

FeedBackLoopBanner-300x300Efforts to build “good enough” feedback loops are underway, aimed at helping small or midsize NGOs measure their impact without having to undertake expensive, long-term randomized control trials.  Thoai Ngo, a senior director of research at Innovations for Poverty Action, talked about an effort called The Goldilocks Project that aims to build “right-fit” evaluation systems, focusing on collecting credible, actionable data in a timely way–that is, feedback to help an NGO change course if needed. Ken Berger, the former chief executive at Charity Navigator, described his work at a firm called Algorhythm which offers impact measurement to small NGOs for as little as $750. “These are organizations that never before had an opportunity to measure what matters most,” Berger said.

Technology is making it much easier to gather feedback, and make sense of it. Louis Dorval is the co-founder of VOTO Mobile, a Ghana-based tech startup that aims to “amplify the voice of the under-heard” by using voice and text messages on mobile devices to survey citizens, as well as send one-way messages. Less than three years old, Voto Mobile has already worked with about 250 organizations, including Unicef and Innovations for Poverty Action. David Bonbright of Keystone Accountability, who is a pioneer in the feedback arena, talked about Feedback Commons, an online platform designed to allow  organizations to “share and compare” the feedback they collect from their constituents.

SUPPLY AND DEMAND

All these initiatives are designed to improve the development of feedback loops. Before long, NGOs should be able to show how collecting feedback generates better outcomes for their clients. That’s all to the good. Think of that as the supply side of the feedback “business.”

But what about the demand side? Who’s going to fund feedback loops? I’m still a newcomer to the development world but my impression (from reporting on water and sanitation projects, and on cookstoves) is that most foundations and NGOs are not as rigorous as they could and should be about measuring their impact.

This brings us back to the fundamental power dynamic of philanthropy, as Caroline Fiennes of Giving Evidence explained during the Feedback Summit.

Why don’t foundations ask for feedback?” Fiennes asked. “Because they don’t have to.” 

“It’s difficult and it’s a bit painful,” she went on. “If you have made a chunk of money and you want to give it away, in general you will feel good about that, and everybody will love you. Once you start asking questions” — questions designed to find out if the work is making as much of a difference as it can–“you might not like the answers.”

In an essay called What do they want? at Aeon, Claire Melamed elaborates:

While most individual aid workers do care, very much, about the people they work with and for, the actual structure of the aid business offers few reasons for anyone to worry about what aid recipients think or want. Staff in aid agencies need to think about what their funders want to pay for. For their own performance reviews, they need to think about how to demonstrate that what they are doing is achieving the best possible results with the smallest amount of money. So the incentives for spending money on expensive surveys to find out what representative samples of poor people think of their operations are just not there.

And besides, the information might be a threat. What if it turned out that people feel patronised by aid workers? Or that they would rather their food didn’t arrive with logos announcing their indebtedness to foreign governments? Or that they resent being given a T-shirt when really they would sooner just have the money? What if people don’t really want another agricultural programme, and they’d rather have a bus ticket to the nearest town and somewhere to stay when they get there? These kinds of discoveries could be quite discomfiting for the agencies themselves – though in the long run, they would presumably do a better job.”

This is why those funders (like the foundations behind the Fund for Shared Insight) who push for feedback loops and rigorous evaluation deserve a lot of credit. Let’s hope their numbers grow.

 

Face our fears! Learn from failure…

Face our fears! Learn from failure…

Global Giving has a nice example of getting participant feedback success/failure of a project their fundraising funded. The organization failed, which was sorrowful to the players in West Africa and funders worldwide (I, too am a soccer mom). Yes, it failed. It is hard to read those words, and all involved faced it, and they learned from it. Thanks to a spirited discussion on Pelican Platform for Evidence-based Learning about a plethora of unintended impacts, I learned about the organization committed to learning from failures: Admitting Failure.

 

Maybe some of you have heard of FailFests, such as the one in Raleigh N.C. US this year that answered, “Why are we doing this?” with “We’re on a mission to erase the stigma around failure. The more we talk about it, the more we can learn from it. Failure doesn’t have to be fatal!” You may know Engineers without Borders Canada has been an early proponent of “openly acknowledging failure is often a catalyst for innovation that takes our work from good to great.”  Such examples are heartening, especially for participatory sustainability evaluation.

Admitting_Failure___Engineers_Without_Borders_Canada

In 2014 a met a fellow evaluator at the American Evaluation Association Conference, where I was presenting on Post-Project Sustainability Evaluation. They told me their organization was waiting for a really successful project to end so they could evaluate its sustainability. Another evaluator told me of a post-project evaluation that was hidden away, unpublished, after the findings were negative. And that’s where the problem lies: our international development industry’s neurosis about presenting anything unsuccessful.

 

No wonder so few projects have been evaluated post-project, as we fear:

* What if activities and outcomes aren’t sustained (note: we didn’t design them that way, often, we opt for quicker wins achievable only with large resource inflows of funds and technical help)?

 

* What if our funders find out that funds had limited impact or partners had no means to continue good programming?

 

* What if we had entirely unintended impacts that favored some over others (well beyond our expected logical frameworks and Theories of Change)?

 

Hallelujah 🙂 Why can’t we see this as good news that propels us to improve our projects by listening to what worked? To learn not to do what didn’t work again?! To design in locally sustainable ways now that we have learned?

* What if we learn that our resources and empowerment led them to succeed on their own terms in ways we couldn’t imagine, far exceeding the planned impacts we had expected?

 

* What if we find that unexpected outcomes showcased ways groups within communities stood on their feet, making development work for them on their terms?

 

Would we design and implement, monitor, and evaluate projects differently? We’ll see. Organizations such as USAID’s Food for Peace funded a four-country study on exit strategies (forthcoming 2015) and are looking at success and failure. Catholic Relief Services hired Valuing Voices to do a sustainability evaluation in Africa. Others may follow…

 

Post-project sustainability evaluations expose us to learning the full range. Let’s be brave and ask, learn and innovate, making aid and philanthropy more effective, and learn from failure for success!