The Disruptive Potential of Feedback (reblog from NonprofitChronicles.com)

The Disruptive Potential of Feedback

by MARC GUNTHER, OCTOBER 18, 2015 ( reblogged http://nonprofitchronicles.com/2015/10/18/the-disruptive-potential-of-feedback/)

Few institutions in the US are as undemocratic as endowed foundations. The executives in charge of foundations answer to, er, no one. They give money away, so people tend to laugh at their jokes, tell them they look well, nod in agreement at their banal remarks. What’s not to like?

As for nonprofits, they pay heed to foundations and donors, but they need not listen to their “beneficiaries,” unless they feel a moral obligation to do so. What if, goodness knows, the people they are trying to serve turn out to be unhappy with the service? Talk about inconvenient truths.

fbl-final-logoLast week in Washington, a group of about 70 people — the generals and foot soldiers of a growing movement to devolve power to mostly poor recipients of aid in the US and abroad — came together to talk about how to turn that power dynamic of philanthropy upside down. They believe that feedback from constituents “has the potential to unleash massive, timely and necessary changes in the way social change and development are pursued,” in the words of Feedback Labs, a DC-based NGO that convened the firstFeedback Summit.

As Dennis Whittle, the executive director of Feedback Labs, has written:

Will aid and philanthropy democratize themselves? Will aid agencies and foundations cede power and sovereignty to the people they are trying to serve?

It’s too soon to say but there were signs during the two-day confab that a half-dozen or so forward-thinking foundations, along with a growing number of nonprofits, are starting to figure how to create tight feedback loops that will enable them to solicit feedback from citizens, listen, analyze and, most important, change their practices as a result of what they learn.

“It’s the right thing to do, morally and ethically, philosophically. It’s the smart thing to do,” Whittle said. Now the goal is to make it “the feasible thing to do, financially and operationally.”

How do feedback loops differ from conventional monitoring and evaluation (M&E)? One attendee told me that feedback loops are the equivalent of diagnosing and treating a disease; a conventional evaluation is more like an autopsy, and thus of limited value to the patient.

Here are three signs that the feedback movement is gathering momentum:

A group called the Fund for Shared Insight, a collaboration of foundations that makes grants to improve philanthropy, has launched an initiative called Listen for Good that intends to fund 50 nonprofits to seek feedback from the people they are trying to help. They will use the now-famous Net Promoter System methodology developed for business by Bain & Co., which is working with the fund to make sure that the simple, elegant and yet rigorous system is deployed effectively. “It needs to be a high velocity loop of feedback, learning and action,” said Vikki Tam, a Bain partner. To the extent possible, the feedback results will be made public, enabling nonprofits to compare their net promoter scores with peers. “It’s so important for foundations to be open about what they do, and what they’re learning,” said Lindsay Austin Louie, a program officer at William and Flora Hewlett Foundation who works closely with the fund. Supporters of the fund include the David and Lucile Packard Foundation, the Ford Foundation, the Gordon and Betty Moore Foundation, the JPB Foundation, Liquidnet (which I wrote about here), the Rita Allen Foundation and the W.K. Kellogg Foundation.

FeedBackLoopBanner-300x300Efforts to build “good enough” feedback loops are underway, aimed at helping small or midsize NGOs measure their impact without having to undertake expensive, long-term randomized control trials.  Thoai Ngo, a senior director of research at Innovations for Poverty Action, talked about an effort called The Goldilocks Project that aims to build “right-fit” evaluation systems, focusing on collecting credible, actionable data in a timely way–that is, feedback to help an NGO change course if needed. Ken Berger, the former chief executive at Charity Navigator, described his work at a firm called Algorhythm which offers impact measurement to small NGOs for as little as $750. “These are organizations that never before had an opportunity to measure what matters most,” Berger said.

Technology is making it much easier to gather feedback, and make sense of it. Louis Dorval is the co-founder of VOTO Mobile, a Ghana-based tech startup that aims to “amplify the voice of the under-heard” by using voice and text messages on mobile devices to survey citizens, as well as send one-way messages. Less than three years old, Voto Mobile has already worked with about 250 organizations, including Unicef and Innovations for Poverty Action. David Bonbright of Keystone Accountability, who is a pioneer in the feedback arena, talked about Feedback Commons, an online platform designed to allow  organizations to “share and compare” the feedback they collect from their constituents.

SUPPLY AND DEMAND

All these initiatives are designed to improve the development of feedback loops. Before long, NGOs should be able to show how collecting feedback generates better outcomes for their clients. That’s all to the good. Think of that as the supply side of the feedback “business.”

But what about the demand side? Who’s going to fund feedback loops? I’m still a newcomer to the development world but my impression (from reporting on water and sanitation projects, and on cookstoves) is that most foundations and NGOs are not as rigorous as they could and should be about measuring their impact.

This brings us back to the fundamental power dynamic of philanthropy, as Caroline Fiennes of Giving Evidence explained during the Feedback Summit.

“Why don’t foundations ask for feedback?” Fiennes asked. “Because they don’t have to.” 

“It’s difficult and it’s a bit painful,” she went on. “If you have made a chunk of money and you want to give it away, in general you will feel good about that, and everybody will love you. Once you start asking questions” — questions designed to find out if the work is making as much of a difference as it can–“you might not like the answers.”

In an essay called What do they want? at Aeon, Claire Melamed elaborates:

While most individual aid workers do care, very much, about the people they work with and for, the actual structure of the aid business offers few reasons for anyone to worry about what aid recipients think or want. Staff in aid agencies need to think about what their funders want to pay for. For their own performance reviews, they need to think about how to demonstrate that what they are doing is achieving the best possible results with the smallest amount of money. So the incentives for spending money on expensive surveys to find out what representative samples of poor people think of their operations are just not there.

And besides, the information might be a threat. What if it turned out that people feel patronised by aid workers? Or that they would rather their food didn’t arrive with logos announcing their indebtedness to foreign governments? Or that they resent being given a T-shirt when really they would sooner just have the money? What if people don’t really want another agricultural programme, and they’d rather have a bus ticket to the nearest town and somewhere to stay when they get there? These kinds of discoveries could be quite discomfiting for the agencies themselves – though in the long run, they would presumably do a better job.”

This is why those funders (like the foundations behind the Fund for Shared Insight) who push for feedback loops and rigorous evaluation deserve a lot of credit. Let’s hope their numbers grow.

 

Sustainable Development Goals and Foreign Aid– How Sustainable and Accountable to Whom?  Reposting Blog from LinkedIn Pulse

 

Sustainable Development Goals and Foreign Aid–
How Sustainable and Accountable to Whom?

 

Jindra Cekan, PhD of ValuingVoices

World leaders will paint New York City red next week at the UN Summit adopting the new post-2015 development agenda.  The agreed plans set 17 new ‘Sustainable’ Development Goals (SDGs) to be achieved by 2030. These are successors to the Millennium Development Goals (MDGs).

While many of us have heard of them, how many of us know whether we met them and what prospects are there for the Sustainable Development Goals to do well or better?  According to Bill Gates the MDGs were “the best idea for focusing the world on fighting global poverty that I have ever seen.” The Brookings Institute goes on to praise the eight MDG targets for aiming high by setting targets such as halving world poverty and reducing child mortality, improving universal primary education and gender equality and empower women etc. [1]. Donor countries pledged three times more than they had until that time (raising the percentage of gross national income for international development assistance from 0.2% to 0.7%, not huge amounts but laudable).  From 2000 to 2015 extreme poverty did fall by half (although some argue China and Asia were well on their way before this aid came) and in some countries (Senegal, Cambodia) child deaths fell by half.  Global health improved via huge coalitions on immunization and HIV/AIDS. Yet while poverty dropped and health increased, hunger, environment and sanitation targets were not met, for instance, and there are 850 million people still hungry worldwide (11% of all people) [2]. Yet gains far outweigh losses.  The new Sustainable Development Goals are to be achieved in 15 years, by 2030. The UN and member nations will track a remarkable 169 indicators, “monitoring progress towards the SDGs at the local, national, regional, and global levels… [3]

 

BusinessWorld___INFOGRAPHIC__MDGs__It’s_2015__Now_what_

 

Overall, one would feel rather tickled by these results — not 100% but still amazing given global disparities. Nancy Birdsall of the Center for Global Development thinks measurement is not the goal: “Growing global interconnectedness means that the problems the world faces, that hold back development, are increasingly shared… we’re making a promise to ourselves that we are one world, one planet, one society, one people, who look out for each other…” [4]

But I’m a fan of measurable results, I must admit. One would logically think that our international development projects funded by U.S. Agency for International Development (USAID), the U.S. Department of Agriculture (USDS), the Millennium Challenge Corporation and others could outline how the caused the good results that some MDGs showed. USAID’s website links its work to the MDGs clearly: “in September 2010, President Obama called for the elevation of development as a key pillar of America’s national security and foreign policy. This set forth a vision of an empowered and robust U.S. Agency for International Development that could lead the world in solving the greatest development challenges of our time and, ultimately, meet the goal of ending extreme poverty in the next generation” [5]. It goes on to talk about work to “Promote sustainable development through high-impact partnerships and local solutions“ [5]. USDA’s Foreign Agricultural Service states their “non-emergency food aid programs help meet recipients’ nutritional needs and also support agricultural development and education. These food assistance programs, combined with trade capacity building efforts, support long-term economic development” [6]. Finally, MCC states they are “committed to delivering results throughout the entire lifecycle of its investments. From before investments begin to their completion and beyond… MCC’s continuum of results is designed to foster learning and accountability” [7].

Maybe. We don’t actually know because 99% of the time we never return to projects after they end to learn how sustainable they actually were. We could be fostering super-sustainability. Or not.

For international development programming works on 1-5 year programming cycles. Multi-million dollar project requests for proposals are designed and sent out by these funders to non-profit or for-profit implementers. These are awarded to one or more organization, quite rigorously monitored, and most have very good results. Then they end. Since 2000, the US Agency for International Development has spent $280 billion on country-to-country development and humanitarian aid projects as well as funding multilateral aid and in spite of much work evaluating the final impact of projects at the end, they never go back [8]. The EU has spent a staggering $1.4 trillion. USAID has funded one evaluation that has gone back to see what communities and partners could sustain… in the last 30 years, and that is about to be published. A handful of international non-profits have taken matters in their own hands and funded such studies privately. The EU’s track record is even more dismal, with policies being proposed but not done [10]. The World Bank, which has funded over 12,000 projects has an independent evaluation arm, the Independent Evaluation Group. They returned after projects closed out to evaluate results only 33 times and we found only three of them systematically talked to project participants about what was sustained.

The bottom line is, how do we know anything we’ve done in international development or SDGs is sustainable unless we go back to see? What amazing or awful results must we know for future design? If we do not return, are we really accountable to our taxpayers and our real clients: the participants and the national countries foreign aid recipients themselves?

The UN has pledged to have an SGD report card to “measure progress towards sustainable development and help ensure the accountability of all stakeholders for achieving the SDGs….[and a] Global Partnership for Sustainable Development Data, to help drive the Data Revolution….by using data we can ensure accountability for the policies that are implemented to reduce global and local inequities” [3]. I completely agree that having citizen generated data at the local, national, regional, and global levels is so very important “to fill gaps in our knowledge, establish global norms and standards and…help countries develop robust national strategies for data development.”  And as the World Bank IEG’s Caroline Heider states, measuring them is complex (e.g. agriculture affected by climate change and measuring changes across sectors is hard) but worthwhile [11].

While SDG data tells us what donor-funded activities and policies work, very few in international development know how sustainably our programming works for our ultimate clients, our participants and partners.  And the price needn’t be high—a recent post-project evaluation we did cost under $120,000 which is a pittance given the project cost over $30 million and reached 500,000 folks.  We found clear (mostly successful) lessons. USAID has, after 30 years, funded one post-project evaluation that also has clear cost-effective lessons (forthcoming). Really, in this era of cost-effectiveness, don’t we want data on what worked best (Note to self: do that more) and what worked least (Note to self: stop doing that)?

Learning what participants and partners could self-sustain after we left is actually all we should care about. They want to get beyond aid. Shouldn’t we know if we are getting them there? Self-sustainability of outcomes is a clear indicator of good Return on Investment of our resources and expertise and their time, effort and expertise. It shows us we want to put ourselves out of a job, having built country-led development that really has a future in-country with their own resources.

Two steps are:

1) Donors to add a funding equivalent to 1% of program value for five years after closeout for all projects over $10 million to support local capacity-building of NGOs and national partners to take over implementation plus to evaluate lessons across different sector’s sustainability outcomes.

2) A cross-donor fund for country-led analysis of such learning plus lessons for what capacity needs to be built in-country to take over programming. This needs support from regionally-based knowledge repositories and learning centers in Africa, Asia, etc. Online and tangible centers could house both implementer reports/ evaluations and analyze/ share lessons learned across sectors and countries from post-project evaluations for projects that closed out 2-7 years ago for future design.

Now that is accountability. Let’s advocate for sustainability funding, data and learning now.

What are your suggestions? How can we improve sustainability?

 

 

Sources:

[1] McArthur, J. (2013, February 21). Own the Goals: What the Millennium Development Goals Have Accomplished. Retrieved from https://www.brookings.edu/articles/own-the-goals-what-the-millennium-development-goals-have-accomplished/

[2] End Poverty 2015 Millennium Campaign. (n.d.). MDG 1: Eradicate Extreme Poverty and Hunger. Retrieved from https://www.endpoverty2015.org/mdg-success-stories/mdg-1-end-hunger/

[3] Sharma, S. (2015, August 20). From Aspirations to Reality: How to Effectively Measure the Sustainable Development Goals. Retrieved from https://www.huffpost.com/entry/measuring-the-peoples-age_b_7999640

[4] Mirchandani, R. (2015, September 22). Does It Matter If We Don’t Achieve the SDGs? A New Podcast with Nancy Birdsall and Michael Elliott. Retrieved from https://www.cgdev.org/blog/does-it-matter-if-we-dont-achieve-sdgs-podcast-nancy-birdsall-and-michael-elliott

[5] USAID. (n.d.). USAID Forward. Retrieved from https://www.usaid.gov/usaidforward/usaid-forward-2014-archive

[6] United States Department of Agriculture (USDA). (n.d.). Food Assistance. Retrieved from https://www.fas.usda.gov/topics/food-assistance

[7] Millennium Challenge Corporation. (n.d.). Our Impact. Retrieved 2016, from https://web.archive.org/web/20160325135258/https://www.mcc.gov/our-impact

[8] Cekan, J. (2015, March 13). When Funders Move On. Retrieved from https://ssir.org/articles/entry/when_funders_move_on

[9] Global Issues. (n.d.). Foreign Aid for Development Assistance: Foreign Aid Numbers in Charts and Graphs. Retrieved from https://www.globalissues.org/print/article/35#globalissues-org

[10] Florio, M. (2009, November/December). Sixth European Conference on Evaluation of Cohesion Policy: Getting Incentives Right — Do We Need Ex Post CBA? Retrieved from https://ec.europa.eu/regional_policy/archive/conferences/evaluation2009/abstracts/florio.doc

[11] Heider, C. (2015, September 15). Evaluation Beyond 2015: Implications of the SDGs for Evaluation. Retrieved from https://ieg.worldbankgroup.org/blog/evaluation-beyond-2015-implications-sdgs-evaluation

 

Face our fears! Learn from failure…

Face our fears! Learn from failure…

Global Giving has a nice example of getting participant feedback success/failure of a project their fundraising funded. The organization failed, which was sorrowful to the players in West Africa and funders worldwide (I, too am a soccer mom). Yes, it failed. It is hard to read those words, and all involved faced it, and they learned from it. Thanks to a spirited discussion on Pelican Platform for Evidence-based Learning about a plethora of unintended impacts, I learned about the organization committed to learning from failures: Admitting Failure.

 

Maybe some of you have heard of FailFests, such as the one in Raleigh N.C. US this year that answered, “Why are we doing this?” with “We’re on a mission to erase the stigma around failure. The more we talk about it, the more we can learn from it. Failure doesn’t have to be fatal!” You may know Engineers without Borders Canada has been an early proponent of “openly acknowledging failure is often a catalyst for innovation that takes our work from good to great.”  Such examples are heartening, especially for participatory sustainability evaluation.

Admitting_Failure___Engineers_Without_Borders_Canada

In 2014 a met a fellow evaluator at the American Evaluation Association Conference, where I was presenting on Post-Project Sustainability Evaluation. They told me their organization was waiting for a really successful project to end so they could evaluate its sustainability. Another evaluator told me of a post-project evaluation that was hidden away, unpublished, after the findings were negative. And that’s where the problem lies: our international development industry’s neurosis about presenting anything unsuccessful.

 

No wonder so few projects have been evaluated post-project, as we fear:

* What if activities and outcomes aren’t sustained (note: we didn’t design them that way, often, we opt for quicker wins achievable only with large resource inflows of funds and technical help)?

 

* What if our funders find out that funds had limited impact or partners had no means to continue good programming?

 

* What if we had entirely unintended impacts that favored some over others (well beyond our expected logical frameworks and Theories of Change)?

 

Hallelujah 🙂 Why can’t we see this as good news that propels us to improve our projects by listening to what worked? To learn not to do what didn’t work again?! To design in locally sustainable ways now that we have learned?

* What if we learn that our resources and empowerment led them to succeed on their own terms in ways we couldn’t imagine, far exceeding the planned impacts we had expected?

 

* What if we find that unexpected outcomes showcased ways groups within communities stood on their feet, making development work for them on their terms?

 

Would we design and implement, monitor, and evaluate projects differently? We’ll see. Organizations such as USAID’s Food for Peace funded a four-country study on exit strategies (forthcoming 2015) and are looking at success and failure. Catholic Relief Services hired Valuing Voices to do a sustainability evaluation in Africa. Others may follow…

 

Post-project sustainability evaluations expose us to learning the full range. Let’s be brave and ask, learn and innovate, making aid and philanthropy more effective, and learn from failure for success!

Altruistic Accountability… for Sustainability

Altruistic Accountability… for Sustainability

Many of us in international development feel a sense of responsibility for others to be well, and for our work to improve their lives as well as for the work to be done in good stewardship of aid resources and optimizing their impact. As Matthieu Ricard writes, "Altruism is a benevolent state of mind. To be altruistic is to be concerned about the fate of all those around us and to wish them well. This should be done together with the determination to act for their benefit. Valuing others is the main state of mind that leads to altruism."  We also feel a responsibility to our international aid donors and taxpayers. We who implement, monitor and evaluate projects work to ensure that the altruism of aid is responsible to both donors and recipients.

Altruism appears most vividly when implementers issue appeals after disasters, with millions donated as a result, but unsung heroes are also development workers. Organizations such as Charity Navigator, ONE and Center for Global Development on how well US organizations spent funds and track donor-country policy accountability. Thoughtful donor studies such as French Development Agency’s OECD study report on the power of AidWatch and Reality of Aid intiatives in Europe for their taxpayers.

But who is pushing for our donor’s accountability to the country’s participants themselves? While USAID funds many program evaluations, some of which “identify promising, effective…strategies and to conduct needs assessments and other research to guide program decisions”, they are always at project end, rather than looking at sustainability of the outcomes and impacts, and focus on Congressional and domestic listeners. This is no funding and no small audience. The US Department of State/ USAID’s FY13 Summary report states that in fiscal year 2013, USAID had $23.8 billion to disburse, over $12 billion for programming. While total beneficiary (participant) numbers were not provided, emergency food assistance alone used $981 million for nearly 21.6 million people in 25 countries.

So who is a watchdog for what results? OXFAM may excellently highlight opportunities for better programming. 3ie does many studies looking at projected impact and does systemic reviews (but only three were post-project). Challenges such as Making All Voices Count may fund channels for country-nationals to hold their own governments responsible, but can in-country project participants ever demand sustainable results from anyone but their own governments? Herein lies the crux of the issue. Unless governments demand it (unlikely in ‘free’ aid), only pressure from donor country nationals (you? we) can push for changes.

globe

At the core of Valuing Voices mission is advocacy for altruistic accountability of the sustainability of projects to country-ownership at all levels. For us, this involves valuing and also giving voice to those supported by, and also tasked with doing ‘sustainable projects’. Unless we know how sustainable our development projects have been, we have only temporarily helped those in greatest need. This means looking beyond whether funding continued to whether the benefits of an activity or even the existence of entire local NGOs tasked with this actually continued after funding was withdrawn. Unless we strive to learn what has continued to work best or failed to be continued after projects left in the views of the participants themselves, we can let down the very people who have entrusted us with hopes of a self-sustainable future of well being. Unless we listen to project staff and local partners to see what program staff felt they did right/wrong, and national partners felt they were supported to do keep doing right, we have minimized success of future projects. While increasing numbers of organizations such as Hewlett Foundation fund work to “increase the responsiveness of governments to their citizens’ needs. We do this by working to make governments more transparent and accountable,” the long-term effectiveness of our donor development assistance is not yet visible.

OECD guidelines on corporate accountability and transparency are illuminating. Adapting it from State-Corporate to Non-profit-State is interesting. For how well have we considered who ‘owns’ these development projects in practical terms from inception onward? Our donors? Implementing agencies? Local partners and communities?

OECD Guidelines on Corporate Governance of State-Owned Enterprises

1: The State Acting as an Owner

2: Equitable Treatment and Relations with Shareholders

3: Ensuring an Effective Legal and Regulatory Framework for State-Owned Enterprises

4: Transparency and Disclosure

How well do we design projects along these lines to do this successfully? Not terrifically:

  • Too often ‘stakeholders’ are not consulted at the very inception of the proposal design, only at design or implementation
  • Too often our work is aimed at making only our ‘client’- our donors- happy with our results rather than the country nationals who are tasked with self-sustaining them.
  • Too often handover is done at the 11th hour, not transferring it throughout implementation or building local capacity for those taking over be true projects’ owners.

But it is coming, through changing societal trends. On the data-access front, USAID (and differently, other European donors) have promised to modernize diplomacy and development by 2017 by “increas[ing] the number and effectiveness of communication and collaboration tools that leverage interactive digital platforms to improve direct engagement with both domestic and foreign publics. This will include increasing the number of publicly available data sets and ensuring that USAID-funded evaluations are published online, expanding publicly available foreign assistance data, increasing the number of repeat users of International Information.” Now to generate and add self- sustainability data to inform future projects!

Second, on the they-are-se front, our basic human nature, according to Ricard, lends itself to altruism. “Let's assume that the majority of us are basically good people who are willing to build a better world. In that case, we can do so together thanks to altruism. If we have more consideration for others, we will promote a more caring economy, and we will promote harmony in society and remedy inequalities.” Let’s get going…

It’s not just Me, it’s We

It's not just Me, it's We

Many of us want to be of service. That's why we go into international development, government, and many other fields. We hope our words and deeds help make others' lives better.

 

For 25 years I've written proposals, designed and evaluated projects, knowing that while I could not live in-country due to my family constraints, I could get resources there and help us learn how well they are used.  I became a consultant so I could raise my kids without being on the road 60% of the time, one who promotes national consultants so that African, Asian, Latin American and European experts evaluate their own projects. I put myself into the shoes of our participants and realized any local person my age wants to leave behind a better, more sustainably viable livelihood for her family, so I looked to see what was most sustained and how we knew it. I took my love of participatory approaches of listening to and learning from the end-users and founded Valuing Voices to promote learning from projects whose activities were most self-sustained. 

 

Yet this is not enough. I am one person with only my views (however great I think they are :), many of us have great views and knowledge about how to best promote sustainable development. For the state of things today seem to me that too often our donors have limited funds for limited time with goals that they limit because they can only assure success by holding the outcome and funding reins so tightly that none of us are fostering self-sustainable development which takes time, faith in one's participants. I have found that the lack of post-project evaluation (see ValuingVoices.com/blogs such as this one on causes and conditions being ripe for sustainability) is a symptom but doing them also provides a huge opportunity to design projects well learning from what communities were able to sustain themselves, based on why/how it worked and how can we do this well again? For instance, from my fieldwork I have realized that questions such as ‘sustainable by whom for how long’ are ones I never asked and don’t think others have ways to go about it well (yet)… unless you have ideas!

How can we foster aid effectiveness, effective philanthropy, community-driven-development, community-driven and NGO-led impact , and effective policy? It takes many of us – giraffes, ostriches, wliderbeast, gazelles, each with our own expertise.

animalhi.com_WE

This takes Time to Listen, respect for local capacities (Doing Development Differently) and an openness to step out of the limelight of 'we saved you' to asking "how can we best work together for a sustainable world?". This takes you, me, WE. One way is to join together in a LinkedIn Group: Sustainable Solutions for Excellent Impact  where we can discuss how can we best design, implement, evaluate, fund, promote (etc!) projects well that are programmatically, financially, institutionally and environmentally sustainable. Please join us!