by Jindra Cekan | Dec 1, 2014 | Beneficiaries, Big data, Evaluation, Evidence-based policy, Foundations, Participants, Philanthropy, Results, Sustainable development
Times are a Changin' in those who Fund Listening… then Doing
So you've been helped by an organization. You think it has a good mission and have actively participated in its activities yet one day (somewhat arbitrarily in your view), it takes you off its list, shuts its doors and moves to another state. What would you feel? Angry? Perplexed? Disappointed?
So a year or two goes by and you get a knock on your door from a similar organization, wanting you to participate with them, that their mission is great, that you will benefit a lot. While you may really want their help, you are understandably wary and wonder if the same will happen. Heck, the last ones didn't tell you why they left, even with unfinished work, nor came back to see how you were faring…
Maybe that won’t happen anymore. Until recently many of our international development participants (some call them beneficiaries) could feel the same way. Our projects came (and went) with set goals, on fixed funding cycles, with little ongoing input from them to influence how projects accomplishes good things, much less learned what happened after projects ended. Rarely have we put into place participant monitoring systems with feedback loops much less listen to participants on how to design for self-sustainability.…
But times are a changin'; there is much to celebrate among funders and implementers, programming and policy makers.
1) There is a happy blizzard of interest in listening to our participants. From Feedback Labs "committed to making governments, NGOs and donors more responsive to the needs of their constituents" and Rita Allen Foundation funding for the Center for Effective Philanthropy's "Hearing from those we seek to Help" to now the Rockefeller and Hewlett Foundation's Effective Philanthropy's beginning a joint Fund for Shared Insight which "provides grants to nonprofit organizations to encourage and incorporate feedback from the people we seek to help; understand the connection between feedback and better results…".
Independent voices abound that are advocating for participants' voices to be heard in design, implementation, monitoring and evaluation: "While we may have a glut of information and even the best of intentions, our initiatives will continue to fall short until we recognize that our ‘beneficiaries’ are really the people who have the solutions that both they and we need." And others call for even more than recognition – participation of the funders in discussions with participants: A recent study by the Center for Effective Philanthropy heard from recipient NGOs that the "funders who best understand our beneficiaries’ needs are the ones who visit us during our programs, meet [those[ served by our organization, spend time talking to them and being with them.”
2) Information and Communication Technologies for Development) has created options of listening to our project participants, learning from them/ with them through mobiles, tablets and other mechanisms (e.g. Catholic Relief Services' ICT4D 6th annual conference with presentations from donors and government as well, Ushahidi which we've celebrated before); IATI, the International Aid Transparency Initiative has spent 6 years fostering sustainable and foreign aid transparent development, now reaching 24 signatory countries, and 290 organizations. A data revolution is taking shape to join donor data, national government statistical data and civil society socio-economic data. There is a brand new initiative at IDS named Doing Development Differently. Listening and learning indeed!
3) And even more importantly, an understanding that development is not a one-size fits all endeavour, is arising. I blogged about Rwanda's success in nutritional impact from allowing communities to address their specific needs and this week New Republic published an excellent article by Michael Hobbes which says "The repeated “success, scale, fail” experience of the last 20 years of development practice suggests something super boring: Development projects thrive or tank according to the specific dynamics of the place in which they’re applied. It’s not that you test something in one place, then scale it up to 50. It’s that you test it in one place, then test it in another, then another." Hobbes goes on to add that what we need is a revision in our expectations of international aid. "The rise of formerly destitute countries into the sweaters-and-smartphones bracket is less a refutation of the impact of development aid than a reality-check of its scale. In 2013, development aid from all the rich countries combined was $134.8 billion, or about $112 per year for each of the world’s 1.2 billion people living on less than $1.25 per day. Did we really expect an extra hundred bucks a year to pull anyone, much less a billion of them, out of poverty?… Even the most wildly successful projects decrease maternal mortality by a few percent here, add an extra year or two of life expectancy there. This isn’t a criticism of the projects themselves. This is how social policy works, in baby steps and trial-and-error and tweaks, not in game changers."
4) What does change the game in the view of Valuing Voices is who we listen to and what we do, for how long. Often, project participants have been the implementers of our solutions rather than the drivers of their own ideas of development; much is lost in translation. As Linda Raftree reports from one Finnish Slush attendee, "“When you think ‘since people are poor, they have nothing, they will really want this thing I’m going to give them,’ you will fail:…“People everywhere already have values, knowledge, relationships, things that they themselves value. This all impacts on what they want and what they are willing to receive. The biggest mistake is assuming that you know what is best, and thinking ‘these people would think like me if I were them.’ That is never the case.” Hallelujah.
Let's listen before we implement our best answers, adapt to specific communities, think of how to foster self-sustainability rather than just successful impact, ask what that is in their terms. Let’s return to listen to participants’ views on sustained impact, on unexpected results… let’s fund this and do development differently!
So how are you listening to participants today?
by valuingvoicesjin | Nov 18, 2014 | Accountability, Africa, Credit, Ethiopia, Evaluation, ex-post evaluation, Food security, impact evaluation, International aid, LIvelihoods, Local Participants, Participation, post-project evaluation, Project design, Rural Development, self-sustainability, Sustainability, Sustainable development
What’s likely to ‘stand’ after we go? A new consideration in project design and evaluation
This spring I had the opportunity to not only evaluate a food security project but also to use the knowledge gleaned for the follow-on project design. This Ethiopian Red Cross (ERCS) project “Building Resilient Community: Integrated Food Security Project to Build the Capacity of Dedba, Dergajen & Shibta Vulnerable People to Food Insecurity” (with Federation and Swedish Red Cross support) was targeted to 2,259 households in Dedba, Dergajen and Shibta through provision of crossbreed cows, ox fattening, sheep/goats, beehives and poultry which were to be repaid in cash over time as well water and agriculture/ seedlings for environmental resilience. ERCS had been working with the Ethiopian government to provide credit for these food security inputs to households in Tigray which were to be repaid in cash over time. During this evaluation, we met with 168 respondents (8% of total project participants).
Not only were we looking for food consumption impacts (which were very good), and income impacts (good), we also probed for self-sustainability of activities. My evaluation team and I asked 52 of these participants more in-depth questions on income and self-sustainability preferences. In Tigray, Ethiopia, we used participatory methods to learn what they felt they could most sustain themselves after they repaid the credit and the project moved on to new participants and communities.
We also asked the to rank what input provided the greatest source of income. The largest income (above 30,000 birr or $1,500) was earned from dairy and oxen fattening, while a range of dairy, oxen, shoats and beehives provided over half of our respondents (40 people) smaller amounts between 1,000-10,000 birr ($50 to $500).
![]()
And even while 87% of total loans were for ox fattening, dairy cows (and beehives) which brought in farm more income, and only 11% of loans were sheep/goats (shoats) and 2% for poultry, the self-sustainability feedback was clear. In the chart below, poultry and shoats (and to a lesser degree, ox fattening) were what men and women felt they could self-sustain. In descending order, the vast majority of participants prioritized these activities:
![]()
To learn more about how we discussed that Ethiopian participants can self-monitor, see blog.
So how can such a listening and learning approach feed program success and sustainability? We need to sit with communities to discuss the project’s objectives during design plus manage our/ our donors’ impact expectations:
1) If raising income in the short-term is the goal, the project could only have offered dairy and ox fattening to the communities as their incomes gained the most. Note, fewer took this risk as the credit for these assets was costly.
2) If they took a longer view, investing in what communities felt they could self-sustain, then poultry and sheep/goats were the activities to promote. This is because more people (especially women, who preferred poultry 15:1 and shoats 2:1 compared to men ) could afford these smaller amounts of credit as well as the feed to sustain them.
3) In order to learn about true impacts we must return post-project close to confirm the extent to which income increases continued, as well as the degree to which communities were truly able to self-sustain the activities the project enabled them to launch. How do our goals fit with the communities’?
What is important is seeing community actors, our participants as the experts. It is their lives and livelihoods, and not one of us in international development is living there except them…
What are your questions and thoughts? Have you seen such tradeoffs? We long to know…
[*NB: There were other inputs (water, health, natural resource conservation) which are separate from this discussion.]
by valuingvoicesjin | Oct 7, 2014 | Accountability, Big data, Donors, Evaluation, ex-post evaluation, impact evaluation, International aid, International non-profits, JICA, Local Participants, OECD, Participation, post-project evaluation, Results, Sustainability, Sustainable development, USAID, World Bank
Data for whose good?
Many of us work in international development because we are driven to serve, to make corners of the world better by improving the lives of those that live there. Many of us are driven by compassion to help directly through working ‘in the field’ with ‘beneficiary’/ participants, some of us manifest our desire to help through staying in our home countries, advocating to powers that be for more funding, while others create new technologies to help improve the lives of others all over the world. Some of us what to use Western funds and report back to our taxpayers that funds were well-spent, others want to create future markets via increasing globally-thriving economies. We use data all the time to prove our case.
USAID has spent millions on USAID Forward and monitoring and evaluation systems. Organizations such as 3ie rigorously document projected impact of projects while they are being implemented. Japan’s JICA and the OECD are two of the rarest kinds of organizations – returning post-project to look at the continued impact (as USAID did 30 years ago and stopped). Sadly the World Bank and USAID have only done one post-project evaluation each in the last 20 years that drew on communities’ opinions. While a handful of non-profits have used private funds to do recent ex-post evaluations, the esteemed American Evaluation Association has (shockingly) not one resource.
Do we not care about sustained impact? Or are we just not looking in the right places with the right perspective? Linda Raftree has a blog on Big Data and Resilience. She says, “instead of large organizations thinking about how they can use data from afar to ‘rescue’ or ‘help’ the poor, organizations should be working together with communities in crisis (or supporting local or nationally based intermediaries to facilitate this process) so that communities can discuss and pull meaning from the data, contextualize it and use it to help themselves….” Respect for communities’ self-determination seems to be a key missing ingredient.
As an article from the Center for Global Development cites the empowerment that data gives citizens and our own international donors knowledge by which to steer: Citizens. When statistical information is released to the public through a vigorous open government mechanism it can help citizens directly. Citizens need data both to hold their government accountable and to improve their private decision-making. (On the CGD website, see discussions of the value of public disclosure for climate policy here and for AIDS foreign assistance here.)
In my experience, most communities have information but are not perceived to have data unless they collect it using 'Western' methods. Having data to support and back information, opinions and demands can serve communities in negotiations with entities that wield more power. (See the book “Who Counts, the power of participatory statistics” on how to work with communities to create ‘data’ from participatory approaches). Even if we codify qualitative (interview) data and quantify it via surveys, this is not enough if there is no political will to make change to respond to the data and to demands being made based on the data. This is due in some part to a lack of foundational respect that communities’ views count.
Occasionally, excellent thinkers at World Bank 'get' this: "In 2000, a study by the World Bank, conducted in fifty developing countries, stated that “there are 2.8 billion poverty experts: the poor themselves. Yet the development discourse about poverty has been dominated by the perspectives and expertise of those who are not poor … The bottom poor, in all their diversity, are excluded, impotent, ignored and neglected; the bottom poor are a blind spot in development." (This came from a session description for the 2014 World Bank Spring Meetings Civil Society Forum meetings, where I presented for Valuing Voices this spring, see photo below).
And as Anju Sharma’s great blog on community empowerment says, “Why do we continue to talk merely of community “participation” in development? Why not community-driven development, or community-driven adaptation, where communities don’t just participate in activities meant to benefit them, but actually lead them?” Valuing Voices would like to add that we need participatory self-sustainability feedback data from communities documenting Global Aid Effectiveness, ‘walking’ Busan’s talk. Rather than our evaluating their effectiveness in carrying out our development objectives, goals, activities and proposed outcomes, let’s shift to manifest theirs!
Our centuries-old love affair with data is hard to break. Fine, data has to inform our actions, so let’s make it as grassroots, community-driven as possible, based on respect for the knowledge of those most affected by projects, where the rubber hits the road. While that may make massive development projects targeted at hundreds of thousands uniformly… messy… but at least projects many be more efficacious, sustainable and theirs. What do you think?
by Jindra Cekan | Jun 2, 2014 | Accountability, Bilateral organizations, Donors, Evaluation, ex-post evaluation, impact evaluation, International aid, International non-profits, Kenya, Local Participants, Open Data, post-project evaluation, Resilience, Results, Rural Development, Sustainability, Sustainable development, USAID
Sustainability SPRINGing out all over the place… and Disrupting
So what is sustainability? You may think it's the climate's long-term wellbeing and how to gauge changes to that. You may think it's linked to sustainable development regarding consumption, trade, education and environment and how to assess it. You may think it's data-driven organizational success as Chelsea Clinton describes, or is it Michael Porter's business' view of Creating Shared Value on social and environmental concerns or is it about people, as hallowed University of Cambridge trains experts in its Institute for Sustainability Leadership (I revel that I was a Fellow there in the '90s). Finally, is it WCED’s lovely definition "Sustainable development is development that meets the needs of the present without compromising the ability of future generations to meet their own needs"? Yes, when applied to communities' abilities to self-sustainably and resiliently chart their own development!
So how are we to get there? A Sustainable Brands Conference this year gets us there through being clear about their own consumption, and USAID is no different. USAID Forward is putting their money where their keyboards are (so to speak), toward more sustainable local delivery by directing a huge 30 percent of its funding to “local solutions” through procurement in coming years. This framework is to “support the ‘new model of development’ that USAID Administrator Rajiv Shah has touted, which entails a shift away from hiring U.S.-based development contractors and NGOs to implement projects, and toward channeling money through host-country governments and local organizations to build their capacity to do the work themselves and sustain programs after funding dries up. I, and others celebrate the investments this will enable local firms to make in their own capacity, in leading development!
Of course all sorts of safeguards are needed, and ideally US firms would be providing capacity development, but shouldn’t we have been doing this all along, to move toward transferring ‘development’ to the countries themselves?
Source: GAO report
Also vital to sustainable development is learning from what works and doing more of it. USAID is finally planning to incorporate more ex-post evaluations into its toolkit of evaluating sustainability! Two weeks ago, PPL/LER shared their great new policy document- “Local systems: A framework for supporting sustained development” on how they can better incorporate local systems thinking into policy as well as DIME (Design, Implementation, Monitoring and Evaluation). Industry insider DevEx tells us "even though the agency plans to use ex-post evaluations to measure whether development projects are successful or not, these evaluations will not focus on “specific contractor performance” but instead consider the “types of approaches that contribute to more sustainable outcomes…to inform USAID’s country strategies and project design." While PVO implementing partners will not [yet?] be required to do ex-post evaluations as part of their projects, having this door cracked open is excitingly opening. Notably, it is a ‘back to the future’ moment, as 30 years ago USAID led the development world in post-project evaluations, yet in the last 24 years has done none (or at least not published any) except for the Food for Peace retrospective below, as I found in our Valuing Voices research of USAID's Development Experience Clearinghouse.
There is far more to watch. In our view, the whole development industry needs to grapple with the perceived barrier that funding ends with projects (note: a trust could be set up to document post-project impact 1, 3, 5 years later and results retained, much as 3ie does now for impact evaluations) and the view that one cannot discern attributable project impact with a time-lag of several years. Yet even the General Accounting Office is asking for longitudinal data; they reviewed USAID’s document and wants to see clear measures of success at Mission and HQ level by different indicators of local institutional sustainability and impact four years on.
Why should we care? As Chelsea Clinton of the Clinton Global Initiative puts it, "you can't measure everything, but you can measure almost everything through quantitative or qualitative means, so that we know what we're disproportionately good at. And, candidly, what we're not so good at, so we can stop doing that.
Yes! Development should be about doing more of what works, sustainably, and less of what doesn’t. USAID’s Local Systems Framework found the best could also be free, as in this one Food For Peace evaluation shows:
Returning to Chelsea Clinton, I’ll conclude by stating something obvious. She "wants to see some evidence of why we're making decisions, as opposed to the anecdotes” which is what getting post-project evaluation data from our true clients, our participants, is all about. Clinton says this will transform CGI into a smart, accountable, and sustainable support system for philanthropic disrupters around the world. USAID is radical for me, today, with their Local Systems investments… my neighborhood disrupter.
Are you such a disrupter too? Who else is one whom we can celebrate together?
by Kelsey Lopez | May 16, 2014 | mobiles, Sustainable development, Unemployment, Youth
Youth Series Part II: How Technology Enables Youth Participation
See Youth Series Part I: Factors Hindering Youth Participation in Development
After discussing the various challenges youth face that inhibit their potential to be fully integrated, effective members of their societies, an analysis of some factors that enable their participation is also necessary to construct a complete picture. An International Monetary Fund (IMF) article, Youth in the Balance, elaborates on the specific challenges created by the economic crisis that spread around the world in 2008, and the subsequent responses by empowered youths – the Arab Spring. The article also notes the role of modern technology in the uprisings, stating that, “amid the turmoil and the uncertainty about their economic future, young people, more than any other group, have turned to new media for information and to communicate with their peers and beyond. Widespread access to the Internet has raised their aspirations, in part by making the young aware of the vast differences in standards of living within their countries and around the world. It has also made them more conscious of the extent of corruption and injustice and how that affects their lives” [1].
Thus, expanding access to technology has allowed for greater communication and awareness of issues, with a disproportionately higher percentage of youth using these technologies as compared to their older counterparts. As a result, there has been a surge of uprisings and political movements, spearheaded by disenfranchised youths, in response to the injustices they face, with technology being a key tool in the success of these movements in promoting change. “Whether it’s the ‘Occupy Wall Street’ movement in the United States or the mass rallies of the Arab world, young people have been jolted into action and are leading the response to diminished opportunities and unfulfilled aspirations,” a trend of empowered self-determination that has been facilitated immensely by access to technology [1]. Modern technology also aids the infectious nature of these political movements.
In cases such as the Arab Spring, youth populations have made the loudest cry for sociopolitical change, and this should come as no surprise, since the high level of unemployment most heavily affected this group in particular. As the IMF states, “youth have a huge stake in bringing about a political and economic system that heeds their aspirations, addresses their need for a decent standard of living, and offers them hope for the future,” and furthermore, “the absence of such a system is a potent recipe for conflict—especially now, with the availability of cheap means of communication such as smartphones and social media” [1].
Despite the catastrophe of youth unemployment that the global economic crisis exacerbated, technology provides a great opportunity for youths to become empowered, enabling them to participate in finding new solutions to the socioeconomic problems that plague them. As Bloom claims in Youth in the Balance, “young people will inevitably play a key role in the recovery thanks to their dynamism and willingness to relocate from labor-surplus to labor-shortage areas, and from low-productivity agriculture to higher-productivity industry and services. Their up-to-date training and education is also often a plus—although too often the education system imparts skills that are out of date or unneeded. Insofar as the expectations that education typically creates are not satisfied, youth can also power a decisive impulse to change institutions and leadership” [1]. With the heightened empowerment that access to the Internet allows, youth can take advantage of a wide variety of technological tools to achieve their own self-identified development solutions and to enable the spread of ideas.
Change is infectious, and the discontent from the ongoing Arab Spring movements is beginning to take root in other regions as well, namely Sub-Saharan Africa. In North Africa and the Middle East, “idealistic young protesters have toppled some of the most ruthless and well-resourced political strongmen on the planet,” and the conditions that led to this massive uprising are mirrored in Sub-Saharan Africa, where, “many of the underlying realities are the same. As food and fuel prices rise, inflation is driving millions of Africans below the poverty line…Radical and growing economic inequality animated much of what was at stake in the various Arab uprisings, and it will play a major role in shaping African politics for years to come” [2]. With marginalized, impoverished youths demanding new forms of government and representation, better economic policies, and greater social justice, technology plays a key role in spreading these ideas globally. Regarding the youth bulge in Africa, “their resentment is only heightened by the tools of the information age, which remind them that they have been excluded from feeding at the trough enjoyed so blatantly by the nouveau riche – a lifestyle that is showcased by the newly minted wealthy on television, Twitter, Facebook and the Web in infuriating detail. Globalization has changed the aspirations of the poor, and their expectations will follow” [2].
The feeling of inequality, or at least a perception of inequality, is a recent development challenge that the world faces, and this is a direct result of the role technology has played in highlighting the existence of inherent societal inequalities. For instance, for informed youths, “resentment of the president’s son’s Ferrari, more than envy of Europe and America’s comparative wealth, is driving young Africans into the streets to challenge their kleptocratic governments… consequently, the struggle to mitigate inequality, rather than “making poverty history” through debt relief has become the most urgent task [of development]” [2]. We live in a unique moment in development history wherein empowered youth suffrage movements were able to force the political change of the Arab Spring to a global movement. The root cause of this outrage is that, “the Arab Spring occurred at a moment when economic development had outpaced political development in much of the region; ossified political systems no longer satisfied a population yearning for modern freedoms. The explosive democratization in the Arab world, therefore, is a result of development’s success, not its failure. An authoritarian country cannot grow itself out of its fundamental underlying political contradictions. Eventually, a democratic deficit sets in” [2]. For countries that are in the midst of drastic societal change, there must be a balance of economic growth while still addressing the deeper issues that will determine the ultimate success or failure of that economic growth; these determining factors include basic freedoms, mitigating inequalities, and addressing the youths’ demands [2].
Technology has the power to harness the collective desire for youth empowerment that can bring countries out of economic crisis and into a new, more prosperous era. Central to the success of technology in aiding development goals is the transparency that it evokes, allowing cell phone and internet users to hold institutions accountable in a previously unheard of way. The challenge that remains, however, is ensuring that equal access to technology proliferates throughout all developing nations, thereby enabling citizens to communicate their ideas freely and without government interference. After establishing this holistic understanding of the ‘youth picture’ as it exists today on an institutional level, what is the challenge to development organizations that hope to combat the factors that hinder and encourage the factors that enable?
What do you think will help this movement to spread?
Sources:
[1] Bloom, D. E. (2012, March). Youth in the Balance. Retrieved from https://www.imf.org/external/pubs/ft/fandd/2012/03/bloom.htm
[2] Roos, J. (2011, July 30). Growing poverty may push Arab Spring into Africa. Retrieved from https://web.archive.org/web/20150225214042/https://roarmag.org/2011/07/growing-poverty-may-push-arab-spring-into-sub-saharan-africa/
by valuingvoicesjin | May 1, 2014 | Accountability, Asia, Bilateral organizations, Donors, Evaluation, impact evaluation, International aid, International non-profits, Local Participants, NGO, Participation, Resilience, Results, Sustainability, Sustainable development, USAID, water
Pineapple, Apple- what differentiates Impact from Sustainability Evaluation?
There is great news. Impact Evaluation is getting attention and being funded to do excellent research, such as by the International Initiative for Impact Evaluation (3ie), by donors such as the World Bank, USAID, UKAid, the Bill and Melinda Gates Foundation in countries around the world. Better Evaluation tell us that “USAID, for example, uses the following definition: “Impact evaluations measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other that the intervention that might account for the observed change.”
William Savedoff of CGD reports in Evaluation Gap reports that whole countries are setting up such evaluation institutes: “Germany’s new independent evaluation institute for the country’s development policies, based in Bonn, is a year old. DEval has a mandate that looks similar to Britain’s Independent Commission for Aid Impact (discussed in a previous newsletter ) because it will not only conduct its own evaluations but also help the Federal Parliament monitor the effectiveness of international assistance programs and policies. DEval’s 2013-2015 work program is ambitious and wide – ranging from specific studies of health programs in Rwanda to overviews of microfinance and studies regarding mitigation of climate change and aid for trade.” There is even a huge compendium of impact evaluation databases.
There is definitely a key place for impact evaluations in analyzing which activities are likely to have the most statistically significant (which means definitive change) impact. One such study in Papua New Guinea found SMS (mobile text) inclusion in teaching made a significant difference in student test scores compared to the non-participating ‘control group’ who did not get the SMS (texts). Another study, the Tuungane I evaluation by a group of Columbia University scholars showed clearly that an International Rescue Committee program on community-level reconstruction did not change participant behaviors. The study was as well designed as an RCT can be, and its conclusions are very convincing. But as the authors note, we don’t actually know why the intervention failed. To find that out, we need the kind of thick descriptive qualitative data that only a mixed methods study can provide.
Economist Kremer from Harvard says ““The vast majority of development projects are not subject to any evaluation of this type, but I’d argue the number should at least be greater than it is now.” Impact evaluations use ‘randomized control trials’, comparing the group that got project assistance to a similar group that didn’t to gauge the change. A recent article that talks about treating poverty as a science experiment says “nongovernmental organizations and governments have been slow to adopt the idea of testing programs to help the poor in this way. But proponents of randomization—“randomistas,” as they’re sometimes called—argue that many programs meant to help the poor are being implemented without sufficient evidence that they’re helping, or even not hurting.” However we get there, we want to know – the real (or at least likely)- impact of our programming, helping us focus funds wisely.
Data gleaned from impact evaluations is excellent information to have before design and during implementation. While impact evaluations are a thorough addition to the evaluation field, experts recommend they be done from the beginning of implementation. While they ask “Are impacts likely to be sustainable?”, and “to what extent did the impacts match the needs of the intended beneficiaries?” and importantly “did participants/key informants believe the intervention had made a difference?” they focus only on possible sustainability, using indicators we expect to see at project end rather than tangible proof of sustainability of the activities and impacts that communities define themselves that we actually return to measure 2-10 years later.
That is the role for something that has rarely been used in 30 years – for post-project (ex-post) evaluations looking at:
- The resilience of expected impacts of the project 2, 5, 10 years after close-out
- The communities’ and NGOs’ ability to sustain which activities themselves
- Positive and negative unintended impacts of the project, especially 2 years after, while still in clear living memory
- Kinds of activities the community and NGOs felt were successes which could not be maintained without further funding
- Lessons for other projects across projects on what was most resilient that communities valued enough to do themselves or NGOs valued enough to get other funding for, as well as what was not resilient.
Where is this systematically happening already? There are our catalysts ex-post evaluation organizations, drawing on communities’ wisdom. Here and there there are other glimpses of ValuingVoices, mainly to inform current programming, such as these two interesting approaches:
- Vijayendra Rao describes how a social observatory approach to monitoring and evaluation in India’s self-help groups leads to “Learning by Doing”– drawing on material from the book Localizing Development: Does Participation Work? The examples show how groups are creating faster feedback loops with more useful information by incorporating approaches commonly used in impact evaluations. Rao writes: “The aim is to balance long-term learning with quick turnaround studies that can inform everyday decision-making.”
- Ned Breslin, CEO of Water For People talks about “Rethinking Social Entrepreneurism: Moving from Bland Rhetoric to Impact (Assessment)”. His new water and sanitation program, Everyone Forever, does not focus on the inputs and outputs, including water provided or girls returning to school. Instead it centers instead on attaining the ideal vision of what a community would look like with improved water and sanitation, and working to achieve that goal. Instead of working on fundraising only, Breslin wants to redefine the meaning of success as a world in which everyone has access to clean water.
We need a combination. We need to know how good our programming is now through rigorous randomized control trials, and we need to ask communities and NGOs how sustainable the impacts are. Remember, 99% of all development projects worth hundreds of millions of dollars a year are not currently evaluated for long-term sustainability by their ultimate consumers, the communities they were designed to help.
We need an Institute of Sustainable Evaluation and a Ministry of Sustainable Development in every emerging nation, funded by donors who support national learning to shape international assistance. We need a sustainability global database, mandatory to be referred to in all future project planning. We need to care enough about the well-being of our true client to listen, learn and act.
