by valuingvoicesjin | May 1, 2014 | Accountability, Asia, Bilateral organizations, Donors, Evaluation, impact evaluation, International aid, International non-profits, Local Participants, NGO, Participation, Resilience, Results, Sustainability, Sustainable development, USAID, water
Pineapple, Apple- what differentiates Impact from Sustainability Evaluation?
There is great news. Impact Evaluation is getting attention and being funded to do excellent research, such as by the International Initiative for Impact Evaluation (3ie), by donors such as the World Bank, USAID, UKAid, the Bill and Melinda Gates Foundation in countries around the world. Better Evaluation tell us that “USAID, for example, uses the following definition: “Impact evaluations measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other that the intervention that might account for the observed change.”
William Savedoff of CGD reports in Evaluation Gap reports that whole countries are setting up such evaluation institutes: “Germany’s new independent evaluation institute for the country’s development policies, based in Bonn, is a year old. DEval has a mandate that looks similar to Britain’s Independent Commission for Aid Impact (discussed in a previous newsletter ) because it will not only conduct its own evaluations but also help the Federal Parliament monitor the effectiveness of international assistance programs and policies. DEval’s 2013-2015 work program is ambitious and wide – ranging from specific studies of health programs in Rwanda to overviews of microfinance and studies regarding mitigation of climate change and aid for trade.” There is even a huge compendium of impact evaluation databases.
There is definitely a key place for impact evaluations in analyzing which activities are likely to have the most statistically significant (which means definitive change) impact. One such study in Papua New Guinea found SMS (mobile text) inclusion in teaching made a significant difference in student test scores compared to the non-participating ‘control group’ who did not get the SMS (texts). Another study, the Tuungane I evaluation by a group of Columbia University scholars showed clearly that an International Rescue Committee program on community-level reconstruction did not change participant behaviors. The study was as well designed as an RCT can be, and its conclusions are very convincing. But as the authors note, we don’t actually know why the intervention failed. To find that out, we need the kind of thick descriptive qualitative data that only a mixed methods study can provide.
Economist Kremer from Harvard says ““The vast majority of development projects are not subject to any evaluation of this type, but I’d argue the number should at least be greater than it is now.” Impact evaluations use ‘randomized control trials’, comparing the group that got project assistance to a similar group that didn’t to gauge the change. A recent article that talks about treating poverty as a science experiment says “nongovernmental organizations and governments have been slow to adopt the idea of testing programs to help the poor in this way. But proponents of randomization—“randomistas,” as they’re sometimes called—argue that many programs meant to help the poor are being implemented without sufficient evidence that they’re helping, or even not hurting.” However we get there, we want to know – the real (or at least likely)- impact of our programming, helping us focus funds wisely.
Data gleaned from impact evaluations is excellent information to have before design and during implementation. While impact evaluations are a thorough addition to the evaluation field, experts recommend they be done from the beginning of implementation. While they ask “Are impacts likely to be sustainable?”, and “to what extent did the impacts match the needs of the intended beneficiaries?” and importantly “did participants/key informants believe the intervention had made a difference?” they focus only on possible sustainability, using indicators we expect to see at project end rather than tangible proof of sustainability of the activities and impacts that communities define themselves that we actually return to measure 2-10 years later.
That is the role for something that has rarely been used in 30 years – for post-project (ex-post) evaluations looking at:
- The resilience of expected impacts of the project 2, 5, 10 years after close-out
- The communities’ and NGOs’ ability to sustain which activities themselves
- Positive and negative unintended impacts of the project, especially 2 years after, while still in clear living memory
- Kinds of activities the community and NGOs felt were successes which could not be maintained without further funding
- Lessons for other projects across projects on what was most resilient that communities valued enough to do themselves or NGOs valued enough to get other funding for, as well as what was not resilient.
Where is this systematically happening already? There are our catalysts ex-post evaluation organizations, drawing on communities’ wisdom. Here and there there are other glimpses of ValuingVoices, mainly to inform current programming, such as these two interesting approaches:
- Vijayendra Rao describes how a social observatory approach to monitoring and evaluation in India’s self-help groups leads to “Learning by Doing”– drawing on material from the book Localizing Development: Does Participation Work? The examples show how groups are creating faster feedback loops with more useful information by incorporating approaches commonly used in impact evaluations. Rao writes: “The aim is to balance long-term learning with quick turnaround studies that can inform everyday decision-making.”
- Ned Breslin, CEO of Water For People talks about “Rethinking Social Entrepreneurism: Moving from Bland Rhetoric to Impact (Assessment)”. His new water and sanitation program, Everyone Forever, does not focus on the inputs and outputs, including water provided or girls returning to school. Instead it centers instead on attaining the ideal vision of what a community would look like with improved water and sanitation, and working to achieve that goal. Instead of working on fundraising only, Breslin wants to redefine the meaning of success as a world in which everyone has access to clean water.
We need a combination. We need to know how good our programming is now through rigorous randomized control trials, and we need to ask communities and NGOs how sustainable the impacts are. Remember, 99% of all development projects worth hundreds of millions of dollars a year are not currently evaluated for long-term sustainability by their ultimate consumers, the communities they were designed to help.
We need an Institute of Sustainable Evaluation and a Ministry of Sustainable Development in every emerging nation, funded by donors who support national learning to shape international assistance. We need a sustainability global database, mandatory to be referred to in all future project planning. We need to care enough about the well-being of our true client to listen, learn and act.
by Jindra Cekan | Jan 31, 2014 | Accountability, Bilateral organizations, Evaluation, International aid, Participation, Results, Sustainability, Transparency, USAID
Twigas (Participant self-sustainability) throughout 'development time'- Design, Implementation, M&E and beyond
What if we saw our true clients (sometimes vulnerable twigas) as our project participants and wanted the return on investment of projects be maximally sustained? How would this change how we design, implement, monitor and evaluate projects with country-leadership? With $1.5 trillion dollars in US and EU foreign aid spent since 2000, don't you think that our industry urgently needs feedback on what communities feel will be sustainable now, what interventions offer the likelihood of positive impact beyond the performance of the project’s planned (log-framed) activities, and what activities stood the test of time and our departure?
Shockingly, such learning is not happening today. YET.
Country-led development through participation has been a clarion call for several years, including USAID Forward and EU’s interest in food security and development finance. The Paris Declaration on Aid Effectiveness is a clarion call supporting country-led development. Multilaterals do some work to evaluate their projects; a handful of international non-profits do a small amount of evaluation post-project; and non-profits do some work using mobile technology to harness community feedback. Africa’s CAADP also manifests country-led and partially country-financed development (5-10% of GDP budgeted for Agriculture). Aid transparency is gaining support, and ValuingVoices adds a much needed local voice in terms of transparent, effective assistance. Much more funding is emerging to foster grassroots participation (e.g., Ashoka, Rockefeller and other funding), ranging from lessons learned from 6,000 aid recipients by Time to Listen to books touting Participatory Statistics, and the astonishing growth of mobile crowdsourcing platforms such as Kenya’s Ushahidi and Ghana’s Esoko. IOCE’s new evaluation initiative (VOPE) that is building capacity of national evaluators will also be a key resource.
So that's the good news coming up the riverbank. 
When should we be talking to communities about their views of what they feel is most sustainable and that they can sustain themselves? ALL THE TIME.
1) During Design
Most of our projects are designed in capital cities by donors who are informed by past projects, research, industry trends, policy imperatives, and are limited by narrowly focused funding streams. Very few projects are designed by those who will live with the results, namely the communities themselves. Donors often develop proposal requirements with little input from locals until start-up, when implementers fine-tune. Communities only get what donors are willing to give and donors want results. Not just temporary results but sustained ones. Such faulty design and lack of learning from evaluations—rarely asking our true clients, our participants, from the very start—leads to rusting tractors, farmers returning to traditional practices without resources to switch, latrines that are not used, etc. We mean well, but these practices are not working well. We know this from working over 25 years in international development. We need to change that.
2) During Implementation, at Midterm and Final Evaluation
Communities participate in midterm and final evaluations but often the knowledge is a one-way street, out. Hardly ever are findings shared with communities and joint re-design done to optimize prospects for self-sustainability of project activities. Ongoing learning from implementation needs to be transparently captured and shared in open-data format for discussion and adaptation (along USAID’s CLA Approach – Collaborating, Learning and Adapting program activities). Often sustainability is planned during the last 6 months of implementation, and handed to partners that are weakly-resourced, may contain untested assumptions, and haven’t had a chance to build up the organizational culture needed. Capacity building of partners to take over implementation also requires resources, which need to be budgeted in, and systems that partners co-create to fit their needs and ability to maintain. Program evaluation data that exists is often locked away in .pdf files in repositories that are not often accessible by countries themselves, and lessons learned within doesn’t often inform future design even among the implementers, much less by locals. We must change that.
3) After Project Close-out
Very rarely do implementing agencies return 3, 5, 10 years after projects close and ask participants what is “still standing” that they managed to sustain themselves. How often do we take community members, local NGOs, national evaluators as the leaders of evaluations of long-term self-sustainability of our projects? Based on my research (www.ValuingVoices/blogs) 99% of international aid projects are not evaluated for sustainability or impact after project close by anyone, much less by the communities they are designed to serve. We can change that.
This is what ValuingVoices is all about.
by Jindra Cekan | Jan 24, 2014 | Accountability, Africa, Bilateral organizations, Caribbean, International aid, Results, Sustainability, Transparency, USAID
'Developing them' fails (Haiti). Long live country-led development! (?)
I am tired of inefficiency and waste of tax dollars and human capacity and time. Some organizations use images of Africans, Asians, Latin Americans and Caribbean people suffering without permission to further their mission to get funds to do programming. United for Sight says it "evokes the idea that the poor are helpless and incapable of helping themselves, thereby cultivating a culture of paternalism." And that's what I am the very most tired of, our paternalism; this is the antithesis of country-led development.
Maybe we've gone about it a bit wrong. A case in point is a recent report on Haiti, that the US spent $2.8 billion and that there is little to show for it. Now, emergency response is truly a different kettle of fish – it is very expensive to intervene and restore even basic services and many nations have low aid-absorptive capacity. Nonetheless, it opens the door to unbridled waste, low capacity building and little sustainability as well. As Global Post and AP found, only 5% of 2012 US government spending went through Haitian organizations (and 95% through US and others) so the following dismal results are on our plate:
* "The number of people still living in grim encampments of quake survivors is now at 170,000" and while "The US promised to build 15,000 permanent homes but completed only 2,649 of them before ending its housing construction program, deciding instead to extend financing to Haitians directly for them to build their own homes." (I think at least we've tried to let people get in debt and find their own way to house themselves :), like us)
* The US is spending $170 million "to attract manufacturing to the new Caracol Industrial Park in Haiti’s North. But the massive investment was to have created 65, 000 jobs but has created fewer than 3,000 jobs, and the project may not have the funds to construct the port needed to export the industrial goods." Further, a US 2013 GAO audit found that there remained a deficit of over $100 million to complete it, with questionable prospects of success. (Not the best Return on Investment)
* “If the dismal record of USAID’s most trusted contractor is any indication, these entities are unreliable to say the least. A government audit of Haiti projects performed by USAID’s largest contractor, Chemonics, shows the company routinely failed to implement its aid projects correctly. In 2012, GlobalPost discovered that a $2-million, US-taxpayer funded USAID/Chemonics project constructed a building for Haiti’s parliament that was unfinished and unusable.”
As one Haitian professional said, "“There have been aid programs for such a long time here, but when you evaluate it, they don’t have durable impacts… [it] isn’t really helping the people with these problems be released from their problems, it only keeps them stuck in poverty.”
So this is what can happen when we know best. Again and again, with great hubris and waste, we know better than the country's nationals. I have worked in "international development" for 25 years and the title has always irked me. Are international development projects named aptly? Isn't it instead a matter of national development projects supported by international expertise and funding instead?!
There is good news. USAID recently launched its Forward Progress program, which aims to increase the amount of money it sends directly to local companies and NGOs (Congress permitting). "we have to support the institutions, private sector partners and civil society organizations that serve as engines of growth and progress for their own nations. USAID Forward is helping us to do that through new models for public-private partnerships and increased investment directly to partner governments and local organizations." The jury's still out about tangible results locally, but this sounds hopeful.
A delightful prospect of real country-led development potentially comes from a collaborative international donor-country partnership, CAADP. Comprehensive Africa Agriculture Development Programme's (CAADP) goal is to eliminate hunger and reduce poverty through agriculture by addressing "policy and capacity issues across the entire agricultural sector and African continent. CAADP is entirely African-led and African-owned and represents African leaders' collective vision for agriculture in Africa." Notably, putting their money where their mouths are, "African governments have agreed to increase public investment in agriculture by a minimum of 10 per cent of their national budgets and to raise agricultural productivity by at least 6 per cent."
So how are is CAADP doing after 10 years? The results are mixed. A 2013 IFPRI evaluation found that
* "Unlike programs that are largely driven by donor priorities, undermining local institutions in the process, CAADP is a product of the locally-sourced development goals of participating countries. This approach has resulted in a sense of ownership and better development outcomes" [yet]
* “The countries with the 10 largest agricultural sectors on the continent… have spent less than 5 percent of their total budget"
But while these results are mixed, so too are donor commitments. The 2011 midterm evaluation found that while there were compacts created in 22 countries, and 6 countries have been awarded GASFP funding totaling $270million, another 8 countries across West Africa (plus Ethiopia) received less than 50% of the funds promised to promote agriculture.
So what will it take for the poor to get the help they need from any of us? Do we value their voices? Is our international aid really helping them? You tell me.
by valuingvoicesjin | Jan 6, 2014 | Accountability, Bilateral organizations, DIFD, Evaluation, International aid, International non-profits, LIvelihoods, Participation, Resilience, Results, Sustainability, USAID
Development= A Jeep (motor optional).. Resilience? If within 5 years!
Imagine being given a lovely new Jeep. You get a driver (remember driving school) to help you learn to steer it around the pothole-strewn, scantly lit roads. Eventually you take over the controls of the Jeep and control the steering wheel directly, driving offroad, with the copilot praising your good driving and steering only to avoid catastrophe. You are told that one day the Jeep will be yours.
That day arrives. The development agency hands you the keys to the Jeep. You wave good bye to them, return to the Jeep, turn the key. Dead.
Looking under the hood, you realize the motor is gone. Checking the rest of the Jeep you realize there is no fuel and the tries are flat. That is what it is like from the community's view of development projects after close-out. The local NGO to whom the project has been 'handed over' has scant financial or human resources to continue (no engine), and in the last few months' scramble to close out, the implemeneting agency put in few systems for communities to continue doing the programming without support by the local NGO or all the resources they had poured in (no fuel). There is little to help you move the Jeep (even on flat tires) except your own feet, other than the capacity building that was learned early on, as it wasn't built to last based on local materials. Sustainability isn't programmed in projects that have set timelines and donor-set markers of success which mandate close-out.
So you own the Jeep but with little power to move, very much like countless well-meant tractor for development agriculture before you.
There are several glimmers of hope. What communities have is the human power that exists locally, fuelled by participation coupled with information transmission such as WorkWithUs and MakingAllVoicesCount (based on the moral imperative of it's Their Development as well) and ALNAP's push to use evaluation for learning in international development.
We could also help donors springboard to learn from sustainability via the latest buzzword in development: "Resilience". At USAID and DIFD this is the "ability of people, households, communities, countries, and systems to mitigate, adapt to, and recover from shocks and stresses in a manner that reduces chronic vulnerability and facilitates inclusive growth" and "helping communities and countries to be better prepared to withstand and rapidly recover from a shock", respectively.
Resilience could be the doorway to getting community-defined sustainable programming to break the cycle of recurrent emergencies that divert resources from long-term development. Imagine: we could ask citizens what will make them resilient! A rare, shining example is a USAID-funded Ethiopia project with a mandate to use participatory impact assessments (process monitoring plus participatory input to capture local perceptions of benefits) to learn from communities. A USAID Solicitation tells us "seventeen impact assessments on different program activities were undertaken to inform best practice and to develop guidelines and policies. A major impact was the development and adoption of Emergency Livestock Guidelines by the Ethiopian government. These were based on best practice assessments in many countries (including Kenya) and action research on different types of interventions. Emergency de-stocking–selling livestock early in a drought to preserve their price and leave more fodder and water for remaining animals — was found to be particularly effective, with a 40:1 benefit cost ratio. Emergency livestock vaccination campaigns, on the other hand, were found to have no impact on livestock mortality, and were dropped in favor of other health interventions including parasite control and de-stocking."
Excellent Valuing of pastoralist Voices! How are such locally-informed excellent processes and findings being widely shared and implemented? What do you think?
