Pineapple, Apple- what differentiates Impact from self-Sustainability Evaluation?
There is great news. Impact Evaluation is getting attention and being funded to do excellent research, such as by the International Initiative for Impact Evaluation (3ie), by donors such as the World Bank, USAID, UKAid, the Bill and Melinda Gates Foundation in countries around the world. Better Evaluation tell us that "USAID, for example, uses the following definition: “Impact evaluations measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other that the intervention that might account for the observed change.”
William Savedoff of CGD reports in Evaluation Gap reports that whole countries are setting up such evaluation institutes: "Germany's new independent evaluation institute for the country's development policies, based in Bonn, is a year old. DEval has a mandate that looks similar to Britain's Independent Commission for Aid Impact (discussed in a previous newsletter ) because it will not only conduct its own evaluations but also help the Federal Parliament monitor the effectiveness of international assistance programs and policies. DEval's 2013-2015 work program is ambitious and wide – ranging from specific studies of health programs in Rwanda to overviews of microfinance and studies regarding mitigation of climate change and aid for trade." There is even a huge compendium of impact evaluation databases.
There is definitely a key place for impact evaluations in analyzing which activities are likely to have the most statistically significant (which means definitive change) impact. One such study in Papua New Guinea found SMS (mobile text) inclusion in teaching made a significant difference in student test scores compared to the non-participating 'control group' who did not get the SMS (texts). Another study, the Tuungane I evaluation by a group of Columbia University scholars showed clearly that an International Rescue Committee program on community-level reconstruction did not change participant behaviors. The study was as well designed as an RCT can be, and its conclusions are very convincing. But as the authors note, we don't actually know why the intervention failed. To find that out, we need the kind of thick descriptive qualitative data that only a mixed methods study can provide.
Economist Kremer from Harvard says "“The vast majority of development projects are not subject to any evaluation of this type, but I’d argue the number should at least be greater than it is now.” Impact evaluations use 'randomized control trials', comparing the group that got project assistance to a similar group that didn't to gauge the change. A recent article that talks about treating poverty as a science experiment says "nongovernmental organizations and governments have been slow to adopt the idea of testing programs to help the poor in this way. But proponents of randomization—“randomistas,” as they’re sometimes called—argue that many programs meant to help the poor are being implemented without sufficient evidence that they’re helping, or even not hurting." However we get there, we want to know – the real (or at least likely)- impact of our programming, helping us focus funds wisely.
Data gleaned from impact evaluations is excellent information to have before design and during implementation. While impact evaluations are a thorough addition to the evaluation field, experts recommend they be done from the beginning of implementation. While they ask “Are impacts likely to be sustainable?”, and “to what extent did the impacts match the needs of the intended beneficiaries?” and importantly “did participants/key informants believe the intervention had made a difference?” they focus only on possible sustainability, using indicators we expect to see at project end rather than tangible proof of sustainability of the activities and impacts that communities define themselves that we actually return to measure 2-10 years later.
That is the role for something that has rarely been used in 30 years – for post-project (ex-post) evaluations looking at:
The resilience of expected impacts of the project 2, 5, 10 years after close-out
The communities’ and NGOs’ ability to self-sustain which activities themselves
Positive and negative unintended impacts of the project, especially 2 years after, while still in clear living memory
Kinds of activities the community and NGOs felt were successes which could not be maintained without further funding
Lessons for other projects across projects on what was most resilient that communities valued enough to do themselves or NGOs valued enough to get other funding for, as well as what was not resilient.
Where is this systematically happening already? There are our catalysts ex-post evaluation organizations, drawing on communities' wisdom. Here and there there are other glimpses of ValuingVoices, mainly to inform current programming, such as these two interesting approaches:
Vijayendra Rao describes how a social observatory approach to monitoring and evaluation in India’s self-help groups leads to “Learning by Doing”– drawing on material from the book Localizing Development: Does Participation Work? The examples show how groups are creating faster feedback loops with more useful information by incorporating approaches commonly used in impact evaluations. Rao writes: “The aim is to balance long-term learning with quick turnaround studies that can inform everyday decision-making.”
Ned Breslin, CEO of Water For People talks about “Rethinking Social Entrepreneurism: Moving from Bland Rhetoric to Impact (Assessment)”. His new water and sanitation program, Everyone Forever, does not focus on the inputs and outputs, including water provided or girls returning to school. Instead it centers instead on attaining the ideal vision of what a community would look like with improved water and sanitation, and working to achieve that goal. Instead of working on fundraising only, Breslin wants to redefine the meaning of success as a world in which everyone has access to clean water.
We need a combination. We need to know how good our programming is now through rigorous randomized control trials, and we need to ask communities and NGOs how sustainable the impacts are. Remember, 99% of all development projects worth hundreds of millions of dollars a year are not currently evaluated for long-term self-sustainability by their ultimate consumers, the communities they were designed to help.
We need an Institute of Self-Sustainable Evaluation and a Ministry of Sustainable Development in every emerging nation, funded by donors who support national learning to shape international assistance. We need a self-sustainability global database, mandatory to be referred to in all future project planning. We need to care enough about the well-being of our true client to listen, learn and act.
Transparency and Accountability via open [Cambodian] data that ValuesVoices
My colleague pointed out this fascinating article about a new transparency program for garment factories, using data collected meticulously by the International Labor Organization (ILO) on working conditions in Cambodia in a publically accessible searchable database: http://betterfactories.org/transparency/. and this story about it:
The key elements I want to point out about this great example are the following:
The data was already being collected by the ILO-thorough interviews and investigations of different factories, at a presumably high cost. But this information was not being widely shared, because the format the data was in (PDF), was not easily accessible by those who could use it, resulting in these reports not having the “shame” impact that is important for voluntary compliance.
The BFC was able to partner with the ILO to open their data, making it easier for anyone to search and use the outcomes of the ILO reports. This increased access has already made an impact in making Cambodian factory owners more focused on compliance.
This pattern is very similar to what we see occurring throughout international development, where evaluation data is being captured and collected, but not shared in formats that would make transparency and accountability easy to access.
The article hinted at the two main challenges to making this data open:
The need for a robust back end to support and maintain it – something that requires investment and thoughtfulness; and
“very public debates between the garment producer’s association and Better Factories Cambodia that ensued for months” – i.e. political concerns about transparency and what the data means.
When we unlock the data that is currently being collected (and paid for) in international development, especially during project evaluations, we find opportunities for increased transparency and accountability, as well as making the data captured more likely to provide insight for those not directly involved in the project.
This effort will take time, effort, and money, and there will be political obstacles to overcome. But the result is information to help us make better decisions on where we spend our money, invest our time, or highlight those who need to improve. The combination of ValuingVoices' debates and data work!
Siobhan Green is a partner company to ValuingVoices and can be found at Sonjara.
Prospects for long-term sustainability… or lack thereof at the Macro level (Part 2)
During 2003-2004, the OECD conducted a very interesting Ex-Post Evaluation Sustainability Summary that synthesizes four separate Regional Rural Development (RRD) projects, which has illuminated sustainability of the impacts of German RRD projects over nearly a 30-year period. The goal of this evaluation was to use the perspectives of local experts (evaluators and national consultants) to inform the Federal Ministry for Economic Cooperation and Development about the true impacts of four rural long-term projects.
The four projects evaluated were:
· Indonesia – Area Development Project in West Pasaman (1980-1992); German contribution: €32 million; Target population: 200,000 urban and rural people
· Sri Lanka – Regional Rural Development Project (RRDP) in Kandy (1987-2000); German contribution: €8.1 million; Target population: 200,000 rural small-scale producers
· Tanzania – Tanga Integrated Rural Development Program (TIRDEP) (1972-1993); German contribution: €75 million; Target population: 700,000 rural small-scale producers
· Zambia – Integrated Rural Development Program (IRDP) in Kabompo (1977-1993); German contribution: €30 million; Target population: 65,000 rural small-scale producers
While these programs “had been implemented in regions with very different underlying general conditions,” involving interventions of varying scales, with different socioeconomic conditions affecting the different countries and have effected a great number of changes in their respective countries, German’s RRD experts also drew similarly positive findings on outcomes and negative findings in terms of processes and the pitfalls of programming without focus on sustainability.
Here is the good news:
· “Living conditions of the target groups have improved in all four project regions,” with specific sustainable project outcomes observed in the “health and education sector, food security, increase in income and employment and the ensuing rise in the standard of living.” The projects were able to create these changes by improving infrastructure, enhancing the private sector economy, and supporting innovations in agriculture.
· Project planning was done largely on the German side, however it was determined that there was still a relatively high acceptance of the objectives by the participating stakeholders (locals) due to the establishment of joint objectives by locals and the implementing agencies.
Yet, there were also some factors that jeopardized the sustainability of positive results, which included the (non)maintenance of the infrastructure as well as the intensification of economic activities that led to negative ecological impacts (such as acidification of the soil and overfishing).
Even more interestingly, the OECD gets kudos from ValuingVoices for analyzing where the projects failed to meet their outcomes and sharing what led to unsustainable outcomes. Here is the bad news:
· There was low institutional sustainability at the level of state executing organizations for all four projects due to inadequate funds, inefficient organizational structures and a lack of coordination. Thus, viable exit and handover was limited. Due to this lack of institutional sustainability, it was concluded that, “the putting into place of new structures by means of development projects runs a very high risk of not being sustainable,” noting significant differences in local expectations of the partners and the concepts of the RRD. Hence imposing structures ex-post rather than designing with sustainability in mind doesn’t seem to work.
· There is also an issue of changing standards over time. These projects were completed from the 1970s-1990s, but older development schemes are not considered as relevant today. “The former RRD project concept is no longer pursued” today as a result of changes, such as economic reforms and decentralization, which led to the adoption of new concepts. This kind of risk bedevils all ex-post evaluations, akin to asking perfectly good donkeys why they are not thoroughbreds.
Further, the processes this collection of ex-post studies illuminates are key:
· “The expenses for the consultancy services of German experts were often considered as disproportionate compared with the hardware supplied. Here, partner expectations obviously were not consistent with those of the German side.” Potentially this is an incentive to use national evaluators. Yet the study concluded: “this serial evaluation has brought about a change in perspectives in part through the deployment of local experts. However, the qualification profile of locally available consultants varies considerable. The results of this serial evaluation do not suggest a general shift of evaluations to local organizations.” This study was 10 years ago so this has also begun to shift, even though pairing a ‘northern’ evaluator with ‘southern’ ones is more the norm.
· Finally, “the termination of German support was often considered abrupt and incomprehensible. Phasing out was done according to German views and did not take sufficient account of the partners’ views and needs as seen by them…[which] provides grounds for a systematic conflict between the interests of the partners and those of the German side.” This is key in reforming international development, from artificial timelines to those informed by adaptive learning, that takes into account and is even led by community and local NGO learning and readiness to take the helm themselves.
There is much to learn from OECD reviews such as this one about both the aims and processes for ex-post evaluation. What remains unclear in all ex-post evaluations Valuing Voices has found is what the organizations and donors have learned from them, and to what degree they have applied the lessons learned to the rest of their programming.
What are your experiences? Have lessons from evaluations been taken in? How? Why not?