by valuingvoicesjin | Dec 12, 2013 | Central Asia, Evaluation, Governance, International aid, International non-profits, Sustainability, Uncategorized
Mercy Corps – early leader in evaluating sustainability… and what donors are funding
Mercy Corps shared their work in post-project sustainability early on, inspiring me it was possible. As they put it, "clearly, a sustained ability for collective problem solving offers the best path to lasting improvement in people's lives and, for donors, the best return on investment" in two conflict resolution projects in Kyrgyztan, Tajikistan and Uzbekistan: Peaceful Communities Initiative $6.5 mil (2002-2007) and Community Action Investment Program $11.8 mil (2002-2005). These two projects were complex community mobilization programs with aims "to empower communities to work together in a participatory manner to address the infrastructure and social needs [while] developing sustainable skills in problem solving, consensus-building and accountability. The process also empowers communities to begin to identify and utilize existing resources within the communities and not to depend only on external assistance."
So what happened? Their report on sustainability in 2007 random-sampled and interviewed youth leaders and in community action groups, 55% of the communities and found promising results:
According to the evaluation:
*93% of surveyed projects are still being actively used by the community after programs closed.
*73% of members of the community action groups (CAG) felt it was still easier to approach local government at least one year after the programs ended and 68% witnessed local government becoming more involved in community activities after the end of the programs as compared with before the programs [participants and partners had] implemented almost 100 infrastructure projects by themselves and independent of donor funds.
* 72% of youth report that they continue to use at least one skill they learned during the programs. Those cited most often include teamwork and communication, as well as practical skills such as sewing, construction, roofing, journalism and cooking
*57% of the communities studied continuing to use one or more of the decision-making practices promoted during the program.
* 42% of CAG members, representing 35 of the 51 communities, reported that the community had worked collectively on new projects or repairs to existing infrastructure
* In total, 40% of general community members interviewed reported that youth had initiated community activities since January 2007, and 68% of these community members recognized that some or all of the activities had not taken place prior to Mercy Corps’ program
This appears excellent! They interviewed youth and key local stakeholders (CAG), used country nationals to evaluate (both of which are wonderful as it is their country) and they distilled best practices in community mobilization in accessible reports shared within the international non-profit world via the web. Quibbling only a little, a difficulty is that while the report states that they "I dentified factors that influence sustainability, through both positive examples and non-sustained projects and practices" our ability to learn from what didn't work was hidden. Mercy Corps discussed what didn't work only by recommending what to do, rather than discuss the extent to which specific activities simply didn't work. While focusing on the positive is the best path forward, ideally Mercy Corps would have also shared what failed, possibly why and whether they had seen this elsewhere (thereby suggesting such activities may not be promising to replicate). Further, what would be valuable is to gauge roughly the percentage of the program value these successes represented versus those that did not work well. Sucha cost-effectiveness ratio would benefit our industry.
Overall, such successful national-level capacity building and program effectiveness learning is terrific, and new focus may lead funding to follow. Two major foundations have lately said that they have an interest in supporting national capacity building and empowerment (Rockefeller) or cost-effectiveness (Bill and Melinda Gates), respectively.
More specifically, Nancy Macpherson of Rockefeller Foundation states the Foundation is "committed to evaluation practices that are rigorous, innovative, inclusive of stakeholders’ voices, and appropriate to the contexts in which the foundation works." This is done by "integrating the views of developing-region evaluators" as well as:
* "strengthening developing country evaluation practice and ownership of results…
* developing innovative methods and approaches to evaluation and learning…
* the empowerment of people; and
* the effectiveness of development interventions by national governments and international partners and, increasingly, by non-state actors—foundations, philanthropists, and agencies that promote investing for impact."
The Bill and Melinda Gates Foundation mentions cost effectiveness
* When evidence is needed to fill a knowledge gap or evaluate a significant policy decision. Evaluation can help to resolve uncertainty and determine the relative cost-effectiveness of different interventions, models, or approaches" (Gates) and
* Both quantitative and qualitative data are relevant in evaluating processes, operations, cost effectiveness, key stakeholders’ perceptions, and enabling contextual factors" (Gates)
USAID seems not to have done any ex-post evaluation since a single one on a Philiiipines loan in 1980, whereas parts of the EU seem to be doing many more such evaluations in agriculture and rural development (2002-06), industrial technologies (2009-11) as recently as 2013 for ICT. Given the size of its portfolio of development assistance, $20.4 billion in development and humanitarian programs in fiscal 2014, learning about sustained impact seems imperative from a return on investment (ROI) perspecive.
Supporting national capacity and evaluating programs' return on investment is pivotal. In Mercy Corps/ USAID's funding case alone, this comes to over $18 million dollars. But Mercy Corps paid for this evaluation themselves, from private funds. If we are in the process of fostering country-led and eventually country-financed development, they need to know how much such investments get them – as we do.
Does anyone know of such great program learning that begins to teach about return on investment? Can you share?
by valuingvoicesjin | Dec 6, 2013 | Africa, Evaluation, International aid, International non-profits, Resilience, Women and gender
Unintended impacts – LWR and Gates Drought Resilience in Niger
Unintended development program impacts – how much do we know about what they are and how we could learn from them for future programming? How often do we even question our assumptions about what we meant the programming to bring and result in, versus what actually happened?
I had the privilege of consulting to Lutheran World Relief from 2005-2007 in Niger on a drought resilience and rehabilitation project. IIED's PLA Notes just published my write-up of this baseline and slightly ex-post (6 months after closeout) final evaluation funded by the Bill and Melinda Gates Foundation. I led a team doing mixed-method interviews (that means tracking numbers finding out how many, how much impact we had, combined with listening to words explaining why things worked out this way, or not). There were numerous lessons lessons learned from targeting sheep, wells and animal fodder to 600 of the poorest women in 10 communities in northern Niger. Among them were that LWR's programming did some real good, especially due to great knowledge of the herding communities via national staff.
The main two project goals had to do with whether the communities were more resilient against future drought and were they more sustainably food secure thanks to the assets (sheep, water, fodder), income (future sheep, sale of fodder) and training (management of sheep and maintenance of water points). The answers seemed to be generally yes. We found that women's share of household income increased from 5% to 25% in some households. This was due to the sheep, as well as time savings used for income generation (a staggering 7-10 hours every other day saved from not having to go fetch water far away or bring animals to drink).
We also found several completely unexpected benefits: resources + time savings generated in harmony and peace. Women reported far more inter-ethnic harmony thanks to collaboration across tribes during the sheep management training and water sharing, and ethnic groups attended each others marriages and baptisms. Households also were far more at peace, some saying "our husbands don't beat us anymore", thanks to increased respect, cleanliness and their ability to be home for their husbands, children and mothers-in-law. Men reported that they sat with women by project's end and there was more intra-household collaboration. Were these planned at all? No! We need to return to learn about what our work engenders, and bring those lessons forward, see if they're replicated or not. What else has emerged in the years since the project ended? What was valued enough to be sustained by the communities themselves?
Finally, a fascinating flaw in our logic which is interesting for international development staff to consider was whether our participants shared the two large goals at all (drought resilience and sustained food security). Some women sold the sheep to buy food, pay their childrens' school fees or their daughters' dowries, even buy themselves beds or pots and pans immediately. Some had their sheep sold by their husbands who used them to buy other animals, pay for ceremonies or other expenses. Spending assets on immediate needs is not at all illogical for a community who can feed itself only 4 months a year; for some households, their pressing needs far outweighed the luxury to wait and buffer seasonal food insecurity way down the line. So while overall the project brought benefit – many saying they didn't have to resort to worse survival strategies during the next hungry season- it illuminated that donors' goals may differ greatly from participants'. These people didn't sign a contract agreeing to abide with donor expectations, yet that is what the evaluation looked for. We need to learn not to assume, and instead explore what goals they do have as expectations of impact. Looking beyond within useful but narrow 'boxes' supported but limited by logical frameworks of inputs-outputs-impacts can help us learn what communities really value and what they got from what we offered.
Valuing their voices is key. What projects have you seen voices valued? What have you learned?
by valuingvoicesjin | Nov 22, 2013 | Africa, Corruption, International aid, International non-profits
Corruption. Now I've said it, fear is our greatest barrier
That wasn't a typo. When I started working in international development, and quite regularly since, the need for us international development agencies as intermediaries has been that we are the defense against corruption. That it is our money, that we must safeguard it. It needs safeguarding, that is true.
Absolutely there are politicians all around the world who have robbed their people blind — Zaire's Mobutu, Philippines' Marcos, Iraq's Hussein, Nicaragua's Garcia, Romania's Ceaucescu and for that matter America's (VP) Agnew. It's absolutely true that they exist, and that corruption exists at all levels if allowed. But who should safeguard it? Most local non-profit staff, local government staff I know are as passionate about doing development well as we are, but their institutional capacity, strategic planning, computer systems/ financial management systems are weak. Most have low capacity to absorb anywhere near the hundreds of thousands of dollars we have on offer.
Yet they can have allies. Most households and communities are extraordinary stewards of resources — remember, many of them, our clients, are living on less than $2.00 per day. Talk about stewardship — managing to live on this amount. The communities I have worked in in Africa, Latin America and the Balkans have no patience for corruption among them – and deal with it swiftly- nor should they have patience for a lack of good stewardship of those coming to help them. Yet these communities and local non-profits aren't often entrusted with the design nor the stewardship. When we flood organizationally weak local non-profits or even ministries with money, when their organizations' absorptive capacity is low, there is no way money will not find other channels. When we start at the top and not the bottom, when we don't build in financial safeguards that ordinary people can check, how else would it turn out, even here in the 'developed' world?
Why do we do it for them? Because we so deeply want to help alleviate suffering. I believe our fear is the key. Fear that it won't be done as well as we would have done it, that working with communities will simply take too long, that our work won't be maintained so we lag and build in sustainability at the very end of the project because we know how to do it best and are too rushed implementing to think beyond the mandated project-funded-close out. We are also afraid of fostering dependency on us, so we shut down fast at the end, after the time we pre-set (2, 3, 5 years later, without analysis of how ready the communities are for shutdown). We sometimes even don't start up projects that are desperately needed to prevent suffering because we are so afraid of creating dependency down the line, but that is for another blog…
Some of us may even fear that we would lose our jobs and our own non-profits would close if we planned for our own phase over to indigenous non-profits, rather than seeing it would immeasurably add value to build capacity and more equity. So the industry is self-maintained, little capacity is built for handover, and the millions of dollars, euros, yen circulate from developed to developing with a lot of short-term wonderful activities and improvements but without much sustained impact.
We are also afraid that if the money is not spent, we won't get more so we sometimes close our eyes to the lack of absorptive capacity, our hurried pace not allowing local government, NGOs, even communities themselves to participate. Time to Listen's research, mentioned in another blog says that that is hundreds of communities' greatest wish – to participate, to captain their own development.
In 1991 when I was doing my doctoral fieldwork in Mali, NORAD (Norway's equivalent to USAID) was in the late stages of transferring its programs to Malian direction. They had a wooden board outside their project transparently showing pages of their project's budget, expenses, and balance of funds by line item (granted in French, not always accessible some to illiterate folks in Bamako, but easily accessible to normal educated folks in the capitol). I had walked over because I saw Malians standing there, reading it. Radical, and sure enough, when I returned in 1995, the program was all-Malian designed and led, and the only Norwegian input were funds being sent from ongoing donors. Yet in the course of writing this blog, I went to look at whether this trend continued, and it hasn't. One-third of NORAD's funding was going to Mali's public sector and local organizations, but two-thirds was going via Norwegian non-profits.
Should this continue? Sigh. What are the clearest incentives for financial and power handover to (hopefully-soon-to-be) well-funded, capacity-built local non-profits and local government and empowered communities? Trust, with safeguards such as robust systems, institutional capacity building and more time. The funds are, after all, for them.
At l'Aquilla in 2009, President Obama committed to investing $20 billion in food security — agricultural development programs to help fight world hunger, where he said "we believe that the purpose of aid must be to create the conditions where it's no longer needed — to help people become self-sufficient, provide for their families, and lift their standards of living. And that's why I proposed a new approach to this issue — one endorsed by all the leaders here — a coordinated effort to support comprehensive plans created by the countries themselves, with help from multilateral institutions like the World Bank when appropriate, along with significant and sustained financial commitments from our nations." This kind of development, country-led and internationally supported is the only sustainable path. Development projects are multi-million dollar investments – don't we owe ourselves and our clients time to optimize the use of these funds? Time and trust are the way forward.
What do you think? What fears do you have or think others have that are barriers to sustainability? What would you do?
by valuingvoicesjin | Nov 19, 2013 | Africa, Bilateral organizations, Governance, Sustainability, United Nations
Sustainable Development… are we done yet?
The UN's Commission on Sustainable Development has just disbanded after 20 years. They have given their convening power and work to a High‐level Political Forum, saying "we must effectively use the new High‐Level Political Forum to ensure that sustainable development continues to be implemented and is integrated into the heart of the post‐2015 development agenda."
Yes, while a few organizations have actually funded some post-project (ex-post) evaluations here and there, this idea that anyone in our industry could say "sustainable development continues to be implemented and integrated" strikes me as absurd. How many communities in Africa, Asia, Latin America and elsewhere feel that they are sustainably developed, that they can finally hand out gilded keys to their stunning cities and villages and never have to invite anyone in for help again? Scant few, I'd wager (Botswana and Asian tigers notwithstanding).
One of the fascinating findings of my preliminary research is that extremely few ex-post evaluations of sustained impact of projects are done at all, and the lions share is by some banks and multilateral organizations. While 30 years ago USAID used to fund such work, today it is UN agencies such as the OECD, the World Bank1 and World Bank2/ IMF, bilateral banks such as the European Investment Bank, IADB, a German Bank, and a few bilateral development organizations such as Japan's JICA. Kudos to them for doing it at all, anytime.
Why do some do it? An Australian government publication notes that:
* "Good ex post evaluation is a potentially powerful tool that can attest to the value of a reform and provide lessons for future reforms.
* Evaluations can improve the understanding of the impacts of regulation on risk —
both the probability of an adverse event and the impact if it arises."
This publication also states that "to be more effective, ex post evaluations need to: be resourced in advance; embed data collection where it is not otherwise available; share the findings; and include governance arrangements which support transparent process to test the findings and to encourage their use." The bit about use is especially compelling, especially if such results are shared with all stakeholders, especially national ones- from communities involved, up to policy decision-makers. Better yet, start discussions about sustainability with local communities to begin with, where rubber hits the road.
Unfortunately, reading some of these reports is a bit surreal; it shows how far from that end-user perspective we currently are. While evaluators ask questions about long-term impact, questions of sustainability often remain 'within the box' of expected outcomes. Reports I have read use an external evaluator who alone assesses OECD's measures of: program relevance, effectiveness, efficiency, development impact and sustainability of the project as it was planned. So far nothing has appeared that implies that participants themselves were asked about sustained impact, rarely are traces of anything 'outside the box' of what was planned (such as the appearance of unintended impacts), or that anything could have happened outside of the project objectives in the elapsed years post-project. This is in spite of our understanding that our world is complex and our lives and livelihoods adaptively evolve all the time in response to external economic, climatic, social opportunities and shocks. "Sustainability" results seem to be reported as if time stopped after the end of project (often many years after the project was planned, the results framework crafted), and little evolved in these countries to affect project sustainability.
Finally, the rose-colored glasses aspect includes (the possible fiction) that those involved in the project shared the Goals and Objectives aspirations held by the donors and implementers. In my fieldwork experience, communities sometimes cannot tell one project from another unless one helps them differentiate specific resources such as wells, seeds, literacy training transferred. Nor do participants necessarily share expectations of outcomes such as they would be more drought-resilient or food-secure in the future. In research I hope to do, I will look for how many shared expectations of impact there are, how communities evaluate long-term impact of the tangible and intangible resources that were transferred and what their perceptions of the real value of projects is, long after implementers and donors have left.
What do you think? Have you seen cases where we are doing Sustainable Development well?
by valuingvoicesjin | Nov 13, 2013 | Accountability, Africa, Evaluation, International aid, International non-profits, Sustainability
Accountability and Solidarity- what does it take?
Who are we accountable to? US agencies would say our taxpayers, international charities would say their donors, but what would ordinary folks like you and me say? Our consciences?
My dear friend Peter Kimeu of Machakos, Kenya is a mzee, Swahili for wise man. He has worked for them in Kenya and around Africa; we met at CRS (successfully) saving 'safety net' programming from being closed out by USAID 20 years ago. He is touring the US this month on behalf of Catholic Relief Services, talking to the charity's supporters about solidarity. Charity Navigator ranks the organization highly, using $0.94 cents of every dollar for their international programs in health, agriculture, economic development, social justice and so on. Peter talks to those in his audience, genuinely calling them 'brother' and 'sister' and calling on them to see their brethren in Kenya and the world as brothers and sisters. Do we feel the same? Do our systems support such thinking and acting? Are we accountable in our choices to our brethren globally?
It got me thinking how we determine what we're accountable for and how we know our funds are well spent. Charity Navigator does respected rankings of international charities. They define accountability as "Accountability is an obligation or willingness by a charity to explain its actions to its stakeholders. For now, Charity Navigator is specifically evaluating the fiduciary actions of charities." Currently the stakeholders are only those who financially invest in the charity. This excludes the stakeholders in the countries in which development happens. But there is excellent news coming.
Charity Navigator has plans to start looking at impact In Results Reporting, "we are specifically focusing on the way charities come to know, use and share their results with stakeholders including donors… We believe effective charities manage their performance and thereby know and act on their results. These high performing organizations use early results as milestones to important longer-term results." This will include "This element assesses whether and how well a charity collects and publishes feedback from it primary constituents."
This will include:
* Does the charity publish feedback data from its primary constituents?
* Does this data indicate if it is representative of all primary constituents?
* Is this data presented in a way that shows changes over time going back at least one year?
* Does this data include questions that speak to the organization’s effectiveness?
* Does the organization report back to its primary constituents what it heard from them?
This is enormously exciting news, and the only measure to add is long-term, post-project sustainability. These measures together will help us see which charities "provide the most meaningful change in communities and people’s lives" during and after the project. It will help us see which ones are learning from their programming with communities leading the way, teaching us what works best for them and what they want more or less of. This will shape future sectoral program investments as well as investor decisions. Kudos to Charity Navigator! They are helping all of us move in a direction of trulyinformed decision-making toward greater global solidarity.
